From 2b4369d7cb5c026c75a1c8c8fcd5d35dbf7ee5a8 Mon Sep 17 00:00:00 2001 From: Guilhem Date: Thu, 20 Aug 2026 10:49:01 +0200 Subject: [PATCH 01/48] fix: reject invalid AI agent tool names when the chat writes a flow (#10756) * fix: reject invalid AI agent tool names when the chat writes a flow Co-Authored-By: Claude Opus 5 (1M context) * fix: address review nits on agent tool name validation Share one AI-agent walk between the providerless-agent and invalid-tool-name collectors, drop the unused validateToolName, and list every reserved id in the tool naming rules. Co-Authored-By: Claude Opus 5 (1M context) * fix: describe an agent tool's summary as the name the agent calls it by The OpenFlow schema described `AgentTool.summary` as a short description of the tool, which is the same schema the flow write tools hand the model, so it pulled against the naming rules. Narrow those rules to flowmodule tools, since websearch and mcp tool names are never regex-checked, and let `kind` take either vocabulary its callers resolve. Co-Authored-By: Claude Opus 5 (1M context) * fix: name-check only the agent tools whose summary the agent calls An mcp tool exposes the MCP server's own tool names and a websearch tool's summary is a plain label, so neither reaches the worker's name check. Both default to an empty summary in the editor, which the chat then refused to write back. Co-Authored-By: Claude Opus 5 (1M context) --------- Co-authored-by: Claude Opus 5 (1M context) --- cli/src/guidance/skills.gen.ts | 67 ++++++++++++++++++- .../lib/components/copilot/MetadataGen.svelte | 2 +- .../lib/components/copilot/chat/flow/core.ts | 39 ----------- .../chat/flow/editableFlowJson.test.ts | 55 +++++++++++++++ .../copilot/chat/flow/editableFlowJson.ts | 21 +++++- .../copilot/chat/flow/openFlow.json | 2 +- .../copilot/chat/flow/openFlowZod.gen.ts | 4 +- .../src/lib/components/flows/agentToolTree.ts | 62 ++++++++++++++--- .../lib/components/flows/agentToolUtils.ts | 31 +++++++++ .../flows/common/FlowCardHeader.svelte | 2 +- .../graph/renderers/nodes/AIToolNode.svelte | 28 +------- openflow.openapi.yaml | 2 +- system_prompts/auto-generated/flow.md | 67 ++++++++++++++++++- system_prompts/auto-generated/prompts.ts | 67 ++++++++++++++++++- .../auto-generated/skills/write-flow/SKILL.md | 67 ++++++++++++++++++- system_prompts/base/flow-base.md | 65 ++++++++++++++++++ 16 files changed, 496 insertions(+), 85 deletions(-) diff --git a/cli/src/guidance/skills.gen.ts b/cli/src/guidance/skills.gen.ts index eb386b3ec8..0df389d1a8 100644 --- a/cli/src/guidance/skills.gen.ts +++ b/cli/src/guidance/skills.gen.ts @@ -5124,11 +5124,75 @@ value: - Use underscores, not spaces (e.g., \`fetch_data\` not \`fetch data\`) - Use descriptive names that reflect the step's purpose +## AI Agent Modules + +An \`aiagent\` module runs an LLM that can call tools. Each entry of \`value.tools\` is a module-shaped +object with an extra \`value.tool_type\`: \`flowmodule\` for a script/flow tool, \`mcp\` for an MCP server +tool, \`websearch\` for web search. + +\`\`\`json +{ + "id": "support_agent", + "summary": "AI agent for customer support", + "value": { + "type": "aiagent", + "input_transforms": { + "provider": { + "type": "static", + "value": { "kind": "openai", "resource": "$res:f/ai_providers/openai", "model": "gpt-4o" } + }, + "output_type": { "type": "static", "value": "text" }, + "user_message": { "type": "javascript", "expr": "flow_input.query" }, + "system_prompt": { "type": "static", "value": "You are a helpful assistant." } + }, + "tools": [ + { + "id": "search_docs", + "summary": "search_documentation", + "description": "Search the product documentation. Use it whenever the user asks how a feature works.", + "value": { + "tool_type": "flowmodule", + "type": "rawscript", + "language": "bun", + "content": "export async function main(query: string) { return ['doc1', 'doc2']; }", + "input_transforms": { "query": { "type": "static", "value": "" } } + } + } + ] + } +} +\`\`\` + +- \`provider\` is a static object, not a bare resource string: \`{ "kind": , + "resource": "$res:", "model": }\`. Required unless the module links to a saved + agent through \`value.agent\` + +### Tool Naming Rules + +These rules cover \`flowmodule\` tools, the ones the agent calls by name. A \`websearch\` tool's +\`summary\` is a plain label (\`Web Search\`), and an \`mcp\` tool exposes the MCP server's own tool +names, so neither is name-checked at all — leave those summaries as they are. + +- A flowmodule tool's \`summary\` is the **name the agent calls it by**, not a human label. Put the + human-readable explanation in \`description\` +- \`summary\` must match \`^[a-zA-Z0-9_]+$\`: letters, numbers and underscores only. No spaces, dashes, + dots or accents — \`search_documentation\`, never \`Search documentation\` +- Always set \`summary\`. It must be unique among that agent's tools, and must not be one of the + reserved ids (\`do\`, \`bg\`, \`ctx\`, \`state\`, \`if\`, \`else\`, \`for\`, \`delete\`, \`while\`, \`new\`, \`in\`, + \`failure\`, \`preprocessor\`, \`as\`, \`Input\`, \`Result\`, \`Trigger\`) +- A tool name outside that character set is rejected: flow write tools refuse it, and a flow that + reaches the worker with one fails every run with \`Invalid tool name\` +- Tool \`id\` follows the same rules as any module ID — unique across the flow, underscores not spaces +- \`description\` is optional free text telling the agent when and how to call the tool. Set it + whenever the name alone does not make that obvious; it overrides the description derived from the + underlying script + ## Common Mistakes to Avoid - Missing \`input_transforms\` - Rawscript parameters won't receive values without them - Referencing future steps - \`results.step_id\` only works for steps that execute before the current one - Duplicate module IDs - Each module ID must be unique in the flow +- AI agent flowmodule tool names with spaces - \`summary\` is the tool name and only accepts letters, numbers and underscores ## Data Flow Between Steps @@ -5347,6 +5411,7 @@ Before finalizing a flow, verify: - any failure handler is in \`value.failure_module\` - any approval step has module-level \`suspend\` - no downstream step references inner branch step ids from outside the branch +- every AI agent flowmodule tool has a unique \`summary\` made only of letters, numbers and underscores ## S3 Object Operations @@ -5404,7 +5469,7 @@ Reference a specific resource using \`$res:\` prefix: ## OpenFlow Schema -{"OpenFlow":{"type":"object","description":"Top-level flow definition containing metadata, configuration, and the flow structure","properties":{"summary":{"type":"string","description":"Short description of what this flow does"},"description":{"type":"string","description":"Detailed documentation for this flow"},"value":{"$ref":"#/components/schemas/FlowValue"},"schema":{"type":"object","description":"JSON Schema for flow inputs. Use this to define input parameters, their types, defaults, and validation. For resource inputs, set type to 'object' and format to 'resource-' (e.g., 'resource-stripe')"},"on_behalf_of_email":{"type":"string","description":"Address of the account the flow runs on behalf of. Derived from on_behalf_of on read; accepted on write, where it is resolved to the account it names."},"on_behalf_of":{"type":"string","description":"The flow runs with the permissions of this identity: u/{username}, g/{group}, or a bare email when the username is itself email-shaped. The only stored half of the identity; on_behalf_of_email is derived from it. Omit it when writing and it is resolved from that address instead."}},"required":["summary","value"]},"FlowValue":{"type":"object","description":"The flow structure containing modules and optional preprocessor/failure handlers","properties":{"modules":{"type":"array","description":"Array of steps that execute in sequence. Each step can be a script, subflow, loop, or branch","items":{"$ref":"#/components/schemas/FlowModule"}},"failure_module":{"description":"Special module that executes when the flow fails. Receives error object with message, name, stack, and step_id. Must have id 'failure'. Only supports script/rawscript types","$ref":"#/components/schemas/FlowModule"},"preprocessor_module":{"description":"Special module that runs before the first step on external triggers. Must have id 'preprocessor'. Only supports script/rawscript types. Cannot reference other step results","$ref":"#/components/schemas/FlowModule"},"same_worker":{"type":"boolean","description":"If true, all steps run on the same worker for better performance"},"preserve_step_tags":{"type":"boolean","description":"If true and the flow runs on a custom worker tag, steps that declare their own non-empty tag run on it instead of inheriting the flow tag. Steps without their own tag still inherit the flow tag."},"concurrent_limit":{"type":"number","description":"Maximum number of concurrent executions of this flow"},"concurrency_key":{"type":"string","description":"Expression to group concurrent executions (e.g., by user ID)"},"concurrency_time_window_s":{"type":"number","description":"Time window in seconds for concurrent_limit"},"debounce_delay_s":{"type":"integer","description":"Delay in seconds to debounce flow executions"},"debounce_key":{"type":"string","description":"Expression to group debounced executions"},"debounce_args_to_accumulate":{"type":"array","description":"Arguments to accumulate across debounced executions","items":{"type":"string"}},"max_total_debouncing_time":{"type":"integer","description":"Maximum total time in seconds that a job can be debounced"},"max_total_debounces_amount":{"type":"integer","description":"Maximum number of times a job can be debounced"},"skip_expr":{"type":"string","description":"JavaScript expression to conditionally skip the entire flow"},"cache_ttl":{"type":"number","description":"Cache duration in seconds for flow results"},"cache_ignore_s3_path":{"type":"boolean"},"delete_after_secs":{"type":"integer","description":"If set, delete the flow job's args, result and logs after this many seconds following job completion"},"flow_env":{"type":"object","description":"Environment variables available to all steps. Values can be strings, JSON values, or special references: '$var:path' (workspace variable) or '$res:path' (resource).","additionalProperties":{}},"priority":{"type":"number","description":"Execution priority (higher numbers run first)"},"early_return":{"type":"string","description":"JavaScript expression to return early from the flow"},"chat_input_enabled":{"type":"boolean","description":"Whether this flow accepts chat-style input"},"notes":{"type":"array","description":"Sticky notes attached to the flow","items":{"$ref":"#/components/schemas/FlowNote"}},"groups":{"type":"array","description":"Semantic groups of modules for organizational purposes","items":{"$ref":"#/components/schemas/FlowGroup"}}},"required":["modules"]},"Retry":{"type":"object","description":"Retry configuration for failed module executions","properties":{"constant":{"type":"object","description":"Retry with constant delay between attempts","properties":{"attempts":{"type":"integer","description":"Number of retry attempts"},"seconds":{"type":"integer","description":"Seconds to wait between retries"}}},"exponential":{"type":"object","description":"Retry with exponential backoff (delay doubles each time)","properties":{"attempts":{"type":"integer","description":"Number of retry attempts"},"multiplier":{"type":"integer","description":"Multiplier for exponential backoff"},"seconds":{"type":"integer","minimum":1,"description":"Initial delay in seconds"},"random_factor":{"type":"integer","minimum":0,"maximum":100,"description":"Random jitter percentage (0-100) to avoid thundering herd"}}},"retry_if":{"$ref":"#/components/schemas/RetryIf"}}},"FlowNote":{"type":"object","description":"A sticky note attached to a flow for documentation and annotation","properties":{"id":{"type":"string","description":"Unique identifier for the note"},"text":{"type":"string","description":"Content of the note"},"position":{"type":"object","description":"Position of the note in the flow editor","properties":{"x":{"type":"number","description":"X coordinate"},"y":{"type":"number","description":"Y coordinate"}},"required":["x","y"]},"size":{"type":"object","description":"Size of the note in the flow editor","properties":{"width":{"type":"number","description":"Width in pixels"},"height":{"type":"number","description":"Height in pixels"}},"required":["width","height"]},"color":{"type":"string","description":"Color of the note (e.g., \\"yellow\\", \\"#ffff00\\")"},"type":{"type":"string","enum":["free","group"],"description":"Type of note - 'free' for standalone notes, 'group' for notes that group other nodes"},"locked":{"type":"boolean","default":false,"description":"Whether the note is locked and cannot be edited or moved"},"contained_node_ids":{"type":"array","items":{"type":"string"},"description":"For group notes, the IDs of nodes contained within this group"}},"required":["id","text","color","type"]},"FlowGroup":{"type":"object","description":"A semantic group of flow modules for organizational purposes. Does not affect execution \\u2014 modules remain in their original position in the flow. Groups provide naming and collapsibility in the editor. Members are computed dynamically from all nodes on paths between start_id and end_id.","properties":{"summary":{"type":"string","description":"Display name for this group"},"note":{"type":"string","description":"Markdown note shown below the group header"},"autocollapse":{"type":"boolean","default":false,"description":"If true, this group is collapsed by default in the flow editor. UI hint only."},"start_id":{"type":"string","description":"ID of the first flow module in this group (topological entry point)"},"end_id":{"type":"string","description":"ID of the last flow module in this group (topological exit point)"},"color":{"type":"string","description":"Color for the group in the flow editor"}},"required":["start_id","end_id"]},"RetryIf":{"type":"object","description":"Conditional retry based on error or result","properties":{"expr":{"type":"string","description":"JavaScript expression that returns true to retry. Has access to 'result' and 'error' variables"}},"required":["expr"]},"StopAfterIf":{"type":"object","description":"Early termination condition for a module","properties":{"skip_if_stopped":{"type":"boolean","description":"If true, following steps are skipped when this condition triggers"},"expr":{"type":"string","description":"JavaScript expression evaluated after the module runs. Can use 'result' (step's result) or 'flow_input'. Return true to stop"},"error_message":{"type":"string","nullable":true,"description":"Custom error message when stopping with an error. Mutually exclusive with skip_if_stopped. If set to a non-empty string, the flow stops with this error. If empty string, a default error message is used. If null or omitted, no error is raised."},"error_include_result":{"type":"boolean","description":"When stopping with an error (error_message set), embed the stopping step's own result inside the raised error object (as error.result) instead of discarding it. The top-level result stays { error }. Defaults to false."}},"required":["expr"]},"FlowModule":{"type":"object","description":"A single step in a flow. Can be a script, subflow, loop, or branch","properties":{"id":{"type":"string","description":"Unique identifier for this step. Used to reference results via 'results.step_id'. Must be a valid identifier (alphanumeric, underscore, hyphen)"},"value":{"$ref":"#/components/schemas/FlowModuleValue"},"stop_after_if":{"description":"Early termination condition evaluated after this step completes","$ref":"#/components/schemas/StopAfterIf"},"stop_after_all_iters_if":{"description":"For loops only - early termination condition evaluated after all iterations complete","$ref":"#/components/schemas/StopAfterIf"},"skip_if":{"type":"object","description":"Conditionally skip this step based on previous results or flow inputs","properties":{"expr":{"type":"string","description":"JavaScript expression that returns true to skip. Can use 'flow_input' or 'results.'"}},"required":["expr"]},"sleep":{"description":"Delay before executing this step (in seconds or as expression)","$ref":"#/components/schemas/InputTransform"},"cache_ttl":{"type":"number","description":"Cache duration in seconds for this step's results"},"cache_ignore_s3_path":{"type":"boolean"},"timeout":{"description":"Maximum execution time in seconds (static value or expression)","$ref":"#/components/schemas/InputTransform"},"delete_after_secs":{"type":"integer","description":"If set, delete the step's args, result and logs after this many seconds following job completion"},"summary":{"type":"string","description":"Short description of what this step does"},"mock":{"type":"object","description":"Mock configuration for testing without executing the actual step","properties":{"enabled":{"type":"boolean","description":"If true, return mock value instead of executing"},"return_value":{"description":"Value to return when mocked"}}},"suspend":{"type":"object","description":"Configuration for approval/resume steps that wait for user input","properties":{"required_events":{"type":"integer","description":"Number of approvals required before continuing"},"timeout":{"type":"integer","description":"Timeout in seconds before auto-continuing or canceling"},"resume_form":{"type":"object","description":"Form schema for collecting input when resuming","properties":{"schema":{"type":"object","description":"JSON Schema for the resume form"}}},"user_auth_required":{"type":"boolean","description":"If true, only authenticated users can approve"},"user_groups_required":{"description":"Expression or list of groups that can approve","$ref":"#/components/schemas/InputTransform"},"self_approval_disabled":{"type":"boolean","description":"If true, the user who started the flow cannot approve"},"hide_cancel":{"type":"boolean","description":"If true, hide the cancel button on the approval form"},"continue_on_disapprove_timeout":{"type":"boolean","description":"If true, continue flow on timeout instead of canceling"}}},"priority":{"type":"number","description":"Execution priority for this step (higher numbers run first)"},"continue_on_error":{"type":"boolean","description":"If true, flow continues even if this step fails"},"retry":{"description":"Retry configuration if this step fails","$ref":"#/components/schemas/Retry"},"debouncing":{"description":"Debounce configuration for this step (EE only)","type":"object","properties":{"debounce_delay_s":{"type":"integer","description":"Delay in seconds to debounce this step's executions across flow runs"},"debounce_key":{"type":"string","description":"Expression to group debounced executions. Supports $workspace and $args[name]. Default: $workspace/flow/-"},"debounce_args_to_accumulate":{"type":"array","description":"Array-type arguments to accumulate across debounced executions","items":{"type":"string"}},"max_total_debouncing_time":{"type":"integer","description":"Maximum total time in seconds before forced execution"},"max_total_debounces_amount":{"type":"integer","description":"Maximum number of debounces before forced execution"}}}},"required":["value","id"]},"InputTransform":{"description":"Maps input parameters for a step. Can be a static value or a JavaScript expression that references previous results or flow inputs","oneOf":[{"$ref":"#/components/schemas/StaticTransform"},{"$ref":"#/components/schemas/JavascriptTransform"},{"$ref":"#/components/schemas/AiTransform"}],"discriminator":{"propertyName":"type","mapping":{"static":"#/components/schemas/StaticTransform","javascript":"#/components/schemas/JavascriptTransform","ai":"#/components/schemas/AiTransform"}}},"StaticTransform":{"type":"object","description":"Static value passed directly to the step. Use for hardcoded values or resource references like '$res:path/to/resource'","properties":{"value":{"description":"The static value. For resources, use format '$res:path/to/resource'"},"type":{"type":"string","enum":["static"]}},"required":["type"]},"JavascriptTransform":{"type":"object","description":"JavaScript expression evaluated at runtime. Can reference previous step results via 'results.step_id' or flow inputs via 'flow_input.property'. Inside for loops, use 'flow_input.iter.value' for the current iteration value (in while loops it equals 'flow_input.iter.index')","properties":{"expr":{"type":"string","description":"JavaScript expression returning the value. Available variables - results (object with all previous step results), flow_input (flow inputs), flow_input.iter (in loops)"},"type":{"type":"string","enum":["javascript"]}},"required":["expr","type"]},"AiTransform":{"type":"object","description":"Value resolved by the AI runtime for this input. The AI engine decides how to satisfy the parameter.","properties":{"type":{"type":"string","enum":["ai"]}},"required":["type"]},"AIProviderKind":{"type":"string","description":"Supported AI provider types","enum":["openai","azure_openai","azure_foundry","anthropic","mistral","deepseek","googleai","groq","openrouter","togetherai","customai","aws_bedrock"]},"ProviderConfig":{"type":"object","description":"Complete AI provider configuration with resource reference and model selection","properties":{"kind":{"$ref":"#/components/schemas/AIProviderKind"},"resource":{"type":"string","description":"Resource reference in format '$res:{resource_path}' pointing to provider credentials"},"model":{"type":"string","description":"Model identifier (e.g., 'gpt-4', 'claude-3-opus-20240229', 'gemini-pro')"},"reasoning_effort":{"type":"string","description":"Provider-native reasoning effort token (e.g. 'low', 'high', 'none') for models that support extended thinking. Optional; unset leaves the provider default."}},"required":["kind","resource","model"]},"StaticProviderTransform":{"type":"object","description":"Static provider configuration passed directly to the AI agent","properties":{"value":{"$ref":"#/components/schemas/ProviderConfig"},"type":{"type":"string","enum":["static"]}},"required":["type","value"]},"ProviderTransform":{"description":"Provider configuration - can be static (ProviderConfig), JavaScript expression, or AI-determined","oneOf":[{"$ref":"#/components/schemas/StaticProviderTransform"},{"$ref":"#/components/schemas/JavascriptTransform"},{"$ref":"#/components/schemas/AiTransform"}],"discriminator":{"propertyName":"type","mapping":{"static":"#/components/schemas/StaticProviderTransform","javascript":"#/components/schemas/JavascriptTransform","ai":"#/components/schemas/AiTransform"}}},"MemoryOff":{"type":"object","description":"No conversation memory/context","properties":{"kind":{"type":"string","enum":["off"]}},"required":["kind"]},"MemoryAuto":{"type":"object","description":"Automatic context management","properties":{"kind":{"type":"string","enum":["auto"]},"context_length":{"type":"integer","description":"Maximum number of messages to retain in context"},"memory_id":{"type":"string","description":"Identifier for persistent memory across agent invocations"}},"required":["kind"]},"MemoryMessage":{"type":"object","description":"A single message in conversation history","properties":{"role":{"type":"string","enum":["user","assistant","system"]},"content":{"type":"string"}},"required":["role","content"]},"MemoryManual":{"type":"object","description":"Explicit message history","properties":{"kind":{"type":"string","enum":["manual"]},"messages":{"type":"array","items":{"$ref":"#/components/schemas/MemoryMessage"}}},"required":["kind","messages"]},"MemoryConfig":{"description":"Conversation memory configuration","oneOf":[{"$ref":"#/components/schemas/MemoryOff"},{"$ref":"#/components/schemas/MemoryAuto"},{"$ref":"#/components/schemas/MemoryManual"}],"discriminator":{"propertyName":"kind","mapping":{"off":"#/components/schemas/MemoryOff","auto":"#/components/schemas/MemoryAuto","manual":"#/components/schemas/MemoryManual"}}},"StaticMemoryTransform":{"type":"object","description":"Static memory configuration passed directly to the AI agent","properties":{"value":{"$ref":"#/components/schemas/MemoryConfig"},"type":{"type":"string","enum":["static"]}},"required":["type","value"]},"MemoryTransform":{"description":"Memory configuration - can be static (MemoryConfig), JavaScript expression, or AI-determined","oneOf":[{"$ref":"#/components/schemas/StaticMemoryTransform"},{"$ref":"#/components/schemas/JavascriptTransform"},{"$ref":"#/components/schemas/AiTransform"}],"discriminator":{"propertyName":"type","mapping":{"static":"#/components/schemas/StaticMemoryTransform","javascript":"#/components/schemas/JavascriptTransform","ai":"#/components/schemas/AiTransform"}}},"FlowModuleValue":{"description":"The actual implementation of a flow step. Can be a script (inline or referenced), subflow, loop, branch, or special module type","oneOf":[{"$ref":"#/components/schemas/RawScript"},{"$ref":"#/components/schemas/PathScript"},{"$ref":"#/components/schemas/PathFlow"},{"$ref":"#/components/schemas/ForloopFlow"},{"$ref":"#/components/schemas/WhileloopFlow"},{"$ref":"#/components/schemas/BranchOne"},{"$ref":"#/components/schemas/BranchAll"},{"$ref":"#/components/schemas/Identity"},{"$ref":"#/components/schemas/AiAgent"}],"discriminator":{"propertyName":"type","mapping":{"rawscript":"#/components/schemas/RawScript","script":"#/components/schemas/PathScript","flow":"#/components/schemas/PathFlow","forloopflow":"#/components/schemas/ForloopFlow","whileloopflow":"#/components/schemas/WhileloopFlow","branchone":"#/components/schemas/BranchOne","branchall":"#/components/schemas/BranchAll","identity":"#/components/schemas/Identity","aiagent":"#/components/schemas/AiAgent"}}},"RawScript":{"type":"object","description":"Inline script with code defined directly in the flow. Use 'bun' as default language if unspecified. The script receives arguments from input_transforms","properties":{"input_transforms":{"type":"object","description":"Map of parameter names to their values (static or JavaScript expressions). These become the script's input arguments","additionalProperties":{"$ref":"#/components/schemas/InputTransform"}},"content":{"type":"string","description":"The script source code. Should export a 'main' function"},"language":{"type":"string","description":"Programming language for this script","enum":["deno","bun","bunnative","python3","go","bash","powershell","postgresql","mysql","bigquery","snowflake","mssql","oracledb","graphql","nativets","php","rust","ansible","csharp","nu","java","ruby","rlang","duckdb"]},"path":{"type":"string","description":"Optional path for saving this script"},"lock":{"type":"string","description":"Lock file content for dependencies"},"type":{"type":"string","enum":["rawscript"]},"tag":{"type":"string","description":"Worker group tag for execution routing"},"concurrent_limit":{"type":"number","description":"Maximum concurrent executions of this script"},"concurrency_time_window_s":{"type":"number","description":"Time window for concurrent_limit"},"custom_concurrency_key":{"type":"string","description":"Custom key for grouping concurrent executions"},"is_trigger":{"type":"boolean","description":"If true, this script is a trigger that can start the flow"},"assets":{"type":"array","description":"External resources this script accesses (S3 objects, resources, etc.)","items":{"type":"object","required":["path","kind"],"properties":{"path":{"type":"string","description":"Path to the asset"},"kind":{"type":"string","description":"Type of asset","enum":["s3object","resource","ducklake","datatable","volume","dbt"]},"access_type":{"type":"string","nullable":true,"description":"Access level for this asset","enum":["r","w","rw"]},"alt_access_type":{"type":"string","nullable":true,"description":"Alternative access level","enum":["r","w","rw"]}}}}},"required":["type","content","language","input_transforms"]},"PathScript":{"type":"object","description":"Reference to an existing script by path. Use this when calling a previously saved script instead of writing inline code","properties":{"input_transforms":{"type":"object","description":"Map of parameter names to their values (static or JavaScript expressions). These become the script's input arguments","additionalProperties":{"$ref":"#/components/schemas/InputTransform"}},"path":{"type":"string","description":"Path to the script in the workspace (e.g., 'f/scripts/send_email')"},"hash":{"type":"string","description":"Optional specific version hash of the script to use"},"type":{"type":"string","enum":["script"]},"tag_override":{"type":"string","description":"Override the script's default worker group tag"},"is_trigger":{"type":"boolean","description":"If true, this script is a trigger that can start the flow"}},"required":["type","path","input_transforms"]},"PathFlow":{"type":"object","description":"Reference to an existing flow by path. Use this to call another flow as a subflow","properties":{"input_transforms":{"type":"object","description":"Map of parameter names to their values (static or JavaScript expressions). These become the subflow's input arguments","additionalProperties":{"$ref":"#/components/schemas/InputTransform"}},"path":{"type":"string","description":"Path to the flow in the workspace (e.g., 'f/flows/process_user')"},"type":{"type":"string","enum":["flow"]}},"required":["type","path","input_transforms"]},"ForloopFlow":{"type":"object","description":"Executes nested modules in a loop over an iterator. Inside the loop, use 'flow_input.iter.value' to access the current iteration value, and 'flow_input.iter.index' for the index. Supports parallel execution for better performance on I/O-bound operations","properties":{"modules":{"type":"array","description":"Steps to execute for each iteration. These can reference the iteration value via 'flow_input.iter.value'","items":{"$ref":"#/components/schemas/FlowModule"}},"iterator":{"description":"JavaScript expression that returns an array to iterate over. Can reference 'results.step_id' or 'flow_input'","$ref":"#/components/schemas/InputTransform"},"skip_failures":{"type":"boolean","description":"If true, iteration failures don't stop the loop. Failed iterations return null"},"type":{"type":"string","enum":["forloopflow"]},"parallel":{"type":"boolean","description":"If true, iterations run concurrently (faster for I/O-bound operations). Use with parallelism to control concurrency"},"parallelism":{"description":"Maximum number of concurrent iterations when parallel=true. Limits resource usage. Can be static number or expression","$ref":"#/components/schemas/InputTransform"},"squash":{"type":"boolean"}},"required":["modules","iterator","skip_failures","type"]},"WhileloopFlow":{"type":"object","description":"Executes nested modules repeatedly until stopped. The implicit iterator is the iteration counter, so 'flow_input.iter.value' equals 'flow_input.iter.index' (0, 1, 2, ...) and never carries state. To carry state across iterations, a step reads its own previous-iteration result via 'results.' with a first-iteration fallback - the loop's stop_after_if must then be on that inner step (a plain single-step body with stop_after_if on the loop module does not resolve 'results' across iterations and never terminates); plain counters can instead be derived from 'flow_input.iter.index', which works in every configuration. stop_after_if is evaluated after each iteration - on the loop module 'result' is the last iteration's result","properties":{"modules":{"type":"array","description":"Steps to execute in each iteration","items":{"$ref":"#/components/schemas/FlowModule"}},"skip_failures":{"type":"boolean","description":"If true, iteration failures don't stop the loop. Failed iterations return null"},"type":{"type":"string","enum":["whileloopflow"]},"parallel":{"type":"boolean","description":"If true, iterations run concurrently (use with caution in while loops)"},"parallelism":{"description":"Maximum number of concurrent iterations when parallel=true","$ref":"#/components/schemas/InputTransform"},"squash":{"type":"boolean"}},"required":["modules","skip_failures","type"]},"BranchOne":{"type":"object","description":"Conditional branching where only the first matching branch executes. Branches are evaluated in order, and the first one with a true expression runs. If no branches match, the default branch executes","properties":{"branches":{"type":"array","description":"Array of branches to evaluate in order. The first branch with expr evaluating to true executes","items":{"type":"object","properties":{"summary":{"type":"string","description":"Short description of this branch condition"},"expr":{"type":"string","description":"JavaScript expression that returns boolean. Can use 'results.step_id' or 'flow_input'. First true expr wins"},"modules":{"type":"array","description":"Steps to execute if this branch's expr is true","items":{"$ref":"#/components/schemas/FlowModule"}}},"required":["modules","expr"]}},"default":{"type":"array","description":"Steps to execute if no branch expressions match","items":{"$ref":"#/components/schemas/FlowModule"}},"type":{"type":"string","enum":["branchone"]}},"required":["branches","default","type"]},"BranchAll":{"type":"object","description":"Parallel branching where all branches execute simultaneously. Unlike BranchOne, all branches run regardless of conditions. Useful for executing independent tasks concurrently","properties":{"branches":{"type":"array","description":"Array of branches that all execute (either in parallel or sequentially)","items":{"type":"object","properties":{"summary":{"type":"string","description":"Short description of this branch's purpose"},"skip_failure":{"type":"boolean","description":"If true, failure in this branch doesn't fail the entire flow"},"modules":{"type":"array","description":"Steps to execute in this branch","items":{"$ref":"#/components/schemas/FlowModule"}}},"required":["modules"]}},"type":{"type":"string","enum":["branchall"]},"parallel":{"type":"boolean","description":"If true, all branches execute concurrently. If false, they execute sequentially"}},"required":["branches","type"]},"AgentTool":{"type":"object","description":"A tool available to an AI agent. Can be a flow module or an external MCP (Model Context Protocol) tool","properties":{"id":{"type":"string","description":"Unique identifier for this tool. Cannot contain spaces - use underscores instead (e.g., 'get_user_data' not 'get user data')"},"summary":{"type":"string","description":"Short description of what this tool does (shown to the AI)"},"description":{"type":"string","description":"Free-text description of the tool given to the AI to decide when and how to call it. Overrides the description auto-derived from the underlying script."},"value":{"$ref":"#/components/schemas/ToolValue"}},"required":["id","value"]},"ToolValue":{"description":"The implementation of a tool. Can be a flow module (script/flow) or an MCP tool reference","oneOf":[{"$ref":"#/components/schemas/FlowModuleTool"},{"$ref":"#/components/schemas/McpToolValue"},{"$ref":"#/components/schemas/WebsearchToolValue"}],"discriminator":{"propertyName":"tool_type","mapping":{"flowmodule":"#/components/schemas/FlowModuleTool","mcp":"#/components/schemas/McpToolValue","websearch":"#/components/schemas/WebsearchToolValue"}}},"FlowModuleTool":{"description":"A tool implemented as a flow module (script, flow, etc.). The AI can call this like any other flow module","allOf":[{"type":"object","properties":{"tool_type":{"type":"string","enum":["flowmodule"]}},"required":["tool_type"]},{"$ref":"#/components/schemas/FlowModuleValue"}]},"WebsearchToolValue":{"type":"object","description":"A tool implemented as a websearch tool. The AI can call this like any other websearch tool","properties":{"tool_type":{"type":"string","enum":["websearch"]}},"required":["tool_type"]},"McpToolValue":{"type":"object","description":"Reference to an external MCP (Model Context Protocol) tool. The AI can call tools from MCP servers","properties":{"tool_type":{"type":"string","enum":["mcp"]},"resource_path":{"type":"string","description":"Path to the MCP resource/server configuration"},"include_tools":{"type":"array","description":"Whitelist of specific tools to include from this MCP server","items":{"type":"string"}},"exclude_tools":{"type":"array","description":"Blacklist of tools to exclude from this MCP server","items":{"type":"string"}}},"required":["tool_type","resource_path"]},"AiAgent":{"type":"object","description":"AI agent step that can use tools to accomplish tasks. The agent receives inputs and can call any of its configured tools to complete the task","properties":{"input_transforms":{"type":"object","description":"Input parameters for the AI agent mapped to their values","properties":{"provider":{"$ref":"#/components/schemas/ProviderTransform"},"output_type":{"allOf":[{"$ref":"#/components/schemas/InputTransform"}],"description":"Output format type.\\nValid values: 'text' (default) - plain text response, 'image' - image generation\\n"},"user_message":{"allOf":[{"$ref":"#/components/schemas/InputTransform"}],"description":"The user's prompt/message to the AI agent. Supports variable interpolation with flow.input syntax."},"system_prompt":{"allOf":[{"$ref":"#/components/schemas/InputTransform"}],"description":"System instructions that guide the AI's behavior, persona, and response style. Optional."},"streaming":{"allOf":[{"$ref":"#/components/schemas/InputTransform"}],"description":"Boolean. If true, stream the AI response incrementally.\\nStreaming events include: token_delta, reasoning_token_delta, tool_call, tool_call_arguments, tool_execution, tool_result\\n"},"memory":{"$ref":"#/components/schemas/MemoryTransform"},"output_schema":{"allOf":[{"$ref":"#/components/schemas/InputTransform"}],"description":"JSON Schema object defining structured output format. Used when you need the AI to return data in a specific shape.\\nSupports standard JSON Schema properties: type, properties, required, items, enum, pattern, minLength, maxLength, minimum, maximum, etc.\\nExample: { type: 'object', properties: { name: { type: 'string' }, age: { type: 'integer' } }, required: ['name'] }\\n"},"user_attachments":{"allOf":[{"$ref":"#/components/schemas/InputTransform"}],"description":"Array of file references (images or PDFs) for the AI agent.\\nFormat: Array<{ bucket: string, key: string }> - S3 object references\\nExample: [{ bucket: 'my-bucket', key: 'documents/report.pdf' }]\\n"},"max_completion_tokens":{"allOf":[{"$ref":"#/components/schemas/InputTransform"}],"description":"Integer. Maximum number of tokens the AI will generate in its response.\\nRange: 1 to 4,294,967,295. Typical values: 256-4096 for most use cases.\\n"},"temperature":{"allOf":[{"$ref":"#/components/schemas/InputTransform"}],"description":"Float. Controls randomness/creativity of responses.\\nRange: 0.0 to 2.0 (provider-dependent)\\n- 0.0 = deterministic, focused responses\\n- 0.7 = balanced (common default)\\n- 1.0+ = more creative/random\\n"},"max_iterations":{"allOf":[{"$ref":"#/components/schemas/InputTransform"}],"description":"Number. Limits how many times the agent can loop through reasoning and tool use.\\nRange: 1-1000.\\n"}},"required":["user_message"]},"tools":{"type":"array","description":"Array of tools the agent can use. The agent decides which tools to call based on the task","items":{"$ref":"#/components/schemas/AgentTool"}},"type":{"type":"string","enum":["aiagent"]},"tag":{"type":"string","description":"Worker group tag for execution routing. If not set, the AI agent step runs on the flow's tag (default \`flow\`)"},"omit_output_from_conversation":{"type":"boolean","default":false,"description":"If true, this AI agent step does not persist its assistant or tool messages to the flow conversation when chat mode is enabled."},"agent":{"type":"string","description":"Path of a reusable \`ai_agent\` resource (hybrid linking). When set, the agent brain\\nconfig (provider/model/system prompt/etc.) and tool set are resolved at runtime from\\nthat resource; the module's input_transforms then only carry the flow-local inputs\\n(user_message/user_attachments).\\n"},"tool_inputs":{"type":"object","description":"Host-local wiring for an agent's tool inputs, keyed by tool id then input key. Binds the\\nreferenced agent's tools to this flow's context (flow_input/results) without mutating the\\nshared resource; overlaid onto the tools' input_transforms at runtime \\u2014 including when\\n\`agent\` is unset, since a step forked for editing keeps these overrides until it is saved\\nback or unlinked.\\n","additionalProperties":{"type":"object","additionalProperties":{"$ref":"#/components/schemas/InputTransform"}}},"parallel":{"type":"boolean","description":"If true, the agent can execute multiple tool calls in parallel"}},"required":["type","input_transforms"]},"Identity":{"type":"object","description":"Pass-through module that returns its input unchanged. Useful for flow structure or as a placeholder","properties":{"type":{"type":"string","enum":["identity"]},"flow":{"type":"boolean","description":"If true, marks this as a flow identity (special handling)"}},"required":["type"]},"FlowStatus":{"type":"object","properties":{"step":{"type":"integer"},"modules":{"type":"array","items":{"$ref":"#/components/schemas/FlowStatusModule"}},"user_states":{"additionalProperties":true},"preprocessor_module":{"allOf":[{"$ref":"#/components/schemas/FlowStatusModule"}]},"failure_module":{"allOf":[{"$ref":"#/components/schemas/FlowStatusModule"},{"type":"object","properties":{"parent_module":{"type":"string"}}}]},"retry":{"type":"object","properties":{"fail_count":{"type":"integer"},"failed_jobs":{"type":"array","items":{"type":"string","format":"uuid"}}}}},"required":["step","modules","failure_module"]},"FlowStatusModule":{"type":"object","properties":{"type":{"type":"string","enum":["WaitingForPriorSteps","WaitingForEvents","WaitingForExecutor","InProgress","Success","Failure"]},"id":{"type":"string"},"job":{"type":"string","format":"uuid"},"count":{"type":"integer"},"progress":{"type":"integer"},"iterator":{"type":"object","properties":{"index":{"type":"integer"},"itered":{"type":"array","items":{}},"itered_len":{"type":"integer"},"args":{}}},"flow_jobs":{"type":"array","items":{"type":"string"}},"flow_jobs_success":{"type":"array","items":{"type":"boolean"}},"flow_jobs_duration":{"type":"object","properties":{"started_at":{"type":"array","items":{"type":"string"}},"duration_ms":{"type":"array","items":{"type":"integer"}}}},"branch_chosen":{"type":"object","properties":{"type":{"type":"string","enum":["branch","default"]},"branch":{"type":"integer"}},"required":["type"]},"branchall":{"type":"object","properties":{"branch":{"type":"integer"},"len":{"type":"integer"}},"required":["branch","len"]},"approvers":{"type":"array","items":{"type":"object","properties":{"resume_id":{"type":"integer"},"approver":{"type":"string"}},"required":["resume_id","approver"]}},"failed_retries":{"type":"array","items":{"type":"string","format":"uuid"}},"skipped":{"type":"boolean"},"agent_actions":{"type":"array","items":{"type":"object","oneOf":[{"type":"object","properties":{"job_id":{"type":"string","format":"uuid"},"function_name":{"type":"string"},"type":{"type":"string","enum":["tool_call"]},"module_id":{"type":"string"}},"required":["job_id","function_name","type","module_id"]},{"type":"object","properties":{"call_id":{"type":"string","format":"uuid"},"function_name":{"type":"string"},"resource_path":{"type":"string"},"type":{"type":"string","enum":["mcp_tool_call"]},"arguments":{"type":"object"}},"required":["call_id","function_name","resource_path","type"]},{"type":"object","properties":{"type":{"type":"string","enum":["web_search"]}},"required":["type"]},{"type":"object","properties":{"type":{"type":"string","enum":["message"]}},"required":["content","type"]}]}},"agent_actions_success":{"type":"array","items":{"type":"boolean"}}},"required":["type"]}}`, +{"OpenFlow":{"type":"object","description":"Top-level flow definition containing metadata, configuration, and the flow structure","properties":{"summary":{"type":"string","description":"Short description of what this flow does"},"description":{"type":"string","description":"Detailed documentation for this flow"},"value":{"$ref":"#/components/schemas/FlowValue"},"schema":{"type":"object","description":"JSON Schema for flow inputs. Use this to define input parameters, their types, defaults, and validation. For resource inputs, set type to 'object' and format to 'resource-' (e.g., 'resource-stripe')"},"on_behalf_of_email":{"type":"string","description":"Address of the account the flow runs on behalf of. Derived from on_behalf_of on read; accepted on write, where it is resolved to the account it names."},"on_behalf_of":{"type":"string","description":"The flow runs with the permissions of this identity: u/{username}, g/{group}, or a bare email when the username is itself email-shaped. The only stored half of the identity; on_behalf_of_email is derived from it. Omit it when writing and it is resolved from that address instead."}},"required":["summary","value"]},"FlowValue":{"type":"object","description":"The flow structure containing modules and optional preprocessor/failure handlers","properties":{"modules":{"type":"array","description":"Array of steps that execute in sequence. Each step can be a script, subflow, loop, or branch","items":{"$ref":"#/components/schemas/FlowModule"}},"failure_module":{"description":"Special module that executes when the flow fails. Receives error object with message, name, stack, and step_id. Must have id 'failure'. Only supports script/rawscript types","$ref":"#/components/schemas/FlowModule"},"preprocessor_module":{"description":"Special module that runs before the first step on external triggers. Must have id 'preprocessor'. Only supports script/rawscript types. Cannot reference other step results","$ref":"#/components/schemas/FlowModule"},"same_worker":{"type":"boolean","description":"If true, all steps run on the same worker for better performance"},"preserve_step_tags":{"type":"boolean","description":"If true and the flow runs on a custom worker tag, steps that declare their own non-empty tag run on it instead of inheriting the flow tag. Steps without their own tag still inherit the flow tag."},"concurrent_limit":{"type":"number","description":"Maximum number of concurrent executions of this flow"},"concurrency_key":{"type":"string","description":"Expression to group concurrent executions (e.g., by user ID)"},"concurrency_time_window_s":{"type":"number","description":"Time window in seconds for concurrent_limit"},"debounce_delay_s":{"type":"integer","description":"Delay in seconds to debounce flow executions"},"debounce_key":{"type":"string","description":"Expression to group debounced executions"},"debounce_args_to_accumulate":{"type":"array","description":"Arguments to accumulate across debounced executions","items":{"type":"string"}},"max_total_debouncing_time":{"type":"integer","description":"Maximum total time in seconds that a job can be debounced"},"max_total_debounces_amount":{"type":"integer","description":"Maximum number of times a job can be debounced"},"skip_expr":{"type":"string","description":"JavaScript expression to conditionally skip the entire flow"},"cache_ttl":{"type":"number","description":"Cache duration in seconds for flow results"},"cache_ignore_s3_path":{"type":"boolean"},"delete_after_secs":{"type":"integer","description":"If set, delete the flow job's args, result and logs after this many seconds following job completion"},"flow_env":{"type":"object","description":"Environment variables available to all steps. Values can be strings, JSON values, or special references: '$var:path' (workspace variable) or '$res:path' (resource).","additionalProperties":{}},"priority":{"type":"number","description":"Execution priority (higher numbers run first)"},"early_return":{"type":"string","description":"JavaScript expression to return early from the flow"},"chat_input_enabled":{"type":"boolean","description":"Whether this flow accepts chat-style input"},"notes":{"type":"array","description":"Sticky notes attached to the flow","items":{"$ref":"#/components/schemas/FlowNote"}},"groups":{"type":"array","description":"Semantic groups of modules for organizational purposes","items":{"$ref":"#/components/schemas/FlowGroup"}}},"required":["modules"]},"Retry":{"type":"object","description":"Retry configuration for failed module executions","properties":{"constant":{"type":"object","description":"Retry with constant delay between attempts","properties":{"attempts":{"type":"integer","description":"Number of retry attempts"},"seconds":{"type":"integer","description":"Seconds to wait between retries"}}},"exponential":{"type":"object","description":"Retry with exponential backoff (delay doubles each time)","properties":{"attempts":{"type":"integer","description":"Number of retry attempts"},"multiplier":{"type":"integer","description":"Multiplier for exponential backoff"},"seconds":{"type":"integer","minimum":1,"description":"Initial delay in seconds"},"random_factor":{"type":"integer","minimum":0,"maximum":100,"description":"Random jitter percentage (0-100) to avoid thundering herd"}}},"retry_if":{"$ref":"#/components/schemas/RetryIf"}}},"FlowNote":{"type":"object","description":"A sticky note attached to a flow for documentation and annotation","properties":{"id":{"type":"string","description":"Unique identifier for the note"},"text":{"type":"string","description":"Content of the note"},"position":{"type":"object","description":"Position of the note in the flow editor","properties":{"x":{"type":"number","description":"X coordinate"},"y":{"type":"number","description":"Y coordinate"}},"required":["x","y"]},"size":{"type":"object","description":"Size of the note in the flow editor","properties":{"width":{"type":"number","description":"Width in pixels"},"height":{"type":"number","description":"Height in pixels"}},"required":["width","height"]},"color":{"type":"string","description":"Color of the note (e.g., \\"yellow\\", \\"#ffff00\\")"},"type":{"type":"string","enum":["free","group"],"description":"Type of note - 'free' for standalone notes, 'group' for notes that group other nodes"},"locked":{"type":"boolean","default":false,"description":"Whether the note is locked and cannot be edited or moved"},"contained_node_ids":{"type":"array","items":{"type":"string"},"description":"For group notes, the IDs of nodes contained within this group"}},"required":["id","text","color","type"]},"FlowGroup":{"type":"object","description":"A semantic group of flow modules for organizational purposes. Does not affect execution \\u2014 modules remain in their original position in the flow. Groups provide naming and collapsibility in the editor. Members are computed dynamically from all nodes on paths between start_id and end_id.","properties":{"summary":{"type":"string","description":"Display name for this group"},"note":{"type":"string","description":"Markdown note shown below the group header"},"autocollapse":{"type":"boolean","default":false,"description":"If true, this group is collapsed by default in the flow editor. UI hint only."},"start_id":{"type":"string","description":"ID of the first flow module in this group (topological entry point)"},"end_id":{"type":"string","description":"ID of the last flow module in this group (topological exit point)"},"color":{"type":"string","description":"Color for the group in the flow editor"}},"required":["start_id","end_id"]},"RetryIf":{"type":"object","description":"Conditional retry based on error or result","properties":{"expr":{"type":"string","description":"JavaScript expression that returns true to retry. Has access to 'result' and 'error' variables"}},"required":["expr"]},"StopAfterIf":{"type":"object","description":"Early termination condition for a module","properties":{"skip_if_stopped":{"type":"boolean","description":"If true, following steps are skipped when this condition triggers"},"expr":{"type":"string","description":"JavaScript expression evaluated after the module runs. Can use 'result' (step's result) or 'flow_input'. Return true to stop"},"error_message":{"type":"string","nullable":true,"description":"Custom error message when stopping with an error. Mutually exclusive with skip_if_stopped. If set to a non-empty string, the flow stops with this error. If empty string, a default error message is used. If null or omitted, no error is raised."},"error_include_result":{"type":"boolean","description":"When stopping with an error (error_message set), embed the stopping step's own result inside the raised error object (as error.result) instead of discarding it. The top-level result stays { error }. Defaults to false."}},"required":["expr"]},"FlowModule":{"type":"object","description":"A single step in a flow. Can be a script, subflow, loop, or branch","properties":{"id":{"type":"string","description":"Unique identifier for this step. Used to reference results via 'results.step_id'. Must be a valid identifier (alphanumeric, underscore, hyphen)"},"value":{"$ref":"#/components/schemas/FlowModuleValue"},"stop_after_if":{"description":"Early termination condition evaluated after this step completes","$ref":"#/components/schemas/StopAfterIf"},"stop_after_all_iters_if":{"description":"For loops only - early termination condition evaluated after all iterations complete","$ref":"#/components/schemas/StopAfterIf"},"skip_if":{"type":"object","description":"Conditionally skip this step based on previous results or flow inputs","properties":{"expr":{"type":"string","description":"JavaScript expression that returns true to skip. Can use 'flow_input' or 'results.'"}},"required":["expr"]},"sleep":{"description":"Delay before executing this step (in seconds or as expression)","$ref":"#/components/schemas/InputTransform"},"cache_ttl":{"type":"number","description":"Cache duration in seconds for this step's results"},"cache_ignore_s3_path":{"type":"boolean"},"timeout":{"description":"Maximum execution time in seconds (static value or expression)","$ref":"#/components/schemas/InputTransform"},"delete_after_secs":{"type":"integer","description":"If set, delete the step's args, result and logs after this many seconds following job completion"},"summary":{"type":"string","description":"Short description of what this step does"},"mock":{"type":"object","description":"Mock configuration for testing without executing the actual step","properties":{"enabled":{"type":"boolean","description":"If true, return mock value instead of executing"},"return_value":{"description":"Value to return when mocked"}}},"suspend":{"type":"object","description":"Configuration for approval/resume steps that wait for user input","properties":{"required_events":{"type":"integer","description":"Number of approvals required before continuing"},"timeout":{"type":"integer","description":"Timeout in seconds before auto-continuing or canceling"},"resume_form":{"type":"object","description":"Form schema for collecting input when resuming","properties":{"schema":{"type":"object","description":"JSON Schema for the resume form"}}},"user_auth_required":{"type":"boolean","description":"If true, only authenticated users can approve"},"user_groups_required":{"description":"Expression or list of groups that can approve","$ref":"#/components/schemas/InputTransform"},"self_approval_disabled":{"type":"boolean","description":"If true, the user who started the flow cannot approve"},"hide_cancel":{"type":"boolean","description":"If true, hide the cancel button on the approval form"},"continue_on_disapprove_timeout":{"type":"boolean","description":"If true, continue flow on timeout instead of canceling"}}},"priority":{"type":"number","description":"Execution priority for this step (higher numbers run first)"},"continue_on_error":{"type":"boolean","description":"If true, flow continues even if this step fails"},"retry":{"description":"Retry configuration if this step fails","$ref":"#/components/schemas/Retry"},"debouncing":{"description":"Debounce configuration for this step (EE only)","type":"object","properties":{"debounce_delay_s":{"type":"integer","description":"Delay in seconds to debounce this step's executions across flow runs"},"debounce_key":{"type":"string","description":"Expression to group debounced executions. Supports $workspace and $args[name]. Default: $workspace/flow/-"},"debounce_args_to_accumulate":{"type":"array","description":"Array-type arguments to accumulate across debounced executions","items":{"type":"string"}},"max_total_debouncing_time":{"type":"integer","description":"Maximum total time in seconds before forced execution"},"max_total_debounces_amount":{"type":"integer","description":"Maximum number of debounces before forced execution"}}}},"required":["value","id"]},"InputTransform":{"description":"Maps input parameters for a step. Can be a static value or a JavaScript expression that references previous results or flow inputs","oneOf":[{"$ref":"#/components/schemas/StaticTransform"},{"$ref":"#/components/schemas/JavascriptTransform"},{"$ref":"#/components/schemas/AiTransform"}],"discriminator":{"propertyName":"type","mapping":{"static":"#/components/schemas/StaticTransform","javascript":"#/components/schemas/JavascriptTransform","ai":"#/components/schemas/AiTransform"}}},"StaticTransform":{"type":"object","description":"Static value passed directly to the step. Use for hardcoded values or resource references like '$res:path/to/resource'","properties":{"value":{"description":"The static value. For resources, use format '$res:path/to/resource'"},"type":{"type":"string","enum":["static"]}},"required":["type"]},"JavascriptTransform":{"type":"object","description":"JavaScript expression evaluated at runtime. Can reference previous step results via 'results.step_id' or flow inputs via 'flow_input.property'. Inside for loops, use 'flow_input.iter.value' for the current iteration value (in while loops it equals 'flow_input.iter.index')","properties":{"expr":{"type":"string","description":"JavaScript expression returning the value. Available variables - results (object with all previous step results), flow_input (flow inputs), flow_input.iter (in loops)"},"type":{"type":"string","enum":["javascript"]}},"required":["expr","type"]},"AiTransform":{"type":"object","description":"Value resolved by the AI runtime for this input. The AI engine decides how to satisfy the parameter.","properties":{"type":{"type":"string","enum":["ai"]}},"required":["type"]},"AIProviderKind":{"type":"string","description":"Supported AI provider types","enum":["openai","azure_openai","azure_foundry","anthropic","mistral","deepseek","googleai","groq","openrouter","togetherai","customai","aws_bedrock"]},"ProviderConfig":{"type":"object","description":"Complete AI provider configuration with resource reference and model selection","properties":{"kind":{"$ref":"#/components/schemas/AIProviderKind"},"resource":{"type":"string","description":"Resource reference in format '$res:{resource_path}' pointing to provider credentials"},"model":{"type":"string","description":"Model identifier (e.g., 'gpt-4', 'claude-3-opus-20240229', 'gemini-pro')"},"reasoning_effort":{"type":"string","description":"Provider-native reasoning effort token (e.g. 'low', 'high', 'none') for models that support extended thinking. Optional; unset leaves the provider default."}},"required":["kind","resource","model"]},"StaticProviderTransform":{"type":"object","description":"Static provider configuration passed directly to the AI agent","properties":{"value":{"$ref":"#/components/schemas/ProviderConfig"},"type":{"type":"string","enum":["static"]}},"required":["type","value"]},"ProviderTransform":{"description":"Provider configuration - can be static (ProviderConfig), JavaScript expression, or AI-determined","oneOf":[{"$ref":"#/components/schemas/StaticProviderTransform"},{"$ref":"#/components/schemas/JavascriptTransform"},{"$ref":"#/components/schemas/AiTransform"}],"discriminator":{"propertyName":"type","mapping":{"static":"#/components/schemas/StaticProviderTransform","javascript":"#/components/schemas/JavascriptTransform","ai":"#/components/schemas/AiTransform"}}},"MemoryOff":{"type":"object","description":"No conversation memory/context","properties":{"kind":{"type":"string","enum":["off"]}},"required":["kind"]},"MemoryAuto":{"type":"object","description":"Automatic context management","properties":{"kind":{"type":"string","enum":["auto"]},"context_length":{"type":"integer","description":"Maximum number of messages to retain in context"},"memory_id":{"type":"string","description":"Identifier for persistent memory across agent invocations"}},"required":["kind"]},"MemoryMessage":{"type":"object","description":"A single message in conversation history","properties":{"role":{"type":"string","enum":["user","assistant","system"]},"content":{"type":"string"}},"required":["role","content"]},"MemoryManual":{"type":"object","description":"Explicit message history","properties":{"kind":{"type":"string","enum":["manual"]},"messages":{"type":"array","items":{"$ref":"#/components/schemas/MemoryMessage"}}},"required":["kind","messages"]},"MemoryConfig":{"description":"Conversation memory configuration","oneOf":[{"$ref":"#/components/schemas/MemoryOff"},{"$ref":"#/components/schemas/MemoryAuto"},{"$ref":"#/components/schemas/MemoryManual"}],"discriminator":{"propertyName":"kind","mapping":{"off":"#/components/schemas/MemoryOff","auto":"#/components/schemas/MemoryAuto","manual":"#/components/schemas/MemoryManual"}}},"StaticMemoryTransform":{"type":"object","description":"Static memory configuration passed directly to the AI agent","properties":{"value":{"$ref":"#/components/schemas/MemoryConfig"},"type":{"type":"string","enum":["static"]}},"required":["type","value"]},"MemoryTransform":{"description":"Memory configuration - can be static (MemoryConfig), JavaScript expression, or AI-determined","oneOf":[{"$ref":"#/components/schemas/StaticMemoryTransform"},{"$ref":"#/components/schemas/JavascriptTransform"},{"$ref":"#/components/schemas/AiTransform"}],"discriminator":{"propertyName":"type","mapping":{"static":"#/components/schemas/StaticMemoryTransform","javascript":"#/components/schemas/JavascriptTransform","ai":"#/components/schemas/AiTransform"}}},"FlowModuleValue":{"description":"The actual implementation of a flow step. Can be a script (inline or referenced), subflow, loop, branch, or special module type","oneOf":[{"$ref":"#/components/schemas/RawScript"},{"$ref":"#/components/schemas/PathScript"},{"$ref":"#/components/schemas/PathFlow"},{"$ref":"#/components/schemas/ForloopFlow"},{"$ref":"#/components/schemas/WhileloopFlow"},{"$ref":"#/components/schemas/BranchOne"},{"$ref":"#/components/schemas/BranchAll"},{"$ref":"#/components/schemas/Identity"},{"$ref":"#/components/schemas/AiAgent"}],"discriminator":{"propertyName":"type","mapping":{"rawscript":"#/components/schemas/RawScript","script":"#/components/schemas/PathScript","flow":"#/components/schemas/PathFlow","forloopflow":"#/components/schemas/ForloopFlow","whileloopflow":"#/components/schemas/WhileloopFlow","branchone":"#/components/schemas/BranchOne","branchall":"#/components/schemas/BranchAll","identity":"#/components/schemas/Identity","aiagent":"#/components/schemas/AiAgent"}}},"RawScript":{"type":"object","description":"Inline script with code defined directly in the flow. Use 'bun' as default language if unspecified. The script receives arguments from input_transforms","properties":{"input_transforms":{"type":"object","description":"Map of parameter names to their values (static or JavaScript expressions). These become the script's input arguments","additionalProperties":{"$ref":"#/components/schemas/InputTransform"}},"content":{"type":"string","description":"The script source code. Should export a 'main' function"},"language":{"type":"string","description":"Programming language for this script","enum":["deno","bun","bunnative","python3","go","bash","powershell","postgresql","mysql","bigquery","snowflake","mssql","oracledb","graphql","nativets","php","rust","ansible","csharp","nu","java","ruby","rlang","duckdb"]},"path":{"type":"string","description":"Optional path for saving this script"},"lock":{"type":"string","description":"Lock file content for dependencies"},"type":{"type":"string","enum":["rawscript"]},"tag":{"type":"string","description":"Worker group tag for execution routing"},"concurrent_limit":{"type":"number","description":"Maximum concurrent executions of this script"},"concurrency_time_window_s":{"type":"number","description":"Time window for concurrent_limit"},"custom_concurrency_key":{"type":"string","description":"Custom key for grouping concurrent executions"},"is_trigger":{"type":"boolean","description":"If true, this script is a trigger that can start the flow"},"assets":{"type":"array","description":"External resources this script accesses (S3 objects, resources, etc.)","items":{"type":"object","required":["path","kind"],"properties":{"path":{"type":"string","description":"Path to the asset"},"kind":{"type":"string","description":"Type of asset","enum":["s3object","resource","ducklake","datatable","volume","dbt"]},"access_type":{"type":"string","nullable":true,"description":"Access level for this asset","enum":["r","w","rw"]},"alt_access_type":{"type":"string","nullable":true,"description":"Alternative access level","enum":["r","w","rw"]}}}}},"required":["type","content","language","input_transforms"]},"PathScript":{"type":"object","description":"Reference to an existing script by path. Use this when calling a previously saved script instead of writing inline code","properties":{"input_transforms":{"type":"object","description":"Map of parameter names to their values (static or JavaScript expressions). These become the script's input arguments","additionalProperties":{"$ref":"#/components/schemas/InputTransform"}},"path":{"type":"string","description":"Path to the script in the workspace (e.g., 'f/scripts/send_email')"},"hash":{"type":"string","description":"Optional specific version hash of the script to use"},"type":{"type":"string","enum":["script"]},"tag_override":{"type":"string","description":"Override the script's default worker group tag"},"is_trigger":{"type":"boolean","description":"If true, this script is a trigger that can start the flow"}},"required":["type","path","input_transforms"]},"PathFlow":{"type":"object","description":"Reference to an existing flow by path. Use this to call another flow as a subflow","properties":{"input_transforms":{"type":"object","description":"Map of parameter names to their values (static or JavaScript expressions). These become the subflow's input arguments","additionalProperties":{"$ref":"#/components/schemas/InputTransform"}},"path":{"type":"string","description":"Path to the flow in the workspace (e.g., 'f/flows/process_user')"},"type":{"type":"string","enum":["flow"]}},"required":["type","path","input_transforms"]},"ForloopFlow":{"type":"object","description":"Executes nested modules in a loop over an iterator. Inside the loop, use 'flow_input.iter.value' to access the current iteration value, and 'flow_input.iter.index' for the index. Supports parallel execution for better performance on I/O-bound operations","properties":{"modules":{"type":"array","description":"Steps to execute for each iteration. These can reference the iteration value via 'flow_input.iter.value'","items":{"$ref":"#/components/schemas/FlowModule"}},"iterator":{"description":"JavaScript expression that returns an array to iterate over. Can reference 'results.step_id' or 'flow_input'","$ref":"#/components/schemas/InputTransform"},"skip_failures":{"type":"boolean","description":"If true, iteration failures don't stop the loop. Failed iterations return null"},"type":{"type":"string","enum":["forloopflow"]},"parallel":{"type":"boolean","description":"If true, iterations run concurrently (faster for I/O-bound operations). Use with parallelism to control concurrency"},"parallelism":{"description":"Maximum number of concurrent iterations when parallel=true. Limits resource usage. Can be static number or expression","$ref":"#/components/schemas/InputTransform"},"squash":{"type":"boolean"}},"required":["modules","iterator","skip_failures","type"]},"WhileloopFlow":{"type":"object","description":"Executes nested modules repeatedly until stopped. The implicit iterator is the iteration counter, so 'flow_input.iter.value' equals 'flow_input.iter.index' (0, 1, 2, ...) and never carries state. To carry state across iterations, a step reads its own previous-iteration result via 'results.' with a first-iteration fallback - the loop's stop_after_if must then be on that inner step (a plain single-step body with stop_after_if on the loop module does not resolve 'results' across iterations and never terminates); plain counters can instead be derived from 'flow_input.iter.index', which works in every configuration. stop_after_if is evaluated after each iteration - on the loop module 'result' is the last iteration's result","properties":{"modules":{"type":"array","description":"Steps to execute in each iteration","items":{"$ref":"#/components/schemas/FlowModule"}},"skip_failures":{"type":"boolean","description":"If true, iteration failures don't stop the loop. Failed iterations return null"},"type":{"type":"string","enum":["whileloopflow"]},"parallel":{"type":"boolean","description":"If true, iterations run concurrently (use with caution in while loops)"},"parallelism":{"description":"Maximum number of concurrent iterations when parallel=true","$ref":"#/components/schemas/InputTransform"},"squash":{"type":"boolean"}},"required":["modules","skip_failures","type"]},"BranchOne":{"type":"object","description":"Conditional branching where only the first matching branch executes. Branches are evaluated in order, and the first one with a true expression runs. If no branches match, the default branch executes","properties":{"branches":{"type":"array","description":"Array of branches to evaluate in order. The first branch with expr evaluating to true executes","items":{"type":"object","properties":{"summary":{"type":"string","description":"Short description of this branch condition"},"expr":{"type":"string","description":"JavaScript expression that returns boolean. Can use 'results.step_id' or 'flow_input'. First true expr wins"},"modules":{"type":"array","description":"Steps to execute if this branch's expr is true","items":{"$ref":"#/components/schemas/FlowModule"}}},"required":["modules","expr"]}},"default":{"type":"array","description":"Steps to execute if no branch expressions match","items":{"$ref":"#/components/schemas/FlowModule"}},"type":{"type":"string","enum":["branchone"]}},"required":["branches","default","type"]},"BranchAll":{"type":"object","description":"Parallel branching where all branches execute simultaneously. Unlike BranchOne, all branches run regardless of conditions. Useful for executing independent tasks concurrently","properties":{"branches":{"type":"array","description":"Array of branches that all execute (either in parallel or sequentially)","items":{"type":"object","properties":{"summary":{"type":"string","description":"Short description of this branch's purpose"},"skip_failure":{"type":"boolean","description":"If true, failure in this branch doesn't fail the entire flow"},"modules":{"type":"array","description":"Steps to execute in this branch","items":{"$ref":"#/components/schemas/FlowModule"}}},"required":["modules"]}},"type":{"type":"string","enum":["branchall"]},"parallel":{"type":"boolean","description":"If true, all branches execute concurrently. If false, they execute sequentially"}},"required":["branches","type"]},"AgentTool":{"type":"object","description":"A tool available to an AI agent. Can be a flow module or an external MCP (Model Context Protocol) tool","properties":{"id":{"type":"string","description":"Unique identifier for this tool. Cannot contain spaces - use underscores instead (e.g., 'get_user_data' not 'get user data')"},"summary":{"type":"string","description":"The name the AI agent calls this tool by, not a human label. On a flowmodule tool it must match ^[a-zA-Z0-9_]+$ - letters, numbers and underscores only (e.g. 'search_documentation', not 'Search documentation') - and always be set; on an mcp or websearch tool it is a plain label. Put the human-readable explanation in 'description'."},"description":{"type":"string","description":"Free-text description of the tool given to the AI to decide when and how to call it. Overrides the description auto-derived from the underlying script."},"value":{"$ref":"#/components/schemas/ToolValue"}},"required":["id","value"]},"ToolValue":{"description":"The implementation of a tool. Can be a flow module (script/flow) or an MCP tool reference","oneOf":[{"$ref":"#/components/schemas/FlowModuleTool"},{"$ref":"#/components/schemas/McpToolValue"},{"$ref":"#/components/schemas/WebsearchToolValue"}],"discriminator":{"propertyName":"tool_type","mapping":{"flowmodule":"#/components/schemas/FlowModuleTool","mcp":"#/components/schemas/McpToolValue","websearch":"#/components/schemas/WebsearchToolValue"}}},"FlowModuleTool":{"description":"A tool implemented as a flow module (script, flow, etc.). The AI can call this like any other flow module","allOf":[{"type":"object","properties":{"tool_type":{"type":"string","enum":["flowmodule"]}},"required":["tool_type"]},{"$ref":"#/components/schemas/FlowModuleValue"}]},"WebsearchToolValue":{"type":"object","description":"A tool implemented as a websearch tool. The AI can call this like any other websearch tool","properties":{"tool_type":{"type":"string","enum":["websearch"]}},"required":["tool_type"]},"McpToolValue":{"type":"object","description":"Reference to an external MCP (Model Context Protocol) tool. The AI can call tools from MCP servers","properties":{"tool_type":{"type":"string","enum":["mcp"]},"resource_path":{"type":"string","description":"Path to the MCP resource/server configuration"},"include_tools":{"type":"array","description":"Whitelist of specific tools to include from this MCP server","items":{"type":"string"}},"exclude_tools":{"type":"array","description":"Blacklist of tools to exclude from this MCP server","items":{"type":"string"}}},"required":["tool_type","resource_path"]},"AiAgent":{"type":"object","description":"AI agent step that can use tools to accomplish tasks. The agent receives inputs and can call any of its configured tools to complete the task","properties":{"input_transforms":{"type":"object","description":"Input parameters for the AI agent mapped to their values","properties":{"provider":{"$ref":"#/components/schemas/ProviderTransform"},"output_type":{"allOf":[{"$ref":"#/components/schemas/InputTransform"}],"description":"Output format type.\\nValid values: 'text' (default) - plain text response, 'image' - image generation\\n"},"user_message":{"allOf":[{"$ref":"#/components/schemas/InputTransform"}],"description":"The user's prompt/message to the AI agent. Supports variable interpolation with flow.input syntax."},"system_prompt":{"allOf":[{"$ref":"#/components/schemas/InputTransform"}],"description":"System instructions that guide the AI's behavior, persona, and response style. Optional."},"streaming":{"allOf":[{"$ref":"#/components/schemas/InputTransform"}],"description":"Boolean. If true, stream the AI response incrementally.\\nStreaming events include: token_delta, reasoning_token_delta, tool_call, tool_call_arguments, tool_execution, tool_result\\n"},"memory":{"$ref":"#/components/schemas/MemoryTransform"},"output_schema":{"allOf":[{"$ref":"#/components/schemas/InputTransform"}],"description":"JSON Schema object defining structured output format. Used when you need the AI to return data in a specific shape.\\nSupports standard JSON Schema properties: type, properties, required, items, enum, pattern, minLength, maxLength, minimum, maximum, etc.\\nExample: { type: 'object', properties: { name: { type: 'string' }, age: { type: 'integer' } }, required: ['name'] }\\n"},"user_attachments":{"allOf":[{"$ref":"#/components/schemas/InputTransform"}],"description":"Array of file references (images or PDFs) for the AI agent.\\nFormat: Array<{ bucket: string, key: string }> - S3 object references\\nExample: [{ bucket: 'my-bucket', key: 'documents/report.pdf' }]\\n"},"max_completion_tokens":{"allOf":[{"$ref":"#/components/schemas/InputTransform"}],"description":"Integer. Maximum number of tokens the AI will generate in its response.\\nRange: 1 to 4,294,967,295. Typical values: 256-4096 for most use cases.\\n"},"temperature":{"allOf":[{"$ref":"#/components/schemas/InputTransform"}],"description":"Float. Controls randomness/creativity of responses.\\nRange: 0.0 to 2.0 (provider-dependent)\\n- 0.0 = deterministic, focused responses\\n- 0.7 = balanced (common default)\\n- 1.0+ = more creative/random\\n"},"max_iterations":{"allOf":[{"$ref":"#/components/schemas/InputTransform"}],"description":"Number. Limits how many times the agent can loop through reasoning and tool use.\\nRange: 1-1000.\\n"}},"required":["user_message"]},"tools":{"type":"array","description":"Array of tools the agent can use. The agent decides which tools to call based on the task","items":{"$ref":"#/components/schemas/AgentTool"}},"type":{"type":"string","enum":["aiagent"]},"tag":{"type":"string","description":"Worker group tag for execution routing. If not set, the AI agent step runs on the flow's tag (default \`flow\`)"},"omit_output_from_conversation":{"type":"boolean","default":false,"description":"If true, this AI agent step does not persist its assistant or tool messages to the flow conversation when chat mode is enabled."},"agent":{"type":"string","description":"Path of a reusable \`ai_agent\` resource (hybrid linking). When set, the agent brain\\nconfig (provider/model/system prompt/etc.) and tool set are resolved at runtime from\\nthat resource; the module's input_transforms then only carry the flow-local inputs\\n(user_message/user_attachments).\\n"},"tool_inputs":{"type":"object","description":"Host-local wiring for an agent's tool inputs, keyed by tool id then input key. Binds the\\nreferenced agent's tools to this flow's context (flow_input/results) without mutating the\\nshared resource; overlaid onto the tools' input_transforms at runtime \\u2014 including when\\n\`agent\` is unset, since a step forked for editing keeps these overrides until it is saved\\nback or unlinked.\\n","additionalProperties":{"type":"object","additionalProperties":{"$ref":"#/components/schemas/InputTransform"}}},"parallel":{"type":"boolean","description":"If true, the agent can execute multiple tool calls in parallel"}},"required":["type","input_transforms"]},"Identity":{"type":"object","description":"Pass-through module that returns its input unchanged. Useful for flow structure or as a placeholder","properties":{"type":{"type":"string","enum":["identity"]},"flow":{"type":"boolean","description":"If true, marks this as a flow identity (special handling)"}},"required":["type"]},"FlowStatus":{"type":"object","properties":{"step":{"type":"integer"},"modules":{"type":"array","items":{"$ref":"#/components/schemas/FlowStatusModule"}},"user_states":{"additionalProperties":true},"preprocessor_module":{"allOf":[{"$ref":"#/components/schemas/FlowStatusModule"}]},"failure_module":{"allOf":[{"$ref":"#/components/schemas/FlowStatusModule"},{"type":"object","properties":{"parent_module":{"type":"string"}}}]},"retry":{"type":"object","properties":{"fail_count":{"type":"integer"},"failed_jobs":{"type":"array","items":{"type":"string","format":"uuid"}}}}},"required":["step","modules","failure_module"]},"FlowStatusModule":{"type":"object","properties":{"type":{"type":"string","enum":["WaitingForPriorSteps","WaitingForEvents","WaitingForExecutor","InProgress","Success","Failure"]},"id":{"type":"string"},"job":{"type":"string","format":"uuid"},"count":{"type":"integer"},"progress":{"type":"integer"},"iterator":{"type":"object","properties":{"index":{"type":"integer"},"itered":{"type":"array","items":{}},"itered_len":{"type":"integer"},"args":{}}},"flow_jobs":{"type":"array","items":{"type":"string"}},"flow_jobs_success":{"type":"array","items":{"type":"boolean"}},"flow_jobs_duration":{"type":"object","properties":{"started_at":{"type":"array","items":{"type":"string"}},"duration_ms":{"type":"array","items":{"type":"integer"}}}},"branch_chosen":{"type":"object","properties":{"type":{"type":"string","enum":["branch","default"]},"branch":{"type":"integer"}},"required":["type"]},"branchall":{"type":"object","properties":{"branch":{"type":"integer"},"len":{"type":"integer"}},"required":["branch","len"]},"approvers":{"type":"array","items":{"type":"object","properties":{"resume_id":{"type":"integer"},"approver":{"type":"string"}},"required":["resume_id","approver"]}},"failed_retries":{"type":"array","items":{"type":"string","format":"uuid"}},"skipped":{"type":"boolean"},"agent_actions":{"type":"array","items":{"type":"object","oneOf":[{"type":"object","properties":{"job_id":{"type":"string","format":"uuid"},"function_name":{"type":"string"},"type":{"type":"string","enum":["tool_call"]},"module_id":{"type":"string"}},"required":["job_id","function_name","type","module_id"]},{"type":"object","properties":{"call_id":{"type":"string","format":"uuid"},"function_name":{"type":"string"},"resource_path":{"type":"string"},"type":{"type":"string","enum":["mcp_tool_call"]},"arguments":{"type":"object"}},"required":["call_id","function_name","resource_path","type"]},{"type":"object","properties":{"type":{"type":"string","enum":["web_search"]}},"required":["type"]},{"type":"object","properties":{"type":{"type":"string","enum":["message"]}},"required":["content","type"]}]}},"agent_actions_success":{"type":"array","items":{"type":"boolean"}}},"required":["type"]}}`, "raw-app": `--- name: raw-app description: MUST use when creating raw apps. diff --git a/frontend/src/lib/components/copilot/MetadataGen.svelte b/frontend/src/lib/components/copilot/MetadataGen.svelte index b541909e1b..52e781e4be 100644 --- a/frontend/src/lib/components/copilot/MetadataGen.svelte +++ b/frontend/src/lib/components/copilot/MetadataGen.svelte @@ -12,7 +12,7 @@ import { yamlStringifyExceptKeys } from './utils' import type { ChatCompletionMessageParam } from 'openai/resources/index.mjs' import { triggerableByAI } from '$lib/actions/triggerableByAI.svelte' - import { getToolNameError } from '$lib/components/graph/renderers/nodes/AIToolNode.svelte' + import { getToolNameError } from '$lib/components/flows/agentToolUtils' import { inputBaseClass, inputBorderClass, diff --git a/frontend/src/lib/components/copilot/chat/flow/core.ts b/frontend/src/lib/components/copilot/chat/flow/core.ts index bff934ba29..db21c4c746 100644 --- a/frontend/src/lib/components/copilot/chat/flow/core.ts +++ b/frontend/src/lib/components/copilot/chat/flow/core.ts @@ -1148,45 +1148,6 @@ Example: Before writing TypeScript/Bun code, call \`get_instructions_for_code_ge 3. **After making code changes, ALWAYS use \`get_lint_errors\` to check for issues.** Fix any errors before proceeding with testing. -### AI Agent Modules - -AI agents can use tools to accomplish tasks. When creating an AI agent module: - -\`\`\`javascript -{ - id: "support_agent", - summary: "AI agent for customer support", - value: { - type: "aiagent", - input_transforms: { - provider: { type: "static", value: "$res:f/ai_providers/openai" }, - output_type: { type: "static", value: "text" }, - user_message: { type: "javascript", expr: "flow_input.query" }, - system_prompt: { type: "static", value: "You are a helpful assistant." } - }, - tools: [ - { - id: "search_docs", - summary: "Search_documentation", - description: "Search the product documentation. Use it whenever the user asks how a feature works.", - value: { - tool_type: "flowmodule", - type: "rawscript", - language: "bun", - content: "export async function main(query: string) { return ['doc1', 'doc2']; }", - input_transforms: { query: { type: "static", value: "" } } - } - } - ] - } -} -\`\`\` - -- **Tool IDs**: Cannot contain spaces - use underscores -- **Tool summaries**: Cannot contain spaces - use underscores. This is the tool *name* the agent sees -- **Tool descriptions**: Optional free text telling the agent when and how to call the tool. Set it whenever the name alone does not make that obvious - it overrides the description derived from the underlying script -- **Tool types**: \`flowmodule\` for scripts/flows, \`mcp\` for MCP server tools - ### Contexts You have access to the following contexts: diff --git a/frontend/src/lib/components/copilot/chat/flow/editableFlowJson.test.ts b/frontend/src/lib/components/copilot/chat/flow/editableFlowJson.test.ts index 9a106f5131..7c47efb715 100644 --- a/frontend/src/lib/components/copilot/chat/flow/editableFlowJson.test.ts +++ b/frontend/src/lib/components/copilot/chat/flow/editableFlowJson.test.ts @@ -100,3 +100,58 @@ describe('flow settings in the compact editable view', () => { ).toThrow(/chat_input_enabled/) }) }) + +describe('AI agent tool names', () => { + function makeAgentFlow(toolSummary: string, extraTools: unknown[] = []) { + return { + modules: [ + { + id: 'support_agent', + value: { + type: 'aiagent', + input_transforms: { + provider: { + type: 'static', + value: { kind: 'openai', resource: '$res:f/ai/openai', model: 'gpt-4o' } + }, + output_type: { type: 'static', value: 'text' }, + user_message: { type: 'static', value: 'hi' } + }, + tools: [ + { + id: 'search_docs', + summary: toolSummary, + value: { + tool_type: 'flowmodule', + type: 'rawscript', + language: 'bun', + content: 'export async function main() { return 1 }', + input_transforms: {} + } + }, + ...extraTools + ] + } + } + ] + } + } + + it('rejects a tool name the worker would refuse at run time', () => { + expect(() => validateEditableFlowJson(makeAgentFlow('Search documentation'))).toThrow( + /Invalid AI agent tool name\(s\).*letters, numbers and underscores/s + ) + }) + + it('accepts an underscored tool name', () => { + expect(() => validateEditableFlowJson(makeAgentFlow('search_documentation'))).not.toThrow() + }) + + it('leaves mcp and websearch summaries alone - the worker never reads them as names', () => { + const flow = makeAgentFlow('search_documentation', [ + { id: 'mcp_tool', summary: '', value: { tool_type: 'mcp', resource_path: 'f/mcp/server' } }, + { id: 'websearch_tool', summary: 'Web Search', value: { tool_type: 'websearch' } } + ]) + expect(() => validateEditableFlowJson(flow)).not.toThrow() + }) +}) diff --git a/frontend/src/lib/components/copilot/chat/flow/editableFlowJson.ts b/frontend/src/lib/components/copilot/chat/flow/editableFlowJson.ts index 1258f94a93..5c551938c4 100644 --- a/frontend/src/lib/components/copilot/chat/flow/editableFlowJson.ts +++ b/frontend/src/lib/components/copilot/chat/flow/editableFlowJson.ts @@ -1,7 +1,10 @@ import { z } from 'zod' import type { FlowModule, FlowValue } from '$lib/gen' import { collectAllFlowModuleIdsFromModules } from '$lib/components/flows/flowTree' -import { collectProviderlessAgentIds } from '$lib/components/flows/agentToolTree' +import { + collectInvalidAgentToolNames, + collectProviderlessAgentIds +} from '$lib/components/flows/agentToolTree' import { SPECIAL_MODULE_IDS } from '../shared' import { findUnresolvedInlineScriptRefs, type InlineScriptSession } from './inlineScriptsUtils' import { @@ -268,6 +271,22 @@ export function validateFlowModules( ) } + // An agent tool's `summary` is the name the LLM sees; the worker rejects anything outside + // `^[a-zA-Z0-9_]+$`, so a flow written with a spaced name saves but fails on every run. + const invalidToolNames = collectInvalidAgentToolNames(parsedModules) + if (invalidToolNames.length > 0) { + throw new Error( + `Invalid AI agent tool name(s): ${invalidToolNames + .map( + (t) => + `agent "${t.agentId}" tool "${t.toolId}" is named ${JSON.stringify(t.name)} - ${t.error}` + ) + .join( + '; ' + )}. The tool's "summary" is the name the agent calls it by: use underscores instead of spaces (e.g. "search_docs").` + ) + } + return parsedModules } diff --git a/frontend/src/lib/components/copilot/chat/flow/openFlow.json b/frontend/src/lib/components/copilot/chat/flow/openFlow.json index da7dd9da5b..cab2839d48 100644 --- a/frontend/src/lib/components/copilot/chat/flow/openFlow.json +++ b/frontend/src/lib/components/copilot/chat/flow/openFlow.json @@ -1 +1 @@ -{"openapi":"3.0.3","info":{"version":"1.775.2","title":"OpenFlow Spec","contact":{"name":"Ruben Fiszel","email":"ruben@windmill.dev","url":"https://windmill.dev"},"license":{"name":"Apache 2.0","url":"https://www.apache.org/licenses/LICENSE-2.0.html"}},"paths":{},"externalDocs":{"description":"documentation portal","url":"https://windmill.dev"},"components":{"schemas":{"OpenFlow":{"type":"object","description":"Top-level flow definition containing metadata, configuration, and the flow structure","properties":{"summary":{"type":"string","description":"Short description of what this flow does"},"description":{"type":"string","description":"Detailed documentation for this flow"},"value":{"$ref":"#/components/schemas/FlowValue"},"schema":{"type":"object","description":"JSON Schema for flow inputs. Use this to define input parameters, their types, defaults, and validation. For resource inputs, set type to 'object' and format to 'resource-' (e.g., 'resource-stripe')"},"on_behalf_of_email":{"type":"string","description":"Address of the account the flow runs on behalf of. Derived from on_behalf_of on read; accepted on write, where it is resolved to the account it names."},"on_behalf_of":{"type":"string","description":"The flow runs with the permissions of this identity: u/{username}, g/{group}, or a bare email when the username is itself email-shaped. The only stored half of the identity; on_behalf_of_email is derived from it. Omit it when writing and it is resolved from that address instead."}},"required":["summary","value"]},"FlowValue":{"type":"object","description":"The flow structure containing modules and optional preprocessor/failure handlers","properties":{"modules":{"type":"array","description":"Array of steps that execute in sequence. Each step can be a script, subflow, loop, or branch","items":{"$ref":"#/components/schemas/FlowModule"}},"failure_module":{"description":"Special module that executes when the flow fails. Receives error object with message, name, stack, and step_id. Must have id 'failure'. Only supports script/rawscript types","$ref":"#/components/schemas/FlowModule"},"preprocessor_module":{"description":"Special module that runs before the first step on external triggers. Must have id 'preprocessor'. Only supports script/rawscript types. Cannot reference other step results","$ref":"#/components/schemas/FlowModule"},"same_worker":{"type":"boolean","description":"If true, all steps run on the same worker for better performance"},"preserve_step_tags":{"type":"boolean","description":"If true and the flow runs on a custom worker tag, steps that declare their own non-empty tag run on it instead of inheriting the flow tag. Steps without their own tag still inherit the flow tag."},"concurrent_limit":{"type":"number","description":"Maximum number of concurrent executions of this flow"},"concurrency_key":{"type":"string","description":"Expression to group concurrent executions (e.g., by user ID)"},"concurrency_time_window_s":{"type":"number","description":"Time window in seconds for concurrent_limit"},"debounce_delay_s":{"type":"integer","description":"Delay in seconds to debounce flow executions"},"debounce_key":{"type":"string","description":"Expression to group debounced executions"},"debounce_args_to_accumulate":{"type":"array","description":"Arguments to accumulate across debounced executions","items":{"type":"string"}},"max_total_debouncing_time":{"type":"integer","description":"Maximum total time in seconds that a job can be debounced"},"max_total_debounces_amount":{"type":"integer","description":"Maximum number of times a job can be debounced"},"skip_expr":{"type":"string","description":"JavaScript expression to conditionally skip the entire flow"},"cache_ttl":{"type":"number","description":"Cache duration in seconds for flow results"},"cache_ignore_s3_path":{"type":"boolean"},"delete_after_secs":{"type":"integer","description":"If set, delete the flow job's args, result and logs after this many seconds following job completion"},"flow_env":{"type":"object","description":"Environment variables available to all steps. Values can be strings, JSON values, or special references: '$var:path' (workspace variable) or '$res:path' (resource).","additionalProperties":{}},"priority":{"type":"number","description":"Execution priority (higher numbers run first)"},"early_return":{"type":"string","description":"JavaScript expression to return early from the flow"},"chat_input_enabled":{"type":"boolean","description":"Whether this flow accepts chat-style input"},"notes":{"type":"array","description":"Sticky notes attached to the flow","items":{"$ref":"#/components/schemas/FlowNote"}},"groups":{"type":"array","description":"Semantic groups of modules for organizational purposes","items":{"$ref":"#/components/schemas/FlowGroup"}}},"required":["modules"]},"Retry":{"type":"object","description":"Retry configuration for failed module executions","properties":{"constant":{"type":"object","description":"Retry with constant delay between attempts","properties":{"attempts":{"type":"integer","description":"Number of retry attempts"},"seconds":{"type":"integer","description":"Seconds to wait between retries"}}},"exponential":{"type":"object","description":"Retry with exponential backoff (delay doubles each time)","properties":{"attempts":{"type":"integer","description":"Number of retry attempts"},"multiplier":{"type":"integer","description":"Multiplier for exponential backoff"},"seconds":{"type":"integer","minimum":1,"description":"Initial delay in seconds"},"random_factor":{"type":"integer","minimum":0,"maximum":100,"description":"Random jitter percentage (0-100) to avoid thundering herd"}}},"retry_if":{"$ref":"#/components/schemas/RetryIf"}}},"FlowNote":{"type":"object","description":"A sticky note attached to a flow for documentation and annotation","properties":{"id":{"type":"string","description":"Unique identifier for the note"},"text":{"type":"string","description":"Content of the note"},"position":{"type":"object","description":"Position of the note in the flow editor","properties":{"x":{"type":"number","description":"X coordinate"},"y":{"type":"number","description":"Y coordinate"}},"required":["x","y"]},"size":{"type":"object","description":"Size of the note in the flow editor","properties":{"width":{"type":"number","description":"Width in pixels"},"height":{"type":"number","description":"Height in pixels"}},"required":["width","height"]},"color":{"type":"string","description":"Color of the note (e.g., \"yellow\", \"#ffff00\")"},"type":{"type":"string","enum":["free","group"],"description":"Type of note - 'free' for standalone notes, 'group' for notes that group other nodes"},"locked":{"type":"boolean","default":false,"description":"Whether the note is locked and cannot be edited or moved"},"contained_node_ids":{"type":"array","items":{"type":"string"},"description":"For group notes, the IDs of nodes contained within this group"}},"required":["id","text","color","type"]},"FlowGroup":{"type":"object","description":"A semantic group of flow modules for organizational purposes. Does not affect execution — modules remain in their original position in the flow. Groups provide naming and collapsibility in the editor. Members are computed dynamically from all nodes on paths between start_id and end_id.","properties":{"summary":{"type":"string","description":"Display name for this group"},"note":{"type":"string","description":"Markdown note shown below the group header"},"autocollapse":{"type":"boolean","default":false,"description":"If true, this group is collapsed by default in the flow editor. UI hint only."},"start_id":{"type":"string","description":"ID of the first flow module in this group (topological entry point)"},"end_id":{"type":"string","description":"ID of the last flow module in this group (topological exit point)"},"color":{"type":"string","description":"Color for the group in the flow editor"}},"required":["start_id","end_id"]},"RetryIf":{"type":"object","description":"Conditional retry based on error or result","properties":{"expr":{"type":"string","description":"JavaScript expression that returns true to retry. Has access to 'result' and 'error' variables"}},"required":["expr"]},"StopAfterIf":{"type":"object","description":"Early termination condition for a module","properties":{"skip_if_stopped":{"type":"boolean","description":"If true, following steps are skipped when this condition triggers"},"expr":{"type":"string","description":"JavaScript expression evaluated after the module runs. Can use 'result' (step's result) or 'flow_input'. Return true to stop"},"error_message":{"type":"string","nullable":true,"description":"Custom error message when stopping with an error. Mutually exclusive with skip_if_stopped. If set to a non-empty string, the flow stops with this error. If empty string, a default error message is used. If null or omitted, no error is raised."},"error_include_result":{"type":"boolean","description":"When stopping with an error (error_message set), embed the stopping step's own result inside the raised error object (as error.result) instead of discarding it. The top-level result stays { error }. Defaults to false."}},"required":["expr"]},"FlowModule":{"type":"object","description":"A single step in a flow. Can be a script, subflow, loop, or branch","properties":{"id":{"type":"string","description":"Unique identifier for this step. Used to reference results via 'results.step_id'. Must be a valid identifier (alphanumeric, underscore, hyphen)"},"value":{"$ref":"#/components/schemas/FlowModuleValue"},"stop_after_if":{"description":"Early termination condition evaluated after this step completes","$ref":"#/components/schemas/StopAfterIf"},"stop_after_all_iters_if":{"description":"For loops only - early termination condition evaluated after all iterations complete","$ref":"#/components/schemas/StopAfterIf"},"skip_if":{"type":"object","description":"Conditionally skip this step based on previous results or flow inputs","properties":{"expr":{"type":"string","description":"JavaScript expression that returns true to skip. Can use 'flow_input' or 'results.'"}},"required":["expr"]},"sleep":{"description":"Delay before executing this step (in seconds or as expression)","$ref":"#/components/schemas/InputTransform"},"cache_ttl":{"type":"number","description":"Cache duration in seconds for this step's results"},"cache_ignore_s3_path":{"type":"boolean"},"timeout":{"description":"Maximum execution time in seconds (static value or expression)","$ref":"#/components/schemas/InputTransform"},"delete_after_secs":{"type":"integer","description":"If set, delete the step's args, result and logs after this many seconds following job completion"},"summary":{"type":"string","description":"Short description of what this step does"},"mock":{"type":"object","description":"Mock configuration for testing without executing the actual step","properties":{"enabled":{"type":"boolean","description":"If true, return mock value instead of executing"},"return_value":{"description":"Value to return when mocked"}}},"suspend":{"type":"object","description":"Configuration for approval/resume steps that wait for user input","properties":{"required_events":{"type":"integer","description":"Number of approvals required before continuing"},"timeout":{"type":"integer","description":"Timeout in seconds before auto-continuing or canceling"},"resume_form":{"type":"object","description":"Form schema for collecting input when resuming","properties":{"schema":{"type":"object","description":"JSON Schema for the resume form"}}},"user_auth_required":{"type":"boolean","description":"If true, only authenticated users can approve"},"user_groups_required":{"description":"Expression or list of groups that can approve","$ref":"#/components/schemas/InputTransform"},"self_approval_disabled":{"type":"boolean","description":"If true, the user who started the flow cannot approve"},"hide_cancel":{"type":"boolean","description":"If true, hide the cancel button on the approval form"},"continue_on_disapprove_timeout":{"type":"boolean","description":"If true, continue flow on timeout instead of canceling"}}},"priority":{"type":"number","description":"Execution priority for this step (higher numbers run first)"},"continue_on_error":{"type":"boolean","description":"If true, flow continues even if this step fails"},"retry":{"description":"Retry configuration if this step fails","$ref":"#/components/schemas/Retry"},"debouncing":{"description":"Debounce configuration for this step (EE only)","type":"object","properties":{"debounce_delay_s":{"type":"integer","description":"Delay in seconds to debounce this step's executions across flow runs"},"debounce_key":{"type":"string","description":"Expression to group debounced executions. Supports $workspace and $args[name]. Default: $workspace/flow/-"},"debounce_args_to_accumulate":{"type":"array","description":"Array-type arguments to accumulate across debounced executions","items":{"type":"string"}},"max_total_debouncing_time":{"type":"integer","description":"Maximum total time in seconds before forced execution"},"max_total_debounces_amount":{"type":"integer","description":"Maximum number of debounces before forced execution"}}}},"required":["value","id"]},"InputTransform":{"description":"Maps input parameters for a step. Can be a static value or a JavaScript expression that references previous results or flow inputs","oneOf":[{"$ref":"#/components/schemas/StaticTransform"},{"$ref":"#/components/schemas/JavascriptTransform"},{"$ref":"#/components/schemas/AiTransform"}],"discriminator":{"propertyName":"type"}},"StaticTransform":{"type":"object","description":"Static value passed directly to the step. Use for hardcoded values or resource references like '$res:path/to/resource'","properties":{"value":{"description":"The static value. For resources, use format '$res:path/to/resource'"},"type":{"type":"string","enum":["static"]}},"required":["type"]},"JavascriptTransform":{"type":"object","description":"JavaScript expression evaluated at runtime. Can reference previous step results via 'results.step_id' or flow inputs via 'flow_input.property'. Inside for loops, use 'flow_input.iter.value' for the current iteration value (in while loops it equals 'flow_input.iter.index')","properties":{"expr":{"type":"string","description":"JavaScript expression returning the value. Available variables - results (object with all previous step results), flow_input (flow inputs), flow_input.iter (in loops)"},"type":{"type":"string","enum":["javascript"]}},"required":["expr","type"]},"AiTransform":{"type":"object","description":"Value resolved by the AI runtime for this input. The AI engine decides how to satisfy the parameter.","properties":{"type":{"type":"string","enum":["ai"]}},"required":["type"]},"AIProviderKind":{"type":"string","description":"Supported AI provider types","enum":["openai","azure_openai","azure_foundry","anthropic","mistral","deepseek","googleai","groq","openrouter","togetherai","customai","aws_bedrock"]},"ProviderConfig":{"type":"object","description":"Complete AI provider configuration with resource reference and model selection","properties":{"kind":{"$ref":"#/components/schemas/AIProviderKind"},"resource":{"type":"string","description":"Resource reference in format '$res:{resource_path}' pointing to provider credentials"},"model":{"type":"string","description":"Model identifier (e.g., 'gpt-4', 'claude-3-opus-20240229', 'gemini-pro')"},"reasoning_effort":{"type":"string","description":"Provider-native reasoning effort token (e.g. 'low', 'high', 'none') for models that support extended thinking. Optional; unset leaves the provider default."}},"required":["kind","resource","model"]},"StaticProviderTransform":{"type":"object","description":"Static provider configuration passed directly to the AI agent","properties":{"value":{"$ref":"#/components/schemas/ProviderConfig"},"type":{"type":"string","enum":["static"]}},"required":["type","value"]},"ProviderTransform":{"description":"Provider configuration - can be static (ProviderConfig), JavaScript expression, or AI-determined","oneOf":[{"$ref":"#/components/schemas/StaticProviderTransform"},{"$ref":"#/components/schemas/JavascriptTransform"},{"$ref":"#/components/schemas/AiTransform"}],"discriminator":{"propertyName":"type"}},"MemoryOff":{"type":"object","description":"No conversation memory/context","properties":{"kind":{"type":"string","enum":["off"]}},"required":["kind"]},"MemoryAuto":{"type":"object","description":"Automatic context management","properties":{"kind":{"type":"string","enum":["auto"]},"context_length":{"type":"integer","description":"Maximum number of messages to retain in context"},"memory_id":{"type":"string","description":"Identifier for persistent memory across agent invocations"}},"required":["kind"]},"MemoryMessage":{"type":"object","description":"A single message in conversation history","properties":{"role":{"type":"string","enum":["user","assistant","system"]},"content":{"type":"string"}},"required":["role","content"]},"MemoryManual":{"type":"object","description":"Explicit message history","properties":{"kind":{"type":"string","enum":["manual"]},"messages":{"type":"array","items":{"$ref":"#/components/schemas/MemoryMessage"}}},"required":["kind","messages"]},"MemoryConfig":{"description":"Conversation memory configuration","oneOf":[{"$ref":"#/components/schemas/MemoryOff"},{"$ref":"#/components/schemas/MemoryAuto"},{"$ref":"#/components/schemas/MemoryManual"}],"discriminator":{"propertyName":"kind"}},"StaticMemoryTransform":{"type":"object","description":"Static memory configuration passed directly to the AI agent","properties":{"value":{"$ref":"#/components/schemas/MemoryConfig"},"type":{"type":"string","enum":["static"]}},"required":["type","value"]},"MemoryTransform":{"description":"Memory configuration - can be static (MemoryConfig), JavaScript expression, or AI-determined","oneOf":[{"$ref":"#/components/schemas/StaticMemoryTransform"},{"$ref":"#/components/schemas/JavascriptTransform"},{"$ref":"#/components/schemas/AiTransform"}],"discriminator":{"propertyName":"type"}},"FlowModuleValue":{"description":"The actual implementation of a flow step. Can be a script (inline or referenced), subflow, loop, branch, or special module type","oneOf":[{"$ref":"#/components/schemas/RawScript"},{"$ref":"#/components/schemas/PathScript"},{"$ref":"#/components/schemas/PathFlow"},{"$ref":"#/components/schemas/ForloopFlow"},{"$ref":"#/components/schemas/WhileloopFlow"},{"$ref":"#/components/schemas/BranchOne"},{"$ref":"#/components/schemas/BranchAll"},{"$ref":"#/components/schemas/Identity"},{"$ref":"#/components/schemas/AiAgent"}],"discriminator":{"propertyName":"type"}},"RawScript":{"type":"object","description":"Inline script with code defined directly in the flow. Use 'bun' as default language if unspecified. The script receives arguments from input_transforms","properties":{"input_transforms":{"type":"object","description":"Map of parameter names to their values (static or JavaScript expressions). These become the script's input arguments","additionalProperties":{"$ref":"#/components/schemas/InputTransform"}},"content":{"type":"string","description":"The script source code. Should export a 'main' function"},"language":{"type":"string","description":"Programming language for this script","enum":["deno","bun","bunnative","python3","go","bash","powershell","postgresql","mysql","bigquery","snowflake","mssql","oracledb","graphql","nativets","php","rust","ansible","csharp","nu","java","ruby","rlang","duckdb"]},"path":{"type":"string","description":"Optional path for saving this script"},"lock":{"type":"string","description":"Lock file content for dependencies"},"type":{"type":"string","enum":["rawscript"]},"tag":{"type":"string","description":"Worker group tag for execution routing"},"concurrent_limit":{"type":"number","description":"Maximum concurrent executions of this script"},"concurrency_time_window_s":{"type":"number","description":"Time window for concurrent_limit"},"custom_concurrency_key":{"type":"string","description":"Custom key for grouping concurrent executions"},"is_trigger":{"type":"boolean","description":"If true, this script is a trigger that can start the flow"},"assets":{"type":"array","description":"External resources this script accesses (S3 objects, resources, etc.)","items":{"type":"object","required":["path","kind"],"properties":{"path":{"type":"string","description":"Path to the asset"},"kind":{"type":"string","description":"Type of asset","enum":["s3object","resource","ducklake","datatable","volume"]},"access_type":{"type":"string","nullable":true,"description":"Access level for this asset","enum":["r","w","rw",null]},"alt_access_type":{"type":"string","nullable":true,"description":"Alternative access level","enum":["r","w","rw",null]}}}}},"required":["type","content","language","input_transforms"]},"PathScript":{"type":"object","description":"Reference to an existing script by path. Use this when calling a previously saved script instead of writing inline code","properties":{"input_transforms":{"type":"object","description":"Map of parameter names to their values (static or JavaScript expressions). These become the script's input arguments","additionalProperties":{"$ref":"#/components/schemas/InputTransform"}},"path":{"type":"string","description":"Path to the script in the workspace (e.g., 'f/scripts/send_email')"},"hash":{"type":"string","description":"Optional specific version hash of the script to use"},"type":{"type":"string","enum":["script"]},"tag_override":{"type":"string","description":"Override the script's default worker group tag"},"is_trigger":{"type":"boolean","description":"If true, this script is a trigger that can start the flow"}},"required":["type","path","input_transforms"]},"PathFlow":{"type":"object","description":"Reference to an existing flow by path. Use this to call another flow as a subflow","properties":{"input_transforms":{"type":"object","description":"Map of parameter names to their values (static or JavaScript expressions). These become the subflow's input arguments","additionalProperties":{"$ref":"#/components/schemas/InputTransform"}},"path":{"type":"string","description":"Path to the flow in the workspace (e.g., 'f/flows/process_user')"},"type":{"type":"string","enum":["flow"]}},"required":["type","path","input_transforms"]},"ForloopFlow":{"type":"object","description":"Executes nested modules in a loop over an iterator. Inside the loop, use 'flow_input.iter.value' to access the current iteration value, and 'flow_input.iter.index' for the index. Supports parallel execution for better performance on I/O-bound operations","properties":{"modules":{"type":"array","description":"Steps to execute for each iteration. These can reference the iteration value via 'flow_input.iter.value'","items":{"$ref":"#/components/schemas/FlowModule"}},"iterator":{"description":"JavaScript expression that returns an array to iterate over. Can reference 'results.step_id' or 'flow_input'","$ref":"#/components/schemas/InputTransform"},"skip_failures":{"type":"boolean","description":"If true, iteration failures don't stop the loop. Failed iterations return null"},"type":{"type":"string","enum":["forloopflow"]},"parallel":{"type":"boolean","description":"If true, iterations run concurrently (faster for I/O-bound operations). Use with parallelism to control concurrency"},"parallelism":{"description":"Maximum number of concurrent iterations when parallel=true. Limits resource usage. Can be static number or expression","$ref":"#/components/schemas/InputTransform"},"squash":{"type":"boolean"}},"required":["modules","iterator","skip_failures","type"]},"WhileloopFlow":{"type":"object","description":"Executes nested modules repeatedly until stopped. The implicit iterator is the iteration counter, so 'flow_input.iter.value' equals 'flow_input.iter.index' (0, 1, 2, ...) and never carries state. To carry state across iterations, a step reads its own previous-iteration result via 'results.' with a first-iteration fallback - the loop's stop_after_if must then be on that inner step (a plain single-step body with stop_after_if on the loop module does not resolve 'results' across iterations and never terminates); plain counters can instead be derived from 'flow_input.iter.index', which works in every configuration. stop_after_if is evaluated after each iteration - on the loop module 'result' is the last iteration's result","properties":{"modules":{"type":"array","description":"Steps to execute in each iteration","items":{"$ref":"#/components/schemas/FlowModule"}},"skip_failures":{"type":"boolean","description":"If true, iteration failures don't stop the loop. Failed iterations return null"},"type":{"type":"string","enum":["whileloopflow"]},"parallel":{"type":"boolean","description":"If true, iterations run concurrently (use with caution in while loops)"},"parallelism":{"description":"Maximum number of concurrent iterations when parallel=true","$ref":"#/components/schemas/InputTransform"},"squash":{"type":"boolean"}},"required":["modules","skip_failures","type"]},"BranchOne":{"type":"object","description":"Conditional branching where only the first matching branch executes. Branches are evaluated in order, and the first one with a true expression runs. If no branches match, the default branch executes","properties":{"branches":{"type":"array","description":"Array of branches to evaluate in order. The first branch with expr evaluating to true executes","items":{"type":"object","properties":{"summary":{"type":"string","description":"Short description of this branch condition"},"expr":{"type":"string","description":"JavaScript expression that returns boolean. Can use 'results.step_id' or 'flow_input'. First true expr wins"},"modules":{"type":"array","description":"Steps to execute if this branch's expr is true","items":{"$ref":"#/components/schemas/FlowModule"}}},"required":["modules","expr"]}},"default":{"type":"array","description":"Steps to execute if no branch expressions match","items":{"$ref":"#/components/schemas/FlowModule"}},"type":{"type":"string","enum":["branchone"]}},"required":["branches","default","type"]},"BranchAll":{"type":"object","description":"Parallel branching where all branches execute simultaneously. Unlike BranchOne, all branches run regardless of conditions. Useful for executing independent tasks concurrently","properties":{"branches":{"type":"array","description":"Array of branches that all execute (either in parallel or sequentially)","items":{"type":"object","properties":{"summary":{"type":"string","description":"Short description of this branch's purpose"},"skip_failure":{"type":"boolean","description":"If true, failure in this branch doesn't fail the entire flow"},"modules":{"type":"array","description":"Steps to execute in this branch","items":{"$ref":"#/components/schemas/FlowModule"}}},"required":["modules"]}},"type":{"type":"string","enum":["branchall"]},"parallel":{"type":"boolean","description":"If true, all branches execute concurrently. If false, they execute sequentially"}},"required":["branches","type"]},"AgentTool":{"type":"object","description":"A tool available to an AI agent. Can be a flow module or an external MCP (Model Context Protocol) tool","properties":{"id":{"type":"string","description":"Unique identifier for this tool. Cannot contain spaces - use underscores instead (e.g., 'get_user_data' not 'get user data')"},"summary":{"type":"string","description":"Short description of what this tool does (shown to the AI)"},"description":{"type":"string","description":"Free-text description of the tool given to the AI to decide when and how to call it. Overrides the description auto-derived from the underlying script."},"value":{"$ref":"#/components/schemas/ToolValue"}},"required":["id","value"]},"ToolValue":{"description":"The implementation of a tool. Can be a flow module (script/flow) or an MCP tool reference","oneOf":[{"$ref":"#/components/schemas/FlowModuleTool"},{"$ref":"#/components/schemas/McpToolValue"},{"$ref":"#/components/schemas/WebsearchToolValue"}]},"FlowModuleTool":{"description":"A tool implemented as a flow module (script, flow, etc.). The AI can call this like any other flow module","allOf":[{"type":"object","properties":{"tool_type":{"type":"string","enum":["flowmodule"]}},"required":["tool_type"]},{"$ref":"#/components/schemas/FlowModuleValue"}]},"WebsearchToolValue":{"type":"object","description":"A tool implemented as a websearch tool. The AI can call this like any other websearch tool","properties":{"tool_type":{"type":"string","enum":["websearch"]}},"required":["tool_type"]},"McpToolValue":{"type":"object","description":"Reference to an external MCP (Model Context Protocol) tool. The AI can call tools from MCP servers","properties":{"tool_type":{"type":"string","enum":["mcp"]},"resource_path":{"type":"string","description":"Path to the MCP resource/server configuration"},"include_tools":{"type":"array","description":"Whitelist of specific tools to include from this MCP server","items":{"type":"string"}},"exclude_tools":{"type":"array","description":"Blacklist of tools to exclude from this MCP server","items":{"type":"string"}}},"required":["tool_type","resource_path"]},"AiAgent":{"type":"object","description":"AI agent step that can use tools to accomplish tasks. The agent receives inputs and can call any of its configured tools to complete the task","properties":{"input_transforms":{"type":"object","description":"Input parameters for the AI agent mapped to their values","properties":{"provider":{"$ref":"#/components/schemas/ProviderTransform"},"output_type":{"allOf":[{"$ref":"#/components/schemas/InputTransform"}],"description":"Output format type.\nValid values: 'text' (default) - plain text response, 'image' - image generation\n"},"user_message":{"allOf":[{"$ref":"#/components/schemas/InputTransform"}],"description":"The user's prompt/message to the AI agent. Supports variable interpolation with flow.input syntax."},"system_prompt":{"allOf":[{"$ref":"#/components/schemas/InputTransform"}],"description":"System instructions that guide the AI's behavior, persona, and response style. Optional."},"streaming":{"allOf":[{"$ref":"#/components/schemas/InputTransform"}],"description":"Boolean. If true, stream the AI response incrementally.\nStreaming events include: token_delta, reasoning_token_delta, tool_call, tool_call_arguments, tool_execution, tool_result\n"},"memory":{"$ref":"#/components/schemas/MemoryTransform"},"output_schema":{"allOf":[{"$ref":"#/components/schemas/InputTransform"}],"description":"JSON Schema object defining structured output format. Used when you need the AI to return data in a specific shape.\nSupports standard JSON Schema properties: type, properties, required, items, enum, pattern, minLength, maxLength, minimum, maximum, etc.\nExample: { type: 'object', properties: { name: { type: 'string' }, age: { type: 'integer' } }, required: ['name'] }\n"},"user_attachments":{"allOf":[{"$ref":"#/components/schemas/InputTransform"}],"description":"Array of file references (images or PDFs) for the AI agent.\nFormat: Array<{ bucket: string, key: string }> - S3 object references\nExample: [{ bucket: 'my-bucket', key: 'documents/report.pdf' }]\n"},"max_completion_tokens":{"allOf":[{"$ref":"#/components/schemas/InputTransform"}],"description":"Integer. Maximum number of tokens the AI will generate in its response.\nRange: 1 to 4,294,967,295. Typical values: 256-4096 for most use cases.\n"},"temperature":{"allOf":[{"$ref":"#/components/schemas/InputTransform"}],"description":"Float. Controls randomness/creativity of responses.\nRange: 0.0 to 2.0 (provider-dependent)\n- 0.0 = deterministic, focused responses\n- 0.7 = balanced (common default)\n- 1.0+ = more creative/random\n"},"max_iterations":{"allOf":[{"$ref":"#/components/schemas/InputTransform"}],"description":"Number. Limits how many times the agent can loop through reasoning and tool use.\nRange: 1-1000.\n"}},"required":["user_message"]},"tools":{"type":"array","description":"Array of tools the agent can use. The agent decides which tools to call based on the task","items":{"$ref":"#/components/schemas/AgentTool"}},"type":{"type":"string","enum":["aiagent"]},"tag":{"type":"string","description":"Worker group tag for execution routing. If not set, the AI agent step runs on the flow's tag (default `flow`)"},"omit_output_from_conversation":{"type":"boolean","default":false,"description":"If true, this AI agent step does not persist its assistant or tool messages to the flow conversation when chat mode is enabled."},"agent":{"type":"string","description":"Path of a reusable `ai_agent` resource (hybrid linking). When set, the agent brain\nconfig (provider/model/system prompt/etc.) and tool set are resolved at runtime from\nthat resource; the module's input_transforms then only carry the flow-local inputs\n(user_message/user_attachments).\n"},"tool_inputs":{"type":"object","description":"Host-local wiring for an agent's tool inputs, keyed by tool id then input key. Binds the\nreferenced agent's tools to this flow's context (flow_input/results) without mutating the\nshared resource; overlaid onto the tools' input_transforms at runtime — including when\n`agent` is unset, since a step forked for editing keeps these overrides until it is saved\nback or unlinked.\n","additionalProperties":{"type":"object","additionalProperties":{"$ref":"#/components/schemas/InputTransform"}}},"parallel":{"type":"boolean","description":"If true, the agent can execute multiple tool calls in parallel"}},"required":["type","input_transforms"]},"Identity":{"type":"object","description":"Pass-through module that returns its input unchanged. Useful for flow structure or as a placeholder","properties":{"type":{"type":"string","enum":["identity"]},"flow":{"type":"boolean","description":"If true, marks this as a flow identity (special handling)"}},"required":["type"]},"FlowStatus":{"type":"object","properties":{"step":{"type":"integer"},"modules":{"type":"array","items":{"$ref":"#/components/schemas/FlowStatusModule"}},"user_states":{"additionalProperties":true},"preprocessor_module":{"allOf":[{"$ref":"#/components/schemas/FlowStatusModule"}]},"failure_module":{"allOf":[{"$ref":"#/components/schemas/FlowStatusModule"},{"type":"object","properties":{"parent_module":{"type":"string"}}}]},"retry":{"type":"object","properties":{"fail_count":{"type":"integer"},"failed_jobs":{"type":"array","items":{"type":"string","format":"uuid"}}}}},"required":["step","modules","failure_module"]},"FlowStatusModule":{"type":"object","properties":{"type":{"type":"string","enum":["WaitingForPriorSteps","WaitingForEvents","WaitingForExecutor","InProgress","Success","Failure"]},"id":{"type":"string"},"job":{"type":"string","format":"uuid"},"count":{"type":"integer"},"progress":{"type":"integer"},"iterator":{"type":"object","properties":{"index":{"type":"integer"},"itered":{"type":"array","items":{}},"itered_len":{"type":"integer"},"args":{}}},"flow_jobs":{"type":"array","items":{"type":"string"}},"flow_jobs_success":{"type":"array","items":{"type":"boolean"}},"flow_jobs_duration":{"type":"object","properties":{"started_at":{"type":"array","items":{"type":"string"}},"duration_ms":{"type":"array","items":{"type":"integer"}}}},"branch_chosen":{"type":"object","properties":{"type":{"type":"string","enum":["branch","default"]},"branch":{"type":"integer"}},"required":["type"]},"branchall":{"type":"object","properties":{"branch":{"type":"integer"},"len":{"type":"integer"}},"required":["branch","len"]},"approvers":{"type":"array","items":{"type":"object","properties":{"resume_id":{"type":"integer"},"approver":{"type":"string"}},"required":["resume_id","approver"]}},"failed_retries":{"type":"array","items":{"type":"string","format":"uuid"}},"skipped":{"type":"boolean"},"agent_actions":{"type":"array","items":{"type":"object","oneOf":[{"type":"object","properties":{"job_id":{"type":"string","format":"uuid"},"function_name":{"type":"string"},"type":{"type":"string","enum":["tool_call"]},"module_id":{"type":"string"}},"required":["job_id","function_name","type","module_id"]},{"type":"object","properties":{"call_id":{"type":"string","format":"uuid"},"function_name":{"type":"string"},"resource_path":{"type":"string"},"type":{"type":"string","enum":["mcp_tool_call"]},"arguments":{"type":"object"}},"required":["call_id","function_name","resource_path","type"]},{"type":"object","properties":{"type":{"type":"string","enum":["web_search"]}},"required":["type"]},{"type":"object","properties":{"type":{"type":"string","enum":["message"]}},"required":["content","type"]}]}},"agent_actions_success":{"type":"array","items":{"type":"boolean"}}},"required":["type"]}}}} \ No newline at end of file +{"openapi":"3.0.3","info":{"version":"1.791.0","title":"OpenFlow Spec","contact":{"name":"Ruben Fiszel","email":"ruben@windmill.dev","url":"https://windmill.dev"},"license":{"name":"Apache 2.0","url":"https://www.apache.org/licenses/LICENSE-2.0.html"}},"paths":{},"externalDocs":{"description":"documentation portal","url":"https://windmill.dev"},"components":{"schemas":{"OpenFlow":{"type":"object","description":"Top-level flow definition containing metadata, configuration, and the flow structure","properties":{"summary":{"type":"string","description":"Short description of what this flow does"},"description":{"type":"string","description":"Detailed documentation for this flow"},"value":{"$ref":"#/components/schemas/FlowValue"},"schema":{"type":"object","description":"JSON Schema for flow inputs. Use this to define input parameters, their types, defaults, and validation. For resource inputs, set type to 'object' and format to 'resource-' (e.g., 'resource-stripe')"},"on_behalf_of_email":{"type":"string","description":"Address of the account the flow runs on behalf of. Derived from on_behalf_of on read; accepted on write, where it is resolved to the account it names."},"on_behalf_of":{"type":"string","description":"The flow runs with the permissions of this identity: u/{username}, g/{group}, or a bare email when the username is itself email-shaped. The only stored half of the identity; on_behalf_of_email is derived from it. Omit it when writing and it is resolved from that address instead."}},"required":["summary","value"]},"FlowValue":{"type":"object","description":"The flow structure containing modules and optional preprocessor/failure handlers","properties":{"modules":{"type":"array","description":"Array of steps that execute in sequence. Each step can be a script, subflow, loop, or branch","items":{"$ref":"#/components/schemas/FlowModule"}},"failure_module":{"description":"Special module that executes when the flow fails. Receives error object with message, name, stack, and step_id. Must have id 'failure'. Only supports script/rawscript types","$ref":"#/components/schemas/FlowModule"},"preprocessor_module":{"description":"Special module that runs before the first step on external triggers. Must have id 'preprocessor'. Only supports script/rawscript types. Cannot reference other step results","$ref":"#/components/schemas/FlowModule"},"same_worker":{"type":"boolean","description":"If true, all steps run on the same worker for better performance"},"preserve_step_tags":{"type":"boolean","description":"If true and the flow runs on a custom worker tag, steps that declare their own non-empty tag run on it instead of inheriting the flow tag. Steps without their own tag still inherit the flow tag."},"concurrent_limit":{"type":"number","description":"Maximum number of concurrent executions of this flow"},"concurrency_key":{"type":"string","description":"Expression to group concurrent executions (e.g., by user ID)"},"concurrency_time_window_s":{"type":"number","description":"Time window in seconds for concurrent_limit"},"debounce_delay_s":{"type":"integer","description":"Delay in seconds to debounce flow executions"},"debounce_key":{"type":"string","description":"Expression to group debounced executions"},"debounce_args_to_accumulate":{"type":"array","description":"Arguments to accumulate across debounced executions","items":{"type":"string"}},"max_total_debouncing_time":{"type":"integer","description":"Maximum total time in seconds that a job can be debounced"},"max_total_debounces_amount":{"type":"integer","description":"Maximum number of times a job can be debounced"},"skip_expr":{"type":"string","description":"JavaScript expression to conditionally skip the entire flow"},"cache_ttl":{"type":"number","description":"Cache duration in seconds for flow results"},"cache_ignore_s3_path":{"type":"boolean"},"delete_after_secs":{"type":"integer","description":"If set, delete the flow job's args, result and logs after this many seconds following job completion"},"flow_env":{"type":"object","description":"Environment variables available to all steps. Values can be strings, JSON values, or special references: '$var:path' (workspace variable) or '$res:path' (resource).","additionalProperties":{}},"priority":{"type":"number","description":"Execution priority (higher numbers run first)"},"early_return":{"type":"string","description":"JavaScript expression to return early from the flow"},"chat_input_enabled":{"type":"boolean","description":"Whether this flow accepts chat-style input"},"notes":{"type":"array","description":"Sticky notes attached to the flow","items":{"$ref":"#/components/schemas/FlowNote"}},"groups":{"type":"array","description":"Semantic groups of modules for organizational purposes","items":{"$ref":"#/components/schemas/FlowGroup"}}},"required":["modules"]},"Retry":{"type":"object","description":"Retry configuration for failed module executions","properties":{"constant":{"type":"object","description":"Retry with constant delay between attempts","properties":{"attempts":{"type":"integer","description":"Number of retry attempts"},"seconds":{"type":"integer","description":"Seconds to wait between retries"}}},"exponential":{"type":"object","description":"Retry with exponential backoff (delay doubles each time)","properties":{"attempts":{"type":"integer","description":"Number of retry attempts"},"multiplier":{"type":"integer","description":"Multiplier for exponential backoff"},"seconds":{"type":"integer","minimum":1,"description":"Initial delay in seconds"},"random_factor":{"type":"integer","minimum":0,"maximum":100,"description":"Random jitter percentage (0-100) to avoid thundering herd"}}},"retry_if":{"$ref":"#/components/schemas/RetryIf"}}},"FlowNote":{"type":"object","description":"A sticky note attached to a flow for documentation and annotation","properties":{"id":{"type":"string","description":"Unique identifier for the note"},"text":{"type":"string","description":"Content of the note"},"position":{"type":"object","description":"Position of the note in the flow editor","properties":{"x":{"type":"number","description":"X coordinate"},"y":{"type":"number","description":"Y coordinate"}},"required":["x","y"]},"size":{"type":"object","description":"Size of the note in the flow editor","properties":{"width":{"type":"number","description":"Width in pixels"},"height":{"type":"number","description":"Height in pixels"}},"required":["width","height"]},"color":{"type":"string","description":"Color of the note (e.g., \"yellow\", \"#ffff00\")"},"type":{"type":"string","enum":["free","group"],"description":"Type of note - 'free' for standalone notes, 'group' for notes that group other nodes"},"locked":{"type":"boolean","default":false,"description":"Whether the note is locked and cannot be edited or moved"},"contained_node_ids":{"type":"array","items":{"type":"string"},"description":"For group notes, the IDs of nodes contained within this group"}},"required":["id","text","color","type"]},"FlowGroup":{"type":"object","description":"A semantic group of flow modules for organizational purposes. Does not affect execution — modules remain in their original position in the flow. Groups provide naming and collapsibility in the editor. Members are computed dynamically from all nodes on paths between start_id and end_id.","properties":{"summary":{"type":"string","description":"Display name for this group"},"note":{"type":"string","description":"Markdown note shown below the group header"},"autocollapse":{"type":"boolean","default":false,"description":"If true, this group is collapsed by default in the flow editor. UI hint only."},"start_id":{"type":"string","description":"ID of the first flow module in this group (topological entry point)"},"end_id":{"type":"string","description":"ID of the last flow module in this group (topological exit point)"},"color":{"type":"string","description":"Color for the group in the flow editor"}},"required":["start_id","end_id"]},"RetryIf":{"type":"object","description":"Conditional retry based on error or result","properties":{"expr":{"type":"string","description":"JavaScript expression that returns true to retry. Has access to 'result' and 'error' variables"}},"required":["expr"]},"StopAfterIf":{"type":"object","description":"Early termination condition for a module","properties":{"skip_if_stopped":{"type":"boolean","description":"If true, following steps are skipped when this condition triggers"},"expr":{"type":"string","description":"JavaScript expression evaluated after the module runs. Can use 'result' (step's result) or 'flow_input'. Return true to stop"},"error_message":{"type":"string","nullable":true,"description":"Custom error message when stopping with an error. Mutually exclusive with skip_if_stopped. If set to a non-empty string, the flow stops with this error. If empty string, a default error message is used. If null or omitted, no error is raised."},"error_include_result":{"type":"boolean","description":"When stopping with an error (error_message set), embed the stopping step's own result inside the raised error object (as error.result) instead of discarding it. The top-level result stays { error }. Defaults to false."}},"required":["expr"]},"FlowModule":{"type":"object","description":"A single step in a flow. Can be a script, subflow, loop, or branch","properties":{"id":{"type":"string","description":"Unique identifier for this step. Used to reference results via 'results.step_id'. Must be a valid identifier (alphanumeric, underscore, hyphen)"},"value":{"$ref":"#/components/schemas/FlowModuleValue"},"stop_after_if":{"description":"Early termination condition evaluated after this step completes","$ref":"#/components/schemas/StopAfterIf"},"stop_after_all_iters_if":{"description":"For loops only - early termination condition evaluated after all iterations complete","$ref":"#/components/schemas/StopAfterIf"},"skip_if":{"type":"object","description":"Conditionally skip this step based on previous results or flow inputs","properties":{"expr":{"type":"string","description":"JavaScript expression that returns true to skip. Can use 'flow_input' or 'results.'"}},"required":["expr"]},"sleep":{"description":"Delay before executing this step (in seconds or as expression)","$ref":"#/components/schemas/InputTransform"},"cache_ttl":{"type":"number","description":"Cache duration in seconds for this step's results"},"cache_ignore_s3_path":{"type":"boolean"},"timeout":{"description":"Maximum execution time in seconds (static value or expression)","$ref":"#/components/schemas/InputTransform"},"delete_after_secs":{"type":"integer","description":"If set, delete the step's args, result and logs after this many seconds following job completion"},"summary":{"type":"string","description":"Short description of what this step does"},"mock":{"type":"object","description":"Mock configuration for testing without executing the actual step","properties":{"enabled":{"type":"boolean","description":"If true, return mock value instead of executing"},"return_value":{"description":"Value to return when mocked"}}},"suspend":{"type":"object","description":"Configuration for approval/resume steps that wait for user input","properties":{"required_events":{"type":"integer","description":"Number of approvals required before continuing"},"timeout":{"type":"integer","description":"Timeout in seconds before auto-continuing or canceling"},"resume_form":{"type":"object","description":"Form schema for collecting input when resuming","properties":{"schema":{"type":"object","description":"JSON Schema for the resume form"}}},"user_auth_required":{"type":"boolean","description":"If true, only authenticated users can approve"},"user_groups_required":{"description":"Expression or list of groups that can approve","$ref":"#/components/schemas/InputTransform"},"self_approval_disabled":{"type":"boolean","description":"If true, the user who started the flow cannot approve"},"hide_cancel":{"type":"boolean","description":"If true, hide the cancel button on the approval form"},"continue_on_disapprove_timeout":{"type":"boolean","description":"If true, continue flow on timeout instead of canceling"}}},"priority":{"type":"number","description":"Execution priority for this step (higher numbers run first)"},"continue_on_error":{"type":"boolean","description":"If true, flow continues even if this step fails"},"retry":{"description":"Retry configuration if this step fails","$ref":"#/components/schemas/Retry"},"debouncing":{"description":"Debounce configuration for this step (EE only)","type":"object","properties":{"debounce_delay_s":{"type":"integer","description":"Delay in seconds to debounce this step's executions across flow runs"},"debounce_key":{"type":"string","description":"Expression to group debounced executions. Supports $workspace and $args[name]. Default: $workspace/flow/-"},"debounce_args_to_accumulate":{"type":"array","description":"Array-type arguments to accumulate across debounced executions","items":{"type":"string"}},"max_total_debouncing_time":{"type":"integer","description":"Maximum total time in seconds before forced execution"},"max_total_debounces_amount":{"type":"integer","description":"Maximum number of debounces before forced execution"}}}},"required":["value","id"]},"InputTransform":{"description":"Maps input parameters for a step. Can be a static value or a JavaScript expression that references previous results or flow inputs","oneOf":[{"$ref":"#/components/schemas/StaticTransform"},{"$ref":"#/components/schemas/JavascriptTransform"},{"$ref":"#/components/schemas/AiTransform"}],"discriminator":{"propertyName":"type"}},"StaticTransform":{"type":"object","description":"Static value passed directly to the step. Use for hardcoded values or resource references like '$res:path/to/resource'","properties":{"value":{"description":"The static value. For resources, use format '$res:path/to/resource'"},"type":{"type":"string","enum":["static"]}},"required":["type"]},"JavascriptTransform":{"type":"object","description":"JavaScript expression evaluated at runtime. Can reference previous step results via 'results.step_id' or flow inputs via 'flow_input.property'. Inside for loops, use 'flow_input.iter.value' for the current iteration value (in while loops it equals 'flow_input.iter.index')","properties":{"expr":{"type":"string","description":"JavaScript expression returning the value. Available variables - results (object with all previous step results), flow_input (flow inputs), flow_input.iter (in loops)"},"type":{"type":"string","enum":["javascript"]}},"required":["expr","type"]},"AiTransform":{"type":"object","description":"Value resolved by the AI runtime for this input. The AI engine decides how to satisfy the parameter.","properties":{"type":{"type":"string","enum":["ai"]}},"required":["type"]},"AIProviderKind":{"type":"string","description":"Supported AI provider types","enum":["openai","azure_openai","azure_foundry","anthropic","mistral","deepseek","googleai","groq","openrouter","togetherai","customai","aws_bedrock"]},"ProviderConfig":{"type":"object","description":"Complete AI provider configuration with resource reference and model selection","properties":{"kind":{"$ref":"#/components/schemas/AIProviderKind"},"resource":{"type":"string","description":"Resource reference in format '$res:{resource_path}' pointing to provider credentials"},"model":{"type":"string","description":"Model identifier (e.g., 'gpt-4', 'claude-3-opus-20240229', 'gemini-pro')"},"reasoning_effort":{"type":"string","description":"Provider-native reasoning effort token (e.g. 'low', 'high', 'none') for models that support extended thinking. Optional; unset leaves the provider default."}},"required":["kind","resource","model"]},"StaticProviderTransform":{"type":"object","description":"Static provider configuration passed directly to the AI agent","properties":{"value":{"$ref":"#/components/schemas/ProviderConfig"},"type":{"type":"string","enum":["static"]}},"required":["type","value"]},"ProviderTransform":{"description":"Provider configuration - can be static (ProviderConfig), JavaScript expression, or AI-determined","oneOf":[{"$ref":"#/components/schemas/StaticProviderTransform"},{"$ref":"#/components/schemas/JavascriptTransform"},{"$ref":"#/components/schemas/AiTransform"}],"discriminator":{"propertyName":"type"}},"MemoryOff":{"type":"object","description":"No conversation memory/context","properties":{"kind":{"type":"string","enum":["off"]}},"required":["kind"]},"MemoryAuto":{"type":"object","description":"Automatic context management","properties":{"kind":{"type":"string","enum":["auto"]},"context_length":{"type":"integer","description":"Maximum number of messages to retain in context"},"memory_id":{"type":"string","description":"Identifier for persistent memory across agent invocations"}},"required":["kind"]},"MemoryMessage":{"type":"object","description":"A single message in conversation history","properties":{"role":{"type":"string","enum":["user","assistant","system"]},"content":{"type":"string"}},"required":["role","content"]},"MemoryManual":{"type":"object","description":"Explicit message history","properties":{"kind":{"type":"string","enum":["manual"]},"messages":{"type":"array","items":{"$ref":"#/components/schemas/MemoryMessage"}}},"required":["kind","messages"]},"MemoryConfig":{"description":"Conversation memory configuration","oneOf":[{"$ref":"#/components/schemas/MemoryOff"},{"$ref":"#/components/schemas/MemoryAuto"},{"$ref":"#/components/schemas/MemoryManual"}],"discriminator":{"propertyName":"kind"}},"StaticMemoryTransform":{"type":"object","description":"Static memory configuration passed directly to the AI agent","properties":{"value":{"$ref":"#/components/schemas/MemoryConfig"},"type":{"type":"string","enum":["static"]}},"required":["type","value"]},"MemoryTransform":{"description":"Memory configuration - can be static (MemoryConfig), JavaScript expression, or AI-determined","oneOf":[{"$ref":"#/components/schemas/StaticMemoryTransform"},{"$ref":"#/components/schemas/JavascriptTransform"},{"$ref":"#/components/schemas/AiTransform"}],"discriminator":{"propertyName":"type"}},"FlowModuleValue":{"description":"The actual implementation of a flow step. Can be a script (inline or referenced), subflow, loop, branch, or special module type","oneOf":[{"$ref":"#/components/schemas/RawScript"},{"$ref":"#/components/schemas/PathScript"},{"$ref":"#/components/schemas/PathFlow"},{"$ref":"#/components/schemas/ForloopFlow"},{"$ref":"#/components/schemas/WhileloopFlow"},{"$ref":"#/components/schemas/BranchOne"},{"$ref":"#/components/schemas/BranchAll"},{"$ref":"#/components/schemas/Identity"},{"$ref":"#/components/schemas/AiAgent"}],"discriminator":{"propertyName":"type"}},"RawScript":{"type":"object","description":"Inline script with code defined directly in the flow. Use 'bun' as default language if unspecified. The script receives arguments from input_transforms","properties":{"input_transforms":{"type":"object","description":"Map of parameter names to their values (static or JavaScript expressions). These become the script's input arguments","additionalProperties":{"$ref":"#/components/schemas/InputTransform"}},"content":{"type":"string","description":"The script source code. Should export a 'main' function"},"language":{"type":"string","description":"Programming language for this script","enum":["deno","bun","bunnative","python3","go","bash","powershell","postgresql","mysql","bigquery","snowflake","mssql","oracledb","graphql","nativets","php","rust","ansible","csharp","nu","java","ruby","rlang","duckdb"]},"path":{"type":"string","description":"Optional path for saving this script"},"lock":{"type":"string","description":"Lock file content for dependencies"},"type":{"type":"string","enum":["rawscript"]},"tag":{"type":"string","description":"Worker group tag for execution routing"},"concurrent_limit":{"type":"number","description":"Maximum concurrent executions of this script"},"concurrency_time_window_s":{"type":"number","description":"Time window for concurrent_limit"},"custom_concurrency_key":{"type":"string","description":"Custom key for grouping concurrent executions"},"is_trigger":{"type":"boolean","description":"If true, this script is a trigger that can start the flow"},"assets":{"type":"array","description":"External resources this script accesses (S3 objects, resources, etc.)","items":{"type":"object","required":["path","kind"],"properties":{"path":{"type":"string","description":"Path to the asset"},"kind":{"type":"string","description":"Type of asset","enum":["s3object","resource","ducklake","datatable","volume","dbt"]},"access_type":{"type":"string","nullable":true,"description":"Access level for this asset","enum":["r","w","rw",null]},"alt_access_type":{"type":"string","nullable":true,"description":"Alternative access level","enum":["r","w","rw",null]}}}}},"required":["type","content","language","input_transforms"]},"PathScript":{"type":"object","description":"Reference to an existing script by path. Use this when calling a previously saved script instead of writing inline code","properties":{"input_transforms":{"type":"object","description":"Map of parameter names to their values (static or JavaScript expressions). These become the script's input arguments","additionalProperties":{"$ref":"#/components/schemas/InputTransform"}},"path":{"type":"string","description":"Path to the script in the workspace (e.g., 'f/scripts/send_email')"},"hash":{"type":"string","description":"Optional specific version hash of the script to use"},"type":{"type":"string","enum":["script"]},"tag_override":{"type":"string","description":"Override the script's default worker group tag"},"is_trigger":{"type":"boolean","description":"If true, this script is a trigger that can start the flow"}},"required":["type","path","input_transforms"]},"PathFlow":{"type":"object","description":"Reference to an existing flow by path. Use this to call another flow as a subflow","properties":{"input_transforms":{"type":"object","description":"Map of parameter names to their values (static or JavaScript expressions). These become the subflow's input arguments","additionalProperties":{"$ref":"#/components/schemas/InputTransform"}},"path":{"type":"string","description":"Path to the flow in the workspace (e.g., 'f/flows/process_user')"},"type":{"type":"string","enum":["flow"]}},"required":["type","path","input_transforms"]},"ForloopFlow":{"type":"object","description":"Executes nested modules in a loop over an iterator. Inside the loop, use 'flow_input.iter.value' to access the current iteration value, and 'flow_input.iter.index' for the index. Supports parallel execution for better performance on I/O-bound operations","properties":{"modules":{"type":"array","description":"Steps to execute for each iteration. These can reference the iteration value via 'flow_input.iter.value'","items":{"$ref":"#/components/schemas/FlowModule"}},"iterator":{"description":"JavaScript expression that returns an array to iterate over. Can reference 'results.step_id' or 'flow_input'","$ref":"#/components/schemas/InputTransform"},"skip_failures":{"type":"boolean","description":"If true, iteration failures don't stop the loop. Failed iterations return null"},"type":{"type":"string","enum":["forloopflow"]},"parallel":{"type":"boolean","description":"If true, iterations run concurrently (faster for I/O-bound operations). Use with parallelism to control concurrency"},"parallelism":{"description":"Maximum number of concurrent iterations when parallel=true. Limits resource usage. Can be static number or expression","$ref":"#/components/schemas/InputTransform"},"squash":{"type":"boolean"}},"required":["modules","iterator","skip_failures","type"]},"WhileloopFlow":{"type":"object","description":"Executes nested modules repeatedly until stopped. The implicit iterator is the iteration counter, so 'flow_input.iter.value' equals 'flow_input.iter.index' (0, 1, 2, ...) and never carries state. To carry state across iterations, a step reads its own previous-iteration result via 'results.' with a first-iteration fallback - the loop's stop_after_if must then be on that inner step (a plain single-step body with stop_after_if on the loop module does not resolve 'results' across iterations and never terminates); plain counters can instead be derived from 'flow_input.iter.index', which works in every configuration. stop_after_if is evaluated after each iteration - on the loop module 'result' is the last iteration's result","properties":{"modules":{"type":"array","description":"Steps to execute in each iteration","items":{"$ref":"#/components/schemas/FlowModule"}},"skip_failures":{"type":"boolean","description":"If true, iteration failures don't stop the loop. Failed iterations return null"},"type":{"type":"string","enum":["whileloopflow"]},"parallel":{"type":"boolean","description":"If true, iterations run concurrently (use with caution in while loops)"},"parallelism":{"description":"Maximum number of concurrent iterations when parallel=true","$ref":"#/components/schemas/InputTransform"},"squash":{"type":"boolean"}},"required":["modules","skip_failures","type"]},"BranchOne":{"type":"object","description":"Conditional branching where only the first matching branch executes. Branches are evaluated in order, and the first one with a true expression runs. If no branches match, the default branch executes","properties":{"branches":{"type":"array","description":"Array of branches to evaluate in order. The first branch with expr evaluating to true executes","items":{"type":"object","properties":{"summary":{"type":"string","description":"Short description of this branch condition"},"expr":{"type":"string","description":"JavaScript expression that returns boolean. Can use 'results.step_id' or 'flow_input'. First true expr wins"},"modules":{"type":"array","description":"Steps to execute if this branch's expr is true","items":{"$ref":"#/components/schemas/FlowModule"}}},"required":["modules","expr"]}},"default":{"type":"array","description":"Steps to execute if no branch expressions match","items":{"$ref":"#/components/schemas/FlowModule"}},"type":{"type":"string","enum":["branchone"]}},"required":["branches","default","type"]},"BranchAll":{"type":"object","description":"Parallel branching where all branches execute simultaneously. Unlike BranchOne, all branches run regardless of conditions. Useful for executing independent tasks concurrently","properties":{"branches":{"type":"array","description":"Array of branches that all execute (either in parallel or sequentially)","items":{"type":"object","properties":{"summary":{"type":"string","description":"Short description of this branch's purpose"},"skip_failure":{"type":"boolean","description":"If true, failure in this branch doesn't fail the entire flow"},"modules":{"type":"array","description":"Steps to execute in this branch","items":{"$ref":"#/components/schemas/FlowModule"}}},"required":["modules"]}},"type":{"type":"string","enum":["branchall"]},"parallel":{"type":"boolean","description":"If true, all branches execute concurrently. If false, they execute sequentially"}},"required":["branches","type"]},"AgentTool":{"type":"object","description":"A tool available to an AI agent. Can be a flow module or an external MCP (Model Context Protocol) tool","properties":{"id":{"type":"string","description":"Unique identifier for this tool. Cannot contain spaces - use underscores instead (e.g., 'get_user_data' not 'get user data')"},"summary":{"type":"string","description":"The name the AI agent calls this tool by, not a human label. On a flowmodule tool it must match ^[a-zA-Z0-9_]+$ - letters, numbers and underscores only (e.g. 'search_documentation', not 'Search documentation') - and always be set; on an mcp or websearch tool it is a plain label. Put the human-readable explanation in 'description'."},"description":{"type":"string","description":"Free-text description of the tool given to the AI to decide when and how to call it. Overrides the description auto-derived from the underlying script."},"value":{"$ref":"#/components/schemas/ToolValue"}},"required":["id","value"]},"ToolValue":{"description":"The implementation of a tool. Can be a flow module (script/flow) or an MCP tool reference","oneOf":[{"$ref":"#/components/schemas/FlowModuleTool"},{"$ref":"#/components/schemas/McpToolValue"},{"$ref":"#/components/schemas/WebsearchToolValue"}]},"FlowModuleTool":{"description":"A tool implemented as a flow module (script, flow, etc.). The AI can call this like any other flow module","allOf":[{"type":"object","properties":{"tool_type":{"type":"string","enum":["flowmodule"]}},"required":["tool_type"]},{"$ref":"#/components/schemas/FlowModuleValue"}]},"WebsearchToolValue":{"type":"object","description":"A tool implemented as a websearch tool. The AI can call this like any other websearch tool","properties":{"tool_type":{"type":"string","enum":["websearch"]}},"required":["tool_type"]},"McpToolValue":{"type":"object","description":"Reference to an external MCP (Model Context Protocol) tool. The AI can call tools from MCP servers","properties":{"tool_type":{"type":"string","enum":["mcp"]},"resource_path":{"type":"string","description":"Path to the MCP resource/server configuration"},"include_tools":{"type":"array","description":"Whitelist of specific tools to include from this MCP server","items":{"type":"string"}},"exclude_tools":{"type":"array","description":"Blacklist of tools to exclude from this MCP server","items":{"type":"string"}}},"required":["tool_type","resource_path"]},"AiAgent":{"type":"object","description":"AI agent step that can use tools to accomplish tasks. The agent receives inputs and can call any of its configured tools to complete the task","properties":{"input_transforms":{"type":"object","description":"Input parameters for the AI agent mapped to their values","properties":{"provider":{"$ref":"#/components/schemas/ProviderTransform"},"output_type":{"allOf":[{"$ref":"#/components/schemas/InputTransform"}],"description":"Output format type.\nValid values: 'text' (default) - plain text response, 'image' - image generation\n"},"user_message":{"allOf":[{"$ref":"#/components/schemas/InputTransform"}],"description":"The user's prompt/message to the AI agent. Supports variable interpolation with flow.input syntax."},"system_prompt":{"allOf":[{"$ref":"#/components/schemas/InputTransform"}],"description":"System instructions that guide the AI's behavior, persona, and response style. Optional."},"streaming":{"allOf":[{"$ref":"#/components/schemas/InputTransform"}],"description":"Boolean. If true, stream the AI response incrementally.\nStreaming events include: token_delta, reasoning_token_delta, tool_call, tool_call_arguments, tool_execution, tool_result\n"},"memory":{"$ref":"#/components/schemas/MemoryTransform"},"output_schema":{"allOf":[{"$ref":"#/components/schemas/InputTransform"}],"description":"JSON Schema object defining structured output format. Used when you need the AI to return data in a specific shape.\nSupports standard JSON Schema properties: type, properties, required, items, enum, pattern, minLength, maxLength, minimum, maximum, etc.\nExample: { type: 'object', properties: { name: { type: 'string' }, age: { type: 'integer' } }, required: ['name'] }\n"},"user_attachments":{"allOf":[{"$ref":"#/components/schemas/InputTransform"}],"description":"Array of file references (images or PDFs) for the AI agent.\nFormat: Array<{ bucket: string, key: string }> - S3 object references\nExample: [{ bucket: 'my-bucket', key: 'documents/report.pdf' }]\n"},"max_completion_tokens":{"allOf":[{"$ref":"#/components/schemas/InputTransform"}],"description":"Integer. Maximum number of tokens the AI will generate in its response.\nRange: 1 to 4,294,967,295. Typical values: 256-4096 for most use cases.\n"},"temperature":{"allOf":[{"$ref":"#/components/schemas/InputTransform"}],"description":"Float. Controls randomness/creativity of responses.\nRange: 0.0 to 2.0 (provider-dependent)\n- 0.0 = deterministic, focused responses\n- 0.7 = balanced (common default)\n- 1.0+ = more creative/random\n"},"max_iterations":{"allOf":[{"$ref":"#/components/schemas/InputTransform"}],"description":"Number. Limits how many times the agent can loop through reasoning and tool use.\nRange: 1-1000.\n"}},"required":["user_message"]},"tools":{"type":"array","description":"Array of tools the agent can use. The agent decides which tools to call based on the task","items":{"$ref":"#/components/schemas/AgentTool"}},"type":{"type":"string","enum":["aiagent"]},"tag":{"type":"string","description":"Worker group tag for execution routing. If not set, the AI agent step runs on the flow's tag (default `flow`)"},"omit_output_from_conversation":{"type":"boolean","default":false,"description":"If true, this AI agent step does not persist its assistant or tool messages to the flow conversation when chat mode is enabled."},"agent":{"type":"string","description":"Path of a reusable `ai_agent` resource (hybrid linking). When set, the agent brain\nconfig (provider/model/system prompt/etc.) and tool set are resolved at runtime from\nthat resource; the module's input_transforms then only carry the flow-local inputs\n(user_message/user_attachments).\n"},"tool_inputs":{"type":"object","description":"Host-local wiring for an agent's tool inputs, keyed by tool id then input key. Binds the\nreferenced agent's tools to this flow's context (flow_input/results) without mutating the\nshared resource; overlaid onto the tools' input_transforms at runtime — including when\n`agent` is unset, since a step forked for editing keeps these overrides until it is saved\nback or unlinked.\n","additionalProperties":{"type":"object","additionalProperties":{"$ref":"#/components/schemas/InputTransform"}}},"parallel":{"type":"boolean","description":"If true, the agent can execute multiple tool calls in parallel"}},"required":["type","input_transforms"]},"Identity":{"type":"object","description":"Pass-through module that returns its input unchanged. Useful for flow structure or as a placeholder","properties":{"type":{"type":"string","enum":["identity"]},"flow":{"type":"boolean","description":"If true, marks this as a flow identity (special handling)"}},"required":["type"]},"FlowStatus":{"type":"object","properties":{"step":{"type":"integer"},"modules":{"type":"array","items":{"$ref":"#/components/schemas/FlowStatusModule"}},"user_states":{"additionalProperties":true},"preprocessor_module":{"allOf":[{"$ref":"#/components/schemas/FlowStatusModule"}]},"failure_module":{"allOf":[{"$ref":"#/components/schemas/FlowStatusModule"},{"type":"object","properties":{"parent_module":{"type":"string"}}}]},"retry":{"type":"object","properties":{"fail_count":{"type":"integer"},"failed_jobs":{"type":"array","items":{"type":"string","format":"uuid"}}}}},"required":["step","modules","failure_module"]},"FlowStatusModule":{"type":"object","properties":{"type":{"type":"string","enum":["WaitingForPriorSteps","WaitingForEvents","WaitingForExecutor","InProgress","Success","Failure"]},"id":{"type":"string"},"job":{"type":"string","format":"uuid"},"count":{"type":"integer"},"progress":{"type":"integer"},"iterator":{"type":"object","properties":{"index":{"type":"integer"},"itered":{"type":"array","items":{}},"itered_len":{"type":"integer"},"args":{}}},"flow_jobs":{"type":"array","items":{"type":"string"}},"flow_jobs_success":{"type":"array","items":{"type":"boolean"}},"flow_jobs_duration":{"type":"object","properties":{"started_at":{"type":"array","items":{"type":"string"}},"duration_ms":{"type":"array","items":{"type":"integer"}}}},"branch_chosen":{"type":"object","properties":{"type":{"type":"string","enum":["branch","default"]},"branch":{"type":"integer"}},"required":["type"]},"branchall":{"type":"object","properties":{"branch":{"type":"integer"},"len":{"type":"integer"}},"required":["branch","len"]},"approvers":{"type":"array","items":{"type":"object","properties":{"resume_id":{"type":"integer"},"approver":{"type":"string"}},"required":["resume_id","approver"]}},"failed_retries":{"type":"array","items":{"type":"string","format":"uuid"}},"skipped":{"type":"boolean"},"agent_actions":{"type":"array","items":{"type":"object","oneOf":[{"type":"object","properties":{"job_id":{"type":"string","format":"uuid"},"function_name":{"type":"string"},"type":{"type":"string","enum":["tool_call"]},"module_id":{"type":"string"}},"required":["job_id","function_name","type","module_id"]},{"type":"object","properties":{"call_id":{"type":"string","format":"uuid"},"function_name":{"type":"string"},"resource_path":{"type":"string"},"type":{"type":"string","enum":["mcp_tool_call"]},"arguments":{"type":"object"}},"required":["call_id","function_name","resource_path","type"]},{"type":"object","properties":{"type":{"type":"string","enum":["web_search"]}},"required":["type"]},{"type":"object","properties":{"type":{"type":"string","enum":["message"]}},"required":["content","type"]}]}},"agent_actions_success":{"type":"array","items":{"type":"boolean"}}},"required":["type"]}}}} \ No newline at end of file diff --git a/frontend/src/lib/components/copilot/chat/flow/openFlowZod.gen.ts b/frontend/src/lib/components/copilot/chat/flow/openFlowZod.gen.ts index 854b768479..5bd36b85ce 100644 --- a/frontend/src/lib/components/copilot/chat/flow/openFlowZod.gen.ts +++ b/frontend/src/lib/components/copilot/chat/flow/openFlowZod.gen.ts @@ -1,6 +1,6 @@ import { z } from "zod" -export const flowModuleValueSchema = z.discriminatedUnion("type", [z.object({ "input_transforms": z.record(z.string(), z.discriminatedUnion("type", [z.object({ "value": z.any().describe("The static value. For resources, use format '$res:path/to/resource'").optional(), "type": z.literal("static") }).describe("Static value passed directly to the step. Use for hardcoded values or resource references like '$res:path/to/resource'"), z.object({ "expr": z.string().describe("JavaScript expression returning the value. Available variables - results (object with all previous step results), flow_input (flow inputs), flow_input.iter (in loops)"), "type": z.literal("javascript") }).describe("JavaScript expression evaluated at runtime. Can reference previous step results via 'results.step_id' or flow inputs via 'flow_input.property'. Inside for loops, use 'flow_input.iter.value' for the current iteration value (in while loops it equals 'flow_input.iter.index')"), z.object({ "type": z.literal("ai") }).describe("Value resolved by the AI runtime for this input. The AI engine decides how to satisfy the parameter.")]).describe("Maps input parameters for a step. Can be a static value or a JavaScript expression that references previous results or flow inputs")).describe("Map of parameter names to their values (static or JavaScript expressions). These become the script's input arguments"), "content": z.string().describe("The script source code. Should export a 'main' function"), "language": z.enum(["deno","bun","bunnative","python3","go","bash","powershell","postgresql","mysql","bigquery","snowflake","mssql","oracledb","graphql","nativets","php","rust","ansible","csharp","nu","java","ruby","rlang","duckdb"]).describe("Programming language for this script"), "path": z.string().describe("Optional path for saving this script").optional(), "lock": z.string().describe("Lock file content for dependencies").optional(), "type": z.literal("rawscript"), "tag": z.string().describe("Worker group tag for execution routing").optional(), "concurrent_limit": z.number().describe("Maximum concurrent executions of this script").optional(), "concurrency_time_window_s": z.number().describe("Time window for concurrent_limit").optional(), "custom_concurrency_key": z.string().describe("Custom key for grouping concurrent executions").optional(), "is_trigger": z.boolean().describe("If true, this script is a trigger that can start the flow").optional(), "assets": z.array(z.object({ "path": z.string().describe("Path to the asset"), "kind": z.enum(["s3object","resource","ducklake","datatable","volume","dbt"]).describe("Type of asset"), "access_type": z.union([z.literal("r"), z.literal("w"), z.literal("rw"), z.literal(null)]).nullable().describe("Access level for this asset").optional(), "alt_access_type": z.union([z.literal("r"), z.literal("w"), z.literal("rw"), z.literal(null)]).nullable().describe("Alternative access level").optional() })).describe("External resources this script accesses (S3 objects, resources, etc.)").optional() }).describe("Inline script with code defined directly in the flow. Use 'bun' as default language if unspecified. The script receives arguments from input_transforms"), z.object({ "input_transforms": z.record(z.string(), z.discriminatedUnion("type", [z.object({ "value": z.any().describe("The static value. For resources, use format '$res:path/to/resource'").optional(), "type": z.literal("static") }).describe("Static value passed directly to the step. Use for hardcoded values or resource references like '$res:path/to/resource'"), z.object({ "expr": z.string().describe("JavaScript expression returning the value. Available variables - results (object with all previous step results), flow_input (flow inputs), flow_input.iter (in loops)"), "type": z.literal("javascript") }).describe("JavaScript expression evaluated at runtime. Can reference previous step results via 'results.step_id' or flow inputs via 'flow_input.property'. Inside for loops, use 'flow_input.iter.value' for the current iteration value (in while loops it equals 'flow_input.iter.index')"), z.object({ "type": z.literal("ai") }).describe("Value resolved by the AI runtime for this input. The AI engine decides how to satisfy the parameter.")]).describe("Maps input parameters for a step. Can be a static value or a JavaScript expression that references previous results or flow inputs")).describe("Map of parameter names to their values (static or JavaScript expressions). These become the script's input arguments"), "path": z.string().describe("Path to the script in the workspace (e.g., 'f/scripts/send_email')"), "hash": z.string().describe("Optional specific version hash of the script to use").optional(), "type": z.literal("script"), "tag_override": z.string().describe("Override the script's default worker group tag").optional(), "is_trigger": z.boolean().describe("If true, this script is a trigger that can start the flow").optional() }).describe("Reference to an existing script by path. Use this when calling a previously saved script instead of writing inline code"), z.object({ "input_transforms": z.record(z.string(), z.discriminatedUnion("type", [z.object({ "value": z.any().describe("The static value. For resources, use format '$res:path/to/resource'").optional(), "type": z.literal("static") }).describe("Static value passed directly to the step. Use for hardcoded values or resource references like '$res:path/to/resource'"), z.object({ "expr": z.string().describe("JavaScript expression returning the value. Available variables - results (object with all previous step results), flow_input (flow inputs), flow_input.iter (in loops)"), "type": z.literal("javascript") }).describe("JavaScript expression evaluated at runtime. Can reference previous step results via 'results.step_id' or flow inputs via 'flow_input.property'. Inside for loops, use 'flow_input.iter.value' for the current iteration value (in while loops it equals 'flow_input.iter.index')"), z.object({ "type": z.literal("ai") }).describe("Value resolved by the AI runtime for this input. The AI engine decides how to satisfy the parameter.")]).describe("Maps input parameters for a step. Can be a static value or a JavaScript expression that references previous results or flow inputs")).describe("Map of parameter names to their values (static or JavaScript expressions). These become the subflow's input arguments"), "path": z.string().describe("Path to the flow in the workspace (e.g., 'f/flows/process_user')"), "type": z.literal("flow") }).describe("Reference to an existing flow by path. Use this to call another flow as a subflow"), z.object({ "modules": z.array(z.object({ "id": z.string().describe("Unique identifier for this step. Used to reference results via 'results.step_id'. Must be a valid identifier (alphanumeric, underscore, hyphen)"), "value": z.lazy(() => flowModuleValueSchema), "stop_after_if": z.object({ "skip_if_stopped": z.boolean().describe("If true, following steps are skipped when this condition triggers").optional(), "expr": z.string().describe("JavaScript expression evaluated after the module runs. Can use 'result' (step's result) or 'flow_input'. Return true to stop"), "error_message": z.string().nullable().describe("Custom error message when stopping with an error. Mutually exclusive with skip_if_stopped. If set to a non-empty string, the flow stops with this error. If empty string, a default error message is used. If null or omitted, no error is raised.").optional(), "error_include_result": z.boolean().describe("When stopping with an error (error_message set), embed the stopping step's own result inside the raised error object (as error.result) instead of discarding it. The top-level result stays { error }. Defaults to false.").optional() }).describe("Early termination condition for a module").optional(), "stop_after_all_iters_if": z.object({ "skip_if_stopped": z.boolean().describe("If true, following steps are skipped when this condition triggers").optional(), "expr": z.string().describe("JavaScript expression evaluated after the module runs. Can use 'result' (step's result) or 'flow_input'. Return true to stop"), "error_message": z.string().nullable().describe("Custom error message when stopping with an error. Mutually exclusive with skip_if_stopped. If set to a non-empty string, the flow stops with this error. If empty string, a default error message is used. If null or omitted, no error is raised.").optional(), "error_include_result": z.boolean().describe("When stopping with an error (error_message set), embed the stopping step's own result inside the raised error object (as error.result) instead of discarding it. The top-level result stays { error }. Defaults to false.").optional() }).describe("Early termination condition for a module").optional(), "skip_if": z.object({ "expr": z.string().describe("JavaScript expression that returns true to skip. Can use 'flow_input' or 'results.'") }).describe("Conditionally skip this step based on previous results or flow inputs").optional(), "sleep": z.discriminatedUnion("type", [z.object({ "value": z.any().describe("The static value. For resources, use format '$res:path/to/resource'").optional(), "type": z.literal("static") }).describe("Static value passed directly to the step. Use for hardcoded values or resource references like '$res:path/to/resource'"), z.object({ "expr": z.string().describe("JavaScript expression returning the value. Available variables - results (object with all previous step results), flow_input (flow inputs), flow_input.iter (in loops)"), "type": z.literal("javascript") }).describe("JavaScript expression evaluated at runtime. Can reference previous step results via 'results.step_id' or flow inputs via 'flow_input.property'. Inside for loops, use 'flow_input.iter.value' for the current iteration value (in while loops it equals 'flow_input.iter.index')"), z.object({ "type": z.literal("ai") }).describe("Value resolved by the AI runtime for this input. The AI engine decides how to satisfy the parameter.")]).describe("Maps input parameters for a step. Can be a static value or a JavaScript expression that references previous results or flow inputs").optional(), "cache_ttl": z.number().describe("Cache duration in seconds for this step's results").optional(), "cache_ignore_s3_path": z.boolean().optional(), "timeout": z.discriminatedUnion("type", [z.object({ "value": z.any().describe("The static value. For resources, use format '$res:path/to/resource'").optional(), "type": z.literal("static") }).describe("Static value passed directly to the step. Use for hardcoded values or resource references like '$res:path/to/resource'"), z.object({ "expr": z.string().describe("JavaScript expression returning the value. Available variables - results (object with all previous step results), flow_input (flow inputs), flow_input.iter (in loops)"), "type": z.literal("javascript") }).describe("JavaScript expression evaluated at runtime. Can reference previous step results via 'results.step_id' or flow inputs via 'flow_input.property'. Inside for loops, use 'flow_input.iter.value' for the current iteration value (in while loops it equals 'flow_input.iter.index')"), z.object({ "type": z.literal("ai") }).describe("Value resolved by the AI runtime for this input. The AI engine decides how to satisfy the parameter.")]).describe("Maps input parameters for a step. Can be a static value or a JavaScript expression that references previous results or flow inputs").optional(), "delete_after_secs": z.number().int().describe("If set, delete the step's args, result and logs after this many seconds following job completion").optional(), "summary": z.string().describe("Short description of what this step does").optional(), "mock": z.object({ "enabled": z.boolean().describe("If true, return mock value instead of executing").optional(), "return_value": z.any().describe("Value to return when mocked").optional() }).describe("Mock configuration for testing without executing the actual step").optional(), "suspend": z.object({ "required_events": z.number().int().describe("Number of approvals required before continuing").optional(), "timeout": z.number().int().describe("Timeout in seconds before auto-continuing or canceling").optional(), "resume_form": z.object({ "schema": z.record(z.string(), z.any()).describe("JSON Schema for the resume form").optional() }).describe("Form schema for collecting input when resuming").optional(), "user_auth_required": z.boolean().describe("If true, only authenticated users can approve").optional(), "user_groups_required": z.discriminatedUnion("type", [z.object({ "value": z.any().describe("The static value. For resources, use format '$res:path/to/resource'").optional(), "type": z.literal("static") }).describe("Static value passed directly to the step. Use for hardcoded values or resource references like '$res:path/to/resource'"), z.object({ "expr": z.string().describe("JavaScript expression returning the value. Available variables - results (object with all previous step results), flow_input (flow inputs), flow_input.iter (in loops)"), "type": z.literal("javascript") }).describe("JavaScript expression evaluated at runtime. Can reference previous step results via 'results.step_id' or flow inputs via 'flow_input.property'. Inside for loops, use 'flow_input.iter.value' for the current iteration value (in while loops it equals 'flow_input.iter.index')"), z.object({ "type": z.literal("ai") }).describe("Value resolved by the AI runtime for this input. The AI engine decides how to satisfy the parameter.")]).describe("Maps input parameters for a step. Can be a static value or a JavaScript expression that references previous results or flow inputs").optional(), "self_approval_disabled": z.boolean().describe("If true, the user who started the flow cannot approve").optional(), "hide_cancel": z.boolean().describe("If true, hide the cancel button on the approval form").optional(), "continue_on_disapprove_timeout": z.boolean().describe("If true, continue flow on timeout instead of canceling").optional() }).describe("Configuration for approval/resume steps that wait for user input").optional(), "priority": z.number().describe("Execution priority for this step (higher numbers run first)").optional(), "continue_on_error": z.boolean().describe("If true, flow continues even if this step fails").optional(), "retry": z.object({ "constant": z.object({ "attempts": z.number().int().describe("Number of retry attempts").optional(), "seconds": z.number().int().describe("Seconds to wait between retries").optional() }).describe("Retry with constant delay between attempts").optional(), "exponential": z.object({ "attempts": z.number().int().describe("Number of retry attempts").optional(), "multiplier": z.number().int().describe("Multiplier for exponential backoff").optional(), "seconds": z.number().int().gte(1).describe("Initial delay in seconds").optional(), "random_factor": z.number().int().gte(0).lte(100).describe("Random jitter percentage (0-100) to avoid thundering herd").optional() }).describe("Retry with exponential backoff (delay doubles each time)").optional(), "retry_if": z.object({ "expr": z.string().describe("JavaScript expression that returns true to retry. Has access to 'result' and 'error' variables") }).describe("Conditional retry based on error or result").optional() }).describe("Retry configuration for failed module executions").optional(), "debouncing": z.object({ "debounce_delay_s": z.number().int().describe("Delay in seconds to debounce this step's executions across flow runs").optional(), "debounce_key": z.string().describe("Expression to group debounced executions. Supports $workspace and $args[name]. Default: $workspace/flow/-").optional(), "debounce_args_to_accumulate": z.array(z.string()).describe("Array-type arguments to accumulate across debounced executions").optional(), "max_total_debouncing_time": z.number().int().describe("Maximum total time in seconds before forced execution").optional(), "max_total_debounces_amount": z.number().int().describe("Maximum number of debounces before forced execution").optional() }).describe("Debounce configuration for this step (EE only)").optional() }).describe("A single step in a flow. Can be a script, subflow, loop, or branch")).describe("Steps to execute for each iteration. These can reference the iteration value via 'flow_input.iter.value'"), "iterator": z.discriminatedUnion("type", [z.object({ "value": z.any().describe("The static value. For resources, use format '$res:path/to/resource'").optional(), "type": z.literal("static") }).describe("Static value passed directly to the step. Use for hardcoded values or resource references like '$res:path/to/resource'"), z.object({ "expr": z.string().describe("JavaScript expression returning the value. Available variables - results (object with all previous step results), flow_input (flow inputs), flow_input.iter (in loops)"), "type": z.literal("javascript") }).describe("JavaScript expression evaluated at runtime. Can reference previous step results via 'results.step_id' or flow inputs via 'flow_input.property'. Inside for loops, use 'flow_input.iter.value' for the current iteration value (in while loops it equals 'flow_input.iter.index')"), z.object({ "type": z.literal("ai") }).describe("Value resolved by the AI runtime for this input. The AI engine decides how to satisfy the parameter.")]).describe("Maps input parameters for a step. Can be a static value or a JavaScript expression that references previous results or flow inputs"), "skip_failures": z.boolean().describe("If true, iteration failures don't stop the loop. Failed iterations return null"), "type": z.literal("forloopflow"), "parallel": z.boolean().describe("If true, iterations run concurrently (faster for I/O-bound operations). Use with parallelism to control concurrency").optional(), "parallelism": z.discriminatedUnion("type", [z.object({ "value": z.any().describe("The static value. For resources, use format '$res:path/to/resource'").optional(), "type": z.literal("static") }).describe("Static value passed directly to the step. Use for hardcoded values or resource references like '$res:path/to/resource'"), z.object({ "expr": z.string().describe("JavaScript expression returning the value. Available variables - results (object with all previous step results), flow_input (flow inputs), flow_input.iter (in loops)"), "type": z.literal("javascript") }).describe("JavaScript expression evaluated at runtime. Can reference previous step results via 'results.step_id' or flow inputs via 'flow_input.property'. Inside for loops, use 'flow_input.iter.value' for the current iteration value (in while loops it equals 'flow_input.iter.index')"), z.object({ "type": z.literal("ai") }).describe("Value resolved by the AI runtime for this input. The AI engine decides how to satisfy the parameter.")]).describe("Maps input parameters for a step. Can be a static value or a JavaScript expression that references previous results or flow inputs").optional(), "squash": z.boolean().optional() }).describe("Executes nested modules in a loop over an iterator. Inside the loop, use 'flow_input.iter.value' to access the current iteration value, and 'flow_input.iter.index' for the index. Supports parallel execution for better performance on I/O-bound operations"), z.object({ "modules": z.array(z.object({ "id": z.string().describe("Unique identifier for this step. Used to reference results via 'results.step_id'. Must be a valid identifier (alphanumeric, underscore, hyphen)"), "value": z.lazy(() => flowModuleValueSchema), "stop_after_if": z.object({ "skip_if_stopped": z.boolean().describe("If true, following steps are skipped when this condition triggers").optional(), "expr": z.string().describe("JavaScript expression evaluated after the module runs. Can use 'result' (step's result) or 'flow_input'. Return true to stop"), "error_message": z.string().nullable().describe("Custom error message when stopping with an error. Mutually exclusive with skip_if_stopped. If set to a non-empty string, the flow stops with this error. If empty string, a default error message is used. If null or omitted, no error is raised.").optional(), "error_include_result": z.boolean().describe("When stopping with an error (error_message set), embed the stopping step's own result inside the raised error object (as error.result) instead of discarding it. The top-level result stays { error }. Defaults to false.").optional() }).describe("Early termination condition for a module").optional(), "stop_after_all_iters_if": z.object({ "skip_if_stopped": z.boolean().describe("If true, following steps are skipped when this condition triggers").optional(), "expr": z.string().describe("JavaScript expression evaluated after the module runs. Can use 'result' (step's result) or 'flow_input'. Return true to stop"), "error_message": z.string().nullable().describe("Custom error message when stopping with an error. Mutually exclusive with skip_if_stopped. If set to a non-empty string, the flow stops with this error. If empty string, a default error message is used. If null or omitted, no error is raised.").optional(), "error_include_result": z.boolean().describe("When stopping with an error (error_message set), embed the stopping step's own result inside the raised error object (as error.result) instead of discarding it. The top-level result stays { error }. Defaults to false.").optional() }).describe("Early termination condition for a module").optional(), "skip_if": z.object({ "expr": z.string().describe("JavaScript expression that returns true to skip. Can use 'flow_input' or 'results.'") }).describe("Conditionally skip this step based on previous results or flow inputs").optional(), "sleep": z.discriminatedUnion("type", [z.object({ "value": z.any().describe("The static value. For resources, use format '$res:path/to/resource'").optional(), "type": z.literal("static") }).describe("Static value passed directly to the step. Use for hardcoded values or resource references like '$res:path/to/resource'"), z.object({ "expr": z.string().describe("JavaScript expression returning the value. Available variables - results (object with all previous step results), flow_input (flow inputs), flow_input.iter (in loops)"), "type": z.literal("javascript") }).describe("JavaScript expression evaluated at runtime. Can reference previous step results via 'results.step_id' or flow inputs via 'flow_input.property'. Inside for loops, use 'flow_input.iter.value' for the current iteration value (in while loops it equals 'flow_input.iter.index')"), z.object({ "type": z.literal("ai") }).describe("Value resolved by the AI runtime for this input. The AI engine decides how to satisfy the parameter.")]).describe("Maps input parameters for a step. Can be a static value or a JavaScript expression that references previous results or flow inputs").optional(), "cache_ttl": z.number().describe("Cache duration in seconds for this step's results").optional(), "cache_ignore_s3_path": z.boolean().optional(), "timeout": z.discriminatedUnion("type", [z.object({ "value": z.any().describe("The static value. For resources, use format '$res:path/to/resource'").optional(), "type": z.literal("static") }).describe("Static value passed directly to the step. Use for hardcoded values or resource references like '$res:path/to/resource'"), z.object({ "expr": z.string().describe("JavaScript expression returning the value. Available variables - results (object with all previous step results), flow_input (flow inputs), flow_input.iter (in loops)"), "type": z.literal("javascript") }).describe("JavaScript expression evaluated at runtime. Can reference previous step results via 'results.step_id' or flow inputs via 'flow_input.property'. Inside for loops, use 'flow_input.iter.value' for the current iteration value (in while loops it equals 'flow_input.iter.index')"), z.object({ "type": z.literal("ai") }).describe("Value resolved by the AI runtime for this input. The AI engine decides how to satisfy the parameter.")]).describe("Maps input parameters for a step. Can be a static value or a JavaScript expression that references previous results or flow inputs").optional(), "delete_after_secs": z.number().int().describe("If set, delete the step's args, result and logs after this many seconds following job completion").optional(), "summary": z.string().describe("Short description of what this step does").optional(), "mock": z.object({ "enabled": z.boolean().describe("If true, return mock value instead of executing").optional(), "return_value": z.any().describe("Value to return when mocked").optional() }).describe("Mock configuration for testing without executing the actual step").optional(), "suspend": z.object({ "required_events": z.number().int().describe("Number of approvals required before continuing").optional(), "timeout": z.number().int().describe("Timeout in seconds before auto-continuing or canceling").optional(), "resume_form": z.object({ "schema": z.record(z.string(), z.any()).describe("JSON Schema for the resume form").optional() }).describe("Form schema for collecting input when resuming").optional(), "user_auth_required": z.boolean().describe("If true, only authenticated users can approve").optional(), "user_groups_required": z.discriminatedUnion("type", [z.object({ "value": z.any().describe("The static value. For resources, use format '$res:path/to/resource'").optional(), "type": z.literal("static") }).describe("Static value passed directly to the step. Use for hardcoded values or resource references like '$res:path/to/resource'"), z.object({ "expr": z.string().describe("JavaScript expression returning the value. Available variables - results (object with all previous step results), flow_input (flow inputs), flow_input.iter (in loops)"), "type": z.literal("javascript") }).describe("JavaScript expression evaluated at runtime. Can reference previous step results via 'results.step_id' or flow inputs via 'flow_input.property'. Inside for loops, use 'flow_input.iter.value' for the current iteration value (in while loops it equals 'flow_input.iter.index')"), z.object({ "type": z.literal("ai") }).describe("Value resolved by the AI runtime for this input. The AI engine decides how to satisfy the parameter.")]).describe("Maps input parameters for a step. Can be a static value or a JavaScript expression that references previous results or flow inputs").optional(), "self_approval_disabled": z.boolean().describe("If true, the user who started the flow cannot approve").optional(), "hide_cancel": z.boolean().describe("If true, hide the cancel button on the approval form").optional(), "continue_on_disapprove_timeout": z.boolean().describe("If true, continue flow on timeout instead of canceling").optional() }).describe("Configuration for approval/resume steps that wait for user input").optional(), "priority": z.number().describe("Execution priority for this step (higher numbers run first)").optional(), "continue_on_error": z.boolean().describe("If true, flow continues even if this step fails").optional(), "retry": z.object({ "constant": z.object({ "attempts": z.number().int().describe("Number of retry attempts").optional(), "seconds": z.number().int().describe("Seconds to wait between retries").optional() }).describe("Retry with constant delay between attempts").optional(), "exponential": z.object({ "attempts": z.number().int().describe("Number of retry attempts").optional(), "multiplier": z.number().int().describe("Multiplier for exponential backoff").optional(), "seconds": z.number().int().gte(1).describe("Initial delay in seconds").optional(), "random_factor": z.number().int().gte(0).lte(100).describe("Random jitter percentage (0-100) to avoid thundering herd").optional() }).describe("Retry with exponential backoff (delay doubles each time)").optional(), "retry_if": z.object({ "expr": z.string().describe("JavaScript expression that returns true to retry. Has access to 'result' and 'error' variables") }).describe("Conditional retry based on error or result").optional() }).describe("Retry configuration for failed module executions").optional(), "debouncing": z.object({ "debounce_delay_s": z.number().int().describe("Delay in seconds to debounce this step's executions across flow runs").optional(), "debounce_key": z.string().describe("Expression to group debounced executions. Supports $workspace and $args[name]. Default: $workspace/flow/-").optional(), "debounce_args_to_accumulate": z.array(z.string()).describe("Array-type arguments to accumulate across debounced executions").optional(), "max_total_debouncing_time": z.number().int().describe("Maximum total time in seconds before forced execution").optional(), "max_total_debounces_amount": z.number().int().describe("Maximum number of debounces before forced execution").optional() }).describe("Debounce configuration for this step (EE only)").optional() }).describe("A single step in a flow. Can be a script, subflow, loop, or branch")).describe("Steps to execute in each iteration"), "skip_failures": z.boolean().describe("If true, iteration failures don't stop the loop. Failed iterations return null"), "type": z.literal("whileloopflow"), "parallel": z.boolean().describe("If true, iterations run concurrently (use with caution in while loops)").optional(), "parallelism": z.discriminatedUnion("type", [z.object({ "value": z.any().describe("The static value. For resources, use format '$res:path/to/resource'").optional(), "type": z.literal("static") }).describe("Static value passed directly to the step. Use for hardcoded values or resource references like '$res:path/to/resource'"), z.object({ "expr": z.string().describe("JavaScript expression returning the value. Available variables - results (object with all previous step results), flow_input (flow inputs), flow_input.iter (in loops)"), "type": z.literal("javascript") }).describe("JavaScript expression evaluated at runtime. Can reference previous step results via 'results.step_id' or flow inputs via 'flow_input.property'. Inside for loops, use 'flow_input.iter.value' for the current iteration value (in while loops it equals 'flow_input.iter.index')"), z.object({ "type": z.literal("ai") }).describe("Value resolved by the AI runtime for this input. The AI engine decides how to satisfy the parameter.")]).describe("Maps input parameters for a step. Can be a static value or a JavaScript expression that references previous results or flow inputs").optional(), "squash": z.boolean().optional() }).describe("Executes nested modules repeatedly until stopped. The implicit iterator is the iteration counter, so 'flow_input.iter.value' equals 'flow_input.iter.index' (0, 1, 2, ...) and never carries state. To carry state across iterations, a step reads its own previous-iteration result via 'results.' with a first-iteration fallback - the loop's stop_after_if must then be on that inner step (a plain single-step body with stop_after_if on the loop module does not resolve 'results' across iterations and never terminates); plain counters can instead be derived from 'flow_input.iter.index', which works in every configuration. stop_after_if is evaluated after each iteration - on the loop module 'result' is the last iteration's result"), z.object({ "branches": z.array(z.object({ "summary": z.string().describe("Short description of this branch condition").optional(), "expr": z.string().describe("JavaScript expression that returns boolean. Can use 'results.step_id' or 'flow_input'. First true expr wins"), "modules": z.array(z.object({ "id": z.string().describe("Unique identifier for this step. Used to reference results via 'results.step_id'. Must be a valid identifier (alphanumeric, underscore, hyphen)"), "value": z.lazy(() => flowModuleValueSchema), "stop_after_if": z.object({ "skip_if_stopped": z.boolean().describe("If true, following steps are skipped when this condition triggers").optional(), "expr": z.string().describe("JavaScript expression evaluated after the module runs. Can use 'result' (step's result) or 'flow_input'. Return true to stop"), "error_message": z.string().nullable().describe("Custom error message when stopping with an error. Mutually exclusive with skip_if_stopped. If set to a non-empty string, the flow stops with this error. If empty string, a default error message is used. If null or omitted, no error is raised.").optional(), "error_include_result": z.boolean().describe("When stopping with an error (error_message set), embed the stopping step's own result inside the raised error object (as error.result) instead of discarding it. The top-level result stays { error }. Defaults to false.").optional() }).describe("Early termination condition for a module").optional(), "stop_after_all_iters_if": z.object({ "skip_if_stopped": z.boolean().describe("If true, following steps are skipped when this condition triggers").optional(), "expr": z.string().describe("JavaScript expression evaluated after the module runs. Can use 'result' (step's result) or 'flow_input'. Return true to stop"), "error_message": z.string().nullable().describe("Custom error message when stopping with an error. Mutually exclusive with skip_if_stopped. If set to a non-empty string, the flow stops with this error. If empty string, a default error message is used. If null or omitted, no error is raised.").optional(), "error_include_result": z.boolean().describe("When stopping with an error (error_message set), embed the stopping step's own result inside the raised error object (as error.result) instead of discarding it. The top-level result stays { error }. Defaults to false.").optional() }).describe("Early termination condition for a module").optional(), "skip_if": z.object({ "expr": z.string().describe("JavaScript expression that returns true to skip. Can use 'flow_input' or 'results.'") }).describe("Conditionally skip this step based on previous results or flow inputs").optional(), "sleep": z.discriminatedUnion("type", [z.object({ "value": z.any().describe("The static value. For resources, use format '$res:path/to/resource'").optional(), "type": z.literal("static") }).describe("Static value passed directly to the step. Use for hardcoded values or resource references like '$res:path/to/resource'"), z.object({ "expr": z.string().describe("JavaScript expression returning the value. Available variables - results (object with all previous step results), flow_input (flow inputs), flow_input.iter (in loops)"), "type": z.literal("javascript") }).describe("JavaScript expression evaluated at runtime. Can reference previous step results via 'results.step_id' or flow inputs via 'flow_input.property'. Inside for loops, use 'flow_input.iter.value' for the current iteration value (in while loops it equals 'flow_input.iter.index')"), z.object({ "type": z.literal("ai") }).describe("Value resolved by the AI runtime for this input. The AI engine decides how to satisfy the parameter.")]).describe("Maps input parameters for a step. Can be a static value or a JavaScript expression that references previous results or flow inputs").optional(), "cache_ttl": z.number().describe("Cache duration in seconds for this step's results").optional(), "cache_ignore_s3_path": z.boolean().optional(), "timeout": z.discriminatedUnion("type", [z.object({ "value": z.any().describe("The static value. For resources, use format '$res:path/to/resource'").optional(), "type": z.literal("static") }).describe("Static value passed directly to the step. Use for hardcoded values or resource references like '$res:path/to/resource'"), z.object({ "expr": z.string().describe("JavaScript expression returning the value. Available variables - results (object with all previous step results), flow_input (flow inputs), flow_input.iter (in loops)"), "type": z.literal("javascript") }).describe("JavaScript expression evaluated at runtime. Can reference previous step results via 'results.step_id' or flow inputs via 'flow_input.property'. Inside for loops, use 'flow_input.iter.value' for the current iteration value (in while loops it equals 'flow_input.iter.index')"), z.object({ "type": z.literal("ai") }).describe("Value resolved by the AI runtime for this input. The AI engine decides how to satisfy the parameter.")]).describe("Maps input parameters for a step. Can be a static value or a JavaScript expression that references previous results or flow inputs").optional(), "delete_after_secs": z.number().int().describe("If set, delete the step's args, result and logs after this many seconds following job completion").optional(), "summary": z.string().describe("Short description of what this step does").optional(), "mock": z.object({ "enabled": z.boolean().describe("If true, return mock value instead of executing").optional(), "return_value": z.any().describe("Value to return when mocked").optional() }).describe("Mock configuration for testing without executing the actual step").optional(), "suspend": z.object({ "required_events": z.number().int().describe("Number of approvals required before continuing").optional(), "timeout": z.number().int().describe("Timeout in seconds before auto-continuing or canceling").optional(), "resume_form": z.object({ "schema": z.record(z.string(), z.any()).describe("JSON Schema for the resume form").optional() }).describe("Form schema for collecting input when resuming").optional(), "user_auth_required": z.boolean().describe("If true, only authenticated users can approve").optional(), "user_groups_required": z.discriminatedUnion("type", [z.object({ "value": z.any().describe("The static value. For resources, use format '$res:path/to/resource'").optional(), "type": z.literal("static") }).describe("Static value passed directly to the step. Use for hardcoded values or resource references like '$res:path/to/resource'"), z.object({ "expr": z.string().describe("JavaScript expression returning the value. Available variables - results (object with all previous step results), flow_input (flow inputs), flow_input.iter (in loops)"), "type": z.literal("javascript") }).describe("JavaScript expression evaluated at runtime. Can reference previous step results via 'results.step_id' or flow inputs via 'flow_input.property'. Inside for loops, use 'flow_input.iter.value' for the current iteration value (in while loops it equals 'flow_input.iter.index')"), z.object({ "type": z.literal("ai") }).describe("Value resolved by the AI runtime for this input. The AI engine decides how to satisfy the parameter.")]).describe("Maps input parameters for a step. Can be a static value or a JavaScript expression that references previous results or flow inputs").optional(), "self_approval_disabled": z.boolean().describe("If true, the user who started the flow cannot approve").optional(), "hide_cancel": z.boolean().describe("If true, hide the cancel button on the approval form").optional(), "continue_on_disapprove_timeout": z.boolean().describe("If true, continue flow on timeout instead of canceling").optional() }).describe("Configuration for approval/resume steps that wait for user input").optional(), "priority": z.number().describe("Execution priority for this step (higher numbers run first)").optional(), "continue_on_error": z.boolean().describe("If true, flow continues even if this step fails").optional(), "retry": z.object({ "constant": z.object({ "attempts": z.number().int().describe("Number of retry attempts").optional(), "seconds": z.number().int().describe("Seconds to wait between retries").optional() }).describe("Retry with constant delay between attempts").optional(), "exponential": z.object({ "attempts": z.number().int().describe("Number of retry attempts").optional(), "multiplier": z.number().int().describe("Multiplier for exponential backoff").optional(), "seconds": z.number().int().gte(1).describe("Initial delay in seconds").optional(), "random_factor": z.number().int().gte(0).lte(100).describe("Random jitter percentage (0-100) to avoid thundering herd").optional() }).describe("Retry with exponential backoff (delay doubles each time)").optional(), "retry_if": z.object({ "expr": z.string().describe("JavaScript expression that returns true to retry. Has access to 'result' and 'error' variables") }).describe("Conditional retry based on error or result").optional() }).describe("Retry configuration for failed module executions").optional(), "debouncing": z.object({ "debounce_delay_s": z.number().int().describe("Delay in seconds to debounce this step's executions across flow runs").optional(), "debounce_key": z.string().describe("Expression to group debounced executions. Supports $workspace and $args[name]. Default: $workspace/flow/-").optional(), "debounce_args_to_accumulate": z.array(z.string()).describe("Array-type arguments to accumulate across debounced executions").optional(), "max_total_debouncing_time": z.number().int().describe("Maximum total time in seconds before forced execution").optional(), "max_total_debounces_amount": z.number().int().describe("Maximum number of debounces before forced execution").optional() }).describe("Debounce configuration for this step (EE only)").optional() }).describe("A single step in a flow. Can be a script, subflow, loop, or branch")).describe("Steps to execute if this branch's expr is true") })).describe("Array of branches to evaluate in order. The first branch with expr evaluating to true executes"), "default": z.array(z.object({ "id": z.string().describe("Unique identifier for this step. Used to reference results via 'results.step_id'. Must be a valid identifier (alphanumeric, underscore, hyphen)"), "value": z.lazy(() => flowModuleValueSchema), "stop_after_if": z.object({ "skip_if_stopped": z.boolean().describe("If true, following steps are skipped when this condition triggers").optional(), "expr": z.string().describe("JavaScript expression evaluated after the module runs. Can use 'result' (step's result) or 'flow_input'. Return true to stop"), "error_message": z.string().nullable().describe("Custom error message when stopping with an error. Mutually exclusive with skip_if_stopped. If set to a non-empty string, the flow stops with this error. If empty string, a default error message is used. If null or omitted, no error is raised.").optional(), "error_include_result": z.boolean().describe("When stopping with an error (error_message set), embed the stopping step's own result inside the raised error object (as error.result) instead of discarding it. The top-level result stays { error }. Defaults to false.").optional() }).describe("Early termination condition for a module").optional(), "stop_after_all_iters_if": z.object({ "skip_if_stopped": z.boolean().describe("If true, following steps are skipped when this condition triggers").optional(), "expr": z.string().describe("JavaScript expression evaluated after the module runs. Can use 'result' (step's result) or 'flow_input'. Return true to stop"), "error_message": z.string().nullable().describe("Custom error message when stopping with an error. Mutually exclusive with skip_if_stopped. If set to a non-empty string, the flow stops with this error. If empty string, a default error message is used. If null or omitted, no error is raised.").optional(), "error_include_result": z.boolean().describe("When stopping with an error (error_message set), embed the stopping step's own result inside the raised error object (as error.result) instead of discarding it. The top-level result stays { error }. Defaults to false.").optional() }).describe("Early termination condition for a module").optional(), "skip_if": z.object({ "expr": z.string().describe("JavaScript expression that returns true to skip. Can use 'flow_input' or 'results.'") }).describe("Conditionally skip this step based on previous results or flow inputs").optional(), "sleep": z.discriminatedUnion("type", [z.object({ "value": z.any().describe("The static value. For resources, use format '$res:path/to/resource'").optional(), "type": z.literal("static") }).describe("Static value passed directly to the step. Use for hardcoded values or resource references like '$res:path/to/resource'"), z.object({ "expr": z.string().describe("JavaScript expression returning the value. Available variables - results (object with all previous step results), flow_input (flow inputs), flow_input.iter (in loops)"), "type": z.literal("javascript") }).describe("JavaScript expression evaluated at runtime. Can reference previous step results via 'results.step_id' or flow inputs via 'flow_input.property'. Inside for loops, use 'flow_input.iter.value' for the current iteration value (in while loops it equals 'flow_input.iter.index')"), z.object({ "type": z.literal("ai") }).describe("Value resolved by the AI runtime for this input. The AI engine decides how to satisfy the parameter.")]).describe("Maps input parameters for a step. Can be a static value or a JavaScript expression that references previous results or flow inputs").optional(), "cache_ttl": z.number().describe("Cache duration in seconds for this step's results").optional(), "cache_ignore_s3_path": z.boolean().optional(), "timeout": z.discriminatedUnion("type", [z.object({ "value": z.any().describe("The static value. For resources, use format '$res:path/to/resource'").optional(), "type": z.literal("static") }).describe("Static value passed directly to the step. Use for hardcoded values or resource references like '$res:path/to/resource'"), z.object({ "expr": z.string().describe("JavaScript expression returning the value. Available variables - results (object with all previous step results), flow_input (flow inputs), flow_input.iter (in loops)"), "type": z.literal("javascript") }).describe("JavaScript expression evaluated at runtime. Can reference previous step results via 'results.step_id' or flow inputs via 'flow_input.property'. Inside for loops, use 'flow_input.iter.value' for the current iteration value (in while loops it equals 'flow_input.iter.index')"), z.object({ "type": z.literal("ai") }).describe("Value resolved by the AI runtime for this input. The AI engine decides how to satisfy the parameter.")]).describe("Maps input parameters for a step. Can be a static value or a JavaScript expression that references previous results or flow inputs").optional(), "delete_after_secs": z.number().int().describe("If set, delete the step's args, result and logs after this many seconds following job completion").optional(), "summary": z.string().describe("Short description of what this step does").optional(), "mock": z.object({ "enabled": z.boolean().describe("If true, return mock value instead of executing").optional(), "return_value": z.any().describe("Value to return when mocked").optional() }).describe("Mock configuration for testing without executing the actual step").optional(), "suspend": z.object({ "required_events": z.number().int().describe("Number of approvals required before continuing").optional(), "timeout": z.number().int().describe("Timeout in seconds before auto-continuing or canceling").optional(), "resume_form": z.object({ "schema": z.record(z.string(), z.any()).describe("JSON Schema for the resume form").optional() }).describe("Form schema for collecting input when resuming").optional(), "user_auth_required": z.boolean().describe("If true, only authenticated users can approve").optional(), "user_groups_required": z.discriminatedUnion("type", [z.object({ "value": z.any().describe("The static value. For resources, use format '$res:path/to/resource'").optional(), "type": z.literal("static") }).describe("Static value passed directly to the step. Use for hardcoded values or resource references like '$res:path/to/resource'"), z.object({ "expr": z.string().describe("JavaScript expression returning the value. Available variables - results (object with all previous step results), flow_input (flow inputs), flow_input.iter (in loops)"), "type": z.literal("javascript") }).describe("JavaScript expression evaluated at runtime. Can reference previous step results via 'results.step_id' or flow inputs via 'flow_input.property'. Inside for loops, use 'flow_input.iter.value' for the current iteration value (in while loops it equals 'flow_input.iter.index')"), z.object({ "type": z.literal("ai") }).describe("Value resolved by the AI runtime for this input. The AI engine decides how to satisfy the parameter.")]).describe("Maps input parameters for a step. Can be a static value or a JavaScript expression that references previous results or flow inputs").optional(), "self_approval_disabled": z.boolean().describe("If true, the user who started the flow cannot approve").optional(), "hide_cancel": z.boolean().describe("If true, hide the cancel button on the approval form").optional(), "continue_on_disapprove_timeout": z.boolean().describe("If true, continue flow on timeout instead of canceling").optional() }).describe("Configuration for approval/resume steps that wait for user input").optional(), "priority": z.number().describe("Execution priority for this step (higher numbers run first)").optional(), "continue_on_error": z.boolean().describe("If true, flow continues even if this step fails").optional(), "retry": z.object({ "constant": z.object({ "attempts": z.number().int().describe("Number of retry attempts").optional(), "seconds": z.number().int().describe("Seconds to wait between retries").optional() }).describe("Retry with constant delay between attempts").optional(), "exponential": z.object({ "attempts": z.number().int().describe("Number of retry attempts").optional(), "multiplier": z.number().int().describe("Multiplier for exponential backoff").optional(), "seconds": z.number().int().gte(1).describe("Initial delay in seconds").optional(), "random_factor": z.number().int().gte(0).lte(100).describe("Random jitter percentage (0-100) to avoid thundering herd").optional() }).describe("Retry with exponential backoff (delay doubles each time)").optional(), "retry_if": z.object({ "expr": z.string().describe("JavaScript expression that returns true to retry. Has access to 'result' and 'error' variables") }).describe("Conditional retry based on error or result").optional() }).describe("Retry configuration for failed module executions").optional(), "debouncing": z.object({ "debounce_delay_s": z.number().int().describe("Delay in seconds to debounce this step's executions across flow runs").optional(), "debounce_key": z.string().describe("Expression to group debounced executions. Supports $workspace and $args[name]. Default: $workspace/flow/-").optional(), "debounce_args_to_accumulate": z.array(z.string()).describe("Array-type arguments to accumulate across debounced executions").optional(), "max_total_debouncing_time": z.number().int().describe("Maximum total time in seconds before forced execution").optional(), "max_total_debounces_amount": z.number().int().describe("Maximum number of debounces before forced execution").optional() }).describe("Debounce configuration for this step (EE only)").optional() }).describe("A single step in a flow. Can be a script, subflow, loop, or branch")).describe("Steps to execute if no branch expressions match"), "type": z.literal("branchone") }).describe("Conditional branching where only the first matching branch executes. Branches are evaluated in order, and the first one with a true expression runs. If no branches match, the default branch executes"), z.object({ "branches": z.array(z.object({ "summary": z.string().describe("Short description of this branch's purpose").optional(), "skip_failure": z.boolean().describe("If true, failure in this branch doesn't fail the entire flow").optional(), "modules": z.array(z.object({ "id": z.string().describe("Unique identifier for this step. Used to reference results via 'results.step_id'. Must be a valid identifier (alphanumeric, underscore, hyphen)"), "value": z.lazy(() => flowModuleValueSchema), "stop_after_if": z.object({ "skip_if_stopped": z.boolean().describe("If true, following steps are skipped when this condition triggers").optional(), "expr": z.string().describe("JavaScript expression evaluated after the module runs. Can use 'result' (step's result) or 'flow_input'. Return true to stop"), "error_message": z.string().nullable().describe("Custom error message when stopping with an error. Mutually exclusive with skip_if_stopped. If set to a non-empty string, the flow stops with this error. If empty string, a default error message is used. If null or omitted, no error is raised.").optional(), "error_include_result": z.boolean().describe("When stopping with an error (error_message set), embed the stopping step's own result inside the raised error object (as error.result) instead of discarding it. The top-level result stays { error }. Defaults to false.").optional() }).describe("Early termination condition for a module").optional(), "stop_after_all_iters_if": z.object({ "skip_if_stopped": z.boolean().describe("If true, following steps are skipped when this condition triggers").optional(), "expr": z.string().describe("JavaScript expression evaluated after the module runs. Can use 'result' (step's result) or 'flow_input'. Return true to stop"), "error_message": z.string().nullable().describe("Custom error message when stopping with an error. Mutually exclusive with skip_if_stopped. If set to a non-empty string, the flow stops with this error. If empty string, a default error message is used. If null or omitted, no error is raised.").optional(), "error_include_result": z.boolean().describe("When stopping with an error (error_message set), embed the stopping step's own result inside the raised error object (as error.result) instead of discarding it. The top-level result stays { error }. Defaults to false.").optional() }).describe("Early termination condition for a module").optional(), "skip_if": z.object({ "expr": z.string().describe("JavaScript expression that returns true to skip. Can use 'flow_input' or 'results.'") }).describe("Conditionally skip this step based on previous results or flow inputs").optional(), "sleep": z.discriminatedUnion("type", [z.object({ "value": z.any().describe("The static value. For resources, use format '$res:path/to/resource'").optional(), "type": z.literal("static") }).describe("Static value passed directly to the step. Use for hardcoded values or resource references like '$res:path/to/resource'"), z.object({ "expr": z.string().describe("JavaScript expression returning the value. Available variables - results (object with all previous step results), flow_input (flow inputs), flow_input.iter (in loops)"), "type": z.literal("javascript") }).describe("JavaScript expression evaluated at runtime. Can reference previous step results via 'results.step_id' or flow inputs via 'flow_input.property'. Inside for loops, use 'flow_input.iter.value' for the current iteration value (in while loops it equals 'flow_input.iter.index')"), z.object({ "type": z.literal("ai") }).describe("Value resolved by the AI runtime for this input. The AI engine decides how to satisfy the parameter.")]).describe("Maps input parameters for a step. Can be a static value or a JavaScript expression that references previous results or flow inputs").optional(), "cache_ttl": z.number().describe("Cache duration in seconds for this step's results").optional(), "cache_ignore_s3_path": z.boolean().optional(), "timeout": z.discriminatedUnion("type", [z.object({ "value": z.any().describe("The static value. For resources, use format '$res:path/to/resource'").optional(), "type": z.literal("static") }).describe("Static value passed directly to the step. Use for hardcoded values or resource references like '$res:path/to/resource'"), z.object({ "expr": z.string().describe("JavaScript expression returning the value. Available variables - results (object with all previous step results), flow_input (flow inputs), flow_input.iter (in loops)"), "type": z.literal("javascript") }).describe("JavaScript expression evaluated at runtime. Can reference previous step results via 'results.step_id' or flow inputs via 'flow_input.property'. Inside for loops, use 'flow_input.iter.value' for the current iteration value (in while loops it equals 'flow_input.iter.index')"), z.object({ "type": z.literal("ai") }).describe("Value resolved by the AI runtime for this input. The AI engine decides how to satisfy the parameter.")]).describe("Maps input parameters for a step. Can be a static value or a JavaScript expression that references previous results or flow inputs").optional(), "delete_after_secs": z.number().int().describe("If set, delete the step's args, result and logs after this many seconds following job completion").optional(), "summary": z.string().describe("Short description of what this step does").optional(), "mock": z.object({ "enabled": z.boolean().describe("If true, return mock value instead of executing").optional(), "return_value": z.any().describe("Value to return when mocked").optional() }).describe("Mock configuration for testing without executing the actual step").optional(), "suspend": z.object({ "required_events": z.number().int().describe("Number of approvals required before continuing").optional(), "timeout": z.number().int().describe("Timeout in seconds before auto-continuing or canceling").optional(), "resume_form": z.object({ "schema": z.record(z.string(), z.any()).describe("JSON Schema for the resume form").optional() }).describe("Form schema for collecting input when resuming").optional(), "user_auth_required": z.boolean().describe("If true, only authenticated users can approve").optional(), "user_groups_required": z.discriminatedUnion("type", [z.object({ "value": z.any().describe("The static value. For resources, use format '$res:path/to/resource'").optional(), "type": z.literal("static") }).describe("Static value passed directly to the step. Use for hardcoded values or resource references like '$res:path/to/resource'"), z.object({ "expr": z.string().describe("JavaScript expression returning the value. Available variables - results (object with all previous step results), flow_input (flow inputs), flow_input.iter (in loops)"), "type": z.literal("javascript") }).describe("JavaScript expression evaluated at runtime. Can reference previous step results via 'results.step_id' or flow inputs via 'flow_input.property'. Inside for loops, use 'flow_input.iter.value' for the current iteration value (in while loops it equals 'flow_input.iter.index')"), z.object({ "type": z.literal("ai") }).describe("Value resolved by the AI runtime for this input. The AI engine decides how to satisfy the parameter.")]).describe("Maps input parameters for a step. Can be a static value or a JavaScript expression that references previous results or flow inputs").optional(), "self_approval_disabled": z.boolean().describe("If true, the user who started the flow cannot approve").optional(), "hide_cancel": z.boolean().describe("If true, hide the cancel button on the approval form").optional(), "continue_on_disapprove_timeout": z.boolean().describe("If true, continue flow on timeout instead of canceling").optional() }).describe("Configuration for approval/resume steps that wait for user input").optional(), "priority": z.number().describe("Execution priority for this step (higher numbers run first)").optional(), "continue_on_error": z.boolean().describe("If true, flow continues even if this step fails").optional(), "retry": z.object({ "constant": z.object({ "attempts": z.number().int().describe("Number of retry attempts").optional(), "seconds": z.number().int().describe("Seconds to wait between retries").optional() }).describe("Retry with constant delay between attempts").optional(), "exponential": z.object({ "attempts": z.number().int().describe("Number of retry attempts").optional(), "multiplier": z.number().int().describe("Multiplier for exponential backoff").optional(), "seconds": z.number().int().gte(1).describe("Initial delay in seconds").optional(), "random_factor": z.number().int().gte(0).lte(100).describe("Random jitter percentage (0-100) to avoid thundering herd").optional() }).describe("Retry with exponential backoff (delay doubles each time)").optional(), "retry_if": z.object({ "expr": z.string().describe("JavaScript expression that returns true to retry. Has access to 'result' and 'error' variables") }).describe("Conditional retry based on error or result").optional() }).describe("Retry configuration for failed module executions").optional(), "debouncing": z.object({ "debounce_delay_s": z.number().int().describe("Delay in seconds to debounce this step's executions across flow runs").optional(), "debounce_key": z.string().describe("Expression to group debounced executions. Supports $workspace and $args[name]. Default: $workspace/flow/-").optional(), "debounce_args_to_accumulate": z.array(z.string()).describe("Array-type arguments to accumulate across debounced executions").optional(), "max_total_debouncing_time": z.number().int().describe("Maximum total time in seconds before forced execution").optional(), "max_total_debounces_amount": z.number().int().describe("Maximum number of debounces before forced execution").optional() }).describe("Debounce configuration for this step (EE only)").optional() }).describe("A single step in a flow. Can be a script, subflow, loop, or branch")).describe("Steps to execute in this branch") })).describe("Array of branches that all execute (either in parallel or sequentially)"), "type": z.literal("branchall"), "parallel": z.boolean().describe("If true, all branches execute concurrently. If false, they execute sequentially").optional() }).describe("Parallel branching where all branches execute simultaneously. Unlike BranchOne, all branches run regardless of conditions. Useful for executing independent tasks concurrently"), z.object({ "type": z.literal("identity"), "flow": z.boolean().describe("If true, marks this as a flow identity (special handling)").optional() }).describe("Pass-through module that returns its input unchanged. Useful for flow structure or as a placeholder"), z.object({ "input_transforms": z.object({ "provider": z.discriminatedUnion("type", [z.object({ "value": z.object({ "kind": z.enum(["openai","azure_openai","azure_foundry","anthropic","mistral","deepseek","googleai","groq","openrouter","togetherai","customai","aws_bedrock"]).describe("Supported AI provider types"), "resource": z.string().describe("Resource reference in format '$res:{resource_path}' pointing to provider credentials"), "model": z.string().describe("Model identifier (e.g., 'gpt-4', 'claude-3-opus-20240229', 'gemini-pro')"), "reasoning_effort": z.string().describe("Provider-native reasoning effort token (e.g. 'low', 'high', 'none') for models that support extended thinking. Optional; unset leaves the provider default.").optional() }).describe("Complete AI provider configuration with resource reference and model selection"), "type": z.literal("static") }).describe("Static provider configuration passed directly to the AI agent"), z.object({ "expr": z.string().describe("JavaScript expression returning the value. Available variables - results (object with all previous step results), flow_input (flow inputs), flow_input.iter (in loops)"), "type": z.literal("javascript") }).describe("JavaScript expression evaluated at runtime. Can reference previous step results via 'results.step_id' or flow inputs via 'flow_input.property'. Inside for loops, use 'flow_input.iter.value' for the current iteration value (in while loops it equals 'flow_input.iter.index')"), z.object({ "type": z.literal("ai") }).describe("Value resolved by the AI runtime for this input. The AI engine decides how to satisfy the parameter.")]).describe("Provider configuration - can be static (ProviderConfig), JavaScript expression, or AI-determined").optional(), "output_type": z.discriminatedUnion("type", [z.object({ "value": z.any().describe("The static value. For resources, use format '$res:path/to/resource'").optional(), "type": z.literal("static") }).describe("Static value passed directly to the step. Use for hardcoded values or resource references like '$res:path/to/resource'"), z.object({ "expr": z.string().describe("JavaScript expression returning the value. Available variables - results (object with all previous step results), flow_input (flow inputs), flow_input.iter (in loops)"), "type": z.literal("javascript") }).describe("JavaScript expression evaluated at runtime. Can reference previous step results via 'results.step_id' or flow inputs via 'flow_input.property'. Inside for loops, use 'flow_input.iter.value' for the current iteration value (in while loops it equals 'flow_input.iter.index')"), z.object({ "type": z.literal("ai") }).describe("Value resolved by the AI runtime for this input. The AI engine decides how to satisfy the parameter.")]).describe("Maps input parameters for a step. Can be a static value or a JavaScript expression that references previous results or flow inputs").describe("Output format type.\nValid values: 'text' (default) - plain text response, 'image' - image generation\n").optional(), "user_message": z.discriminatedUnion("type", [z.object({ "value": z.any().describe("The static value. For resources, use format '$res:path/to/resource'").optional(), "type": z.literal("static") }).describe("Static value passed directly to the step. Use for hardcoded values or resource references like '$res:path/to/resource'"), z.object({ "expr": z.string().describe("JavaScript expression returning the value. Available variables - results (object with all previous step results), flow_input (flow inputs), flow_input.iter (in loops)"), "type": z.literal("javascript") }).describe("JavaScript expression evaluated at runtime. Can reference previous step results via 'results.step_id' or flow inputs via 'flow_input.property'. Inside for loops, use 'flow_input.iter.value' for the current iteration value (in while loops it equals 'flow_input.iter.index')"), z.object({ "type": z.literal("ai") }).describe("Value resolved by the AI runtime for this input. The AI engine decides how to satisfy the parameter.")]).describe("Maps input parameters for a step. Can be a static value or a JavaScript expression that references previous results or flow inputs").describe("The user's prompt/message to the AI agent. Supports variable interpolation with flow.input syntax."), "system_prompt": z.discriminatedUnion("type", [z.object({ "value": z.any().describe("The static value. For resources, use format '$res:path/to/resource'").optional(), "type": z.literal("static") }).describe("Static value passed directly to the step. Use for hardcoded values or resource references like '$res:path/to/resource'"), z.object({ "expr": z.string().describe("JavaScript expression returning the value. Available variables - results (object with all previous step results), flow_input (flow inputs), flow_input.iter (in loops)"), "type": z.literal("javascript") }).describe("JavaScript expression evaluated at runtime. Can reference previous step results via 'results.step_id' or flow inputs via 'flow_input.property'. Inside for loops, use 'flow_input.iter.value' for the current iteration value (in while loops it equals 'flow_input.iter.index')"), z.object({ "type": z.literal("ai") }).describe("Value resolved by the AI runtime for this input. The AI engine decides how to satisfy the parameter.")]).describe("Maps input parameters for a step. Can be a static value or a JavaScript expression that references previous results or flow inputs").describe("System instructions that guide the AI's behavior, persona, and response style. Optional.").optional(), "streaming": z.discriminatedUnion("type", [z.object({ "value": z.any().describe("The static value. For resources, use format '$res:path/to/resource'").optional(), "type": z.literal("static") }).describe("Static value passed directly to the step. Use for hardcoded values or resource references like '$res:path/to/resource'"), z.object({ "expr": z.string().describe("JavaScript expression returning the value. Available variables - results (object with all previous step results), flow_input (flow inputs), flow_input.iter (in loops)"), "type": z.literal("javascript") }).describe("JavaScript expression evaluated at runtime. Can reference previous step results via 'results.step_id' or flow inputs via 'flow_input.property'. Inside for loops, use 'flow_input.iter.value' for the current iteration value (in while loops it equals 'flow_input.iter.index')"), z.object({ "type": z.literal("ai") }).describe("Value resolved by the AI runtime for this input. The AI engine decides how to satisfy the parameter.")]).describe("Maps input parameters for a step. Can be a static value or a JavaScript expression that references previous results or flow inputs").describe("Boolean. If true, stream the AI response incrementally.\nStreaming events include: token_delta, reasoning_token_delta, tool_call, tool_call_arguments, tool_execution, tool_result\n").optional(), "memory": z.discriminatedUnion("type", [z.object({ "value": z.discriminatedUnion("kind", [z.object({ "kind": z.literal("off") }).describe("No conversation memory/context"), z.object({ "kind": z.literal("auto"), "context_length": z.number().int().describe("Maximum number of messages to retain in context").optional(), "memory_id": z.string().describe("Identifier for persistent memory across agent invocations").optional() }).describe("Automatic context management"), z.object({ "kind": z.literal("manual"), "messages": z.array(z.object({ "role": z.enum(["user","assistant","system"]), "content": z.string() }).describe("A single message in conversation history")) }).describe("Explicit message history")]).describe("Conversation memory configuration"), "type": z.literal("static") }).describe("Static memory configuration passed directly to the AI agent"), z.object({ "expr": z.string().describe("JavaScript expression returning the value. Available variables - results (object with all previous step results), flow_input (flow inputs), flow_input.iter (in loops)"), "type": z.literal("javascript") }).describe("JavaScript expression evaluated at runtime. Can reference previous step results via 'results.step_id' or flow inputs via 'flow_input.property'. Inside for loops, use 'flow_input.iter.value' for the current iteration value (in while loops it equals 'flow_input.iter.index')"), z.object({ "type": z.literal("ai") }).describe("Value resolved by the AI runtime for this input. The AI engine decides how to satisfy the parameter.")]).describe("Memory configuration - can be static (MemoryConfig), JavaScript expression, or AI-determined").optional(), "output_schema": z.discriminatedUnion("type", [z.object({ "value": z.any().describe("The static value. For resources, use format '$res:path/to/resource'").optional(), "type": z.literal("static") }).describe("Static value passed directly to the step. Use for hardcoded values or resource references like '$res:path/to/resource'"), z.object({ "expr": z.string().describe("JavaScript expression returning the value. Available variables - results (object with all previous step results), flow_input (flow inputs), flow_input.iter (in loops)"), "type": z.literal("javascript") }).describe("JavaScript expression evaluated at runtime. Can reference previous step results via 'results.step_id' or flow inputs via 'flow_input.property'. Inside for loops, use 'flow_input.iter.value' for the current iteration value (in while loops it equals 'flow_input.iter.index')"), z.object({ "type": z.literal("ai") }).describe("Value resolved by the AI runtime for this input. The AI engine decides how to satisfy the parameter.")]).describe("Maps input parameters for a step. Can be a static value or a JavaScript expression that references previous results or flow inputs").describe("JSON Schema object defining structured output format. Used when you need the AI to return data in a specific shape.\nSupports standard JSON Schema properties: type, properties, required, items, enum, pattern, minLength, maxLength, minimum, maximum, etc.\nExample: { type: 'object', properties: { name: { type: 'string' }, age: { type: 'integer' } }, required: ['name'] }\n").optional(), "user_attachments": z.discriminatedUnion("type", [z.object({ "value": z.any().describe("The static value. For resources, use format '$res:path/to/resource'").optional(), "type": z.literal("static") }).describe("Static value passed directly to the step. Use for hardcoded values or resource references like '$res:path/to/resource'"), z.object({ "expr": z.string().describe("JavaScript expression returning the value. Available variables - results (object with all previous step results), flow_input (flow inputs), flow_input.iter (in loops)"), "type": z.literal("javascript") }).describe("JavaScript expression evaluated at runtime. Can reference previous step results via 'results.step_id' or flow inputs via 'flow_input.property'. Inside for loops, use 'flow_input.iter.value' for the current iteration value (in while loops it equals 'flow_input.iter.index')"), z.object({ "type": z.literal("ai") }).describe("Value resolved by the AI runtime for this input. The AI engine decides how to satisfy the parameter.")]).describe("Maps input parameters for a step. Can be a static value or a JavaScript expression that references previous results or flow inputs").describe("Array of file references (images or PDFs) for the AI agent.\nFormat: Array<{ bucket: string, key: string }> - S3 object references\nExample: [{ bucket: 'my-bucket', key: 'documents/report.pdf' }]\n").optional(), "max_completion_tokens": z.discriminatedUnion("type", [z.object({ "value": z.any().describe("The static value. For resources, use format '$res:path/to/resource'").optional(), "type": z.literal("static") }).describe("Static value passed directly to the step. Use for hardcoded values or resource references like '$res:path/to/resource'"), z.object({ "expr": z.string().describe("JavaScript expression returning the value. Available variables - results (object with all previous step results), flow_input (flow inputs), flow_input.iter (in loops)"), "type": z.literal("javascript") }).describe("JavaScript expression evaluated at runtime. Can reference previous step results via 'results.step_id' or flow inputs via 'flow_input.property'. Inside for loops, use 'flow_input.iter.value' for the current iteration value (in while loops it equals 'flow_input.iter.index')"), z.object({ "type": z.literal("ai") }).describe("Value resolved by the AI runtime for this input. The AI engine decides how to satisfy the parameter.")]).describe("Maps input parameters for a step. Can be a static value or a JavaScript expression that references previous results or flow inputs").describe("Integer. Maximum number of tokens the AI will generate in its response.\nRange: 1 to 4,294,967,295. Typical values: 256-4096 for most use cases.\n").optional(), "temperature": z.discriminatedUnion("type", [z.object({ "value": z.any().describe("The static value. For resources, use format '$res:path/to/resource'").optional(), "type": z.literal("static") }).describe("Static value passed directly to the step. Use for hardcoded values or resource references like '$res:path/to/resource'"), z.object({ "expr": z.string().describe("JavaScript expression returning the value. Available variables - results (object with all previous step results), flow_input (flow inputs), flow_input.iter (in loops)"), "type": z.literal("javascript") }).describe("JavaScript expression evaluated at runtime. Can reference previous step results via 'results.step_id' or flow inputs via 'flow_input.property'. Inside for loops, use 'flow_input.iter.value' for the current iteration value (in while loops it equals 'flow_input.iter.index')"), z.object({ "type": z.literal("ai") }).describe("Value resolved by the AI runtime for this input. The AI engine decides how to satisfy the parameter.")]).describe("Maps input parameters for a step. Can be a static value or a JavaScript expression that references previous results or flow inputs").describe("Float. Controls randomness/creativity of responses.\nRange: 0.0 to 2.0 (provider-dependent)\n- 0.0 = deterministic, focused responses\n- 0.7 = balanced (common default)\n- 1.0+ = more creative/random\n").optional(), "max_iterations": z.discriminatedUnion("type", [z.object({ "value": z.any().describe("The static value. For resources, use format '$res:path/to/resource'").optional(), "type": z.literal("static") }).describe("Static value passed directly to the step. Use for hardcoded values or resource references like '$res:path/to/resource'"), z.object({ "expr": z.string().describe("JavaScript expression returning the value. Available variables - results (object with all previous step results), flow_input (flow inputs), flow_input.iter (in loops)"), "type": z.literal("javascript") }).describe("JavaScript expression evaluated at runtime. Can reference previous step results via 'results.step_id' or flow inputs via 'flow_input.property'. Inside for loops, use 'flow_input.iter.value' for the current iteration value (in while loops it equals 'flow_input.iter.index')"), z.object({ "type": z.literal("ai") }).describe("Value resolved by the AI runtime for this input. The AI engine decides how to satisfy the parameter.")]).describe("Maps input parameters for a step. Can be a static value or a JavaScript expression that references previous results or flow inputs").describe("Number. Limits how many times the agent can loop through reasoning and tool use.\nRange: 1-1000.\n").optional() }).describe("Input parameters for the AI agent mapped to their values"), "tools": z.array(z.object({ "id": z.string().describe("Unique identifier for this tool. Cannot contain spaces - use underscores instead (e.g., 'get_user_data' not 'get user data')"), "summary": z.string().describe("Short description of what this tool does (shown to the AI)").optional(), "description": z.string().describe("Free-text description of the tool given to the AI to decide when and how to call it. Overrides the description auto-derived from the underlying script.").optional(), "value": z.any().superRefine((x, ctx) => { +export const flowModuleValueSchema = z.discriminatedUnion("type", [z.object({ "input_transforms": z.record(z.string(), z.discriminatedUnion("type", [z.object({ "value": z.any().describe("The static value. For resources, use format '$res:path/to/resource'").optional(), "type": z.literal("static") }).describe("Static value passed directly to the step. Use for hardcoded values or resource references like '$res:path/to/resource'"), z.object({ "expr": z.string().describe("JavaScript expression returning the value. Available variables - results (object with all previous step results), flow_input (flow inputs), flow_input.iter (in loops)"), "type": z.literal("javascript") }).describe("JavaScript expression evaluated at runtime. Can reference previous step results via 'results.step_id' or flow inputs via 'flow_input.property'. Inside for loops, use 'flow_input.iter.value' for the current iteration value (in while loops it equals 'flow_input.iter.index')"), z.object({ "type": z.literal("ai") }).describe("Value resolved by the AI runtime for this input. The AI engine decides how to satisfy the parameter.")]).describe("Maps input parameters for a step. Can be a static value or a JavaScript expression that references previous results or flow inputs")).describe("Map of parameter names to their values (static or JavaScript expressions). These become the script's input arguments"), "content": z.string().describe("The script source code. Should export a 'main' function"), "language": z.enum(["deno","bun","bunnative","python3","go","bash","powershell","postgresql","mysql","bigquery","snowflake","mssql","oracledb","graphql","nativets","php","rust","ansible","csharp","nu","java","ruby","rlang","duckdb"]).describe("Programming language for this script"), "path": z.string().describe("Optional path for saving this script").optional(), "lock": z.string().describe("Lock file content for dependencies").optional(), "type": z.literal("rawscript"), "tag": z.string().describe("Worker group tag for execution routing").optional(), "concurrent_limit": z.number().describe("Maximum concurrent executions of this script").optional(), "concurrency_time_window_s": z.number().describe("Time window for concurrent_limit").optional(), "custom_concurrency_key": z.string().describe("Custom key for grouping concurrent executions").optional(), "is_trigger": z.boolean().describe("If true, this script is a trigger that can start the flow").optional(), "assets": z.array(z.object({ "path": z.string().describe("Path to the asset"), "kind": z.enum(["s3object","resource","ducklake","datatable","volume","dbt"]).describe("Type of asset"), "access_type": z.union([z.literal("r"), z.literal("w"), z.literal("rw"), z.literal(null)]).nullable().describe("Access level for this asset").optional(), "alt_access_type": z.union([z.literal("r"), z.literal("w"), z.literal("rw"), z.literal(null)]).nullable().describe("Alternative access level").optional() })).describe("External resources this script accesses (S3 objects, resources, etc.)").optional() }).describe("Inline script with code defined directly in the flow. Use 'bun' as default language if unspecified. The script receives arguments from input_transforms"), z.object({ "input_transforms": z.record(z.string(), z.discriminatedUnion("type", [z.object({ "value": z.any().describe("The static value. For resources, use format '$res:path/to/resource'").optional(), "type": z.literal("static") }).describe("Static value passed directly to the step. Use for hardcoded values or resource references like '$res:path/to/resource'"), z.object({ "expr": z.string().describe("JavaScript expression returning the value. Available variables - results (object with all previous step results), flow_input (flow inputs), flow_input.iter (in loops)"), "type": z.literal("javascript") }).describe("JavaScript expression evaluated at runtime. Can reference previous step results via 'results.step_id' or flow inputs via 'flow_input.property'. Inside for loops, use 'flow_input.iter.value' for the current iteration value (in while loops it equals 'flow_input.iter.index')"), z.object({ "type": z.literal("ai") }).describe("Value resolved by the AI runtime for this input. The AI engine decides how to satisfy the parameter.")]).describe("Maps input parameters for a step. Can be a static value or a JavaScript expression that references previous results or flow inputs")).describe("Map of parameter names to their values (static or JavaScript expressions). These become the script's input arguments"), "path": z.string().describe("Path to the script in the workspace (e.g., 'f/scripts/send_email')"), "hash": z.string().describe("Optional specific version hash of the script to use").optional(), "type": z.literal("script"), "tag_override": z.string().describe("Override the script's default worker group tag").optional(), "is_trigger": z.boolean().describe("If true, this script is a trigger that can start the flow").optional() }).describe("Reference to an existing script by path. Use this when calling a previously saved script instead of writing inline code"), z.object({ "input_transforms": z.record(z.string(), z.discriminatedUnion("type", [z.object({ "value": z.any().describe("The static value. For resources, use format '$res:path/to/resource'").optional(), "type": z.literal("static") }).describe("Static value passed directly to the step. Use for hardcoded values or resource references like '$res:path/to/resource'"), z.object({ "expr": z.string().describe("JavaScript expression returning the value. Available variables - results (object with all previous step results), flow_input (flow inputs), flow_input.iter (in loops)"), "type": z.literal("javascript") }).describe("JavaScript expression evaluated at runtime. Can reference previous step results via 'results.step_id' or flow inputs via 'flow_input.property'. Inside for loops, use 'flow_input.iter.value' for the current iteration value (in while loops it equals 'flow_input.iter.index')"), z.object({ "type": z.literal("ai") }).describe("Value resolved by the AI runtime for this input. The AI engine decides how to satisfy the parameter.")]).describe("Maps input parameters for a step. Can be a static value or a JavaScript expression that references previous results or flow inputs")).describe("Map of parameter names to their values (static or JavaScript expressions). These become the subflow's input arguments"), "path": z.string().describe("Path to the flow in the workspace (e.g., 'f/flows/process_user')"), "type": z.literal("flow") }).describe("Reference to an existing flow by path. Use this to call another flow as a subflow"), z.object({ "modules": z.array(z.object({ "id": z.string().describe("Unique identifier for this step. Used to reference results via 'results.step_id'. Must be a valid identifier (alphanumeric, underscore, hyphen)"), "value": z.lazy(() => flowModuleValueSchema), "stop_after_if": z.object({ "skip_if_stopped": z.boolean().describe("If true, following steps are skipped when this condition triggers").optional(), "expr": z.string().describe("JavaScript expression evaluated after the module runs. Can use 'result' (step's result) or 'flow_input'. Return true to stop"), "error_message": z.string().nullable().describe("Custom error message when stopping with an error. Mutually exclusive with skip_if_stopped. If set to a non-empty string, the flow stops with this error. If empty string, a default error message is used. If null or omitted, no error is raised.").optional(), "error_include_result": z.boolean().describe("When stopping with an error (error_message set), embed the stopping step's own result inside the raised error object (as error.result) instead of discarding it. The top-level result stays { error }. Defaults to false.").optional() }).describe("Early termination condition for a module").optional(), "stop_after_all_iters_if": z.object({ "skip_if_stopped": z.boolean().describe("If true, following steps are skipped when this condition triggers").optional(), "expr": z.string().describe("JavaScript expression evaluated after the module runs. Can use 'result' (step's result) or 'flow_input'. Return true to stop"), "error_message": z.string().nullable().describe("Custom error message when stopping with an error. Mutually exclusive with skip_if_stopped. If set to a non-empty string, the flow stops with this error. If empty string, a default error message is used. If null or omitted, no error is raised.").optional(), "error_include_result": z.boolean().describe("When stopping with an error (error_message set), embed the stopping step's own result inside the raised error object (as error.result) instead of discarding it. The top-level result stays { error }. Defaults to false.").optional() }).describe("Early termination condition for a module").optional(), "skip_if": z.object({ "expr": z.string().describe("JavaScript expression that returns true to skip. Can use 'flow_input' or 'results.'") }).describe("Conditionally skip this step based on previous results or flow inputs").optional(), "sleep": z.discriminatedUnion("type", [z.object({ "value": z.any().describe("The static value. For resources, use format '$res:path/to/resource'").optional(), "type": z.literal("static") }).describe("Static value passed directly to the step. Use for hardcoded values or resource references like '$res:path/to/resource'"), z.object({ "expr": z.string().describe("JavaScript expression returning the value. Available variables - results (object with all previous step results), flow_input (flow inputs), flow_input.iter (in loops)"), "type": z.literal("javascript") }).describe("JavaScript expression evaluated at runtime. Can reference previous step results via 'results.step_id' or flow inputs via 'flow_input.property'. Inside for loops, use 'flow_input.iter.value' for the current iteration value (in while loops it equals 'flow_input.iter.index')"), z.object({ "type": z.literal("ai") }).describe("Value resolved by the AI runtime for this input. The AI engine decides how to satisfy the parameter.")]).describe("Maps input parameters for a step. Can be a static value or a JavaScript expression that references previous results or flow inputs").optional(), "cache_ttl": z.number().describe("Cache duration in seconds for this step's results").optional(), "cache_ignore_s3_path": z.boolean().optional(), "timeout": z.discriminatedUnion("type", [z.object({ "value": z.any().describe("The static value. For resources, use format '$res:path/to/resource'").optional(), "type": z.literal("static") }).describe("Static value passed directly to the step. Use for hardcoded values or resource references like '$res:path/to/resource'"), z.object({ "expr": z.string().describe("JavaScript expression returning the value. Available variables - results (object with all previous step results), flow_input (flow inputs), flow_input.iter (in loops)"), "type": z.literal("javascript") }).describe("JavaScript expression evaluated at runtime. Can reference previous step results via 'results.step_id' or flow inputs via 'flow_input.property'. Inside for loops, use 'flow_input.iter.value' for the current iteration value (in while loops it equals 'flow_input.iter.index')"), z.object({ "type": z.literal("ai") }).describe("Value resolved by the AI runtime for this input. The AI engine decides how to satisfy the parameter.")]).describe("Maps input parameters for a step. Can be a static value or a JavaScript expression that references previous results or flow inputs").optional(), "delete_after_secs": z.number().int().describe("If set, delete the step's args, result and logs after this many seconds following job completion").optional(), "summary": z.string().describe("Short description of what this step does").optional(), "mock": z.object({ "enabled": z.boolean().describe("If true, return mock value instead of executing").optional(), "return_value": z.any().describe("Value to return when mocked").optional() }).describe("Mock configuration for testing without executing the actual step").optional(), "suspend": z.object({ "required_events": z.number().int().describe("Number of approvals required before continuing").optional(), "timeout": z.number().int().describe("Timeout in seconds before auto-continuing or canceling").optional(), "resume_form": z.object({ "schema": z.record(z.string(), z.any()).describe("JSON Schema for the resume form").optional() }).describe("Form schema for collecting input when resuming").optional(), "user_auth_required": z.boolean().describe("If true, only authenticated users can approve").optional(), "user_groups_required": z.discriminatedUnion("type", [z.object({ "value": z.any().describe("The static value. For resources, use format '$res:path/to/resource'").optional(), "type": z.literal("static") }).describe("Static value passed directly to the step. Use for hardcoded values or resource references like '$res:path/to/resource'"), z.object({ "expr": z.string().describe("JavaScript expression returning the value. Available variables - results (object with all previous step results), flow_input (flow inputs), flow_input.iter (in loops)"), "type": z.literal("javascript") }).describe("JavaScript expression evaluated at runtime. Can reference previous step results via 'results.step_id' or flow inputs via 'flow_input.property'. Inside for loops, use 'flow_input.iter.value' for the current iteration value (in while loops it equals 'flow_input.iter.index')"), z.object({ "type": z.literal("ai") }).describe("Value resolved by the AI runtime for this input. The AI engine decides how to satisfy the parameter.")]).describe("Maps input parameters for a step. Can be a static value or a JavaScript expression that references previous results or flow inputs").optional(), "self_approval_disabled": z.boolean().describe("If true, the user who started the flow cannot approve").optional(), "hide_cancel": z.boolean().describe("If true, hide the cancel button on the approval form").optional(), "continue_on_disapprove_timeout": z.boolean().describe("If true, continue flow on timeout instead of canceling").optional() }).describe("Configuration for approval/resume steps that wait for user input").optional(), "priority": z.number().describe("Execution priority for this step (higher numbers run first)").optional(), "continue_on_error": z.boolean().describe("If true, flow continues even if this step fails").optional(), "retry": z.object({ "constant": z.object({ "attempts": z.number().int().describe("Number of retry attempts").optional(), "seconds": z.number().int().describe("Seconds to wait between retries").optional() }).describe("Retry with constant delay between attempts").optional(), "exponential": z.object({ "attempts": z.number().int().describe("Number of retry attempts").optional(), "multiplier": z.number().int().describe("Multiplier for exponential backoff").optional(), "seconds": z.number().int().gte(1).describe("Initial delay in seconds").optional(), "random_factor": z.number().int().gte(0).lte(100).describe("Random jitter percentage (0-100) to avoid thundering herd").optional() }).describe("Retry with exponential backoff (delay doubles each time)").optional(), "retry_if": z.object({ "expr": z.string().describe("JavaScript expression that returns true to retry. Has access to 'result' and 'error' variables") }).describe("Conditional retry based on error or result").optional() }).describe("Retry configuration for failed module executions").optional(), "debouncing": z.object({ "debounce_delay_s": z.number().int().describe("Delay in seconds to debounce this step's executions across flow runs").optional(), "debounce_key": z.string().describe("Expression to group debounced executions. Supports $workspace and $args[name]. Default: $workspace/flow/-").optional(), "debounce_args_to_accumulate": z.array(z.string()).describe("Array-type arguments to accumulate across debounced executions").optional(), "max_total_debouncing_time": z.number().int().describe("Maximum total time in seconds before forced execution").optional(), "max_total_debounces_amount": z.number().int().describe("Maximum number of debounces before forced execution").optional() }).describe("Debounce configuration for this step (EE only)").optional() }).describe("A single step in a flow. Can be a script, subflow, loop, or branch")).describe("Steps to execute for each iteration. These can reference the iteration value via 'flow_input.iter.value'"), "iterator": z.discriminatedUnion("type", [z.object({ "value": z.any().describe("The static value. For resources, use format '$res:path/to/resource'").optional(), "type": z.literal("static") }).describe("Static value passed directly to the step. Use for hardcoded values or resource references like '$res:path/to/resource'"), z.object({ "expr": z.string().describe("JavaScript expression returning the value. Available variables - results (object with all previous step results), flow_input (flow inputs), flow_input.iter (in loops)"), "type": z.literal("javascript") }).describe("JavaScript expression evaluated at runtime. Can reference previous step results via 'results.step_id' or flow inputs via 'flow_input.property'. Inside for loops, use 'flow_input.iter.value' for the current iteration value (in while loops it equals 'flow_input.iter.index')"), z.object({ "type": z.literal("ai") }).describe("Value resolved by the AI runtime for this input. The AI engine decides how to satisfy the parameter.")]).describe("Maps input parameters for a step. Can be a static value or a JavaScript expression that references previous results or flow inputs"), "skip_failures": z.boolean().describe("If true, iteration failures don't stop the loop. Failed iterations return null"), "type": z.literal("forloopflow"), "parallel": z.boolean().describe("If true, iterations run concurrently (faster for I/O-bound operations). Use with parallelism to control concurrency").optional(), "parallelism": z.discriminatedUnion("type", [z.object({ "value": z.any().describe("The static value. For resources, use format '$res:path/to/resource'").optional(), "type": z.literal("static") }).describe("Static value passed directly to the step. Use for hardcoded values or resource references like '$res:path/to/resource'"), z.object({ "expr": z.string().describe("JavaScript expression returning the value. Available variables - results (object with all previous step results), flow_input (flow inputs), flow_input.iter (in loops)"), "type": z.literal("javascript") }).describe("JavaScript expression evaluated at runtime. Can reference previous step results via 'results.step_id' or flow inputs via 'flow_input.property'. Inside for loops, use 'flow_input.iter.value' for the current iteration value (in while loops it equals 'flow_input.iter.index')"), z.object({ "type": z.literal("ai") }).describe("Value resolved by the AI runtime for this input. The AI engine decides how to satisfy the parameter.")]).describe("Maps input parameters for a step. Can be a static value or a JavaScript expression that references previous results or flow inputs").optional(), "squash": z.boolean().optional() }).describe("Executes nested modules in a loop over an iterator. Inside the loop, use 'flow_input.iter.value' to access the current iteration value, and 'flow_input.iter.index' for the index. Supports parallel execution for better performance on I/O-bound operations"), z.object({ "modules": z.array(z.object({ "id": z.string().describe("Unique identifier for this step. Used to reference results via 'results.step_id'. Must be a valid identifier (alphanumeric, underscore, hyphen)"), "value": z.lazy(() => flowModuleValueSchema), "stop_after_if": z.object({ "skip_if_stopped": z.boolean().describe("If true, following steps are skipped when this condition triggers").optional(), "expr": z.string().describe("JavaScript expression evaluated after the module runs. Can use 'result' (step's result) or 'flow_input'. Return true to stop"), "error_message": z.string().nullable().describe("Custom error message when stopping with an error. Mutually exclusive with skip_if_stopped. If set to a non-empty string, the flow stops with this error. If empty string, a default error message is used. If null or omitted, no error is raised.").optional(), "error_include_result": z.boolean().describe("When stopping with an error (error_message set), embed the stopping step's own result inside the raised error object (as error.result) instead of discarding it. The top-level result stays { error }. Defaults to false.").optional() }).describe("Early termination condition for a module").optional(), "stop_after_all_iters_if": z.object({ "skip_if_stopped": z.boolean().describe("If true, following steps are skipped when this condition triggers").optional(), "expr": z.string().describe("JavaScript expression evaluated after the module runs. Can use 'result' (step's result) or 'flow_input'. Return true to stop"), "error_message": z.string().nullable().describe("Custom error message when stopping with an error. Mutually exclusive with skip_if_stopped. If set to a non-empty string, the flow stops with this error. If empty string, a default error message is used. If null or omitted, no error is raised.").optional(), "error_include_result": z.boolean().describe("When stopping with an error (error_message set), embed the stopping step's own result inside the raised error object (as error.result) instead of discarding it. The top-level result stays { error }. Defaults to false.").optional() }).describe("Early termination condition for a module").optional(), "skip_if": z.object({ "expr": z.string().describe("JavaScript expression that returns true to skip. Can use 'flow_input' or 'results.'") }).describe("Conditionally skip this step based on previous results or flow inputs").optional(), "sleep": z.discriminatedUnion("type", [z.object({ "value": z.any().describe("The static value. For resources, use format '$res:path/to/resource'").optional(), "type": z.literal("static") }).describe("Static value passed directly to the step. Use for hardcoded values or resource references like '$res:path/to/resource'"), z.object({ "expr": z.string().describe("JavaScript expression returning the value. Available variables - results (object with all previous step results), flow_input (flow inputs), flow_input.iter (in loops)"), "type": z.literal("javascript") }).describe("JavaScript expression evaluated at runtime. Can reference previous step results via 'results.step_id' or flow inputs via 'flow_input.property'. Inside for loops, use 'flow_input.iter.value' for the current iteration value (in while loops it equals 'flow_input.iter.index')"), z.object({ "type": z.literal("ai") }).describe("Value resolved by the AI runtime for this input. The AI engine decides how to satisfy the parameter.")]).describe("Maps input parameters for a step. Can be a static value or a JavaScript expression that references previous results or flow inputs").optional(), "cache_ttl": z.number().describe("Cache duration in seconds for this step's results").optional(), "cache_ignore_s3_path": z.boolean().optional(), "timeout": z.discriminatedUnion("type", [z.object({ "value": z.any().describe("The static value. For resources, use format '$res:path/to/resource'").optional(), "type": z.literal("static") }).describe("Static value passed directly to the step. Use for hardcoded values or resource references like '$res:path/to/resource'"), z.object({ "expr": z.string().describe("JavaScript expression returning the value. Available variables - results (object with all previous step results), flow_input (flow inputs), flow_input.iter (in loops)"), "type": z.literal("javascript") }).describe("JavaScript expression evaluated at runtime. Can reference previous step results via 'results.step_id' or flow inputs via 'flow_input.property'. Inside for loops, use 'flow_input.iter.value' for the current iteration value (in while loops it equals 'flow_input.iter.index')"), z.object({ "type": z.literal("ai") }).describe("Value resolved by the AI runtime for this input. The AI engine decides how to satisfy the parameter.")]).describe("Maps input parameters for a step. Can be a static value or a JavaScript expression that references previous results or flow inputs").optional(), "delete_after_secs": z.number().int().describe("If set, delete the step's args, result and logs after this many seconds following job completion").optional(), "summary": z.string().describe("Short description of what this step does").optional(), "mock": z.object({ "enabled": z.boolean().describe("If true, return mock value instead of executing").optional(), "return_value": z.any().describe("Value to return when mocked").optional() }).describe("Mock configuration for testing without executing the actual step").optional(), "suspend": z.object({ "required_events": z.number().int().describe("Number of approvals required before continuing").optional(), "timeout": z.number().int().describe("Timeout in seconds before auto-continuing or canceling").optional(), "resume_form": z.object({ "schema": z.record(z.string(), z.any()).describe("JSON Schema for the resume form").optional() }).describe("Form schema for collecting input when resuming").optional(), "user_auth_required": z.boolean().describe("If true, only authenticated users can approve").optional(), "user_groups_required": z.discriminatedUnion("type", [z.object({ "value": z.any().describe("The static value. For resources, use format '$res:path/to/resource'").optional(), "type": z.literal("static") }).describe("Static value passed directly to the step. Use for hardcoded values or resource references like '$res:path/to/resource'"), z.object({ "expr": z.string().describe("JavaScript expression returning the value. Available variables - results (object with all previous step results), flow_input (flow inputs), flow_input.iter (in loops)"), "type": z.literal("javascript") }).describe("JavaScript expression evaluated at runtime. Can reference previous step results via 'results.step_id' or flow inputs via 'flow_input.property'. Inside for loops, use 'flow_input.iter.value' for the current iteration value (in while loops it equals 'flow_input.iter.index')"), z.object({ "type": z.literal("ai") }).describe("Value resolved by the AI runtime for this input. The AI engine decides how to satisfy the parameter.")]).describe("Maps input parameters for a step. Can be a static value or a JavaScript expression that references previous results or flow inputs").optional(), "self_approval_disabled": z.boolean().describe("If true, the user who started the flow cannot approve").optional(), "hide_cancel": z.boolean().describe("If true, hide the cancel button on the approval form").optional(), "continue_on_disapprove_timeout": z.boolean().describe("If true, continue flow on timeout instead of canceling").optional() }).describe("Configuration for approval/resume steps that wait for user input").optional(), "priority": z.number().describe("Execution priority for this step (higher numbers run first)").optional(), "continue_on_error": z.boolean().describe("If true, flow continues even if this step fails").optional(), "retry": z.object({ "constant": z.object({ "attempts": z.number().int().describe("Number of retry attempts").optional(), "seconds": z.number().int().describe("Seconds to wait between retries").optional() }).describe("Retry with constant delay between attempts").optional(), "exponential": z.object({ "attempts": z.number().int().describe("Number of retry attempts").optional(), "multiplier": z.number().int().describe("Multiplier for exponential backoff").optional(), "seconds": z.number().int().gte(1).describe("Initial delay in seconds").optional(), "random_factor": z.number().int().gte(0).lte(100).describe("Random jitter percentage (0-100) to avoid thundering herd").optional() }).describe("Retry with exponential backoff (delay doubles each time)").optional(), "retry_if": z.object({ "expr": z.string().describe("JavaScript expression that returns true to retry. Has access to 'result' and 'error' variables") }).describe("Conditional retry based on error or result").optional() }).describe("Retry configuration for failed module executions").optional(), "debouncing": z.object({ "debounce_delay_s": z.number().int().describe("Delay in seconds to debounce this step's executions across flow runs").optional(), "debounce_key": z.string().describe("Expression to group debounced executions. Supports $workspace and $args[name]. Default: $workspace/flow/-").optional(), "debounce_args_to_accumulate": z.array(z.string()).describe("Array-type arguments to accumulate across debounced executions").optional(), "max_total_debouncing_time": z.number().int().describe("Maximum total time in seconds before forced execution").optional(), "max_total_debounces_amount": z.number().int().describe("Maximum number of debounces before forced execution").optional() }).describe("Debounce configuration for this step (EE only)").optional() }).describe("A single step in a flow. Can be a script, subflow, loop, or branch")).describe("Steps to execute in each iteration"), "skip_failures": z.boolean().describe("If true, iteration failures don't stop the loop. Failed iterations return null"), "type": z.literal("whileloopflow"), "parallel": z.boolean().describe("If true, iterations run concurrently (use with caution in while loops)").optional(), "parallelism": z.discriminatedUnion("type", [z.object({ "value": z.any().describe("The static value. For resources, use format '$res:path/to/resource'").optional(), "type": z.literal("static") }).describe("Static value passed directly to the step. Use for hardcoded values or resource references like '$res:path/to/resource'"), z.object({ "expr": z.string().describe("JavaScript expression returning the value. Available variables - results (object with all previous step results), flow_input (flow inputs), flow_input.iter (in loops)"), "type": z.literal("javascript") }).describe("JavaScript expression evaluated at runtime. Can reference previous step results via 'results.step_id' or flow inputs via 'flow_input.property'. Inside for loops, use 'flow_input.iter.value' for the current iteration value (in while loops it equals 'flow_input.iter.index')"), z.object({ "type": z.literal("ai") }).describe("Value resolved by the AI runtime for this input. The AI engine decides how to satisfy the parameter.")]).describe("Maps input parameters for a step. Can be a static value or a JavaScript expression that references previous results or flow inputs").optional(), "squash": z.boolean().optional() }).describe("Executes nested modules repeatedly until stopped. The implicit iterator is the iteration counter, so 'flow_input.iter.value' equals 'flow_input.iter.index' (0, 1, 2, ...) and never carries state. To carry state across iterations, a step reads its own previous-iteration result via 'results.' with a first-iteration fallback - the loop's stop_after_if must then be on that inner step (a plain single-step body with stop_after_if on the loop module does not resolve 'results' across iterations and never terminates); plain counters can instead be derived from 'flow_input.iter.index', which works in every configuration. stop_after_if is evaluated after each iteration - on the loop module 'result' is the last iteration's result"), z.object({ "branches": z.array(z.object({ "summary": z.string().describe("Short description of this branch condition").optional(), "expr": z.string().describe("JavaScript expression that returns boolean. Can use 'results.step_id' or 'flow_input'. First true expr wins"), "modules": z.array(z.object({ "id": z.string().describe("Unique identifier for this step. Used to reference results via 'results.step_id'. Must be a valid identifier (alphanumeric, underscore, hyphen)"), "value": z.lazy(() => flowModuleValueSchema), "stop_after_if": z.object({ "skip_if_stopped": z.boolean().describe("If true, following steps are skipped when this condition triggers").optional(), "expr": z.string().describe("JavaScript expression evaluated after the module runs. Can use 'result' (step's result) or 'flow_input'. Return true to stop"), "error_message": z.string().nullable().describe("Custom error message when stopping with an error. Mutually exclusive with skip_if_stopped. If set to a non-empty string, the flow stops with this error. If empty string, a default error message is used. If null or omitted, no error is raised.").optional(), "error_include_result": z.boolean().describe("When stopping with an error (error_message set), embed the stopping step's own result inside the raised error object (as error.result) instead of discarding it. The top-level result stays { error }. Defaults to false.").optional() }).describe("Early termination condition for a module").optional(), "stop_after_all_iters_if": z.object({ "skip_if_stopped": z.boolean().describe("If true, following steps are skipped when this condition triggers").optional(), "expr": z.string().describe("JavaScript expression evaluated after the module runs. Can use 'result' (step's result) or 'flow_input'. Return true to stop"), "error_message": z.string().nullable().describe("Custom error message when stopping with an error. Mutually exclusive with skip_if_stopped. If set to a non-empty string, the flow stops with this error. If empty string, a default error message is used. If null or omitted, no error is raised.").optional(), "error_include_result": z.boolean().describe("When stopping with an error (error_message set), embed the stopping step's own result inside the raised error object (as error.result) instead of discarding it. The top-level result stays { error }. Defaults to false.").optional() }).describe("Early termination condition for a module").optional(), "skip_if": z.object({ "expr": z.string().describe("JavaScript expression that returns true to skip. Can use 'flow_input' or 'results.'") }).describe("Conditionally skip this step based on previous results or flow inputs").optional(), "sleep": z.discriminatedUnion("type", [z.object({ "value": z.any().describe("The static value. For resources, use format '$res:path/to/resource'").optional(), "type": z.literal("static") }).describe("Static value passed directly to the step. Use for hardcoded values or resource references like '$res:path/to/resource'"), z.object({ "expr": z.string().describe("JavaScript expression returning the value. Available variables - results (object with all previous step results), flow_input (flow inputs), flow_input.iter (in loops)"), "type": z.literal("javascript") }).describe("JavaScript expression evaluated at runtime. Can reference previous step results via 'results.step_id' or flow inputs via 'flow_input.property'. Inside for loops, use 'flow_input.iter.value' for the current iteration value (in while loops it equals 'flow_input.iter.index')"), z.object({ "type": z.literal("ai") }).describe("Value resolved by the AI runtime for this input. The AI engine decides how to satisfy the parameter.")]).describe("Maps input parameters for a step. Can be a static value or a JavaScript expression that references previous results or flow inputs").optional(), "cache_ttl": z.number().describe("Cache duration in seconds for this step's results").optional(), "cache_ignore_s3_path": z.boolean().optional(), "timeout": z.discriminatedUnion("type", [z.object({ "value": z.any().describe("The static value. For resources, use format '$res:path/to/resource'").optional(), "type": z.literal("static") }).describe("Static value passed directly to the step. Use for hardcoded values or resource references like '$res:path/to/resource'"), z.object({ "expr": z.string().describe("JavaScript expression returning the value. Available variables - results (object with all previous step results), flow_input (flow inputs), flow_input.iter (in loops)"), "type": z.literal("javascript") }).describe("JavaScript expression evaluated at runtime. Can reference previous step results via 'results.step_id' or flow inputs via 'flow_input.property'. Inside for loops, use 'flow_input.iter.value' for the current iteration value (in while loops it equals 'flow_input.iter.index')"), z.object({ "type": z.literal("ai") }).describe("Value resolved by the AI runtime for this input. The AI engine decides how to satisfy the parameter.")]).describe("Maps input parameters for a step. Can be a static value or a JavaScript expression that references previous results or flow inputs").optional(), "delete_after_secs": z.number().int().describe("If set, delete the step's args, result and logs after this many seconds following job completion").optional(), "summary": z.string().describe("Short description of what this step does").optional(), "mock": z.object({ "enabled": z.boolean().describe("If true, return mock value instead of executing").optional(), "return_value": z.any().describe("Value to return when mocked").optional() }).describe("Mock configuration for testing without executing the actual step").optional(), "suspend": z.object({ "required_events": z.number().int().describe("Number of approvals required before continuing").optional(), "timeout": z.number().int().describe("Timeout in seconds before auto-continuing or canceling").optional(), "resume_form": z.object({ "schema": z.record(z.string(), z.any()).describe("JSON Schema for the resume form").optional() }).describe("Form schema for collecting input when resuming").optional(), "user_auth_required": z.boolean().describe("If true, only authenticated users can approve").optional(), "user_groups_required": z.discriminatedUnion("type", [z.object({ "value": z.any().describe("The static value. For resources, use format '$res:path/to/resource'").optional(), "type": z.literal("static") }).describe("Static value passed directly to the step. Use for hardcoded values or resource references like '$res:path/to/resource'"), z.object({ "expr": z.string().describe("JavaScript expression returning the value. Available variables - results (object with all previous step results), flow_input (flow inputs), flow_input.iter (in loops)"), "type": z.literal("javascript") }).describe("JavaScript expression evaluated at runtime. Can reference previous step results via 'results.step_id' or flow inputs via 'flow_input.property'. Inside for loops, use 'flow_input.iter.value' for the current iteration value (in while loops it equals 'flow_input.iter.index')"), z.object({ "type": z.literal("ai") }).describe("Value resolved by the AI runtime for this input. The AI engine decides how to satisfy the parameter.")]).describe("Maps input parameters for a step. Can be a static value or a JavaScript expression that references previous results or flow inputs").optional(), "self_approval_disabled": z.boolean().describe("If true, the user who started the flow cannot approve").optional(), "hide_cancel": z.boolean().describe("If true, hide the cancel button on the approval form").optional(), "continue_on_disapprove_timeout": z.boolean().describe("If true, continue flow on timeout instead of canceling").optional() }).describe("Configuration for approval/resume steps that wait for user input").optional(), "priority": z.number().describe("Execution priority for this step (higher numbers run first)").optional(), "continue_on_error": z.boolean().describe("If true, flow continues even if this step fails").optional(), "retry": z.object({ "constant": z.object({ "attempts": z.number().int().describe("Number of retry attempts").optional(), "seconds": z.number().int().describe("Seconds to wait between retries").optional() }).describe("Retry with constant delay between attempts").optional(), "exponential": z.object({ "attempts": z.number().int().describe("Number of retry attempts").optional(), "multiplier": z.number().int().describe("Multiplier for exponential backoff").optional(), "seconds": z.number().int().gte(1).describe("Initial delay in seconds").optional(), "random_factor": z.number().int().gte(0).lte(100).describe("Random jitter percentage (0-100) to avoid thundering herd").optional() }).describe("Retry with exponential backoff (delay doubles each time)").optional(), "retry_if": z.object({ "expr": z.string().describe("JavaScript expression that returns true to retry. Has access to 'result' and 'error' variables") }).describe("Conditional retry based on error or result").optional() }).describe("Retry configuration for failed module executions").optional(), "debouncing": z.object({ "debounce_delay_s": z.number().int().describe("Delay in seconds to debounce this step's executions across flow runs").optional(), "debounce_key": z.string().describe("Expression to group debounced executions. Supports $workspace and $args[name]. Default: $workspace/flow/-").optional(), "debounce_args_to_accumulate": z.array(z.string()).describe("Array-type arguments to accumulate across debounced executions").optional(), "max_total_debouncing_time": z.number().int().describe("Maximum total time in seconds before forced execution").optional(), "max_total_debounces_amount": z.number().int().describe("Maximum number of debounces before forced execution").optional() }).describe("Debounce configuration for this step (EE only)").optional() }).describe("A single step in a flow. Can be a script, subflow, loop, or branch")).describe("Steps to execute if this branch's expr is true") })).describe("Array of branches to evaluate in order. The first branch with expr evaluating to true executes"), "default": z.array(z.object({ "id": z.string().describe("Unique identifier for this step. Used to reference results via 'results.step_id'. Must be a valid identifier (alphanumeric, underscore, hyphen)"), "value": z.lazy(() => flowModuleValueSchema), "stop_after_if": z.object({ "skip_if_stopped": z.boolean().describe("If true, following steps are skipped when this condition triggers").optional(), "expr": z.string().describe("JavaScript expression evaluated after the module runs. Can use 'result' (step's result) or 'flow_input'. Return true to stop"), "error_message": z.string().nullable().describe("Custom error message when stopping with an error. Mutually exclusive with skip_if_stopped. If set to a non-empty string, the flow stops with this error. If empty string, a default error message is used. If null or omitted, no error is raised.").optional(), "error_include_result": z.boolean().describe("When stopping with an error (error_message set), embed the stopping step's own result inside the raised error object (as error.result) instead of discarding it. The top-level result stays { error }. Defaults to false.").optional() }).describe("Early termination condition for a module").optional(), "stop_after_all_iters_if": z.object({ "skip_if_stopped": z.boolean().describe("If true, following steps are skipped when this condition triggers").optional(), "expr": z.string().describe("JavaScript expression evaluated after the module runs. Can use 'result' (step's result) or 'flow_input'. Return true to stop"), "error_message": z.string().nullable().describe("Custom error message when stopping with an error. Mutually exclusive with skip_if_stopped. If set to a non-empty string, the flow stops with this error. If empty string, a default error message is used. If null or omitted, no error is raised.").optional(), "error_include_result": z.boolean().describe("When stopping with an error (error_message set), embed the stopping step's own result inside the raised error object (as error.result) instead of discarding it. The top-level result stays { error }. Defaults to false.").optional() }).describe("Early termination condition for a module").optional(), "skip_if": z.object({ "expr": z.string().describe("JavaScript expression that returns true to skip. Can use 'flow_input' or 'results.'") }).describe("Conditionally skip this step based on previous results or flow inputs").optional(), "sleep": z.discriminatedUnion("type", [z.object({ "value": z.any().describe("The static value. For resources, use format '$res:path/to/resource'").optional(), "type": z.literal("static") }).describe("Static value passed directly to the step. Use for hardcoded values or resource references like '$res:path/to/resource'"), z.object({ "expr": z.string().describe("JavaScript expression returning the value. Available variables - results (object with all previous step results), flow_input (flow inputs), flow_input.iter (in loops)"), "type": z.literal("javascript") }).describe("JavaScript expression evaluated at runtime. Can reference previous step results via 'results.step_id' or flow inputs via 'flow_input.property'. Inside for loops, use 'flow_input.iter.value' for the current iteration value (in while loops it equals 'flow_input.iter.index')"), z.object({ "type": z.literal("ai") }).describe("Value resolved by the AI runtime for this input. The AI engine decides how to satisfy the parameter.")]).describe("Maps input parameters for a step. Can be a static value or a JavaScript expression that references previous results or flow inputs").optional(), "cache_ttl": z.number().describe("Cache duration in seconds for this step's results").optional(), "cache_ignore_s3_path": z.boolean().optional(), "timeout": z.discriminatedUnion("type", [z.object({ "value": z.any().describe("The static value. For resources, use format '$res:path/to/resource'").optional(), "type": z.literal("static") }).describe("Static value passed directly to the step. Use for hardcoded values or resource references like '$res:path/to/resource'"), z.object({ "expr": z.string().describe("JavaScript expression returning the value. Available variables - results (object with all previous step results), flow_input (flow inputs), flow_input.iter (in loops)"), "type": z.literal("javascript") }).describe("JavaScript expression evaluated at runtime. Can reference previous step results via 'results.step_id' or flow inputs via 'flow_input.property'. Inside for loops, use 'flow_input.iter.value' for the current iteration value (in while loops it equals 'flow_input.iter.index')"), z.object({ "type": z.literal("ai") }).describe("Value resolved by the AI runtime for this input. The AI engine decides how to satisfy the parameter.")]).describe("Maps input parameters for a step. Can be a static value or a JavaScript expression that references previous results or flow inputs").optional(), "delete_after_secs": z.number().int().describe("If set, delete the step's args, result and logs after this many seconds following job completion").optional(), "summary": z.string().describe("Short description of what this step does").optional(), "mock": z.object({ "enabled": z.boolean().describe("If true, return mock value instead of executing").optional(), "return_value": z.any().describe("Value to return when mocked").optional() }).describe("Mock configuration for testing without executing the actual step").optional(), "suspend": z.object({ "required_events": z.number().int().describe("Number of approvals required before continuing").optional(), "timeout": z.number().int().describe("Timeout in seconds before auto-continuing or canceling").optional(), "resume_form": z.object({ "schema": z.record(z.string(), z.any()).describe("JSON Schema for the resume form").optional() }).describe("Form schema for collecting input when resuming").optional(), "user_auth_required": z.boolean().describe("If true, only authenticated users can approve").optional(), "user_groups_required": z.discriminatedUnion("type", [z.object({ "value": z.any().describe("The static value. For resources, use format '$res:path/to/resource'").optional(), "type": z.literal("static") }).describe("Static value passed directly to the step. Use for hardcoded values or resource references like '$res:path/to/resource'"), z.object({ "expr": z.string().describe("JavaScript expression returning the value. Available variables - results (object with all previous step results), flow_input (flow inputs), flow_input.iter (in loops)"), "type": z.literal("javascript") }).describe("JavaScript expression evaluated at runtime. Can reference previous step results via 'results.step_id' or flow inputs via 'flow_input.property'. Inside for loops, use 'flow_input.iter.value' for the current iteration value (in while loops it equals 'flow_input.iter.index')"), z.object({ "type": z.literal("ai") }).describe("Value resolved by the AI runtime for this input. The AI engine decides how to satisfy the parameter.")]).describe("Maps input parameters for a step. Can be a static value or a JavaScript expression that references previous results or flow inputs").optional(), "self_approval_disabled": z.boolean().describe("If true, the user who started the flow cannot approve").optional(), "hide_cancel": z.boolean().describe("If true, hide the cancel button on the approval form").optional(), "continue_on_disapprove_timeout": z.boolean().describe("If true, continue flow on timeout instead of canceling").optional() }).describe("Configuration for approval/resume steps that wait for user input").optional(), "priority": z.number().describe("Execution priority for this step (higher numbers run first)").optional(), "continue_on_error": z.boolean().describe("If true, flow continues even if this step fails").optional(), "retry": z.object({ "constant": z.object({ "attempts": z.number().int().describe("Number of retry attempts").optional(), "seconds": z.number().int().describe("Seconds to wait between retries").optional() }).describe("Retry with constant delay between attempts").optional(), "exponential": z.object({ "attempts": z.number().int().describe("Number of retry attempts").optional(), "multiplier": z.number().int().describe("Multiplier for exponential backoff").optional(), "seconds": z.number().int().gte(1).describe("Initial delay in seconds").optional(), "random_factor": z.number().int().gte(0).lte(100).describe("Random jitter percentage (0-100) to avoid thundering herd").optional() }).describe("Retry with exponential backoff (delay doubles each time)").optional(), "retry_if": z.object({ "expr": z.string().describe("JavaScript expression that returns true to retry. Has access to 'result' and 'error' variables") }).describe("Conditional retry based on error or result").optional() }).describe("Retry configuration for failed module executions").optional(), "debouncing": z.object({ "debounce_delay_s": z.number().int().describe("Delay in seconds to debounce this step's executions across flow runs").optional(), "debounce_key": z.string().describe("Expression to group debounced executions. Supports $workspace and $args[name]. Default: $workspace/flow/-").optional(), "debounce_args_to_accumulate": z.array(z.string()).describe("Array-type arguments to accumulate across debounced executions").optional(), "max_total_debouncing_time": z.number().int().describe("Maximum total time in seconds before forced execution").optional(), "max_total_debounces_amount": z.number().int().describe("Maximum number of debounces before forced execution").optional() }).describe("Debounce configuration for this step (EE only)").optional() }).describe("A single step in a flow. Can be a script, subflow, loop, or branch")).describe("Steps to execute if no branch expressions match"), "type": z.literal("branchone") }).describe("Conditional branching where only the first matching branch executes. Branches are evaluated in order, and the first one with a true expression runs. If no branches match, the default branch executes"), z.object({ "branches": z.array(z.object({ "summary": z.string().describe("Short description of this branch's purpose").optional(), "skip_failure": z.boolean().describe("If true, failure in this branch doesn't fail the entire flow").optional(), "modules": z.array(z.object({ "id": z.string().describe("Unique identifier for this step. Used to reference results via 'results.step_id'. Must be a valid identifier (alphanumeric, underscore, hyphen)"), "value": z.lazy(() => flowModuleValueSchema), "stop_after_if": z.object({ "skip_if_stopped": z.boolean().describe("If true, following steps are skipped when this condition triggers").optional(), "expr": z.string().describe("JavaScript expression evaluated after the module runs. Can use 'result' (step's result) or 'flow_input'. Return true to stop"), "error_message": z.string().nullable().describe("Custom error message when stopping with an error. Mutually exclusive with skip_if_stopped. If set to a non-empty string, the flow stops with this error. If empty string, a default error message is used. If null or omitted, no error is raised.").optional(), "error_include_result": z.boolean().describe("When stopping with an error (error_message set), embed the stopping step's own result inside the raised error object (as error.result) instead of discarding it. The top-level result stays { error }. Defaults to false.").optional() }).describe("Early termination condition for a module").optional(), "stop_after_all_iters_if": z.object({ "skip_if_stopped": z.boolean().describe("If true, following steps are skipped when this condition triggers").optional(), "expr": z.string().describe("JavaScript expression evaluated after the module runs. Can use 'result' (step's result) or 'flow_input'. Return true to stop"), "error_message": z.string().nullable().describe("Custom error message when stopping with an error. Mutually exclusive with skip_if_stopped. If set to a non-empty string, the flow stops with this error. If empty string, a default error message is used. If null or omitted, no error is raised.").optional(), "error_include_result": z.boolean().describe("When stopping with an error (error_message set), embed the stopping step's own result inside the raised error object (as error.result) instead of discarding it. The top-level result stays { error }. Defaults to false.").optional() }).describe("Early termination condition for a module").optional(), "skip_if": z.object({ "expr": z.string().describe("JavaScript expression that returns true to skip. Can use 'flow_input' or 'results.'") }).describe("Conditionally skip this step based on previous results or flow inputs").optional(), "sleep": z.discriminatedUnion("type", [z.object({ "value": z.any().describe("The static value. For resources, use format '$res:path/to/resource'").optional(), "type": z.literal("static") }).describe("Static value passed directly to the step. Use for hardcoded values or resource references like '$res:path/to/resource'"), z.object({ "expr": z.string().describe("JavaScript expression returning the value. Available variables - results (object with all previous step results), flow_input (flow inputs), flow_input.iter (in loops)"), "type": z.literal("javascript") }).describe("JavaScript expression evaluated at runtime. Can reference previous step results via 'results.step_id' or flow inputs via 'flow_input.property'. Inside for loops, use 'flow_input.iter.value' for the current iteration value (in while loops it equals 'flow_input.iter.index')"), z.object({ "type": z.literal("ai") }).describe("Value resolved by the AI runtime for this input. The AI engine decides how to satisfy the parameter.")]).describe("Maps input parameters for a step. Can be a static value or a JavaScript expression that references previous results or flow inputs").optional(), "cache_ttl": z.number().describe("Cache duration in seconds for this step's results").optional(), "cache_ignore_s3_path": z.boolean().optional(), "timeout": z.discriminatedUnion("type", [z.object({ "value": z.any().describe("The static value. For resources, use format '$res:path/to/resource'").optional(), "type": z.literal("static") }).describe("Static value passed directly to the step. Use for hardcoded values or resource references like '$res:path/to/resource'"), z.object({ "expr": z.string().describe("JavaScript expression returning the value. Available variables - results (object with all previous step results), flow_input (flow inputs), flow_input.iter (in loops)"), "type": z.literal("javascript") }).describe("JavaScript expression evaluated at runtime. Can reference previous step results via 'results.step_id' or flow inputs via 'flow_input.property'. Inside for loops, use 'flow_input.iter.value' for the current iteration value (in while loops it equals 'flow_input.iter.index')"), z.object({ "type": z.literal("ai") }).describe("Value resolved by the AI runtime for this input. The AI engine decides how to satisfy the parameter.")]).describe("Maps input parameters for a step. Can be a static value or a JavaScript expression that references previous results or flow inputs").optional(), "delete_after_secs": z.number().int().describe("If set, delete the step's args, result and logs after this many seconds following job completion").optional(), "summary": z.string().describe("Short description of what this step does").optional(), "mock": z.object({ "enabled": z.boolean().describe("If true, return mock value instead of executing").optional(), "return_value": z.any().describe("Value to return when mocked").optional() }).describe("Mock configuration for testing without executing the actual step").optional(), "suspend": z.object({ "required_events": z.number().int().describe("Number of approvals required before continuing").optional(), "timeout": z.number().int().describe("Timeout in seconds before auto-continuing or canceling").optional(), "resume_form": z.object({ "schema": z.record(z.string(), z.any()).describe("JSON Schema for the resume form").optional() }).describe("Form schema for collecting input when resuming").optional(), "user_auth_required": z.boolean().describe("If true, only authenticated users can approve").optional(), "user_groups_required": z.discriminatedUnion("type", [z.object({ "value": z.any().describe("The static value. For resources, use format '$res:path/to/resource'").optional(), "type": z.literal("static") }).describe("Static value passed directly to the step. Use for hardcoded values or resource references like '$res:path/to/resource'"), z.object({ "expr": z.string().describe("JavaScript expression returning the value. Available variables - results (object with all previous step results), flow_input (flow inputs), flow_input.iter (in loops)"), "type": z.literal("javascript") }).describe("JavaScript expression evaluated at runtime. Can reference previous step results via 'results.step_id' or flow inputs via 'flow_input.property'. Inside for loops, use 'flow_input.iter.value' for the current iteration value (in while loops it equals 'flow_input.iter.index')"), z.object({ "type": z.literal("ai") }).describe("Value resolved by the AI runtime for this input. The AI engine decides how to satisfy the parameter.")]).describe("Maps input parameters for a step. Can be a static value or a JavaScript expression that references previous results or flow inputs").optional(), "self_approval_disabled": z.boolean().describe("If true, the user who started the flow cannot approve").optional(), "hide_cancel": z.boolean().describe("If true, hide the cancel button on the approval form").optional(), "continue_on_disapprove_timeout": z.boolean().describe("If true, continue flow on timeout instead of canceling").optional() }).describe("Configuration for approval/resume steps that wait for user input").optional(), "priority": z.number().describe("Execution priority for this step (higher numbers run first)").optional(), "continue_on_error": z.boolean().describe("If true, flow continues even if this step fails").optional(), "retry": z.object({ "constant": z.object({ "attempts": z.number().int().describe("Number of retry attempts").optional(), "seconds": z.number().int().describe("Seconds to wait between retries").optional() }).describe("Retry with constant delay between attempts").optional(), "exponential": z.object({ "attempts": z.number().int().describe("Number of retry attempts").optional(), "multiplier": z.number().int().describe("Multiplier for exponential backoff").optional(), "seconds": z.number().int().gte(1).describe("Initial delay in seconds").optional(), "random_factor": z.number().int().gte(0).lte(100).describe("Random jitter percentage (0-100) to avoid thundering herd").optional() }).describe("Retry with exponential backoff (delay doubles each time)").optional(), "retry_if": z.object({ "expr": z.string().describe("JavaScript expression that returns true to retry. Has access to 'result' and 'error' variables") }).describe("Conditional retry based on error or result").optional() }).describe("Retry configuration for failed module executions").optional(), "debouncing": z.object({ "debounce_delay_s": z.number().int().describe("Delay in seconds to debounce this step's executions across flow runs").optional(), "debounce_key": z.string().describe("Expression to group debounced executions. Supports $workspace and $args[name]. Default: $workspace/flow/-").optional(), "debounce_args_to_accumulate": z.array(z.string()).describe("Array-type arguments to accumulate across debounced executions").optional(), "max_total_debouncing_time": z.number().int().describe("Maximum total time in seconds before forced execution").optional(), "max_total_debounces_amount": z.number().int().describe("Maximum number of debounces before forced execution").optional() }).describe("Debounce configuration for this step (EE only)").optional() }).describe("A single step in a flow. Can be a script, subflow, loop, or branch")).describe("Steps to execute in this branch") })).describe("Array of branches that all execute (either in parallel or sequentially)"), "type": z.literal("branchall"), "parallel": z.boolean().describe("If true, all branches execute concurrently. If false, they execute sequentially").optional() }).describe("Parallel branching where all branches execute simultaneously. Unlike BranchOne, all branches run regardless of conditions. Useful for executing independent tasks concurrently"), z.object({ "type": z.literal("identity"), "flow": z.boolean().describe("If true, marks this as a flow identity (special handling)").optional() }).describe("Pass-through module that returns its input unchanged. Useful for flow structure or as a placeholder"), z.object({ "input_transforms": z.object({ "provider": z.discriminatedUnion("type", [z.object({ "value": z.object({ "kind": z.enum(["openai","azure_openai","azure_foundry","anthropic","mistral","deepseek","googleai","groq","openrouter","togetherai","customai","aws_bedrock"]).describe("Supported AI provider types"), "resource": z.string().describe("Resource reference in format '$res:{resource_path}' pointing to provider credentials"), "model": z.string().describe("Model identifier (e.g., 'gpt-4', 'claude-3-opus-20240229', 'gemini-pro')"), "reasoning_effort": z.string().describe("Provider-native reasoning effort token (e.g. 'low', 'high', 'none') for models that support extended thinking. Optional; unset leaves the provider default.").optional() }).describe("Complete AI provider configuration with resource reference and model selection"), "type": z.literal("static") }).describe("Static provider configuration passed directly to the AI agent"), z.object({ "expr": z.string().describe("JavaScript expression returning the value. Available variables - results (object with all previous step results), flow_input (flow inputs), flow_input.iter (in loops)"), "type": z.literal("javascript") }).describe("JavaScript expression evaluated at runtime. Can reference previous step results via 'results.step_id' or flow inputs via 'flow_input.property'. Inside for loops, use 'flow_input.iter.value' for the current iteration value (in while loops it equals 'flow_input.iter.index')"), z.object({ "type": z.literal("ai") }).describe("Value resolved by the AI runtime for this input. The AI engine decides how to satisfy the parameter.")]).describe("Provider configuration - can be static (ProviderConfig), JavaScript expression, or AI-determined").optional(), "output_type": z.discriminatedUnion("type", [z.object({ "value": z.any().describe("The static value. For resources, use format '$res:path/to/resource'").optional(), "type": z.literal("static") }).describe("Static value passed directly to the step. Use for hardcoded values or resource references like '$res:path/to/resource'"), z.object({ "expr": z.string().describe("JavaScript expression returning the value. Available variables - results (object with all previous step results), flow_input (flow inputs), flow_input.iter (in loops)"), "type": z.literal("javascript") }).describe("JavaScript expression evaluated at runtime. Can reference previous step results via 'results.step_id' or flow inputs via 'flow_input.property'. Inside for loops, use 'flow_input.iter.value' for the current iteration value (in while loops it equals 'flow_input.iter.index')"), z.object({ "type": z.literal("ai") }).describe("Value resolved by the AI runtime for this input. The AI engine decides how to satisfy the parameter.")]).describe("Maps input parameters for a step. Can be a static value or a JavaScript expression that references previous results or flow inputs").describe("Output format type.\nValid values: 'text' (default) - plain text response, 'image' - image generation\n").optional(), "user_message": z.discriminatedUnion("type", [z.object({ "value": z.any().describe("The static value. For resources, use format '$res:path/to/resource'").optional(), "type": z.literal("static") }).describe("Static value passed directly to the step. Use for hardcoded values or resource references like '$res:path/to/resource'"), z.object({ "expr": z.string().describe("JavaScript expression returning the value. Available variables - results (object with all previous step results), flow_input (flow inputs), flow_input.iter (in loops)"), "type": z.literal("javascript") }).describe("JavaScript expression evaluated at runtime. Can reference previous step results via 'results.step_id' or flow inputs via 'flow_input.property'. Inside for loops, use 'flow_input.iter.value' for the current iteration value (in while loops it equals 'flow_input.iter.index')"), z.object({ "type": z.literal("ai") }).describe("Value resolved by the AI runtime for this input. The AI engine decides how to satisfy the parameter.")]).describe("Maps input parameters for a step. Can be a static value or a JavaScript expression that references previous results or flow inputs").describe("The user's prompt/message to the AI agent. Supports variable interpolation with flow.input syntax."), "system_prompt": z.discriminatedUnion("type", [z.object({ "value": z.any().describe("The static value. For resources, use format '$res:path/to/resource'").optional(), "type": z.literal("static") }).describe("Static value passed directly to the step. Use for hardcoded values or resource references like '$res:path/to/resource'"), z.object({ "expr": z.string().describe("JavaScript expression returning the value. Available variables - results (object with all previous step results), flow_input (flow inputs), flow_input.iter (in loops)"), "type": z.literal("javascript") }).describe("JavaScript expression evaluated at runtime. Can reference previous step results via 'results.step_id' or flow inputs via 'flow_input.property'. Inside for loops, use 'flow_input.iter.value' for the current iteration value (in while loops it equals 'flow_input.iter.index')"), z.object({ "type": z.literal("ai") }).describe("Value resolved by the AI runtime for this input. The AI engine decides how to satisfy the parameter.")]).describe("Maps input parameters for a step. Can be a static value or a JavaScript expression that references previous results or flow inputs").describe("System instructions that guide the AI's behavior, persona, and response style. Optional.").optional(), "streaming": z.discriminatedUnion("type", [z.object({ "value": z.any().describe("The static value. For resources, use format '$res:path/to/resource'").optional(), "type": z.literal("static") }).describe("Static value passed directly to the step. Use for hardcoded values or resource references like '$res:path/to/resource'"), z.object({ "expr": z.string().describe("JavaScript expression returning the value. Available variables - results (object with all previous step results), flow_input (flow inputs), flow_input.iter (in loops)"), "type": z.literal("javascript") }).describe("JavaScript expression evaluated at runtime. Can reference previous step results via 'results.step_id' or flow inputs via 'flow_input.property'. Inside for loops, use 'flow_input.iter.value' for the current iteration value (in while loops it equals 'flow_input.iter.index')"), z.object({ "type": z.literal("ai") }).describe("Value resolved by the AI runtime for this input. The AI engine decides how to satisfy the parameter.")]).describe("Maps input parameters for a step. Can be a static value or a JavaScript expression that references previous results or flow inputs").describe("Boolean. If true, stream the AI response incrementally.\nStreaming events include: token_delta, reasoning_token_delta, tool_call, tool_call_arguments, tool_execution, tool_result\n").optional(), "memory": z.discriminatedUnion("type", [z.object({ "value": z.discriminatedUnion("kind", [z.object({ "kind": z.literal("off") }).describe("No conversation memory/context"), z.object({ "kind": z.literal("auto"), "context_length": z.number().int().describe("Maximum number of messages to retain in context").optional(), "memory_id": z.string().describe("Identifier for persistent memory across agent invocations").optional() }).describe("Automatic context management"), z.object({ "kind": z.literal("manual"), "messages": z.array(z.object({ "role": z.enum(["user","assistant","system"]), "content": z.string() }).describe("A single message in conversation history")) }).describe("Explicit message history")]).describe("Conversation memory configuration"), "type": z.literal("static") }).describe("Static memory configuration passed directly to the AI agent"), z.object({ "expr": z.string().describe("JavaScript expression returning the value. Available variables - results (object with all previous step results), flow_input (flow inputs), flow_input.iter (in loops)"), "type": z.literal("javascript") }).describe("JavaScript expression evaluated at runtime. Can reference previous step results via 'results.step_id' or flow inputs via 'flow_input.property'. Inside for loops, use 'flow_input.iter.value' for the current iteration value (in while loops it equals 'flow_input.iter.index')"), z.object({ "type": z.literal("ai") }).describe("Value resolved by the AI runtime for this input. The AI engine decides how to satisfy the parameter.")]).describe("Memory configuration - can be static (MemoryConfig), JavaScript expression, or AI-determined").optional(), "output_schema": z.discriminatedUnion("type", [z.object({ "value": z.any().describe("The static value. For resources, use format '$res:path/to/resource'").optional(), "type": z.literal("static") }).describe("Static value passed directly to the step. Use for hardcoded values or resource references like '$res:path/to/resource'"), z.object({ "expr": z.string().describe("JavaScript expression returning the value. Available variables - results (object with all previous step results), flow_input (flow inputs), flow_input.iter (in loops)"), "type": z.literal("javascript") }).describe("JavaScript expression evaluated at runtime. Can reference previous step results via 'results.step_id' or flow inputs via 'flow_input.property'. Inside for loops, use 'flow_input.iter.value' for the current iteration value (in while loops it equals 'flow_input.iter.index')"), z.object({ "type": z.literal("ai") }).describe("Value resolved by the AI runtime for this input. The AI engine decides how to satisfy the parameter.")]).describe("Maps input parameters for a step. Can be a static value or a JavaScript expression that references previous results or flow inputs").describe("JSON Schema object defining structured output format. Used when you need the AI to return data in a specific shape.\nSupports standard JSON Schema properties: type, properties, required, items, enum, pattern, minLength, maxLength, minimum, maximum, etc.\nExample: { type: 'object', properties: { name: { type: 'string' }, age: { type: 'integer' } }, required: ['name'] }\n").optional(), "user_attachments": z.discriminatedUnion("type", [z.object({ "value": z.any().describe("The static value. For resources, use format '$res:path/to/resource'").optional(), "type": z.literal("static") }).describe("Static value passed directly to the step. Use for hardcoded values or resource references like '$res:path/to/resource'"), z.object({ "expr": z.string().describe("JavaScript expression returning the value. Available variables - results (object with all previous step results), flow_input (flow inputs), flow_input.iter (in loops)"), "type": z.literal("javascript") }).describe("JavaScript expression evaluated at runtime. Can reference previous step results via 'results.step_id' or flow inputs via 'flow_input.property'. Inside for loops, use 'flow_input.iter.value' for the current iteration value (in while loops it equals 'flow_input.iter.index')"), z.object({ "type": z.literal("ai") }).describe("Value resolved by the AI runtime for this input. The AI engine decides how to satisfy the parameter.")]).describe("Maps input parameters for a step. Can be a static value or a JavaScript expression that references previous results or flow inputs").describe("Array of file references (images or PDFs) for the AI agent.\nFormat: Array<{ bucket: string, key: string }> - S3 object references\nExample: [{ bucket: 'my-bucket', key: 'documents/report.pdf' }]\n").optional(), "max_completion_tokens": z.discriminatedUnion("type", [z.object({ "value": z.any().describe("The static value. For resources, use format '$res:path/to/resource'").optional(), "type": z.literal("static") }).describe("Static value passed directly to the step. Use for hardcoded values or resource references like '$res:path/to/resource'"), z.object({ "expr": z.string().describe("JavaScript expression returning the value. Available variables - results (object with all previous step results), flow_input (flow inputs), flow_input.iter (in loops)"), "type": z.literal("javascript") }).describe("JavaScript expression evaluated at runtime. Can reference previous step results via 'results.step_id' or flow inputs via 'flow_input.property'. Inside for loops, use 'flow_input.iter.value' for the current iteration value (in while loops it equals 'flow_input.iter.index')"), z.object({ "type": z.literal("ai") }).describe("Value resolved by the AI runtime for this input. The AI engine decides how to satisfy the parameter.")]).describe("Maps input parameters for a step. Can be a static value or a JavaScript expression that references previous results or flow inputs").describe("Integer. Maximum number of tokens the AI will generate in its response.\nRange: 1 to 4,294,967,295. Typical values: 256-4096 for most use cases.\n").optional(), "temperature": z.discriminatedUnion("type", [z.object({ "value": z.any().describe("The static value. For resources, use format '$res:path/to/resource'").optional(), "type": z.literal("static") }).describe("Static value passed directly to the step. Use for hardcoded values or resource references like '$res:path/to/resource'"), z.object({ "expr": z.string().describe("JavaScript expression returning the value. Available variables - results (object with all previous step results), flow_input (flow inputs), flow_input.iter (in loops)"), "type": z.literal("javascript") }).describe("JavaScript expression evaluated at runtime. Can reference previous step results via 'results.step_id' or flow inputs via 'flow_input.property'. Inside for loops, use 'flow_input.iter.value' for the current iteration value (in while loops it equals 'flow_input.iter.index')"), z.object({ "type": z.literal("ai") }).describe("Value resolved by the AI runtime for this input. The AI engine decides how to satisfy the parameter.")]).describe("Maps input parameters for a step. Can be a static value or a JavaScript expression that references previous results or flow inputs").describe("Float. Controls randomness/creativity of responses.\nRange: 0.0 to 2.0 (provider-dependent)\n- 0.0 = deterministic, focused responses\n- 0.7 = balanced (common default)\n- 1.0+ = more creative/random\n").optional(), "max_iterations": z.discriminatedUnion("type", [z.object({ "value": z.any().describe("The static value. For resources, use format '$res:path/to/resource'").optional(), "type": z.literal("static") }).describe("Static value passed directly to the step. Use for hardcoded values or resource references like '$res:path/to/resource'"), z.object({ "expr": z.string().describe("JavaScript expression returning the value. Available variables - results (object with all previous step results), flow_input (flow inputs), flow_input.iter (in loops)"), "type": z.literal("javascript") }).describe("JavaScript expression evaluated at runtime. Can reference previous step results via 'results.step_id' or flow inputs via 'flow_input.property'. Inside for loops, use 'flow_input.iter.value' for the current iteration value (in while loops it equals 'flow_input.iter.index')"), z.object({ "type": z.literal("ai") }).describe("Value resolved by the AI runtime for this input. The AI engine decides how to satisfy the parameter.")]).describe("Maps input parameters for a step. Can be a static value or a JavaScript expression that references previous results or flow inputs").describe("Number. Limits how many times the agent can loop through reasoning and tool use.\nRange: 1-1000.\n").optional() }).describe("Input parameters for the AI agent mapped to their values"), "tools": z.array(z.object({ "id": z.string().describe("Unique identifier for this tool. Cannot contain spaces - use underscores instead (e.g., 'get_user_data' not 'get user data')"), "summary": z.string().describe("The name the AI agent calls this tool by, not a human label. On a flowmodule tool it must match ^[a-zA-Z0-9_]+$ - letters, numbers and underscores only (e.g. 'search_documentation', not 'Search documentation') - and always be set; on an mcp or websearch tool it is a plain label. Put the human-readable explanation in 'description'.").optional(), "description": z.string().describe("Free-text description of the tool given to the AI to decide when and how to call it. Overrides the description auto-derived from the underlying script.").optional(), "value": z.any().superRefine((x, ctx) => { const schemas = [z.intersection(z.object({ "tool_type": z.literal("flowmodule") }), z.lazy(() => flowModuleValueSchema)).describe("A tool implemented as a flow module (script, flow, etc.). The AI can call this like any other flow module"), z.object({ "tool_type": z.literal("mcp"), "resource_path": z.string().describe("Path to the MCP resource/server configuration"), "include_tools": z.array(z.string()).describe("Whitelist of specific tools to include from this MCP server").optional(), "exclude_tools": z.array(z.string()).describe("Blacklist of tools to exclude from this MCP server").optional() }).describe("Reference to an external MCP (Model Context Protocol) tool. The AI can call tools from MCP servers"), z.object({ "tool_type": z.literal("websearch") }).describe("A tool implemented as a websearch tool. The AI can call this like any other websearch tool")]; const errors = schemas.reduce( (errors, schema) => @@ -20,7 +20,7 @@ export const flowModuleValueSchema = z.discriminatedUnion("type", [z.object({ "i }).describe("The implementation of a tool. Can be a flow module (script/flow) or an MCP tool reference") }).describe("A tool available to an AI agent. Can be a flow module or an external MCP (Model Context Protocol) tool")).describe("Array of tools the agent can use. The agent decides which tools to call based on the task").optional(), "type": z.literal("aiagent"), "tag": z.string().describe("Worker group tag for execution routing. If not set, the AI agent step runs on the flow's tag (default `flow`)").optional(), "omit_output_from_conversation": z.boolean().describe("If true, this AI agent step does not persist its assistant or tool messages to the flow conversation when chat mode is enabled.").default(false), "agent": z.string().describe("Path of a reusable `ai_agent` resource (hybrid linking). When set, the agent brain\nconfig (provider/model/system prompt/etc.) and tool set are resolved at runtime from\nthat resource; the module's input_transforms then only carry the flow-local inputs\n(user_message/user_attachments).\n").optional(), "tool_inputs": z.record(z.string(), z.record(z.string(), z.discriminatedUnion("type", [z.object({ "value": z.any().describe("The static value. For resources, use format '$res:path/to/resource'").optional(), "type": z.literal("static") }).describe("Static value passed directly to the step. Use for hardcoded values or resource references like '$res:path/to/resource'"), z.object({ "expr": z.string().describe("JavaScript expression returning the value. Available variables - results (object with all previous step results), flow_input (flow inputs), flow_input.iter (in loops)"), "type": z.literal("javascript") }).describe("JavaScript expression evaluated at runtime. Can reference previous step results via 'results.step_id' or flow inputs via 'flow_input.property'. Inside for loops, use 'flow_input.iter.value' for the current iteration value (in while loops it equals 'flow_input.iter.index')"), z.object({ "type": z.literal("ai") }).describe("Value resolved by the AI runtime for this input. The AI engine decides how to satisfy the parameter.")]).describe("Maps input parameters for a step. Can be a static value or a JavaScript expression that references previous results or flow inputs"))).describe("Host-local wiring for an agent's tool inputs, keyed by tool id then input key. Binds the\nreferenced agent's tools to this flow's context (flow_input/results) without mutating the\nshared resource; overlaid onto the tools' input_transforms at runtime — including when\n`agent` is unset, since a step forked for editing keeps these overrides until it is saved\nback or unlinked.\n").optional(), "parallel": z.boolean().describe("If true, the agent can execute multiple tool calls in parallel").optional() }).describe("AI agent step that can use tools to accomplish tasks. The agent receives inputs and can call any of its configured tools to complete the task")]).describe("The actual implementation of a flow step. Can be a script (inline or referenced), subflow, loop, branch, or special module type") -export const flowModuleSchema = z.object({ "id": z.string().describe("Unique identifier for this step. Used to reference results via 'results.step_id'. Must be a valid identifier (alphanumeric, underscore, hyphen)"), "value": z.discriminatedUnion("type", [z.object({ "input_transforms": z.record(z.string(), z.discriminatedUnion("type", [z.object({ "value": z.any().describe("The static value. For resources, use format '$res:path/to/resource'").optional(), "type": z.literal("static") }).describe("Static value passed directly to the step. Use for hardcoded values or resource references like '$res:path/to/resource'"), z.object({ "expr": z.string().describe("JavaScript expression returning the value. Available variables - results (object with all previous step results), flow_input (flow inputs), flow_input.iter (in loops)"), "type": z.literal("javascript") }).describe("JavaScript expression evaluated at runtime. Can reference previous step results via 'results.step_id' or flow inputs via 'flow_input.property'. Inside for loops, use 'flow_input.iter.value' for the current iteration value (in while loops it equals 'flow_input.iter.index')"), z.object({ "type": z.literal("ai") }).describe("Value resolved by the AI runtime for this input. The AI engine decides how to satisfy the parameter.")]).describe("Maps input parameters for a step. Can be a static value or a JavaScript expression that references previous results or flow inputs")).describe("Map of parameter names to their values (static or JavaScript expressions). These become the script's input arguments"), "content": z.string().describe("The script source code. Should export a 'main' function"), "language": z.enum(["deno","bun","bunnative","python3","go","bash","powershell","postgresql","mysql","bigquery","snowflake","mssql","oracledb","graphql","nativets","php","rust","ansible","csharp","nu","java","ruby","rlang","duckdb"]).describe("Programming language for this script"), "path": z.string().describe("Optional path for saving this script").optional(), "lock": z.string().describe("Lock file content for dependencies").optional(), "type": z.literal("rawscript"), "tag": z.string().describe("Worker group tag for execution routing").optional(), "concurrent_limit": z.number().describe("Maximum concurrent executions of this script").optional(), "concurrency_time_window_s": z.number().describe("Time window for concurrent_limit").optional(), "custom_concurrency_key": z.string().describe("Custom key for grouping concurrent executions").optional(), "is_trigger": z.boolean().describe("If true, this script is a trigger that can start the flow").optional(), "assets": z.array(z.object({ "path": z.string().describe("Path to the asset"), "kind": z.enum(["s3object","resource","ducklake","datatable","volume","dbt"]).describe("Type of asset"), "access_type": z.union([z.literal("r"), z.literal("w"), z.literal("rw"), z.literal(null)]).nullable().describe("Access level for this asset").optional(), "alt_access_type": z.union([z.literal("r"), z.literal("w"), z.literal("rw"), z.literal(null)]).nullable().describe("Alternative access level").optional() })).describe("External resources this script accesses (S3 objects, resources, etc.)").optional() }).describe("Inline script with code defined directly in the flow. Use 'bun' as default language if unspecified. The script receives arguments from input_transforms"), z.object({ "input_transforms": z.record(z.string(), z.discriminatedUnion("type", [z.object({ "value": z.any().describe("The static value. For resources, use format '$res:path/to/resource'").optional(), "type": z.literal("static") }).describe("Static value passed directly to the step. Use for hardcoded values or resource references like '$res:path/to/resource'"), z.object({ "expr": z.string().describe("JavaScript expression returning the value. Available variables - results (object with all previous step results), flow_input (flow inputs), flow_input.iter (in loops)"), "type": z.literal("javascript") }).describe("JavaScript expression evaluated at runtime. Can reference previous step results via 'results.step_id' or flow inputs via 'flow_input.property'. Inside for loops, use 'flow_input.iter.value' for the current iteration value (in while loops it equals 'flow_input.iter.index')"), z.object({ "type": z.literal("ai") }).describe("Value resolved by the AI runtime for this input. The AI engine decides how to satisfy the parameter.")]).describe("Maps input parameters for a step. Can be a static value or a JavaScript expression that references previous results or flow inputs")).describe("Map of parameter names to their values (static or JavaScript expressions). These become the script's input arguments"), "path": z.string().describe("Path to the script in the workspace (e.g., 'f/scripts/send_email')"), "hash": z.string().describe("Optional specific version hash of the script to use").optional(), "type": z.literal("script"), "tag_override": z.string().describe("Override the script's default worker group tag").optional(), "is_trigger": z.boolean().describe("If true, this script is a trigger that can start the flow").optional() }).describe("Reference to an existing script by path. Use this when calling a previously saved script instead of writing inline code"), z.object({ "input_transforms": z.record(z.string(), z.discriminatedUnion("type", [z.object({ "value": z.any().describe("The static value. For resources, use format '$res:path/to/resource'").optional(), "type": z.literal("static") }).describe("Static value passed directly to the step. Use for hardcoded values or resource references like '$res:path/to/resource'"), z.object({ "expr": z.string().describe("JavaScript expression returning the value. Available variables - results (object with all previous step results), flow_input (flow inputs), flow_input.iter (in loops)"), "type": z.literal("javascript") }).describe("JavaScript expression evaluated at runtime. Can reference previous step results via 'results.step_id' or flow inputs via 'flow_input.property'. Inside for loops, use 'flow_input.iter.value' for the current iteration value (in while loops it equals 'flow_input.iter.index')"), z.object({ "type": z.literal("ai") }).describe("Value resolved by the AI runtime for this input. The AI engine decides how to satisfy the parameter.")]).describe("Maps input parameters for a step. Can be a static value or a JavaScript expression that references previous results or flow inputs")).describe("Map of parameter names to their values (static or JavaScript expressions). These become the subflow's input arguments"), "path": z.string().describe("Path to the flow in the workspace (e.g., 'f/flows/process_user')"), "type": z.literal("flow") }).describe("Reference to an existing flow by path. Use this to call another flow as a subflow"), z.object({ "modules": z.array(z.lazy(() => flowModuleSchema)).describe("Steps to execute for each iteration. These can reference the iteration value via 'flow_input.iter.value'"), "iterator": z.discriminatedUnion("type", [z.object({ "value": z.any().describe("The static value. For resources, use format '$res:path/to/resource'").optional(), "type": z.literal("static") }).describe("Static value passed directly to the step. Use for hardcoded values or resource references like '$res:path/to/resource'"), z.object({ "expr": z.string().describe("JavaScript expression returning the value. Available variables - results (object with all previous step results), flow_input (flow inputs), flow_input.iter (in loops)"), "type": z.literal("javascript") }).describe("JavaScript expression evaluated at runtime. Can reference previous step results via 'results.step_id' or flow inputs via 'flow_input.property'. Inside for loops, use 'flow_input.iter.value' for the current iteration value (in while loops it equals 'flow_input.iter.index')"), z.object({ "type": z.literal("ai") }).describe("Value resolved by the AI runtime for this input. The AI engine decides how to satisfy the parameter.")]).describe("Maps input parameters for a step. Can be a static value or a JavaScript expression that references previous results or flow inputs"), "skip_failures": z.boolean().describe("If true, iteration failures don't stop the loop. Failed iterations return null"), "type": z.literal("forloopflow"), "parallel": z.boolean().describe("If true, iterations run concurrently (faster for I/O-bound operations). Use with parallelism to control concurrency").optional(), "parallelism": z.discriminatedUnion("type", [z.object({ "value": z.any().describe("The static value. For resources, use format '$res:path/to/resource'").optional(), "type": z.literal("static") }).describe("Static value passed directly to the step. Use for hardcoded values or resource references like '$res:path/to/resource'"), z.object({ "expr": z.string().describe("JavaScript expression returning the value. Available variables - results (object with all previous step results), flow_input (flow inputs), flow_input.iter (in loops)"), "type": z.literal("javascript") }).describe("JavaScript expression evaluated at runtime. Can reference previous step results via 'results.step_id' or flow inputs via 'flow_input.property'. Inside for loops, use 'flow_input.iter.value' for the current iteration value (in while loops it equals 'flow_input.iter.index')"), z.object({ "type": z.literal("ai") }).describe("Value resolved by the AI runtime for this input. The AI engine decides how to satisfy the parameter.")]).describe("Maps input parameters for a step. Can be a static value or a JavaScript expression that references previous results or flow inputs").optional(), "squash": z.boolean().optional() }).describe("Executes nested modules in a loop over an iterator. Inside the loop, use 'flow_input.iter.value' to access the current iteration value, and 'flow_input.iter.index' for the index. Supports parallel execution for better performance on I/O-bound operations"), z.object({ "modules": z.array(z.lazy(() => flowModuleSchema)).describe("Steps to execute in each iteration"), "skip_failures": z.boolean().describe("If true, iteration failures don't stop the loop. Failed iterations return null"), "type": z.literal("whileloopflow"), "parallel": z.boolean().describe("If true, iterations run concurrently (use with caution in while loops)").optional(), "parallelism": z.discriminatedUnion("type", [z.object({ "value": z.any().describe("The static value. For resources, use format '$res:path/to/resource'").optional(), "type": z.literal("static") }).describe("Static value passed directly to the step. Use for hardcoded values or resource references like '$res:path/to/resource'"), z.object({ "expr": z.string().describe("JavaScript expression returning the value. Available variables - results (object with all previous step results), flow_input (flow inputs), flow_input.iter (in loops)"), "type": z.literal("javascript") }).describe("JavaScript expression evaluated at runtime. Can reference previous step results via 'results.step_id' or flow inputs via 'flow_input.property'. Inside for loops, use 'flow_input.iter.value' for the current iteration value (in while loops it equals 'flow_input.iter.index')"), z.object({ "type": z.literal("ai") }).describe("Value resolved by the AI runtime for this input. The AI engine decides how to satisfy the parameter.")]).describe("Maps input parameters for a step. Can be a static value or a JavaScript expression that references previous results or flow inputs").optional(), "squash": z.boolean().optional() }).describe("Executes nested modules repeatedly until stopped. The implicit iterator is the iteration counter, so 'flow_input.iter.value' equals 'flow_input.iter.index' (0, 1, 2, ...) and never carries state. To carry state across iterations, a step reads its own previous-iteration result via 'results.' with a first-iteration fallback - the loop's stop_after_if must then be on that inner step (a plain single-step body with stop_after_if on the loop module does not resolve 'results' across iterations and never terminates); plain counters can instead be derived from 'flow_input.iter.index', which works in every configuration. stop_after_if is evaluated after each iteration - on the loop module 'result' is the last iteration's result"), z.object({ "branches": z.array(z.object({ "summary": z.string().describe("Short description of this branch condition").optional(), "expr": z.string().describe("JavaScript expression that returns boolean. Can use 'results.step_id' or 'flow_input'. First true expr wins"), "modules": z.array(z.lazy(() => flowModuleSchema)).describe("Steps to execute if this branch's expr is true") })).describe("Array of branches to evaluate in order. The first branch with expr evaluating to true executes"), "default": z.array(z.lazy(() => flowModuleSchema)).describe("Steps to execute if no branch expressions match"), "type": z.literal("branchone") }).describe("Conditional branching where only the first matching branch executes. Branches are evaluated in order, and the first one with a true expression runs. If no branches match, the default branch executes"), z.object({ "branches": z.array(z.object({ "summary": z.string().describe("Short description of this branch's purpose").optional(), "skip_failure": z.boolean().describe("If true, failure in this branch doesn't fail the entire flow").optional(), "modules": z.array(z.lazy(() => flowModuleSchema)).describe("Steps to execute in this branch") })).describe("Array of branches that all execute (either in parallel or sequentially)"), "type": z.literal("branchall"), "parallel": z.boolean().describe("If true, all branches execute concurrently. If false, they execute sequentially").optional() }).describe("Parallel branching where all branches execute simultaneously. Unlike BranchOne, all branches run regardless of conditions. Useful for executing independent tasks concurrently"), z.object({ "type": z.literal("identity"), "flow": z.boolean().describe("If true, marks this as a flow identity (special handling)").optional() }).describe("Pass-through module that returns its input unchanged. Useful for flow structure or as a placeholder"), z.object({ "input_transforms": z.object({ "provider": z.discriminatedUnion("type", [z.object({ "value": z.object({ "kind": z.enum(["openai","azure_openai","azure_foundry","anthropic","mistral","deepseek","googleai","groq","openrouter","togetherai","customai","aws_bedrock"]).describe("Supported AI provider types"), "resource": z.string().describe("Resource reference in format '$res:{resource_path}' pointing to provider credentials"), "model": z.string().describe("Model identifier (e.g., 'gpt-4', 'claude-3-opus-20240229', 'gemini-pro')"), "reasoning_effort": z.string().describe("Provider-native reasoning effort token (e.g. 'low', 'high', 'none') for models that support extended thinking. Optional; unset leaves the provider default.").optional() }).describe("Complete AI provider configuration with resource reference and model selection"), "type": z.literal("static") }).describe("Static provider configuration passed directly to the AI agent"), z.object({ "expr": z.string().describe("JavaScript expression returning the value. Available variables - results (object with all previous step results), flow_input (flow inputs), flow_input.iter (in loops)"), "type": z.literal("javascript") }).describe("JavaScript expression evaluated at runtime. Can reference previous step results via 'results.step_id' or flow inputs via 'flow_input.property'. Inside for loops, use 'flow_input.iter.value' for the current iteration value (in while loops it equals 'flow_input.iter.index')"), z.object({ "type": z.literal("ai") }).describe("Value resolved by the AI runtime for this input. The AI engine decides how to satisfy the parameter.")]).describe("Provider configuration - can be static (ProviderConfig), JavaScript expression, or AI-determined").optional(), "output_type": z.discriminatedUnion("type", [z.object({ "value": z.any().describe("The static value. For resources, use format '$res:path/to/resource'").optional(), "type": z.literal("static") }).describe("Static value passed directly to the step. Use for hardcoded values or resource references like '$res:path/to/resource'"), z.object({ "expr": z.string().describe("JavaScript expression returning the value. Available variables - results (object with all previous step results), flow_input (flow inputs), flow_input.iter (in loops)"), "type": z.literal("javascript") }).describe("JavaScript expression evaluated at runtime. Can reference previous step results via 'results.step_id' or flow inputs via 'flow_input.property'. Inside for loops, use 'flow_input.iter.value' for the current iteration value (in while loops it equals 'flow_input.iter.index')"), z.object({ "type": z.literal("ai") }).describe("Value resolved by the AI runtime for this input. The AI engine decides how to satisfy the parameter.")]).describe("Maps input parameters for a step. Can be a static value or a JavaScript expression that references previous results or flow inputs").describe("Output format type.\nValid values: 'text' (default) - plain text response, 'image' - image generation\n").optional(), "user_message": z.discriminatedUnion("type", [z.object({ "value": z.any().describe("The static value. For resources, use format '$res:path/to/resource'").optional(), "type": z.literal("static") }).describe("Static value passed directly to the step. Use for hardcoded values or resource references like '$res:path/to/resource'"), z.object({ "expr": z.string().describe("JavaScript expression returning the value. Available variables - results (object with all previous step results), flow_input (flow inputs), flow_input.iter (in loops)"), "type": z.literal("javascript") }).describe("JavaScript expression evaluated at runtime. Can reference previous step results via 'results.step_id' or flow inputs via 'flow_input.property'. Inside for loops, use 'flow_input.iter.value' for the current iteration value (in while loops it equals 'flow_input.iter.index')"), z.object({ "type": z.literal("ai") }).describe("Value resolved by the AI runtime for this input. The AI engine decides how to satisfy the parameter.")]).describe("Maps input parameters for a step. Can be a static value or a JavaScript expression that references previous results or flow inputs").describe("The user's prompt/message to the AI agent. Supports variable interpolation with flow.input syntax."), "system_prompt": z.discriminatedUnion("type", [z.object({ "value": z.any().describe("The static value. For resources, use format '$res:path/to/resource'").optional(), "type": z.literal("static") }).describe("Static value passed directly to the step. Use for hardcoded values or resource references like '$res:path/to/resource'"), z.object({ "expr": z.string().describe("JavaScript expression returning the value. Available variables - results (object with all previous step results), flow_input (flow inputs), flow_input.iter (in loops)"), "type": z.literal("javascript") }).describe("JavaScript expression evaluated at runtime. Can reference previous step results via 'results.step_id' or flow inputs via 'flow_input.property'. Inside for loops, use 'flow_input.iter.value' for the current iteration value (in while loops it equals 'flow_input.iter.index')"), z.object({ "type": z.literal("ai") }).describe("Value resolved by the AI runtime for this input. The AI engine decides how to satisfy the parameter.")]).describe("Maps input parameters for a step. Can be a static value or a JavaScript expression that references previous results or flow inputs").describe("System instructions that guide the AI's behavior, persona, and response style. Optional.").optional(), "streaming": z.discriminatedUnion("type", [z.object({ "value": z.any().describe("The static value. For resources, use format '$res:path/to/resource'").optional(), "type": z.literal("static") }).describe("Static value passed directly to the step. Use for hardcoded values or resource references like '$res:path/to/resource'"), z.object({ "expr": z.string().describe("JavaScript expression returning the value. Available variables - results (object with all previous step results), flow_input (flow inputs), flow_input.iter (in loops)"), "type": z.literal("javascript") }).describe("JavaScript expression evaluated at runtime. Can reference previous step results via 'results.step_id' or flow inputs via 'flow_input.property'. Inside for loops, use 'flow_input.iter.value' for the current iteration value (in while loops it equals 'flow_input.iter.index')"), z.object({ "type": z.literal("ai") }).describe("Value resolved by the AI runtime for this input. The AI engine decides how to satisfy the parameter.")]).describe("Maps input parameters for a step. Can be a static value or a JavaScript expression that references previous results or flow inputs").describe("Boolean. If true, stream the AI response incrementally.\nStreaming events include: token_delta, reasoning_token_delta, tool_call, tool_call_arguments, tool_execution, tool_result\n").optional(), "memory": z.discriminatedUnion("type", [z.object({ "value": z.discriminatedUnion("kind", [z.object({ "kind": z.literal("off") }).describe("No conversation memory/context"), z.object({ "kind": z.literal("auto"), "context_length": z.number().int().describe("Maximum number of messages to retain in context").optional(), "memory_id": z.string().describe("Identifier for persistent memory across agent invocations").optional() }).describe("Automatic context management"), z.object({ "kind": z.literal("manual"), "messages": z.array(z.object({ "role": z.enum(["user","assistant","system"]), "content": z.string() }).describe("A single message in conversation history")) }).describe("Explicit message history")]).describe("Conversation memory configuration"), "type": z.literal("static") }).describe("Static memory configuration passed directly to the AI agent"), z.object({ "expr": z.string().describe("JavaScript expression returning the value. Available variables - results (object with all previous step results), flow_input (flow inputs), flow_input.iter (in loops)"), "type": z.literal("javascript") }).describe("JavaScript expression evaluated at runtime. Can reference previous step results via 'results.step_id' or flow inputs via 'flow_input.property'. Inside for loops, use 'flow_input.iter.value' for the current iteration value (in while loops it equals 'flow_input.iter.index')"), z.object({ "type": z.literal("ai") }).describe("Value resolved by the AI runtime for this input. The AI engine decides how to satisfy the parameter.")]).describe("Memory configuration - can be static (MemoryConfig), JavaScript expression, or AI-determined").optional(), "output_schema": z.discriminatedUnion("type", [z.object({ "value": z.any().describe("The static value. For resources, use format '$res:path/to/resource'").optional(), "type": z.literal("static") }).describe("Static value passed directly to the step. Use for hardcoded values or resource references like '$res:path/to/resource'"), z.object({ "expr": z.string().describe("JavaScript expression returning the value. Available variables - results (object with all previous step results), flow_input (flow inputs), flow_input.iter (in loops)"), "type": z.literal("javascript") }).describe("JavaScript expression evaluated at runtime. Can reference previous step results via 'results.step_id' or flow inputs via 'flow_input.property'. Inside for loops, use 'flow_input.iter.value' for the current iteration value (in while loops it equals 'flow_input.iter.index')"), z.object({ "type": z.literal("ai") }).describe("Value resolved by the AI runtime for this input. The AI engine decides how to satisfy the parameter.")]).describe("Maps input parameters for a step. Can be a static value or a JavaScript expression that references previous results or flow inputs").describe("JSON Schema object defining structured output format. Used when you need the AI to return data in a specific shape.\nSupports standard JSON Schema properties: type, properties, required, items, enum, pattern, minLength, maxLength, minimum, maximum, etc.\nExample: { type: 'object', properties: { name: { type: 'string' }, age: { type: 'integer' } }, required: ['name'] }\n").optional(), "user_attachments": z.discriminatedUnion("type", [z.object({ "value": z.any().describe("The static value. For resources, use format '$res:path/to/resource'").optional(), "type": z.literal("static") }).describe("Static value passed directly to the step. Use for hardcoded values or resource references like '$res:path/to/resource'"), z.object({ "expr": z.string().describe("JavaScript expression returning the value. Available variables - results (object with all previous step results), flow_input (flow inputs), flow_input.iter (in loops)"), "type": z.literal("javascript") }).describe("JavaScript expression evaluated at runtime. Can reference previous step results via 'results.step_id' or flow inputs via 'flow_input.property'. Inside for loops, use 'flow_input.iter.value' for the current iteration value (in while loops it equals 'flow_input.iter.index')"), z.object({ "type": z.literal("ai") }).describe("Value resolved by the AI runtime for this input. The AI engine decides how to satisfy the parameter.")]).describe("Maps input parameters for a step. Can be a static value or a JavaScript expression that references previous results or flow inputs").describe("Array of file references (images or PDFs) for the AI agent.\nFormat: Array<{ bucket: string, key: string }> - S3 object references\nExample: [{ bucket: 'my-bucket', key: 'documents/report.pdf' }]\n").optional(), "max_completion_tokens": z.discriminatedUnion("type", [z.object({ "value": z.any().describe("The static value. For resources, use format '$res:path/to/resource'").optional(), "type": z.literal("static") }).describe("Static value passed directly to the step. Use for hardcoded values or resource references like '$res:path/to/resource'"), z.object({ "expr": z.string().describe("JavaScript expression returning the value. Available variables - results (object with all previous step results), flow_input (flow inputs), flow_input.iter (in loops)"), "type": z.literal("javascript") }).describe("JavaScript expression evaluated at runtime. Can reference previous step results via 'results.step_id' or flow inputs via 'flow_input.property'. Inside for loops, use 'flow_input.iter.value' for the current iteration value (in while loops it equals 'flow_input.iter.index')"), z.object({ "type": z.literal("ai") }).describe("Value resolved by the AI runtime for this input. The AI engine decides how to satisfy the parameter.")]).describe("Maps input parameters for a step. Can be a static value or a JavaScript expression that references previous results or flow inputs").describe("Integer. Maximum number of tokens the AI will generate in its response.\nRange: 1 to 4,294,967,295. Typical values: 256-4096 for most use cases.\n").optional(), "temperature": z.discriminatedUnion("type", [z.object({ "value": z.any().describe("The static value. For resources, use format '$res:path/to/resource'").optional(), "type": z.literal("static") }).describe("Static value passed directly to the step. Use for hardcoded values or resource references like '$res:path/to/resource'"), z.object({ "expr": z.string().describe("JavaScript expression returning the value. Available variables - results (object with all previous step results), flow_input (flow inputs), flow_input.iter (in loops)"), "type": z.literal("javascript") }).describe("JavaScript expression evaluated at runtime. Can reference previous step results via 'results.step_id' or flow inputs via 'flow_input.property'. Inside for loops, use 'flow_input.iter.value' for the current iteration value (in while loops it equals 'flow_input.iter.index')"), z.object({ "type": z.literal("ai") }).describe("Value resolved by the AI runtime for this input. The AI engine decides how to satisfy the parameter.")]).describe("Maps input parameters for a step. Can be a static value or a JavaScript expression that references previous results or flow inputs").describe("Float. Controls randomness/creativity of responses.\nRange: 0.0 to 2.0 (provider-dependent)\n- 0.0 = deterministic, focused responses\n- 0.7 = balanced (common default)\n- 1.0+ = more creative/random\n").optional(), "max_iterations": z.discriminatedUnion("type", [z.object({ "value": z.any().describe("The static value. For resources, use format '$res:path/to/resource'").optional(), "type": z.literal("static") }).describe("Static value passed directly to the step. Use for hardcoded values or resource references like '$res:path/to/resource'"), z.object({ "expr": z.string().describe("JavaScript expression returning the value. Available variables - results (object with all previous step results), flow_input (flow inputs), flow_input.iter (in loops)"), "type": z.literal("javascript") }).describe("JavaScript expression evaluated at runtime. Can reference previous step results via 'results.step_id' or flow inputs via 'flow_input.property'. Inside for loops, use 'flow_input.iter.value' for the current iteration value (in while loops it equals 'flow_input.iter.index')"), z.object({ "type": z.literal("ai") }).describe("Value resolved by the AI runtime for this input. The AI engine decides how to satisfy the parameter.")]).describe("Maps input parameters for a step. Can be a static value or a JavaScript expression that references previous results or flow inputs").describe("Number. Limits how many times the agent can loop through reasoning and tool use.\nRange: 1-1000.\n").optional() }).describe("Input parameters for the AI agent mapped to their values"), "tools": z.array(z.object({ "id": z.string().describe("Unique identifier for this tool. Cannot contain spaces - use underscores instead (e.g., 'get_user_data' not 'get user data')"), "summary": z.string().describe("Short description of what this tool does (shown to the AI)").optional(), "description": z.string().describe("Free-text description of the tool given to the AI to decide when and how to call it. Overrides the description auto-derived from the underlying script.").optional(), "value": z.any().superRefine((x, ctx) => { +export const flowModuleSchema = z.object({ "id": z.string().describe("Unique identifier for this step. Used to reference results via 'results.step_id'. Must be a valid identifier (alphanumeric, underscore, hyphen)"), "value": z.discriminatedUnion("type", [z.object({ "input_transforms": z.record(z.string(), z.discriminatedUnion("type", [z.object({ "value": z.any().describe("The static value. For resources, use format '$res:path/to/resource'").optional(), "type": z.literal("static") }).describe("Static value passed directly to the step. Use for hardcoded values or resource references like '$res:path/to/resource'"), z.object({ "expr": z.string().describe("JavaScript expression returning the value. Available variables - results (object with all previous step results), flow_input (flow inputs), flow_input.iter (in loops)"), "type": z.literal("javascript") }).describe("JavaScript expression evaluated at runtime. Can reference previous step results via 'results.step_id' or flow inputs via 'flow_input.property'. Inside for loops, use 'flow_input.iter.value' for the current iteration value (in while loops it equals 'flow_input.iter.index')"), z.object({ "type": z.literal("ai") }).describe("Value resolved by the AI runtime for this input. The AI engine decides how to satisfy the parameter.")]).describe("Maps input parameters for a step. Can be a static value or a JavaScript expression that references previous results or flow inputs")).describe("Map of parameter names to their values (static or JavaScript expressions). These become the script's input arguments"), "content": z.string().describe("The script source code. Should export a 'main' function"), "language": z.enum(["deno","bun","bunnative","python3","go","bash","powershell","postgresql","mysql","bigquery","snowflake","mssql","oracledb","graphql","nativets","php","rust","ansible","csharp","nu","java","ruby","rlang","duckdb"]).describe("Programming language for this script"), "path": z.string().describe("Optional path for saving this script").optional(), "lock": z.string().describe("Lock file content for dependencies").optional(), "type": z.literal("rawscript"), "tag": z.string().describe("Worker group tag for execution routing").optional(), "concurrent_limit": z.number().describe("Maximum concurrent executions of this script").optional(), "concurrency_time_window_s": z.number().describe("Time window for concurrent_limit").optional(), "custom_concurrency_key": z.string().describe("Custom key for grouping concurrent executions").optional(), "is_trigger": z.boolean().describe("If true, this script is a trigger that can start the flow").optional(), "assets": z.array(z.object({ "path": z.string().describe("Path to the asset"), "kind": z.enum(["s3object","resource","ducklake","datatable","volume","dbt"]).describe("Type of asset"), "access_type": z.union([z.literal("r"), z.literal("w"), z.literal("rw"), z.literal(null)]).nullable().describe("Access level for this asset").optional(), "alt_access_type": z.union([z.literal("r"), z.literal("w"), z.literal("rw"), z.literal(null)]).nullable().describe("Alternative access level").optional() })).describe("External resources this script accesses (S3 objects, resources, etc.)").optional() }).describe("Inline script with code defined directly in the flow. Use 'bun' as default language if unspecified. The script receives arguments from input_transforms"), z.object({ "input_transforms": z.record(z.string(), z.discriminatedUnion("type", [z.object({ "value": z.any().describe("The static value. For resources, use format '$res:path/to/resource'").optional(), "type": z.literal("static") }).describe("Static value passed directly to the step. Use for hardcoded values or resource references like '$res:path/to/resource'"), z.object({ "expr": z.string().describe("JavaScript expression returning the value. Available variables - results (object with all previous step results), flow_input (flow inputs), flow_input.iter (in loops)"), "type": z.literal("javascript") }).describe("JavaScript expression evaluated at runtime. Can reference previous step results via 'results.step_id' or flow inputs via 'flow_input.property'. Inside for loops, use 'flow_input.iter.value' for the current iteration value (in while loops it equals 'flow_input.iter.index')"), z.object({ "type": z.literal("ai") }).describe("Value resolved by the AI runtime for this input. The AI engine decides how to satisfy the parameter.")]).describe("Maps input parameters for a step. Can be a static value or a JavaScript expression that references previous results or flow inputs")).describe("Map of parameter names to their values (static or JavaScript expressions). These become the script's input arguments"), "path": z.string().describe("Path to the script in the workspace (e.g., 'f/scripts/send_email')"), "hash": z.string().describe("Optional specific version hash of the script to use").optional(), "type": z.literal("script"), "tag_override": z.string().describe("Override the script's default worker group tag").optional(), "is_trigger": z.boolean().describe("If true, this script is a trigger that can start the flow").optional() }).describe("Reference to an existing script by path. Use this when calling a previously saved script instead of writing inline code"), z.object({ "input_transforms": z.record(z.string(), z.discriminatedUnion("type", [z.object({ "value": z.any().describe("The static value. For resources, use format '$res:path/to/resource'").optional(), "type": z.literal("static") }).describe("Static value passed directly to the step. Use for hardcoded values or resource references like '$res:path/to/resource'"), z.object({ "expr": z.string().describe("JavaScript expression returning the value. Available variables - results (object with all previous step results), flow_input (flow inputs), flow_input.iter (in loops)"), "type": z.literal("javascript") }).describe("JavaScript expression evaluated at runtime. Can reference previous step results via 'results.step_id' or flow inputs via 'flow_input.property'. Inside for loops, use 'flow_input.iter.value' for the current iteration value (in while loops it equals 'flow_input.iter.index')"), z.object({ "type": z.literal("ai") }).describe("Value resolved by the AI runtime for this input. The AI engine decides how to satisfy the parameter.")]).describe("Maps input parameters for a step. Can be a static value or a JavaScript expression that references previous results or flow inputs")).describe("Map of parameter names to their values (static or JavaScript expressions). These become the subflow's input arguments"), "path": z.string().describe("Path to the flow in the workspace (e.g., 'f/flows/process_user')"), "type": z.literal("flow") }).describe("Reference to an existing flow by path. Use this to call another flow as a subflow"), z.object({ "modules": z.array(z.lazy(() => flowModuleSchema)).describe("Steps to execute for each iteration. These can reference the iteration value via 'flow_input.iter.value'"), "iterator": z.discriminatedUnion("type", [z.object({ "value": z.any().describe("The static value. For resources, use format '$res:path/to/resource'").optional(), "type": z.literal("static") }).describe("Static value passed directly to the step. Use for hardcoded values or resource references like '$res:path/to/resource'"), z.object({ "expr": z.string().describe("JavaScript expression returning the value. Available variables - results (object with all previous step results), flow_input (flow inputs), flow_input.iter (in loops)"), "type": z.literal("javascript") }).describe("JavaScript expression evaluated at runtime. Can reference previous step results via 'results.step_id' or flow inputs via 'flow_input.property'. Inside for loops, use 'flow_input.iter.value' for the current iteration value (in while loops it equals 'flow_input.iter.index')"), z.object({ "type": z.literal("ai") }).describe("Value resolved by the AI runtime for this input. The AI engine decides how to satisfy the parameter.")]).describe("Maps input parameters for a step. Can be a static value or a JavaScript expression that references previous results or flow inputs"), "skip_failures": z.boolean().describe("If true, iteration failures don't stop the loop. Failed iterations return null"), "type": z.literal("forloopflow"), "parallel": z.boolean().describe("If true, iterations run concurrently (faster for I/O-bound operations). Use with parallelism to control concurrency").optional(), "parallelism": z.discriminatedUnion("type", [z.object({ "value": z.any().describe("The static value. For resources, use format '$res:path/to/resource'").optional(), "type": z.literal("static") }).describe("Static value passed directly to the step. Use for hardcoded values or resource references like '$res:path/to/resource'"), z.object({ "expr": z.string().describe("JavaScript expression returning the value. Available variables - results (object with all previous step results), flow_input (flow inputs), flow_input.iter (in loops)"), "type": z.literal("javascript") }).describe("JavaScript expression evaluated at runtime. Can reference previous step results via 'results.step_id' or flow inputs via 'flow_input.property'. Inside for loops, use 'flow_input.iter.value' for the current iteration value (in while loops it equals 'flow_input.iter.index')"), z.object({ "type": z.literal("ai") }).describe("Value resolved by the AI runtime for this input. The AI engine decides how to satisfy the parameter.")]).describe("Maps input parameters for a step. Can be a static value or a JavaScript expression that references previous results or flow inputs").optional(), "squash": z.boolean().optional() }).describe("Executes nested modules in a loop over an iterator. Inside the loop, use 'flow_input.iter.value' to access the current iteration value, and 'flow_input.iter.index' for the index. Supports parallel execution for better performance on I/O-bound operations"), z.object({ "modules": z.array(z.lazy(() => flowModuleSchema)).describe("Steps to execute in each iteration"), "skip_failures": z.boolean().describe("If true, iteration failures don't stop the loop. Failed iterations return null"), "type": z.literal("whileloopflow"), "parallel": z.boolean().describe("If true, iterations run concurrently (use with caution in while loops)").optional(), "parallelism": z.discriminatedUnion("type", [z.object({ "value": z.any().describe("The static value. For resources, use format '$res:path/to/resource'").optional(), "type": z.literal("static") }).describe("Static value passed directly to the step. Use for hardcoded values or resource references like '$res:path/to/resource'"), z.object({ "expr": z.string().describe("JavaScript expression returning the value. Available variables - results (object with all previous step results), flow_input (flow inputs), flow_input.iter (in loops)"), "type": z.literal("javascript") }).describe("JavaScript expression evaluated at runtime. Can reference previous step results via 'results.step_id' or flow inputs via 'flow_input.property'. Inside for loops, use 'flow_input.iter.value' for the current iteration value (in while loops it equals 'flow_input.iter.index')"), z.object({ "type": z.literal("ai") }).describe("Value resolved by the AI runtime for this input. The AI engine decides how to satisfy the parameter.")]).describe("Maps input parameters for a step. Can be a static value or a JavaScript expression that references previous results or flow inputs").optional(), "squash": z.boolean().optional() }).describe("Executes nested modules repeatedly until stopped. The implicit iterator is the iteration counter, so 'flow_input.iter.value' equals 'flow_input.iter.index' (0, 1, 2, ...) and never carries state. To carry state across iterations, a step reads its own previous-iteration result via 'results.' with a first-iteration fallback - the loop's stop_after_if must then be on that inner step (a plain single-step body with stop_after_if on the loop module does not resolve 'results' across iterations and never terminates); plain counters can instead be derived from 'flow_input.iter.index', which works in every configuration. stop_after_if is evaluated after each iteration - on the loop module 'result' is the last iteration's result"), z.object({ "branches": z.array(z.object({ "summary": z.string().describe("Short description of this branch condition").optional(), "expr": z.string().describe("JavaScript expression that returns boolean. Can use 'results.step_id' or 'flow_input'. First true expr wins"), "modules": z.array(z.lazy(() => flowModuleSchema)).describe("Steps to execute if this branch's expr is true") })).describe("Array of branches to evaluate in order. The first branch with expr evaluating to true executes"), "default": z.array(z.lazy(() => flowModuleSchema)).describe("Steps to execute if no branch expressions match"), "type": z.literal("branchone") }).describe("Conditional branching where only the first matching branch executes. Branches are evaluated in order, and the first one with a true expression runs. If no branches match, the default branch executes"), z.object({ "branches": z.array(z.object({ "summary": z.string().describe("Short description of this branch's purpose").optional(), "skip_failure": z.boolean().describe("If true, failure in this branch doesn't fail the entire flow").optional(), "modules": z.array(z.lazy(() => flowModuleSchema)).describe("Steps to execute in this branch") })).describe("Array of branches that all execute (either in parallel or sequentially)"), "type": z.literal("branchall"), "parallel": z.boolean().describe("If true, all branches execute concurrently. If false, they execute sequentially").optional() }).describe("Parallel branching where all branches execute simultaneously. Unlike BranchOne, all branches run regardless of conditions. Useful for executing independent tasks concurrently"), z.object({ "type": z.literal("identity"), "flow": z.boolean().describe("If true, marks this as a flow identity (special handling)").optional() }).describe("Pass-through module that returns its input unchanged. Useful for flow structure or as a placeholder"), z.object({ "input_transforms": z.object({ "provider": z.discriminatedUnion("type", [z.object({ "value": z.object({ "kind": z.enum(["openai","azure_openai","azure_foundry","anthropic","mistral","deepseek","googleai","groq","openrouter","togetherai","customai","aws_bedrock"]).describe("Supported AI provider types"), "resource": z.string().describe("Resource reference in format '$res:{resource_path}' pointing to provider credentials"), "model": z.string().describe("Model identifier (e.g., 'gpt-4', 'claude-3-opus-20240229', 'gemini-pro')"), "reasoning_effort": z.string().describe("Provider-native reasoning effort token (e.g. 'low', 'high', 'none') for models that support extended thinking. Optional; unset leaves the provider default.").optional() }).describe("Complete AI provider configuration with resource reference and model selection"), "type": z.literal("static") }).describe("Static provider configuration passed directly to the AI agent"), z.object({ "expr": z.string().describe("JavaScript expression returning the value. Available variables - results (object with all previous step results), flow_input (flow inputs), flow_input.iter (in loops)"), "type": z.literal("javascript") }).describe("JavaScript expression evaluated at runtime. Can reference previous step results via 'results.step_id' or flow inputs via 'flow_input.property'. Inside for loops, use 'flow_input.iter.value' for the current iteration value (in while loops it equals 'flow_input.iter.index')"), z.object({ "type": z.literal("ai") }).describe("Value resolved by the AI runtime for this input. The AI engine decides how to satisfy the parameter.")]).describe("Provider configuration - can be static (ProviderConfig), JavaScript expression, or AI-determined").optional(), "output_type": z.discriminatedUnion("type", [z.object({ "value": z.any().describe("The static value. For resources, use format '$res:path/to/resource'").optional(), "type": z.literal("static") }).describe("Static value passed directly to the step. Use for hardcoded values or resource references like '$res:path/to/resource'"), z.object({ "expr": z.string().describe("JavaScript expression returning the value. Available variables - results (object with all previous step results), flow_input (flow inputs), flow_input.iter (in loops)"), "type": z.literal("javascript") }).describe("JavaScript expression evaluated at runtime. Can reference previous step results via 'results.step_id' or flow inputs via 'flow_input.property'. Inside for loops, use 'flow_input.iter.value' for the current iteration value (in while loops it equals 'flow_input.iter.index')"), z.object({ "type": z.literal("ai") }).describe("Value resolved by the AI runtime for this input. The AI engine decides how to satisfy the parameter.")]).describe("Maps input parameters for a step. Can be a static value or a JavaScript expression that references previous results or flow inputs").describe("Output format type.\nValid values: 'text' (default) - plain text response, 'image' - image generation\n").optional(), "user_message": z.discriminatedUnion("type", [z.object({ "value": z.any().describe("The static value. For resources, use format '$res:path/to/resource'").optional(), "type": z.literal("static") }).describe("Static value passed directly to the step. Use for hardcoded values or resource references like '$res:path/to/resource'"), z.object({ "expr": z.string().describe("JavaScript expression returning the value. Available variables - results (object with all previous step results), flow_input (flow inputs), flow_input.iter (in loops)"), "type": z.literal("javascript") }).describe("JavaScript expression evaluated at runtime. Can reference previous step results via 'results.step_id' or flow inputs via 'flow_input.property'. Inside for loops, use 'flow_input.iter.value' for the current iteration value (in while loops it equals 'flow_input.iter.index')"), z.object({ "type": z.literal("ai") }).describe("Value resolved by the AI runtime for this input. The AI engine decides how to satisfy the parameter.")]).describe("Maps input parameters for a step. Can be a static value or a JavaScript expression that references previous results or flow inputs").describe("The user's prompt/message to the AI agent. Supports variable interpolation with flow.input syntax."), "system_prompt": z.discriminatedUnion("type", [z.object({ "value": z.any().describe("The static value. For resources, use format '$res:path/to/resource'").optional(), "type": z.literal("static") }).describe("Static value passed directly to the step. Use for hardcoded values or resource references like '$res:path/to/resource'"), z.object({ "expr": z.string().describe("JavaScript expression returning the value. Available variables - results (object with all previous step results), flow_input (flow inputs), flow_input.iter (in loops)"), "type": z.literal("javascript") }).describe("JavaScript expression evaluated at runtime. Can reference previous step results via 'results.step_id' or flow inputs via 'flow_input.property'. Inside for loops, use 'flow_input.iter.value' for the current iteration value (in while loops it equals 'flow_input.iter.index')"), z.object({ "type": z.literal("ai") }).describe("Value resolved by the AI runtime for this input. The AI engine decides how to satisfy the parameter.")]).describe("Maps input parameters for a step. Can be a static value or a JavaScript expression that references previous results or flow inputs").describe("System instructions that guide the AI's behavior, persona, and response style. Optional.").optional(), "streaming": z.discriminatedUnion("type", [z.object({ "value": z.any().describe("The static value. For resources, use format '$res:path/to/resource'").optional(), "type": z.literal("static") }).describe("Static value passed directly to the step. Use for hardcoded values or resource references like '$res:path/to/resource'"), z.object({ "expr": z.string().describe("JavaScript expression returning the value. Available variables - results (object with all previous step results), flow_input (flow inputs), flow_input.iter (in loops)"), "type": z.literal("javascript") }).describe("JavaScript expression evaluated at runtime. Can reference previous step results via 'results.step_id' or flow inputs via 'flow_input.property'. Inside for loops, use 'flow_input.iter.value' for the current iteration value (in while loops it equals 'flow_input.iter.index')"), z.object({ "type": z.literal("ai") }).describe("Value resolved by the AI runtime for this input. The AI engine decides how to satisfy the parameter.")]).describe("Maps input parameters for a step. Can be a static value or a JavaScript expression that references previous results or flow inputs").describe("Boolean. If true, stream the AI response incrementally.\nStreaming events include: token_delta, reasoning_token_delta, tool_call, tool_call_arguments, tool_execution, tool_result\n").optional(), "memory": z.discriminatedUnion("type", [z.object({ "value": z.discriminatedUnion("kind", [z.object({ "kind": z.literal("off") }).describe("No conversation memory/context"), z.object({ "kind": z.literal("auto"), "context_length": z.number().int().describe("Maximum number of messages to retain in context").optional(), "memory_id": z.string().describe("Identifier for persistent memory across agent invocations").optional() }).describe("Automatic context management"), z.object({ "kind": z.literal("manual"), "messages": z.array(z.object({ "role": z.enum(["user","assistant","system"]), "content": z.string() }).describe("A single message in conversation history")) }).describe("Explicit message history")]).describe("Conversation memory configuration"), "type": z.literal("static") }).describe("Static memory configuration passed directly to the AI agent"), z.object({ "expr": z.string().describe("JavaScript expression returning the value. Available variables - results (object with all previous step results), flow_input (flow inputs), flow_input.iter (in loops)"), "type": z.literal("javascript") }).describe("JavaScript expression evaluated at runtime. Can reference previous step results via 'results.step_id' or flow inputs via 'flow_input.property'. Inside for loops, use 'flow_input.iter.value' for the current iteration value (in while loops it equals 'flow_input.iter.index')"), z.object({ "type": z.literal("ai") }).describe("Value resolved by the AI runtime for this input. The AI engine decides how to satisfy the parameter.")]).describe("Memory configuration - can be static (MemoryConfig), JavaScript expression, or AI-determined").optional(), "output_schema": z.discriminatedUnion("type", [z.object({ "value": z.any().describe("The static value. For resources, use format '$res:path/to/resource'").optional(), "type": z.literal("static") }).describe("Static value passed directly to the step. Use for hardcoded values or resource references like '$res:path/to/resource'"), z.object({ "expr": z.string().describe("JavaScript expression returning the value. Available variables - results (object with all previous step results), flow_input (flow inputs), flow_input.iter (in loops)"), "type": z.literal("javascript") }).describe("JavaScript expression evaluated at runtime. Can reference previous step results via 'results.step_id' or flow inputs via 'flow_input.property'. Inside for loops, use 'flow_input.iter.value' for the current iteration value (in while loops it equals 'flow_input.iter.index')"), z.object({ "type": z.literal("ai") }).describe("Value resolved by the AI runtime for this input. The AI engine decides how to satisfy the parameter.")]).describe("Maps input parameters for a step. Can be a static value or a JavaScript expression that references previous results or flow inputs").describe("JSON Schema object defining structured output format. Used when you need the AI to return data in a specific shape.\nSupports standard JSON Schema properties: type, properties, required, items, enum, pattern, minLength, maxLength, minimum, maximum, etc.\nExample: { type: 'object', properties: { name: { type: 'string' }, age: { type: 'integer' } }, required: ['name'] }\n").optional(), "user_attachments": z.discriminatedUnion("type", [z.object({ "value": z.any().describe("The static value. For resources, use format '$res:path/to/resource'").optional(), "type": z.literal("static") }).describe("Static value passed directly to the step. Use for hardcoded values or resource references like '$res:path/to/resource'"), z.object({ "expr": z.string().describe("JavaScript expression returning the value. Available variables - results (object with all previous step results), flow_input (flow inputs), flow_input.iter (in loops)"), "type": z.literal("javascript") }).describe("JavaScript expression evaluated at runtime. Can reference previous step results via 'results.step_id' or flow inputs via 'flow_input.property'. Inside for loops, use 'flow_input.iter.value' for the current iteration value (in while loops it equals 'flow_input.iter.index')"), z.object({ "type": z.literal("ai") }).describe("Value resolved by the AI runtime for this input. The AI engine decides how to satisfy the parameter.")]).describe("Maps input parameters for a step. Can be a static value or a JavaScript expression that references previous results or flow inputs").describe("Array of file references (images or PDFs) for the AI agent.\nFormat: Array<{ bucket: string, key: string }> - S3 object references\nExample: [{ bucket: 'my-bucket', key: 'documents/report.pdf' }]\n").optional(), "max_completion_tokens": z.discriminatedUnion("type", [z.object({ "value": z.any().describe("The static value. For resources, use format '$res:path/to/resource'").optional(), "type": z.literal("static") }).describe("Static value passed directly to the step. Use for hardcoded values or resource references like '$res:path/to/resource'"), z.object({ "expr": z.string().describe("JavaScript expression returning the value. Available variables - results (object with all previous step results), flow_input (flow inputs), flow_input.iter (in loops)"), "type": z.literal("javascript") }).describe("JavaScript expression evaluated at runtime. Can reference previous step results via 'results.step_id' or flow inputs via 'flow_input.property'. Inside for loops, use 'flow_input.iter.value' for the current iteration value (in while loops it equals 'flow_input.iter.index')"), z.object({ "type": z.literal("ai") }).describe("Value resolved by the AI runtime for this input. The AI engine decides how to satisfy the parameter.")]).describe("Maps input parameters for a step. Can be a static value or a JavaScript expression that references previous results or flow inputs").describe("Integer. Maximum number of tokens the AI will generate in its response.\nRange: 1 to 4,294,967,295. Typical values: 256-4096 for most use cases.\n").optional(), "temperature": z.discriminatedUnion("type", [z.object({ "value": z.any().describe("The static value. For resources, use format '$res:path/to/resource'").optional(), "type": z.literal("static") }).describe("Static value passed directly to the step. Use for hardcoded values or resource references like '$res:path/to/resource'"), z.object({ "expr": z.string().describe("JavaScript expression returning the value. Available variables - results (object with all previous step results), flow_input (flow inputs), flow_input.iter (in loops)"), "type": z.literal("javascript") }).describe("JavaScript expression evaluated at runtime. Can reference previous step results via 'results.step_id' or flow inputs via 'flow_input.property'. Inside for loops, use 'flow_input.iter.value' for the current iteration value (in while loops it equals 'flow_input.iter.index')"), z.object({ "type": z.literal("ai") }).describe("Value resolved by the AI runtime for this input. The AI engine decides how to satisfy the parameter.")]).describe("Maps input parameters for a step. Can be a static value or a JavaScript expression that references previous results or flow inputs").describe("Float. Controls randomness/creativity of responses.\nRange: 0.0 to 2.0 (provider-dependent)\n- 0.0 = deterministic, focused responses\n- 0.7 = balanced (common default)\n- 1.0+ = more creative/random\n").optional(), "max_iterations": z.discriminatedUnion("type", [z.object({ "value": z.any().describe("The static value. For resources, use format '$res:path/to/resource'").optional(), "type": z.literal("static") }).describe("Static value passed directly to the step. Use for hardcoded values or resource references like '$res:path/to/resource'"), z.object({ "expr": z.string().describe("JavaScript expression returning the value. Available variables - results (object with all previous step results), flow_input (flow inputs), flow_input.iter (in loops)"), "type": z.literal("javascript") }).describe("JavaScript expression evaluated at runtime. Can reference previous step results via 'results.step_id' or flow inputs via 'flow_input.property'. Inside for loops, use 'flow_input.iter.value' for the current iteration value (in while loops it equals 'flow_input.iter.index')"), z.object({ "type": z.literal("ai") }).describe("Value resolved by the AI runtime for this input. The AI engine decides how to satisfy the parameter.")]).describe("Maps input parameters for a step. Can be a static value or a JavaScript expression that references previous results or flow inputs").describe("Number. Limits how many times the agent can loop through reasoning and tool use.\nRange: 1-1000.\n").optional() }).describe("Input parameters for the AI agent mapped to their values"), "tools": z.array(z.object({ "id": z.string().describe("Unique identifier for this tool. Cannot contain spaces - use underscores instead (e.g., 'get_user_data' not 'get user data')"), "summary": z.string().describe("The name the AI agent calls this tool by, not a human label. On a flowmodule tool it must match ^[a-zA-Z0-9_]+$ - letters, numbers and underscores only (e.g. 'search_documentation', not 'Search documentation') - and always be set; on an mcp or websearch tool it is a plain label. Put the human-readable explanation in 'description'.").optional(), "description": z.string().describe("Free-text description of the tool given to the AI to decide when and how to call it. Overrides the description auto-derived from the underlying script.").optional(), "value": z.any().superRefine((x, ctx) => { const schemas = [z.intersection(z.object({ "tool_type": z.literal("flowmodule") }), z.lazy(() => flowModuleValueSchema)).describe("A tool implemented as a flow module (script, flow, etc.). The AI can call this like any other flow module"), z.object({ "tool_type": z.literal("mcp"), "resource_path": z.string().describe("Path to the MCP resource/server configuration"), "include_tools": z.array(z.string()).describe("Whitelist of specific tools to include from this MCP server").optional(), "exclude_tools": z.array(z.string()).describe("Blacklist of tools to exclude from this MCP server").optional() }).describe("Reference to an external MCP (Model Context Protocol) tool. The AI can call tools from MCP servers"), z.object({ "tool_type": z.literal("websearch") }).describe("A tool implemented as a websearch tool. The AI can call this like any other websearch tool")]; const errors = schemas.reduce( (errors, schema) => diff --git a/frontend/src/lib/components/flows/agentToolTree.ts b/frontend/src/lib/components/flows/agentToolTree.ts index e1f29f7b23..7ccf68eb8f 100644 --- a/frontend/src/lib/components/flows/agentToolTree.ts +++ b/frontend/src/lib/components/flows/agentToolTree.ts @@ -1,5 +1,10 @@ import type { FlowModule } from '$lib/gen' -import { isFlowModuleTool, type AgentTool, type FlowModuleTool } from './agentToolUtils' +import { + getToolNameError, + isFlowModuleTool, + type AgentTool, + type FlowModuleTool +} from './agentToolUtils' type FlowNodeLike = Pick @@ -163,19 +168,19 @@ function collectFlowNodeIdsFromNode(node: FlowNodeLike): string[] { return ids } -/** Ids of AI agent modules that neither link to a saved agent nor set a provider — they pass schema - * validation (a linked step legitimately has no provider of its own) but fail on every run. */ -export function collectProviderlessAgentIds(modules: unknown): string[] { - const ids: string[] = [] +/** Walks every AI agent module of a flow, including agents nested in loops, branches and in another + * agent's tools. */ +function visitAgentModules( + modules: unknown, + cb: (mod: FlowModule, value: Record) => void +) { const visit = (mods: unknown) => { if (!Array.isArray(mods)) return for (const mod of mods) { const v = (mod as FlowModule | undefined)?.value as Record | undefined if (!v) continue if (v.type === 'aiagent') { - if (!v.agent && !v.input_transforms?.provider) { - ids.push((mod as FlowModule).id) - } + cb(mod as FlowModule, v) visit(v.tools) } else if (v.type === 'forloopflow' || v.type === 'whileloopflow') { visit(v.modules) @@ -188,5 +193,46 @@ export function collectProviderlessAgentIds(modules: unknown): string[] { } } visit(modules) +} + +/** Ids of AI agent modules that neither link to a saved agent nor set a provider — they pass schema + * validation (a linked step legitimately has no provider of its own) but fail on every run. */ +export function collectProviderlessAgentIds(modules: unknown): string[] { + const ids: string[] = [] + visitAgentModules(modules, (mod, v) => { + if (!v.agent && !v.input_transforms?.provider) { + ids.push(mod.id) + } + }) return ids } + +export type InvalidAgentToolName = { + agentId: string + toolId: string + name: string + error: string +} + +/** Agent tools whose name (their `summary`) the worker rejects — the flow saves fine but every run + * of the agent step fails, so writers must catch this before the flow is stored. */ +export function collectInvalidAgentToolNames(modules: unknown): InvalidAgentToolName[] { + const invalid: InvalidAgentToolName[] = [] + visitAgentModules(modules, (mod, v) => { + const tools: AgentTool[] = Array.isArray(v.tools) ? v.tools : [] + // Only a flowmodule tool's summary is a callable name: the worker never reads an mcp or + // websearch summary, so a blank one there must stay writable (both default to ''). + const named = tools.filter( + (tool) => tool.value?.tool_type !== 'mcp' && tool.value?.tool_type !== 'websearch' + ) + const siblingNames = named.map((tool) => tool.summary ?? '') + for (const tool of named) { + const name = tool.summary ?? '' + const error = getToolNameError(name, tool.value?.tool_type, siblingNames) + if (error) { + invalid.push({ agentId: mod.id, toolId: tool.id, name, error }) + } + } + }) + return invalid +} diff --git a/frontend/src/lib/components/flows/agentToolUtils.ts b/frontend/src/lib/components/flows/agentToolUtils.ts index 5ac2c9ae2d..f8f8d41413 100644 --- a/frontend/src/lib/components/flows/agentToolUtils.ts +++ b/frontend/src/lib/components/flows/agentToolUtils.ts @@ -1,6 +1,37 @@ import type { AiAgent, FlowModule, FlowModuleValue, InputTransform } from '$lib/gen' import { loadStoredConfig } from '../aiProviderStorage' import { AI_AGENT_SCHEMA } from './flowInfers' +import { forbiddenIds } from './idUtils' + +/** + * A tool's `summary` is the name the LLM sees, and the worker rejects any name that does not match + * `^[a-zA-Z0-9_]+$` (`ai_executor.rs`), so an unvalidated name fails on every run of the flow. + * + * `kind` only has to tell the three tool kinds apart, so callers may pass either the raw + * `value.tool_type` or the module type they resolved it to: anything other than `'mcp'` and + * `'websearch'` — including `undefined` on a legacy tool — is checked as a flow module tool, which + * is what the worker does too. + */ +export function getToolNameError( + name: string, + kind?: 'mcp' | 'websearch' | (string & {}), + siblingNames?: string[] +): string | undefined { + if (kind === 'websearch') return undefined + if (kind === 'mcp') { + return name.length > 0 ? undefined : 'Tool name must not be empty' + } + if (!/^[a-zA-Z0-9_]+$/.test(name)) { + return 'Tool name must only contain letters, numbers and underscores' + } + if (forbiddenIds.includes(name)) { + return `'${name}' is a reserved name` + } + if (siblingNames && siblingNames.filter((n) => n === name).length > 1) { + return 'Duplicate tool name' + } + return undefined +} export const SPECIAL_TOOL_KINDS = ['mcpTool', 'websearchTool', 'aiAgentTool'] as const export type SpecialToolKind = (typeof SPECIAL_TOOL_KINDS)[number] diff --git a/frontend/src/lib/components/flows/common/FlowCardHeader.svelte b/frontend/src/lib/components/flows/common/FlowCardHeader.svelte index 8c9e57fa10..33b47a9cb5 100644 --- a/frontend/src/lib/components/flows/common/FlowCardHeader.svelte +++ b/frontend/src/lib/components/flows/common/FlowCardHeader.svelte @@ -24,7 +24,7 @@ import { getLatestHashForScript } from '$lib/scripts' import { sendUserToast, type Item } from '$lib/utils' import { twMerge } from 'tailwind-merge' - import { getToolNameError } from '$lib/components/graph/renderers/nodes/AIToolNode.svelte' + import { getToolNameError } from '$lib/components/flows/agentToolUtils' import autosize from '$lib/autosize' interface Props { diff --git a/frontend/src/lib/components/graph/renderers/nodes/AIToolNode.svelte b/frontend/src/lib/components/graph/renderers/nodes/AIToolNode.svelte index 9465f5b49f..3de9c9b839 100644 --- a/frontend/src/lib/components/graph/renderers/nodes/AIToolNode.svelte +++ b/frontend/src/lib/components/graph/renderers/nodes/AIToolNode.svelte @@ -1,31 +1,5 @@ + {/each} +
+
Group by
+ {#each GROUP_BY_OPTIONS as option (option.value)} + + {/each} {/if} diff --git a/frontend/src/lib/components/sessions/SessionPicker.svelte b/frontend/src/lib/components/sessions/SessionPicker.svelte index b93d350dfe..67b53aeff4 100644 --- a/frontend/src/lib/components/sessions/SessionPicker.svelte +++ b/frontend/src/lib/components/sessions/SessionPicker.svelte @@ -1,18 +1,22 @@ + +{#snippet bar(q: Quota)} +
+
+
+{/snippet} + +{#snippet ring(q: Quota)} + +{/snippet} + +{#if isCloudHosted() && tightest} +
+ + {#snippet text()} + {tightest.label} this month: {fmt(tightest.used)}/{fmt(tightest.cap)}. + {EXECUTIONS_HINT} + {/snippet} + + +
+ + +
+ {#each quotas as quota (quota.key)} +
+
+ {quota.label} + {fmt(quota.used)}/{fmt(quota.cap)} +
+ {@render bar(quota)} +
+ {/each} +

+ {EXECUTIONS_HINT} Counters reset at the start of every calendar month. +

+ {#if $isPremiumStore} +

+ Your {seats} seat{seats === 1 ? '' : 's'} include {fmt( + (seats ?? 0) * SEAT_EXECUTION_QUOTA + )} executions per month. Every extra {fmt(SEAT_EXECUTION_QUOTA)} executions beyond that add + one billed seat for the month. +

+ {:else} +

+ Either quota reaching {fmt(FREE_EXECUTION_QUOTA)} stops jobs from running for the rest of the + month. Team and Enterprise plans lift both limits. + {#if !$userStore?.is_admin} + Ask a workspace admin to change the plan. + {/if} +

+ {/if} +
+ {#snippet actions()} + {#if $userStore?.is_admin} + + {/if} + {/snippet} + +{/if} diff --git a/frontend/src/lib/components/sidebar/UserMenu.svelte b/frontend/src/lib/components/sidebar/UserMenu.svelte index 1ec649cf12..43440249ae 100644 --- a/frontend/src/lib/components/sidebar/UserMenu.svelte +++ b/frontend/src/lib/components/sidebar/UserMenu.svelte @@ -14,8 +14,9 @@ import { Crown, ServerCog, LogOut, Moon, Settings, Sun, User } from 'lucide-svelte' import DarkModeObserver from '../DarkModeObserver.svelte' import MenuButton from './MenuButton.svelte' - import { Menu, MenuItem } from '$lib/components/meltComponents' + import { Menu, MenuItem, Tooltip } from '$lib/components/meltComponents' import { type MenubarBuilders } from '@melt-ui/svelte' + import { EXECUTIONS_HINT } from './executionsHint' let darkMode: boolean = $state(false) @@ -103,27 +104,41 @@ - {#if isCloudHosted()} + + {#if isCloudHosted() && $isPremiumStore !== undefined}
- {#if !$isPremiumStore} - {$usageStore}/1000 user execs + {#if $isPremiumStore === false} + + {$usageStore ?? '—'}/1000 user execs + + {#snippet text()} + {EXECUTIONS_HINT} + {/snippet} + +
-
{#if $workspaceStore != 'demo'} - {$workspaceUsageStore}/1000 free workspace execs + + {$workspaceUsageStore ?? '—'}/1000 free workspace execs + + {#snippet text()} + {EXECUTIONS_HINT} + {/snippet} + +
diff --git a/frontend/src/lib/components/sidebar/WorkspaceMenu.svelte b/frontend/src/lib/components/sidebar/WorkspaceMenu.svelte index 286e044e6b..1555884eff 100644 --- a/frontend/src/lib/components/sidebar/WorkspaceMenu.svelte +++ b/frontend/src/lib/components/sidebar/WorkspaceMenu.svelte @@ -2,6 +2,7 @@ import { workspaceMenuHref } from './workspaceMenuHref' import { isPremiumStore, + maybePremium, superadmin, userStore, userWorkspaces, @@ -16,7 +17,8 @@ import { SvelteSet } from 'svelte/reactivity' import { Badge, CopyButton, NameIdTooltip } from '$lib/components/common' import MenuButton from '$lib/components/sidebar/MenuButton.svelte' - import { Menu, MenuItem } from '$lib/components/meltComponents' + import { Menu, MenuItem, Tooltip } from '$lib/components/meltComponents' + import { EXECUTIONS_HINT } from './executionsHint' import WorkspaceIcon from '$lib/components/workspace/WorkspaceIcon.svelte' import { fixupUrlAfterWorkspaceSwitch } from './workspaceSwitchUrl' import { goto } from '$lib/navigation' @@ -114,9 +116,7 @@ // modal carries its own base-workspace picker). Hidden on non-premium cloud, // in the admins workspace, or when forking is disabled. const canForkHere = $derived( - (!isCloudHosted() || $isPremiumStore) && - $workspaceStore !== 'admins' && - canCreateFork($userStore) + (!isCloudHosted() || $maybePremium) && $workspaceStore !== 'admins' && canCreateFork($userStore) ) const familyWorkspaces = $derived.by(() => { if (strictWorkspaceSelect) return hierarchy @@ -422,14 +422,21 @@
{/if} - {#if isCloudHosted() && !$isPremiumStore && !strictWorkspaceSelect} + {#if isCloudHosted() && $isPremiumStore === false && !strictWorkspaceSelect}
{#if $workspaceStore != 'demo'} - {$workspaceUsageStore}/1000 free workspace execs + + {$workspaceUsageStore ?? '—'}/1000 free workspace execs + + {#snippet text()} + {EXECUTIONS_HINT} + {/snippet} + +
-
{/if} diff --git a/frontend/src/lib/components/sidebar/executionsHint.ts b/frontend/src/lib/components/sidebar/executionsHint.ts new file mode 100644 index 0000000000..6530a6a6cd --- /dev/null +++ b/frontend/src/lib/components/sidebar/executionsHint.ts @@ -0,0 +1,7 @@ +export const FREE_EXECUTION_QUOTA = 1000 + +/** Executions each paid seat includes per month (mirrors the billing page). */ +export const SEAT_EXECUTION_QUOTA = 10000 + +export const EXECUTIONS_HINT = + 'An execution is one second of compute, not one job run: a job counts as 1 execution, plus 1 more for each additional second it runs.' as const diff --git a/frontend/src/lib/stores.ts b/frontend/src/lib/stores.ts index 0058cc8dd0..8710b6d20b 100644 --- a/frontend/src/lib/stores.ts +++ b/frontend/src/lib/stores.ts @@ -80,8 +80,10 @@ export const whitelabelNameStore = derived([enterpriseLicense], ([enterpriseLice return undefined }) export const workerTags = writable(undefined) -export const usageStore = writable(0) -export const workspaceUsageStore = writable(0) +// `undefined` while unresolved. `0` is a real usage value, so a placeholder that +// reads as one lets a failed or in-flight fetch render as "no executions used". +export const usageStore = writable(undefined) +export const workspaceUsageStore = writable(undefined) export const initialArgsStore = writable(undefined) export const oauthStore = writable(undefined) export const userStore = writable(undefined) @@ -90,7 +92,25 @@ export const workspaceStore = writable( ) export const defaultScripts = writable(undefined) export const dbClockDrift = writable(undefined) -export const isPremiumStore = writable(false) +// `undefined` until the active workspace's tier is known — a tier belongs to a +// workspace, so consumers rendering a number from it must not read the previous +// one's value across a switch. `false` is a claim, not a safe default: it meters a +// paid workspace against the free cap, so a failed fetch leaves this `undefined`. +export const isPremiumStore = writable(undefined) +// Set when the tier fetch for the active workspace failed, which is indistinguishable +// from "still pending" in `isPremiumStore` alone. +export const premiumFetchFailed = writable(false) +// For affordances rather than numbers: gate on this so a paid→paid switch doesn't +// retract a button for the length of the fetch, while a failed fetch still fails +// closed instead of leaving it enabled for the session. +export const maybePremium: Readable = derived( + [isPremiumStore, premiumFetchFailed], + ([premium, failed]) => premium !== false && !failed +) +// Bumped when the active workspace's membership is seen to have changed, so anything +// deriving a number from the member count (paid seats) can re-resolve it without +// polling or owning its own invalidation. +export const workspaceMembershipVersion = writable(0) export const usersWorkspaceStore = writable(undefined) export const superadmin = writable(undefined) export const devopsRole = writable(undefined) diff --git a/frontend/src/lib/usage.svelte.ts b/frontend/src/lib/usage.svelte.ts new file mode 100644 index 0000000000..d64934e94b --- /dev/null +++ b/frontend/src/lib/usage.svelte.ts @@ -0,0 +1,99 @@ +import { resource } from 'runed' +import { UserService, WorkspaceService } from '$lib/gen' +import { isCloudHosted } from '$lib/cloud' +import { scopedValue, tagged } from '$lib/utils/scopedValue' +import { + isPremiumStore, + premiumFetchFailed, + usageStore, + workspaceUsageStore, + type UserExt +} from '$lib/stores' + +/** + * The cloud execution counters and the workspace's plan tier. Call once, at layout init: + * these are app-wide values, and the logged-in layout outlives every in-app navigation. + */ +export function createUsageResources(args: { + workspace: () => string | undefined + user: () => UserExt | undefined +}) { + // All three need an authenticated membership, so they key on the user being loaded + // *for this workspace* — a switch must not fire them against the workspace we left. + const readyWorkspace = () => { + const workspace = args.workspace() + if (!isCloudHosted() || !workspace) return undefined + return args.user()?.workspace_id === workspace ? workspace : undefined + } + // The user counter is account-wide, so its key is the account: a workspace switch + // is not a change of key and must not re-fetch or clear it. + const readyUser = () => (isCloudHosted() ? args.user()?.email : undefined) + + // `Number(...)`: both usage endpoints serve text/plain, so the client hands back a + // string despite the generated `number` type. Interpolation and arithmetic coerce + // it, but `toLocaleString` on a string returns it unchanged — the thousands + // separator would silently go missing above 999. + const fetchWorkspaceExecutions = tagged(async (workspace: string) => + Number(await WorkspaceService.getWorkspaceUsage({ workspace })) + ) + const fetchUserExecutions = tagged(async (_email: string) => Number(await UserService.getUsage())) + const fetchPremium = tagged((workspace: string) => WorkspaceService.getIsPremium({ workspace })) + + const workspaceExecutions = resource(readyWorkspace, async (workspace) => + workspace ? await fetchWorkspaceExecutions(workspace) : undefined + ) + + const userExecutions = resource(readyUser, async (email) => + email ? await fetchUserExecutions(email) : undefined + ) + + const premium = resource(readyWorkspace, async (workspace) => + workspace ? await fetchPremium(workspace) : undefined + ) + + const scopedWorkspaceExecutions = scopedValue() + const scopedUserExecutions = scopedValue() + const scopedPremium = scopedValue() + + // The only place any of this reaches a store. `undefined` until a value for the + // active scope has arrived — never a stand-in like `0` or `false`, both of which are + // legal values a consumer would render as real. + $effect(() => { + workspaceUsageStore.set( + scopedWorkspaceExecutions(args.workspace(), workspaceExecutions.current) + ) + }) + + $effect(() => { + usageStore.set(scopedUserExecutions(args.user()?.email, userExecutions.current)) + }) + + $effect(() => { + const tier = scopedPremium(args.workspace(), premium.current) + isPremiumStore.set(tier) + // Only a failure that left us with no tier for this workspace counts: a late + // rejection for a workspace we left must not retract affordances here. + premiumFetchFailed.set(!!premium.error && tier === undefined) + }) + + return { + /** Re-reads the counters. Executions accrue continuously, so anything displaying + * them needs this — the workspace-change refetch alone leaves an open tab stale. */ + refreshExecutions() { + void workspaceExecutions.refetch() + void userExecutions.refetch() + } + } +} + +// Registered by the layout so components can ask for a re-read without owning the +// resources or reaching back into the layout. +let handle: ReturnType | undefined = undefined + +export function registerUsageResources(h: ReturnType): void { + handle = h +} + +export function refreshExecutions(): void { + handle?.refreshExecutions() +} diff --git a/frontend/src/lib/utils/scopedValue.test.ts b/frontend/src/lib/utils/scopedValue.test.ts new file mode 100644 index 0000000000..202547693c --- /dev/null +++ b/frontend/src/lib/utils/scopedValue.test.ts @@ -0,0 +1,75 @@ +import { describe, expect, it } from 'vitest' +import { scopedValue, tagged } from './scopedValue' + +// The ordering these assert is the one `resource` does not provide, and the one whose +// absence produced the stale-workspace and A→B→A defects this guard replaced. +describe('scopedValue', () => { + it('holds a value only for the key it describes', () => { + const held = scopedValue() + expect(held('a', undefined)).toBe(undefined) + expect(held('a', { key: 'a', seq: 1, value: 1 })).toBe(1) + // Switched to b, nothing fetched for it yet: a's value must not stand in. + expect(held('b', { key: 'a', seq: 1, value: 1 })).toBe(undefined) + expect(held('b', { key: 'b', seq: 2, value: 2 })).toBe(2) + }) + + it('ignores an answer for a scope we left, instead of publishing or erasing', () => { + const held = scopedValue() + held('a', { key: 'a', seq: 1, value: 1 }) + held('b', { key: 'b', seq: 2, value: 2 }) + // A's slow response lands after B resolved: neither replaces B's value nor blanks it. + expect(held('b', { key: 'a', seq: 1, value: 99 })).toBe(2) + }) + + it('ignores an answer overtaken by a newer one for the same key', () => { + const held = scopedValue() + // Two fetches for one key — a refetch landing on an in-flight load, or a second + // invalidation — resolving inverted. The later-issued value must win. + expect(held('a', { key: 'a', seq: 2, value: 20 })).toBe(20) + expect(held('a', { key: 'a', seq: 1, value: 10 })).toBe(20) + }) + + it('keeps the value across a re-read of the same key', () => { + const held = scopedValue() + held('a', { key: 'a', seq: 1, value: 1 }) + // A refetch leaves the previous value in place until the new one lands, so the + // display never blanks mid-refresh. + expect(held('a', { key: 'a', seq: 1, value: 1 })).toBe(1) + expect(held('a', { key: 'a', seq: 2, value: 5 })).toBe(5) + }) + + it('treats returning to a key as unknown until it is fetched again', () => { + const held = scopedValue() + held('a', { key: 'a', seq: 1, value: 1 }) + held('b', { key: 'b', seq: 2, value: 2 }) + expect(held('a', undefined)).toBe(undefined) + }) + + it('orders a late answer against the read issued on returning to its key', () => { + const held = scopedValue() + // A's first read is still in flight when we leave for B, so nothing for A is held. + expect(held('b', { key: 'b', seq: 2, value: 2 })).toBe(2) + // Back on A, that late answer is the only value describing A, so it stands... + expect(held('a', { key: 'a', seq: 1, value: 10 })).toBe(10) + // ...until the read issued on returning lands, and cannot come back afterwards. + expect(held('a', { key: 'a', seq: 3, value: 30 })).toBe(30) + expect(held('a', { key: 'a', seq: 1, value: 10 })).toBe(30) + }) + + it('stamps issue order even when responses resolve inverted', async () => { + const settle: Array<(v: number) => void> = [] + const fetch = tagged((_key: string) => new Promise((r) => settle.push(r))) + const first = fetch('a') + const second = fetch('a') + // Resolve the second request first, then the first: the seq must reflect the + // order they were *issued*, not the order they came back. + settle[1](20) + settle[0](10) + expect(await first).toEqual({ key: 'a', seq: 1, value: 10 }) + expect(await second).toEqual({ key: 'a', seq: 2, value: 20 }) + + const held = scopedValue() + expect(held('a', await second)).toBe(20) + expect(held('a', await first)).toBe(20) + }) +}) diff --git a/frontend/src/lib/utils/scopedValue.ts b/frontend/src/lib/utils/scopedValue.ts new file mode 100644 index 0000000000..86bdfb8a7d --- /dev/null +++ b/frontend/src/lib/utils/scopedValue.ts @@ -0,0 +1,33 @@ +export type Tagged = { key: string; seq: number; value: T } + +/** + * Stamps each result with the scope it describes and the order its request was issued in. + * `resource` orders nothing: it assigns `current` unconditionally on resolve, and cancels + * through an `AbortSignal` the generated client cannot consume. The scope alone cannot + * order two requests for one scope, so the issue order travels alongside it. + */ +export function tagged( + fetch: (key: K) => Promise +): (key: K) => Promise> { + let issued = 0 + return async (key: K) => { + const seq = ++issued + return { key, seq, value: await fetch(key) } + } +} + +/** + * Holds the newest value fetched for `key`; a late answer for a scope we left, or one + * overtaken for this scope, neither publishes nor erases. A failed refresh leaves the + * last successful value standing, which is why `loading` cannot gate this: it is true + * throughout a re-read whose held value is still the right one to show. + */ +export function scopedValue() { + let held: Tagged | undefined = undefined + return (key: string | undefined, fetched: Tagged | undefined) => { + if (fetched && fetched.key === key && (held?.key !== key || fetched.seq > held.seq)) { + held = fetched + } + return held && held.key === key ? held.value : undefined + } +} diff --git a/frontend/src/routes/(root)/(logged)/+layout.svelte b/frontend/src/routes/(root)/(logged)/+layout.svelte index ead6fa6d8d..e55cfd54a9 100644 --- a/frontend/src/routes/(root)/(logged)/+layout.svelte +++ b/frontend/src/routes/(root)/(logged)/+layout.svelte @@ -16,6 +16,7 @@ import WorkspaceMenu from '$lib/components/sidebar/WorkspaceMenu.svelte' import SidebarContent from '$lib/components/sidebar/SidebarContent.svelte' import SettingsMenu from '$lib/components/sidebar/SettingsMenu.svelte' + import SidebarUsage from '$lib/components/sidebar/SidebarUsage.svelte' import SidebarScrollArea from '$lib/components/sidebar/SidebarScrollArea.svelte' import { SIDEBAR_BG, SIDEBAR_BG_DARK } from '$lib/components/sidebar/sidebarChrome' import CriticalAlertModal from '$lib/components/sidebar/CriticalAlertModal.svelte' @@ -23,10 +24,7 @@ import UpdateDevWorkspaceModal from '$lib/components/UpdateDevWorkspaceModal.svelte' import { enterpriseLicense, - isPremiumStore, superadmin, - usageStore, - workspaceUsageStore, userStore, workspaceStore, userWorkspaces, @@ -72,6 +70,7 @@ import MenuButton from '$lib/components/sidebar/MenuButton.svelte' import MenuLink from '$lib/components/sidebar/MenuLink.svelte' import { loadProtectionRules } from '$lib/workspaceProtectionRules.svelte' + import { createUsageResources, registerUsageResources } from '$lib/usage.svelte' import { purgeLegacyUserDrafts } from '$lib/userDraftLegacyMigration' import { migrateUserDraftsToDb } from '$lib/userDraftDbMigration' import DraftMigrationErrorModal from '$lib/components/DraftMigrationErrorModal.svelte' @@ -108,6 +107,15 @@ let { children }: Props = $props() OpenAPI.WITH_CREDENTIALS = true + // Owned here because the logged-in layout is the app's lifetime: it outlives every + // in-app navigation, so the counters and tier resolve once per workspace rather than + // per mounting component, and no detached `$effect.root` is needed to hold them. + registerUsageResources( + createUsageResources({ + workspace: () => $workspaceStore, + user: () => $userStore + }) + ) let menuOpen = $state(false) // Set by the workspace⇄session switch before it navigates, so the mobile menu // drawer stays open across a mode toggle (unlike a normal link navigation, @@ -327,16 +335,6 @@ } catch (e) { console.error('Could not persist username to local storage', e) } - // Populate for all members (not just admins) so non-admin developers also get premium-gated - // affordances like the fork entry points on cloud. The `is_premium` endpoint is a boolean - // and no longer admin-gated. Best-effort: a failure here must not block user-store init. - if (isCloudHosted()) { - try { - isPremiumStore.set(await WorkspaceService.getIsPremium({ workspace })) - } catch (e) { - console.error('Could not fetch premium status', e) - } - } } else { userStore.set(undefined) } @@ -460,7 +458,6 @@ function onLoad() { loadFavorites() - loadUsage() syncTutorialsTodos() loadHubBaseUrl() loadWsBaseUrl() @@ -468,15 +465,6 @@ loadUsedTriggerKinds() } - async function loadUsage() { - if (isCloudHosted() && $workspaceStore) { - $usageStore = await UserService.getUsage() - $workspaceUsageStore = await WorkspaceService.getWorkspaceUsage({ - workspace: $workspaceStore! - }) - } - } - async function loadHubBaseUrl() { $hubBaseUrlStore = ((await SettingService.getGlobal({ key: 'hub_accessible_url' })) as string) || @@ -1090,6 +1078,10 @@ {/if} +
+ +
+
{@render brandMark(false)}
@@ -1224,6 +1216,10 @@ {/if} +
+ +
+
"] edition = "2021" diff --git a/backend/parsers/windmill-parser-wasm/Cargo.lock b/backend/parsers/windmill-parser-wasm/Cargo.lock index 9f7870863e..7c78701d80 100644 --- a/backend/parsers/windmill-parser-wasm/Cargo.lock +++ b/backend/parsers/windmill-parser-wasm/Cargo.lock @@ -6191,7 +6191,7 @@ checksum = "712e227841d057c1ee1cd2fb22fa7e5a5461ae8e48fa2ca79ec42cfc1931183f" [[package]] name = "windmill-common" -version = "1.792.2" +version = "1.793.0" dependencies = [ "aho-corasick", "anyhow", @@ -6274,7 +6274,7 @@ dependencies = [ [[package]] name = "windmill-macros" -version = "1.792.2" +version = "1.793.0" dependencies = [ "proc-macro2", "quote", @@ -6286,7 +6286,7 @@ dependencies = [ [[package]] name = "windmill-parser" -version = "1.792.2" +version = "1.793.0" dependencies = [ "convert_case", "serde", @@ -6295,7 +6295,7 @@ dependencies = [ [[package]] name = "windmill-parser-bash" -version = "1.792.2" +version = "1.793.0" dependencies = [ "anyhow", "lazy_static", @@ -6307,7 +6307,7 @@ dependencies = [ [[package]] name = "windmill-parser-csharp" -version = "1.792.2" +version = "1.793.0" dependencies = [ "anyhow", "serde_json", @@ -6319,7 +6319,7 @@ dependencies = [ [[package]] name = "windmill-parser-go" -version = "1.792.2" +version = "1.793.0" dependencies = [ "anyhow", "gosyn", @@ -6331,7 +6331,7 @@ dependencies = [ [[package]] name = "windmill-parser-graphql" -version = "1.792.2" +version = "1.793.0" dependencies = [ "anyhow", "lazy_static", @@ -6343,7 +6343,7 @@ dependencies = [ [[package]] name = "windmill-parser-java" -version = "1.792.2" +version = "1.793.0" dependencies = [ "anyhow", "serde_json", @@ -6355,7 +6355,7 @@ dependencies = [ [[package]] name = "windmill-parser-nu" -version = "1.792.2" +version = "1.793.0" dependencies = [ "anyhow", "nu-parser", @@ -6366,7 +6366,7 @@ dependencies = [ [[package]] name = "windmill-parser-php" -version = "1.792.2" +version = "1.793.0" dependencies = [ "anyhow", "itertools 0.14.0", @@ -6377,7 +6377,7 @@ dependencies = [ [[package]] name = "windmill-parser-py" -version = "1.792.2" +version = "1.793.0" dependencies = [ "anyhow", "itertools 0.14.0", @@ -6389,7 +6389,7 @@ dependencies = [ [[package]] name = "windmill-parser-py-asset" -version = "1.792.2" +version = "1.793.0" dependencies = [ "anyhow", "rustpython-ast", @@ -6400,7 +6400,7 @@ dependencies = [ [[package]] name = "windmill-parser-py-imports" -version = "1.792.2" +version = "1.793.0" dependencies = [ "anyhow", "async-recursion", @@ -6422,7 +6422,7 @@ dependencies = [ [[package]] name = "windmill-parser-r" -version = "1.792.2" +version = "1.793.0" dependencies = [ "anyhow", "serde_json", @@ -6434,7 +6434,7 @@ dependencies = [ [[package]] name = "windmill-parser-ruby" -version = "1.792.2" +version = "1.793.0" dependencies = [ "anyhow", "lazy_static", @@ -6448,7 +6448,7 @@ dependencies = [ [[package]] name = "windmill-parser-rust" -version = "1.792.2" +version = "1.793.0" dependencies = [ "anyhow", "convert_case", @@ -6465,7 +6465,7 @@ dependencies = [ [[package]] name = "windmill-parser-sql" -version = "1.792.2" +version = "1.793.0" dependencies = [ "anyhow", "lazy_static", @@ -6478,7 +6478,7 @@ dependencies = [ [[package]] name = "windmill-parser-sql-asset" -version = "1.792.2" +version = "1.793.0" dependencies = [ "anyhow", "serde", @@ -6490,7 +6490,7 @@ dependencies = [ [[package]] name = "windmill-parser-ts" -version = "1.792.2" +version = "1.793.0" dependencies = [ "anyhow", "lazy_static", @@ -6508,7 +6508,7 @@ dependencies = [ [[package]] name = "windmill-parser-ts-asset" -version = "1.792.2" +version = "1.793.0" dependencies = [ "anyhow", "serde-wasm-bindgen", @@ -6524,7 +6524,7 @@ dependencies = [ [[package]] name = "windmill-parser-wac" -version = "1.792.2" +version = "1.793.0" dependencies = [ "anyhow", "rustpython-ast", @@ -6540,7 +6540,7 @@ dependencies = [ [[package]] name = "windmill-parser-wasm" -version = "1.792.2" +version = "1.793.0" dependencies = [ "anyhow", "getrandom 0.2.17", @@ -6572,7 +6572,7 @@ dependencies = [ [[package]] name = "windmill-parser-yaml" -version = "1.792.2" +version = "1.793.0" dependencies = [ "anyhow", "lazy_static", @@ -6586,7 +6586,7 @@ dependencies = [ [[package]] name = "windmill-types" -version = "1.792.2" +version = "1.793.0" dependencies = [ "anyhow", "bitflags", diff --git a/backend/parsers/windmill-parser-wasm/Cargo.toml b/backend/parsers/windmill-parser-wasm/Cargo.toml index 03243d5824..5477baf13d 100644 --- a/backend/parsers/windmill-parser-wasm/Cargo.toml +++ b/backend/parsers/windmill-parser-wasm/Cargo.toml @@ -12,7 +12,7 @@ resolver = "2" members = ["."] [workspace.package] -version = "1.792.2" +version = "1.793.0" edition = "2021" authors = ["Ruben Fiszel "] diff --git a/backend/windmill-api/openapi.yaml b/backend/windmill-api/openapi.yaml index d16bbf0389..375c15f2fb 100644 --- a/backend/windmill-api/openapi.yaml +++ b/backend/windmill-api/openapi.yaml @@ -1,7 +1,7 @@ openapi: "3.0.3" info: - version: 1.792.2 + version: 1.793.0 title: Windmill API contact: diff --git a/benchmarks/lib.ts b/benchmarks/lib.ts index f5dffe015f..816abbf5b4 100644 --- a/benchmarks/lib.ts +++ b/benchmarks/lib.ts @@ -2,7 +2,7 @@ import { sleep } from "https://deno.land/x/sleep@v1.2.1/mod.ts"; import * as windmill from "https://deno.land/x/windmill@v1.174.0/mod.ts"; import * as api from "https://deno.land/x/windmill@v1.174.0/windmill-api/index.ts"; -export const VERSION = "v1.792.2"; +export const VERSION = "v1.793.0"; export async function login(email: string, password: string): Promise { return await windmill.UserService.login({ diff --git a/cli/src/core/constants.ts b/cli/src/core/constants.ts index 11cf3f2430..7883ce0876 100644 --- a/cli/src/core/constants.ts +++ b/cli/src/core/constants.ts @@ -10,4 +10,4 @@ export const WM_FORK_PREFIX = "wm-fork"; // (e.g. utils.ts) can read it without importing main.ts and creating a circular // dependency (main → workspace → utils → main) that triggers a TDZ. // Re-exported from main.ts for backwards compatibility. -export const VERSION = "1.792.2"; +export const VERSION = "1.793.0"; diff --git a/frontend/package-lock.json b/frontend/package-lock.json index 4bfeac0b46..530af912bb 100644 --- a/frontend/package-lock.json +++ b/frontend/package-lock.json @@ -1,12 +1,12 @@ { "name": "@windmill-labs/components", - "version": "1.792.2", + "version": "1.793.0", "lockfileVersion": 3, "requires": true, "packages": { "": { "name": "@windmill-labs/components", - "version": "1.792.2", + "version": "1.793.0", "hasInstallScript": true, "license": "AGPL-3.0", "dependencies": { diff --git a/frontend/package.json b/frontend/package.json index 0e16aeb90a..3fa2ce0c72 100644 --- a/frontend/package.json +++ b/frontend/package.json @@ -1,6 +1,6 @@ { "name": "@windmill-labs/components", - "version": "1.792.2", + "version": "1.793.0", "scripts": { "dev": "vite dev", "dev:ui-builder": "mv static/ui_builder static/ui_builder.dev-disabled 2>/dev/null || true ; trap 'mv static/ui_builder.dev-disabled static/ui_builder 2>/dev/null || true' EXIT ; vite dev", diff --git a/lsp/Pipfile b/lsp/Pipfile index c3cc60549f..8175909e88 100644 --- a/lsp/Pipfile +++ b/lsp/Pipfile @@ -4,7 +4,7 @@ verify_ssl = true name = "pypi" [packages] -wmill = ">=1.792.2" +wmill = ">=1.793.0" sendgrid = "*" mysql-connector-python = "*" pymongo = "*" diff --git a/openflow.openapi.yaml b/openflow.openapi.yaml index 7b33d4d3a7..8b93ee844f 100644 --- a/openflow.openapi.yaml +++ b/openflow.openapi.yaml @@ -1,7 +1,7 @@ openapi: '3.0.3' info: - version: 1.792.2 + version: 1.793.0 title: OpenFlow Spec contact: name: Ruben Fiszel diff --git a/powershell-client/WindmillClient/WindmillClient.psd1 b/powershell-client/WindmillClient/WindmillClient.psd1 index 032e60a069..0919ab4cbf 100644 --- a/powershell-client/WindmillClient/WindmillClient.psd1 +++ b/powershell-client/WindmillClient/WindmillClient.psd1 @@ -12,7 +12,7 @@ RootModule = 'WindmillClient.psm1' # Version number of this module. - ModuleVersion = '1.792.2' + ModuleVersion = '1.793.0' # Supported PSEditions # CompatiblePSEditions = @() diff --git a/python-client/wmill/pyproject.toml b/python-client/wmill/pyproject.toml index e766ec3ff1..e12a91bf8d 100644 --- a/python-client/wmill/pyproject.toml +++ b/python-client/wmill/pyproject.toml @@ -1,6 +1,6 @@ [tool.poetry] name = "wmill" -version = "1.792.2" +version = "1.793.0" description = "A client library for accessing Windmill server wrapping the Windmill client API" license = "Apache-2.0" homepage = "https://windmill.dev" diff --git a/typescript-client/jsr.json b/typescript-client/jsr.json index 3c89ca6bde..a7a3656a60 100644 --- a/typescript-client/jsr.json +++ b/typescript-client/jsr.json @@ -1,6 +1,6 @@ { "name": "@windmill/windmill", - "version": "1.792.2", + "version": "1.793.0", "exports": "./src/index.ts", "publish": { "exclude": ["!src", "./s3Types.ts", "./sqlUtils.ts", "./client.ts", "./wacError.ts"] diff --git a/typescript-client/package.json b/typescript-client/package.json index a5add6bee3..84caebe95a 100644 --- a/typescript-client/package.json +++ b/typescript-client/package.json @@ -1,7 +1,7 @@ { "name": "windmill-client", "description": "Windmill SDK client for browsers and Node.js", - "version": "1.792.2", + "version": "1.793.0", "author": "Ruben Fiszel", "license": "Apache 2.0", "homepage": "https://github.com/windmill-labs/windmill/tree/main/typescript-client#readme", diff --git a/version.txt b/version.txt index b73809024c..374b8cbb71 100644 --- a/version.txt +++ b/version.txt @@ -1 +1 @@ -1.792.2 +1.793.0 diff --git a/windmill-yaml-validator/package-lock.json b/windmill-yaml-validator/package-lock.json index bc0ffa9005..6a70bdf8e3 100644 --- a/windmill-yaml-validator/package-lock.json +++ b/windmill-yaml-validator/package-lock.json @@ -1,12 +1,12 @@ { "name": "windmill-yaml-validator", - "version": "1.792.2", + "version": "1.793.0", "lockfileVersion": 3, "requires": true, "packages": { "": { "name": "windmill-yaml-validator", - "version": "1.792.2", + "version": "1.793.0", "license": "Apache 2.0", "dependencies": { "@stoplight/yaml": "^4.3.0", diff --git a/windmill-yaml-validator/package.json b/windmill-yaml-validator/package.json index 39db260ea2..b8ed46b0bc 100644 --- a/windmill-yaml-validator/package.json +++ b/windmill-yaml-validator/package.json @@ -1,6 +1,6 @@ { "name": "windmill-yaml-validator", - "version": "1.792.2", + "version": "1.793.0", "description": "YAML validator for Windmill flow, schedule, and trigger files", "main": "dist/index.js", "types": "dist/index.d.ts", From a9112b72a527af06a204827fbdff5ff9cb451f5d Mon Sep 17 00:00:00 2001 From: Ruben Fiszel Date: Fri, 21 Aug 2026 01:05:26 +0200 Subject: [PATCH 12/48] fix: make workspace preprocessor scripts selectable in flow preprocessor steps (#10786) * fix: pick workspace preprocessor scripts in the flow preprocessor step Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01QqKXVsBXynMFtMZ26uvLw7 * fix: explain the empty preprocessor list and keep the editor bar hub populated Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01QqKXVsBXynMFtMZ26uvLw7 * fix: derive the editor bar's script kind from the preprocessor slot Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01QqKXVsBXynMFtMZ26uvLw7 * fix: keep the preprocessor entrypoint when resetting a step's content Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01QqKXVsBXynMFtMZ26uvLw7 * docs: state the preprocessor reset invariant instead of the old control flow Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01QqKXVsBXynMFtMZ26uvLw7 --------- Co-authored-by: Claude Opus 5 --- frontend/src/lib/components/EditorBar.svelte | 10 ++- .../flows/content/FlowInputs.svelte | 36 +++++---- .../flows/content/FlowModuleComponent.svelte | 9 ++- .../flows/content/FlowModuleWrapper.svelte | 2 +- .../pickers/WorkspaceScriptPicker.svelte | 74 +++++++++++-------- .../pickers/WorkspaceScriptPickerQuick.svelte | 5 ++ frontend/src/lib/script_helpers.test.ts | 25 ++++++- frontend/src/lib/script_helpers.ts | 5 ++ 8 files changed, 115 insertions(+), 51 deletions(-) diff --git a/frontend/src/lib/components/EditorBar.svelte b/frontend/src/lib/components/EditorBar.svelte index d0e6d05a1a..91516843ab 100644 --- a/frontend/src/lib/components/EditorBar.svelte +++ b/frontend/src/lib/components/EditorBar.svelte @@ -94,7 +94,7 @@ // editor). `undefined` = not applicable; `false` makes the badge red // even if the main function parses. validAssets?: boolean | undefined - kind?: 'script' | 'trigger' | 'approval' + kind?: 'script' | 'trigger' | 'approval' | 'preprocessor' template?: | 'pgsql' | 'mysql' @@ -580,7 +580,13 @@ {#if pick_existing == 'hub'} - + + {:else} diff --git a/frontend/src/lib/components/flows/content/FlowInputs.svelte b/frontend/src/lib/components/flows/content/FlowInputs.svelte index e275df3976..f86d870a1a 100644 --- a/frontend/src/lib/components/flows/content/FlowInputs.svelte +++ b/frontend/src/lib/components/flows/content/FlowInputs.svelte @@ -44,6 +44,11 @@ ? 'approval' : 'script' ) + // The preprocessor slot shows no kind toggle, so `kind` stays 'script' there. Everything that + // tags a script (inline template, pre-made list) must key off this, never off `kind`. + let scriptKind: 'script' | 'failure' | 'approval' | 'trigger' | 'preprocessor' = $derived( + preprocessorModule ? 'preprocessor' : kind + ) let pick_existing: 'workspace' | 'hub' = $state('hub') let filter = $state('') @@ -56,7 +61,7 @@ ) function displayLang(lang: SupportedLanguage | 'docker', kind: string) { - if (preprocessorModule) { + if (kind === 'preprocessor') { return canHavePreprocessor(lang as SupportedLanguage) } @@ -218,21 +223,23 @@

Inline new {kind == 'script' ? 'action' : kind}{scriptKind == 'script' ? 'action' : scriptKind} script - Embed {kind == 'script' ? 'action' : kind} script directly inside a flow instead - of saving the script into your workspace for reuse. You can always save an inline script to - your workspace later. + Embed {scriptKind == 'script' ? 'action' : scriptKind} script directly inside + a flow instead of saving the script into your workspace for reuse. You can always save an inline + script to your workspace later.
@@ -250,7 +257,7 @@ {/if}
{#each langs.filter((lang) => customUi?.languages == undefined || customUi?.languages?.includes(lang?.[1])) as [label, lang] (lang)} - {#if displayLang(lang, kind)} + {#if displayLang(lang, scriptKind)} { dispatch('new', { language: lang == 'docker' ? 'bash' : lang, - kind, + kind: scriptKind, subkind: lang == 'docker' ? 'docker' : preprocessorModule ? 'preprocessor' : 'flow', summary }) @@ -289,10 +296,13 @@

Use pre-made {kind == 'script' ? 'action' : kind}{scriptKind == 'script' ? 'action' : scriptKind} script

- {#if pick_existing == 'hub'} + {#if preprocessorModule} + + + {:else if pick_existing == 'hub'} diff --git a/frontend/src/lib/components/flows/content/FlowModuleComponent.svelte b/frontend/src/lib/components/flows/content/FlowModuleComponent.svelte index d1fdf8f412..d7a8c4effa 100644 --- a/frontend/src/lib/components/flows/content/FlowModuleComponent.svelte +++ b/frontend/src/lib/components/flows/content/FlowModuleComponent.svelte @@ -109,7 +109,7 @@ preprocessorModule?: boolean parentModule?: FlowModule | undefined previousModule: FlowModule | undefined - scriptKind?: 'script' | 'trigger' | 'approval' + scriptKind?: 'script' | 'trigger' | 'approval' | 'preprocessor' scriptTemplate?: 'pgsql' | 'mysql' | 'script' | 'docker' | 'powershell' noEditor: boolean enableAi: boolean @@ -171,6 +171,11 @@ shellcheck: false }) + // `scriptKind` only records how a step was created this session, so it is back to 'script' on + // any remount. Being a preprocessor is a property of the slot, and the editor bar's reset code + // and script library depend on it, so derive it rather than reading the stale state. + let editorScriptKind = $derived(preprocessorModule ? 'preprocessor' : scriptKind) + let selected = $state(untrack(() => preprocessorModule) ? 'test' : 'inputs') let canShowChatTab = $derived( !preprocessorModule && @@ -864,7 +869,7 @@ {websocketAlive} iconOnly={width < EDITOR_BAR_WIDTH_THRESHOLD} compactHelpers={width < EDITOR_BAR_HELPERS_INLINE_THRESHOLD} - kind={scriptKind} + kind={editorScriptKind} template={scriptTemplate} args={Object.entries(flowModule.value.input_transforms).reduce((acc, [key, obj]) => { acc[key] = obj.type === 'static' ? obj.value : undefined diff --git a/frontend/src/lib/components/flows/content/FlowModuleWrapper.svelte b/frontend/src/lib/components/flows/content/FlowModuleWrapper.svelte index 6f832902d0..c5ae6901ac 100644 --- a/frontend/src/lib/components/flows/content/FlowModuleWrapper.svelte +++ b/frontend/src/lib/components/flows/content/FlowModuleWrapper.svelte @@ -32,7 +32,7 @@ const { triggersState, triggersCount } = getContext('TriggerContext') - let scriptKind: 'script' | 'trigger' | 'approval' = $state('script') + let scriptKind: 'script' | 'trigger' | 'approval' | 'preprocessor' = $state('script') let scriptTemplate: 'pgsql' | 'mysql' | 'script' | 'docker' | 'powershell' = $state('script') // These pointers are used to easily access previewArgs of parent module, and previous module diff --git a/frontend/src/lib/components/flows/pickers/WorkspaceScriptPicker.svelte b/frontend/src/lib/components/flows/pickers/WorkspaceScriptPicker.svelte index 8659fd5404..575744cb04 100644 --- a/frontend/src/lib/components/flows/pickers/WorkspaceScriptPicker.svelte +++ b/frontend/src/lib/components/flows/pickers/WorkspaceScriptPicker.svelte @@ -26,7 +26,7 @@ const flowEditorContext = getContext('FlowEditorContext') let opWs = $derived(flowEditorContext?.opWorkspace?.() ?? $workspaceStore) interface Props { - kind?: 'script' | 'trigger' | 'approval' | 'failure' + kind?: 'script' | 'trigger' | 'approval' | 'failure' | 'preprocessor' isTemplate?: boolean | undefined displayLock?: boolean filter?: string @@ -122,38 +122,48 @@ />
{/if} - {#if filter.length > 0 && filteredItems.length == 0} - - {/if} -
    - {#each filteredItems as { path, hash, summary, description, marked }} -
  • -

- {#if lockHash}{truncateHash(hash ?? '')}{/if} - - - {/each} - + {#if lockHash}{truncateHash(hash ?? '')}{/if} + + + {/each} + + {/if} {:else}
diff --git a/frontend/src/lib/components/flows/pickers/WorkspaceScriptPickerQuick.svelte b/frontend/src/lib/components/flows/pickers/WorkspaceScriptPickerQuick.svelte index 07df5bd5c5..9e3a20b4a7 100644 --- a/frontend/src/lib/components/flows/pickers/WorkspaceScriptPickerQuick.svelte +++ b/frontend/src/lib/components/flows/pickers/WorkspaceScriptPickerQuick.svelte @@ -165,6 +165,11 @@ {#if filteredItems.length == 0}
{kind == 'flow' ? 'No flows found.' : 'No scripts found.'} + {#if kind == 'preprocessor'} +
+ Only workspace scripts whose kind is set to Preprocessor are listed here. +
+ {/if}
{/if}
    diff --git a/frontend/src/lib/script_helpers.test.ts b/frontend/src/lib/script_helpers.test.ts index 1badf03e11..92370f5d2e 100644 --- a/frontend/src/lib/script_helpers.test.ts +++ b/frontend/src/lib/script_helpers.test.ts @@ -1,5 +1,5 @@ import { describe, it, expect } from 'vitest' -import { bashRunsInCustomImage } from './script_helpers' +import { bashRunsInCustomImage, getResetCode } from './script_helpers' // bashRunsInCustomImage decides whether the +Variable/+Resource pickers insert a // curl/wget snippet (custom image, no wmill CLI) or the wmill CLI snippet. It must @@ -41,3 +41,26 @@ describe('bashRunsInCustomImage', () => { expect(bashRunsInCustomImage('# shellcheck shell=bash\necho hi')).toBe(false) }) }) + +// Preprocessors must bypass getResetCode's per-language `main` templates: a preprocessor step +// resets through the same button as an action script, and a `main` body cannot run under the +// preprocessor entrypoint. +describe('getResetCode for preprocessors', () => { + // The concrete `language` values a preprocessor step can carry. PREPROCESSOR_SUPPORTED_LANGUAGES + // also holds the 'typescript'/'python' aliases, which no script is ever stored with. + const langs = ['deno', 'bun', 'python3', 'php'] as const + + it('keeps the preprocessor entrypoint in every language that can have one', () => { + for (const lang of langs) { + const code = getResetCode(lang, 'preprocessor', undefined) + expect(code, lang).toContain('preprocessor') + expect(code, lang).not.toContain('function main') + expect(code, lang).not.toContain('def main') + } + }) + + it('still resets action scripts to a main stub', () => { + expect(getResetCode('python3', 'script', undefined)).toContain('def main') + expect(getResetCode('bun', 'script', undefined)).toContain('function main') + }) +}) diff --git a/frontend/src/lib/script_helpers.ts b/frontend/src/lib/script_helpers.ts index 178f4474e7..76ecfa59d8 100644 --- a/frontend/src/lib/script_helpers.ts +++ b/frontend/src/lib/script_helpers.ts @@ -1703,6 +1703,11 @@ export function getResetCode( | 'ci_test_python' | undefined ) { + // Every *_INIT_CODE_CLEAR below is a `main` stub, which cannot run under the preprocessor + // entrypoint. Preprocessors must go through initialCode to keep theirs. + if (kind === 'preprocessor') { + return initialCode(language, kind, subkind) + } if (language === 'deno') { return DENO_INIT_CODE_CLEAR } else if (language === 'python3') { From d85050f505b3ddc3f3c82f43dd3a9e4c32a1ee34 Mon Sep 17 00:00:00 2001 From: Ruben Fiszel Date: Fri, 21 Aug 2026 01:16:59 +0200 Subject: [PATCH 13/48] feat: upgrade bun to 1.4.0 and demote deno in the language picker (#10784) * chore: upgrade bun to 1.4.0 in dockerfiles and CI pins * chore: move deno last in the language picker and relabel it Deno * chore: move deno last in the pipeline language picker too * chore: pin debugger image to bun 1.4.0 and trim the deno picker comment * chore: state the deno picker constraint without referencing the old order * fix: stamp bun lockfiles back to v1 while the fleet predates bun 1.4 * fix: ask bun for a v1 lockfile instead of rewriting one, and refuse an escalated lock * chore: warn instead of silently storing a lockfile with no readable version --- .github/DockerfileBackendTests | 2 +- .github/workflows/ai-agent-tests.yml | 2 +- .github/workflows/ai-evals-test.yml | 2 +- .github/workflows/backend-test-windows.yml | 2 +- .github/workflows/backend-test.yml | 2 +- .github/workflows/git-sync-test.yml | 2 +- Dockerfile | 2 +- backend/windmill-common/src/min_version.rs | 7 ++ backend/windmill-worker/src/bun_executor.rs | 82 +++++++++++++++++++ debugger/Dockerfile | 2 +- docker/DockerfileSlim | 2 +- docker/DockerfileSlimEe | 2 +- .../assets/AssetGraph/pipelineLanguages.ts | 6 +- frontend/src/lib/scripts.ts | 6 +- 14 files changed, 106 insertions(+), 15 deletions(-) diff --git a/.github/DockerfileBackendTests b/.github/DockerfileBackendTests index 9b29f72a64..87a65214ab 100644 --- a/.github/DockerfileBackendTests +++ b/.github/DockerfileBackendTests @@ -42,7 +42,7 @@ RUN wget https://www.python.org/ftp/python/${PYTHON_VERSION}/Python-${PYTHON_VER RUN /usr/local/bin/python3 -m pip install pip-tools # Bun -COPY --from=oven/bun:1.3.10 /usr/local/bin/bun /usr/bin/bun +COPY --from=oven/bun:1.4.0 /usr/local/bin/bun /usr/bin/bun # Install windmill CLI RUN bun install -g windmill-cli \ diff --git a/.github/workflows/ai-agent-tests.yml b/.github/workflows/ai-agent-tests.yml index d3e695267b..cd07602109 100644 --- a/.github/workflows/ai-agent-tests.yml +++ b/.github/workflows/ai-agent-tests.yml @@ -61,7 +61,7 @@ jobs: - uses: oven-sh/setup-bun@v2 with: - bun-version: 1.3.10 + bun-version: 1.4.0 - uses: actions/setup-node@v4 with: diff --git a/.github/workflows/ai-evals-test.yml b/.github/workflows/ai-evals-test.yml index 71e79395f3..87eefab0f3 100644 --- a/.github/workflows/ai-evals-test.yml +++ b/.github/workflows/ai-evals-test.yml @@ -75,7 +75,7 @@ jobs: - uses: oven-sh/setup-bun@v2 with: - bun-version: 1.3.10 + bun-version: 1.4.0 - uses: actions/setup-node@v4 with: diff --git a/.github/workflows/backend-test-windows.yml b/.github/workflows/backend-test-windows.yml index 5042ed9bfe..c43e4ece79 100644 --- a/.github/workflows/backend-test-windows.yml +++ b/.github/workflows/backend-test-windows.yml @@ -73,7 +73,7 @@ jobs: - uses: oven-sh/setup-bun@v2 with: - bun-version: 1.3.10 + bun-version: 1.4.0 - uses: actions/setup-node@v4 with: diff --git a/.github/workflows/backend-test.yml b/.github/workflows/backend-test.yml index 49bede8a4d..5b6a461668 100644 --- a/.github/workflows/backend-test.yml +++ b/.github/workflows/backend-test.yml @@ -58,7 +58,7 @@ jobs: go-version: 1.21.5 - uses: oven-sh/setup-bun@v2 with: - bun-version: 1.3.10 + bun-version: 1.4.0 - uses: actions/setup-node@v4 with: node-version: "20" diff --git a/.github/workflows/git-sync-test.yml b/.github/workflows/git-sync-test.yml index 4b99c62eae..ee732569dd 100644 --- a/.github/workflows/git-sync-test.yml +++ b/.github/workflows/git-sync-test.yml @@ -133,7 +133,7 @@ jobs: - uses: oven-sh/setup-bun@v2 with: - bun-version: 1.3.10 + bun-version: 1.4.0 - uses: denoland/setup-deno@v2 with: diff --git a/Dockerfile b/Dockerfile index 6d25fc2a13..8ecee96725 100644 --- a/Dockerfile +++ b/Dockerfile @@ -287,7 +287,7 @@ COPY --from=windmill_duckdb_ffi_internal_builder /windmill-duckdb-ffi-internal/t COPY --from=denoland/deno:2.2.1 --chmod=755 /usr/bin/deno /usr/bin/deno -COPY --from=oven/bun:1.3.10 /usr/local/bin/bun /usr/bin/bun +COPY --from=oven/bun:1.4.0 /usr/local/bin/bun /usr/bin/bun # Install windmill CLI RUN bun install -g windmill-cli \ diff --git a/backend/windmill-common/src/min_version.rs b/backend/windmill-common/src/min_version.rs index 90a120d7ae..2ae666c14e 100644 --- a/backend/windmill-common/src/min_version.rs +++ b/backend/windmill-common/src/min_version.rs @@ -12,6 +12,13 @@ pub const MIN_VERSION_SUPPORTS_ON_BEHALF_OF_PRINCIPAL: VC = vc(1, 776, 0, "On-be // already-deployed hash (duplicate git-sync commit, duplicate fork tally, a re-triggered // relative-import cascade). Must name the release this ships in. pub const MIN_VERSION_SUPPORTS_BINARY_PREBUILD: VC = vc(1, 789, 0, "Auto-build binary on deploy"); +// The release that ships bun 1.4, which writes `bun.lock` v2 (v3 with overrides or catalogs). +// Bun 1.3 reports `error: Unknown lockfile version`, then installs anyway from `package.json` +// alone — and the dependency job writes every dependency as `"latest"`, so the lockfile is the +// only pin and an older worker resolves whatever is newest instead. Below this version the +// dependency job asks bun for a v1 lockfile, and refuses to store one bun raised anyway. +// Must name the release this ships in. +pub const MIN_VERSION_SUPPORTS_BUN_LOCKFILE_V2: VC = vc(1, 794, 0, "Bun v2 lockfiles"); pub const MIN_VERSION_SUPPORTS_NODE_DEBOUNCING: VC = vc(1, 658, 0, "Flow node debouncing"); pub const MIN_VERSION_SUPPORTS_TOKEN_HASH: VC = vc(1, 659, 0, "Token hash storage"); pub const MIN_VERSION_SUPPORTS_SYNC_JOBS_DEBOUNCING: VC = vc(1, 602, 0, "Sync jobs debouncing"); diff --git a/backend/windmill-worker/src/bun_executor.rs b/backend/windmill-worker/src/bun_executor.rs index e5134e8a70..1fb566e57f 100644 --- a/backend/windmill-worker/src/bun_executor.rs +++ b/backend/windmill-worker/src/bun_executor.rs @@ -35,6 +35,7 @@ use windmill_common::{ cache, client::AuthedClient, jobs::JobKind, + min_version::MIN_VERSION_SUPPORTS_BUN_LOCKFILE_V2, scripts::{id_to_codebase_info, CodebaseInfo, ScriptHash, ScriptLang}, utils::WarnAfterExt, workspace_dependencies::WorkspaceDependenciesPrefetched, @@ -377,6 +378,44 @@ pub(crate) fn split_lockfile(lockfile: &str) -> (&str, Option<&str>, bool, bool) } } +/// An empty `lockfileVersion: 1` lockfile, planted before `bun install` so bun writes v1. +/// +/// bun keeps whichever version the lockfile it found already had, and only raises it when the +/// dependencies genuinely need newer syntax. Seeding therefore gets a real v1 lockfile — written +/// by bun, not rewritten by us — whenever v1 can express the resolution, and lets bun escalate +/// when it cannot. [`bun_lockfile_version`] catches the escalation afterwards. +const EMPTY_V1_BUN_LOCK: &str = r#"{ + "lockfileVersion": 1, + "configVersion": 1, + "workspaces": { + "": {}, + }, + "packages": {} +}"#; + +async fn seed_v1_bun_lockfile(job_dir: &str) -> Result<()> { + let path = format!("{job_dir}/bun.lock"); + if tokio::fs::metadata(&path).await.is_ok() { + return Ok(()); + } + write_file(job_dir, "bun.lock", EMPTY_V1_BUN_LOCK)?; + Ok(()) +} + +/// The `lockfileVersion` a bun text lockfile declares, if it declares one. +fn bun_lockfile_version(lockfile: &str) -> Option { + let i = lockfile.find("\"lockfileVersion\"")?; + let rest = &lockfile[i + "\"lockfileVersion\"".len()..]; + let digits = rest + .trim_start() + .strip_prefix(':')? + .trim_start() + .chars() + .take_while(char::is_ascii_digit) + .collect::(); + digits.parse().ok() +} + pub async fn gen_bun_lockfile( mem_peak: &mut i32, canceled_by: &mut Option, @@ -496,6 +535,9 @@ pub async fn gen_bun_lockfile( } if !empty_deps { + if !npm_mode && !MIN_VERSION_SUPPORTS_BUN_LOCKFILE_V2.met_conservatively() { + seed_v1_bun_lockfile(job_dir).await?; + } install_bun_lockfile( mem_peak, canceled_by, @@ -539,6 +581,30 @@ pub async fn gen_bun_lockfile( let mut file = File::open(&file).await?; let mut buf = String::default(); file.read_to_string(&mut buf).await?; + if !MIN_VERSION_SUPPORTS_BUN_LOCKFILE_V2.met_conservatively() { + // Seeding asked bun for v1; a higher version back means these + // dependencies cannot be expressed in one. Storing it anyway would not + // fail on an older worker — it would install from package.json alone and + // silently resolve different versions. + match bun_lockfile_version(&buf) { + Some(1) => {} + Some(v) => { + return Err(error::Error::ExecutionErr(format!( + "bun produced a v{v} lockfile, which workers older than {} \ + cannot read. Finish upgrading every worker before deploying \ + dependencies that need it (overrides, catalogs).", + MIN_VERSION_SUPPORTS_BUN_LOCKFILE_V2.version() + ))); + } + // The guard cannot classify this one, so it must not pass silently: + // a bun that stops writing the header would reopen the drift hole + // with nothing in the logs. + None => tracing::warn!( + "bun wrote a lockfile with no readable lockfileVersion; storing \ + it unchecked for job {job_id}" + ), + } + } content.push_str(&buf); } else { content.push_str(&EMPTY_FILE); @@ -4125,6 +4191,22 @@ pub async fn start_worker( mod tests { use super::*; + #[test] + fn test_bun_lockfile_version() { + assert_eq!(bun_lockfile_version(EMPTY_V1_BUN_LOCK), Some(1)); + assert_eq!( + bun_lockfile_version("{\n \"lockfileVersion\": 2,\n \"packages\": {}\n}"), + Some(2) + ); + // bun raises the version on its own for overrides/catalogs, so any version has to parse + assert_eq!(bun_lockfile_version("{\"lockfileVersion\":3}"), Some(3)); + assert_eq!( + bun_lockfile_version("{ \"lockfileVersion\" : 42 }"), + Some(42) + ); + assert_eq!(bun_lockfile_version("{\"packages\":{}}"), None); + } + #[test] fn test_split_lockfile_text_unix() { let lockfile = r#"{"dependencies":{"lodash":"^4.17.21"}} diff --git a/debugger/Dockerfile b/debugger/Dockerfile index a80643e823..6b423920a0 100644 --- a/debugger/Dockerfile +++ b/debugger/Dockerfile @@ -19,7 +19,7 @@ FROM ghcr.io/windmill-labs/windmill-ee:main AS windmill-source # Stage 2: Build the debug service -FROM oven/bun:1 AS runtime +FROM oven/bun:1.4.0 AS runtime # Install Python and required system dependencies RUN apt-get update && apt-get install -y --no-install-recommends \ diff --git a/docker/DockerfileSlim b/docker/DockerfileSlim index 915815f935..64fd1b735e 100644 --- a/docker/DockerfileSlim +++ b/docker/DockerfileSlim @@ -70,7 +70,7 @@ RUN mkdir -p /tmp/windmill/cache && \ rm -rf /tmp/build_cache && \ mkdir -p -m 777 /tmp/windmill/cache/uv -COPY --from=oven/bun:1.3.10 /usr/local/bin/bun /usr/bin/bun +COPY --from=oven/bun:1.4.0 /usr/local/bin/bun /usr/bin/bun # Install windmill CLI (node symlink needed for bun install) RUN ln -s /usr/bin/bun /usr/bin/node \ diff --git a/docker/DockerfileSlimEe b/docker/DockerfileSlimEe index d4ac4bd21c..de0d61a780 100644 --- a/docker/DockerfileSlimEe +++ b/docker/DockerfileSlimEe @@ -70,7 +70,7 @@ RUN mkdir -p /tmp/windmill/cache && \ rm -rf /tmp/build_cache && \ mkdir -p -m 777 /tmp/windmill/cache/uv -COPY --from=oven/bun:1.3.10 /usr/local/bin/bun /usr/bin/bun +COPY --from=oven/bun:1.4.0 /usr/local/bin/bun /usr/bin/bun # Install windmill CLI (node symlink needed for bun install) RUN ln -s /usr/bin/bun /usr/bin/node \ diff --git a/frontend/src/lib/components/assets/AssetGraph/pipelineLanguages.ts b/frontend/src/lib/components/assets/AssetGraph/pipelineLanguages.ts index 809ccb8ce2..26d3e24d84 100644 --- a/frontend/src/lib/components/assets/AssetGraph/pipelineLanguages.ts +++ b/frontend/src/lib/components/assets/AssetGraph/pipelineLanguages.ts @@ -4,7 +4,7 @@ import type { ScriptLang } from '$lib/gen' // actually reach for first: duckdb for in-place SQL on parquet/s3 (the // default), bun for ergonomic data wrangling, python for ML/pandas, then // the sql dialects for warehouse-resident transforms. Everything else -// (deno/bash/go) sits below — still creatable, just not the default +// (bash/go/deno) sits below — still creatable, just not the default // suggestion. export const PIPELINE_LANGUAGES: Array<{ label: string; lang: ScriptLang }> = [ { label: 'DuckDB', lang: 'duckdb' }, @@ -15,7 +15,7 @@ export const PIPELINE_LANGUAGES: Array<{ label: string; lang: ScriptLang }> = [ { label: 'Snowflake', lang: 'snowflake' }, { label: 'MySQL', lang: 'mysql' }, { label: 'MS SQL', lang: 'mssql' }, - { label: 'TypeScript (Deno)', lang: 'deno' }, { label: 'Bash', lang: 'bash' }, - { label: 'Go', lang: 'go' } + { label: 'Go', lang: 'go' }, + { label: 'Deno', lang: 'deno' } ] diff --git a/frontend/src/lib/scripts.ts b/frontend/src/lib/scripts.ts index a86ebea43f..2d905d690a 100644 --- a/frontend/src/lib/scripts.ts +++ b/frontend/src/lib/scripts.ts @@ -152,7 +152,6 @@ export function flowPathToHref(path: string, hubBaseUrl: string = get(hubBaseUrl const scriptLanguagesArray: [SupportedLanguage | 'docker' | 'bunnative', string][] = [ ['bun', 'TypeScript (Bun)'], ['python3', 'Python'], - ['deno', 'TypeScript (Deno)'], ['bash', 'Bash'], ['go', 'Go'], ['nativets', 'REST'], @@ -175,7 +174,10 @@ const scriptLanguagesArray: [SupportedLanguage | 'docker' | 'bunnative', string] ['duckdb', 'DuckDB'], ['ruby', 'Ruby'], ['rlang', 'R'], - ['dbt', 'dbt'] + ['dbt', 'dbt'], + // This array's order is the picker order. Deno is de-emphasized ahead of + // deprecation, so it stays last. + ['deno', 'Deno'] // for related places search: ADD_NEW_LANG ] /** From 75d0c29586a617f2cbfbf66720bdd0e47d6f92b4 Mon Sep 17 00:00:00 2001 From: Ruben Fiszel Date: Fri, 21 Aug 2026 02:15:40 +0200 Subject: [PATCH 14/48] fix: apply the first script kind selection in the script editor (#10789) * fix: apply the first script kind selection in the script editor Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01QqKXVsBXynMFtMZ26uvLw7 * docs: record why the kind setter's early return is load-bearing Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01QqKXVsBXynMFtMZ26uvLw7 --------- Co-authored-by: Claude Opus 5 --- .../src/lib/components/ScriptBuilder.svelte | 21 +++++++++++++------ 1 file changed, 15 insertions(+), 6 deletions(-) diff --git a/frontend/src/lib/components/ScriptBuilder.svelte b/frontend/src/lib/components/ScriptBuilder.svelte index 5c6cb3e5a8..4a1007d15a 100644 --- a/frontend/src/lib/components/ScriptBuilder.svelte +++ b/frontend/src/lib/components/ScriptBuilder.svelte @@ -1466,13 +1466,22 @@ corresponding action. {/snippet} + { - template = 'script' - script.kind = detail - initContent(script.language, detail, template) - }} + bind:selected={ + () => script.kind ?? 'script', + (kind) => { + // Load-bearing: any write to script.kind echoes back through the + // group, and initContent replaces the editor content outright. + if (kind === (script.kind ?? 'script')) return + template = 'script' + script.kind = kind as Script['kind'] + initContent(script.language, script.kind, template) + } + } > {#snippet children({ item })} {#each scriptKindOptions as { value, title, desc, documentationLink, Icon }} From c7e3537da8b3a3be01c99c6b8d234b840fe009dc Mon Sep 17 00:00:00 2001 From: Guilhem Date: Fri, 21 Aug 2026 10:15:19 +0200 Subject: [PATCH 15/48] give every brand icon the lucide safe area and centre its artwork (#10790) Co-authored-by: Claude Opus 5 (1M context) --- .../src/lib/components/icons/AblyIcon.svelte | 2 +- .../components/icons/AbstractApiIcon.svelte | 2 +- .../lib/components/icons/AcceloIcon.svelte | 2 +- .../icons/ActiveCampaignIcon.svelte | 2 +- .../components/icons/ActivitypubIcon.svelte | 2 +- .../components/icons/AcumbamailIcon.svelte | 2 +- .../lib/components/icons/AdhookIcon.svelte | 2 +- .../icons/AgentInstructionsIcon.svelte | 2 +- .../src/lib/components/icons/Ai21Icon.svelte | 2 +- .../lib/components/icons/AiAgentIcon.svelte | 2 +- .../lib/components/icons/AirtableIcon.svelte | 2 +- .../lib/components/icons/AlgoliaIcon.svelte | 2 +- .../src/lib/components/icons/AmqpIcon.svelte | 2 +- .../lib/components/icons/AnsibleIcon.svelte | 2 +- .../lib/components/icons/AnthropicIcon.svelte | 2 +- .../components/icons/ApiKeyAuthIcon.svelte | 2 +- .../src/lib/components/icons/ApifyIcon.svelte | 8 +++- .../lib/components/icons/ApolloIcon.svelte | 2 +- .../lib/components/icons/AppwriteIcon.svelte | 2 +- .../lib/components/icons/ArcGisIcon.svelte | 8 +++- .../src/lib/components/icons/AsanaIcon.svelte | 2 +- .../components/icons/AssemblyAiIcon.svelte | 2 +- .../components/icons/AssetDatabaseIcon.svelte | 2 +- .../components/icons/AssetDucklakeIcon.svelte | 2 +- .../lib/components/icons/AssetResIcon.svelte | 2 +- .../lib/components/icons/AssetS3Icon.svelte | 2 +- .../src/lib/components/icons/AttioIcon.svelte | 2 +- .../src/lib/components/icons/Auth0Icon.svelte | 2 +- .../lib/components/icons/AutheliaIcon.svelte | 7 +++- .../lib/components/icons/AuthentikIcon.svelte | 2 +- .../lib/components/icons/AwsEcrIcon.svelte | 2 +- .../src/lib/components/icons/AwsIcon.svelte | 2 +- .../src/lib/components/icons/AzureIcon.svelte | 2 +- .../lib/components/icons/BambooHrIcon.svelte | 2 +- .../components/icons/BaremetricsIcon.svelte | 8 +++- .../lib/components/icons/BarsStaggered.svelte | 2 +- .../lib/components/icons/BaserowIcon.svelte | 8 +++- .../components/icons/BasicHttpAuthIcon.svelte | 2 +- .../components/icons/BasisTheoryIcon.svelte | 2 +- .../lib/components/icons/BcryptIcon.svelte | 8 +++- .../lib/components/icons/BeamerIcon.svelte | 2 +- .../lib/components/icons/BigQueryIcon.svelte | 2 +- .../lib/components/icons/BitbucketIcon.svelte | 2 +- .../src/lib/components/icons/BitlyIcon.svelte | 2 +- .../lib/components/icons/BloggerIcon.svelte | 2 +- .../lib/components/icons/BlueskyIcon.svelte | 2 +- .../lib/components/icons/BotifyIcon.svelte | 2 +- .../src/lib/components/icons/BoxIcon.svelte | 2 +- .../components/icons/BrandLetterIcon.svelte | 2 +- .../src/lib/components/icons/BrevoIcon.svelte | 8 +++- .../src/lib/components/icons/BrexIcon.svelte | 2 +- .../components/icons/BrowserlessIcon.svelte | 2 +- .../lib/components/icons/BubbleIcon.svelte | 2 +- .../lib/components/icons/BuildkiteIcon.svelte | 8 +++- .../src/lib/components/icons/BunIcon.svelte | 2 +- .../components/icons/ButtondownIcon.svelte | 2 +- .../lib/components/icons/CACertificate.svelte | 8 +++- .../lib/components/icons/CSharpIcon.svelte | 2 +- .../lib/components/icons/CalcomIcon.svelte | 2 +- .../lib/components/icons/CalendlyIcon.svelte | 2 +- .../lib/components/icons/CampaynIcon.svelte | 2 +- .../lib/components/icons/CertopusIcon.svelte | 2 +- .../lib/components/icons/ChromaIcon.svelte | 2 +- .../lib/components/icons/CircleCiIcon.svelte | 2 +- .../src/lib/components/icons/CiscoIcon.svelte | 2 +- .../lib/components/icons/ClaudeIcon.svelte | 2 +- .../lib/components/icons/ClearbitIcon.svelte | 2 +- .../src/lib/components/icons/ClerkIcon.svelte | 8 +++- .../components/icons/ClickhouseIcon.svelte | 2 +- .../lib/components/icons/ClickupIcon.svelte | 2 +- .../src/lib/components/icons/CloseIcon.svelte | 8 +++- .../components/icons/CloudflareIcon.svelte | 2 +- .../components/icons/CloudinaryIcon.svelte | 2 +- .../components/icons/CockroachDbIcon.svelte | 2 +- .../src/lib/components/icons/CodaIcon.svelte | 8 +++- .../src/lib/components/icons/CodatIcon.svelte | 2 +- .../lib/components/icons/CohereIcon.svelte | 2 +- .../components/icons/CoinMarketCapIcon.svelte | 2 +- .../lib/components/icons/CoinbaseIcon.svelte | 2 +- .../lib/components/icons/ComapeoIcon.svelte | 2 +- .../components/icons/ConfluenceIcon.svelte | 2 +- .../components/icons/ContentfulIcon.svelte | 2 +- .../components/icons/ContiguityIcon.svelte | 2 +- .../components/icons/ConvertKitIcon.svelte | 2 +- .../src/lib/components/icons/CoupaIcon.svelte | 2 +- .../src/lib/components/icons/CssIcon.svelte | 2 +- .../components/icons/CurrencyApiIcon.svelte | 8 +++- .../lib/components/icons/CustomAiIcon.svelte | 2 +- .../components/icons/DatabricksIcon.svelte | 2 +- .../lib/components/icons/DatadogIcon.svelte | 4 +- .../lib/components/icons/DatoCmsIcon.svelte | 8 +++- .../src/lib/components/icons/DbIcon.svelte | 2 +- .../src/lib/components/icons/DbtIcon.svelte | 2 +- .../src/lib/components/icons/DeelIcon.svelte | 2 +- .../lib/components/icons/DeepInfraIcon.svelte | 2 +- .../src/lib/components/icons/DeepLIcon.svelte | 4 +- .../lib/components/icons/DeepSeekIcon.svelte | 2 +- .../src/lib/components/icons/DenoIcon.svelte | 4 +- .../components/icons/DigitalOceanIcon.svelte | 8 +++- .../lib/components/icons/DiscordIcon.svelte | 2 +- .../lib/components/icons/DiscourseIcon.svelte | 2 +- .../lib/components/icons/DocSpringIcon.svelte | 2 +- .../lib/components/icons/DockerIcon.svelte | 2 +- .../lib/components/icons/DocusignIcon.svelte | 2 +- .../lib/components/icons/DropboxIcon.svelte | 8 +++- .../lib/components/icons/DuckDbIcon.svelte | 2 +- .../lib/components/icons/DucklakeIcon.svelte | 2 +- .../src/lib/components/icons/DustIcon.svelte | 2 +- .../lib/components/icons/DynatraceIcon.svelte | 2 +- .../lib/components/icons/EdgeDbIcon.svelte | 8 +++- .../src/lib/components/icons/EnodeIcon.svelte | 2 +- .../components/icons/EventbriteIcon.svelte | 7 +++- .../src/lib/components/icons/ExaIcon.svelte | 2 +- .../lib/components/icons/FaunadbIcon.svelte | 2 +- .../src/lib/components/icons/FigmaIcon.svelte | 8 +++- .../lib/components/icons/FirebaseIcon.svelte | 2 +- .../src/lib/components/icons/FlyIcon.svelte | 2 +- .../lib/components/icons/FormInputIcon.svelte | 2 +- .../lib/components/icons/FormstackIcon.svelte | 8 +++- .../lib/components/icons/FoxentryIcon.svelte | 2 +- .../lib/components/icons/FreshdeskIcon.svelte | 2 +- .../lib/components/icons/FrontAppIcon.svelte | 8 +++- .../lib/components/icons/FunkwhaleIcon.svelte | 2 +- .../src/lib/components/icons/FunnelCog.svelte | 2 +- .../lib/components/icons/GSheetsIcon.svelte | 8 +++- .../src/lib/components/icons/GcalIcon.svelte | 8 +++- .../src/lib/components/icons/GdocsIcon.svelte | 2 +- .../lib/components/icons/GdriveIcon.svelte | 8 +++- .../lib/components/icons/GhostCmsIcon.svelte | 2 +- .../src/lib/components/icons/GiphyIcon.svelte | 2 +- .../lib/components/icons/GitBookIcon.svelte | 2 +- .../src/lib/components/icons/GitIcon.svelte | 2 +- .../lib/components/icons/GithubIcon.svelte | 2 +- .../lib/components/icons/GitlabIcon.svelte | 2 +- .../src/lib/components/icons/GmailIcon.svelte | 8 +++- .../lib/components/icons/GoogleAiIcon.svelte | 8 +++- .../icons/GoogleCalendarIcon.svelte | 8 +++- .../components/icons/GoogleCloudIcon.svelte | 2 +- .../components/icons/GoogleDriveIcon.svelte | 8 +++- .../components/icons/GoogleFormsIcon.svelte | 2 +- .../lib/components/icons/GoogleIcon.svelte | 2 +- .../lib/components/icons/GorgiasIcon.svelte | 2 +- .../lib/components/icons/GpgKeyIcon.svelte | 2 +- .../lib/components/icons/GraphqlIcon.svelte | 2 +- .../src/lib/components/icons/GreipIcon.svelte | 2 +- .../src/lib/components/icons/GristIcon.svelte | 2 +- .../src/lib/components/icons/GroqIcon.svelte | 2 +- .../components/icons/HackernewsIcon.svelte | 8 +++- .../lib/components/icons/HoldedIcon.svelte | 8 +++- .../components/icons/HoneybadgerIcon.svelte | 8 +++- .../src/lib/components/icons/HtmlIcon.svelte | 2 +- .../src/lib/components/icons/HttpIcon.svelte | 2 +- .../lib/components/icons/HubspotIcon.svelte | 2 +- .../src/lib/components/icons/IfsIcon.svelte | 8 +++- .../src/lib/components/icons/IftttIcon.svelte | 2 +- .../lib/components/icons/InkeepIcon.svelte | 2 +- .../lib/components/icons/IntercomIcon.svelte | 2 +- .../lib/components/icons/IpinfoIcon.svelte | 2 +- .../src/lib/components/icons/JavaIcon.svelte | 42 +++++++++++-------- .../components/icons/JavaScriptIcon.svelte | 2 +- .../src/lib/components/icons/JiraIcon.svelte | 2 +- .../lib/components/icons/JoomlaIcon.svelte | 2 +- .../lib/components/icons/JotformIcon.svelte | 2 +- .../src/lib/components/icons/JsonIcon.svelte | 2 +- .../components/icons/JsonSchemaIcon.svelte | 2 +- .../lib/components/icons/JumpCloudIcon.svelte | 2 +- .../src/lib/components/icons/KafkaIcon.svelte | 2 +- .../lib/components/icons/KanidmIcon.svelte | 8 +++- .../lib/components/icons/KeycloakIcon.svelte | 8 +++- .../lib/components/icons/KlaviyoIcon.svelte | 2 +- .../components/icons/KoboToolboxIcon.svelte | 2 +- .../lib/components/icons/KustomerIcon.svelte | 2 +- .../lib/components/icons/LangfuseIcon.svelte | 8 +++- .../src/lib/components/icons/LdapIcon.svelte | 2 +- .../src/lib/components/icons/LessIcon.svelte | 2 +- .../src/lib/components/icons/LineIcon.svelte | 8 +++- .../lib/components/icons/LinearIcon.svelte | 2 +- .../lib/components/icons/LinkdingIcon.svelte | 2 +- .../lib/components/icons/LinkedinIcon.svelte | 8 +++- .../lib/components/icons/LinodeIcon.svelte | 2 +- .../lib/components/icons/LumaAiIcon.svelte | 2 +- .../components/icons/MSSqlServerIcon.svelte | 8 +++- .../lib/components/icons/MSTeamsIcon.svelte | 2 +- .../lib/components/icons/MagentoIcon.svelte | 2 +- frontend/src/lib/components/icons/Mail.svelte | 2 +- .../lib/components/icons/MailchimpIcon.svelte | 8 +++- .../components/icons/MailerLiteIcon.svelte | 2 +- .../lib/components/icons/MailgunIcon.svelte | 2 +- .../lib/components/icons/MandrillIcon.svelte | 2 +- .../lib/components/icons/MapboxIcon.svelte | 2 +- .../lib/components/icons/MarkdownIcon.svelte | 2 +- .../lib/components/icons/MastodonIcon.svelte | 8 +++- .../lib/components/icons/MatrixIcon.svelte | 2 +- .../lib/components/icons/MatteroomIcon.svelte | 2 +- .../lib/components/icons/MauticIcon.svelte | 2 +- .../src/lib/components/icons/McpIcon.svelte | 2 +- .../lib/components/icons/MediumIcon.svelte | 2 +- .../components/icons/MeteosourceIcon.svelte | 2 +- .../src/lib/components/icons/MezmoIcon.svelte | 2 +- .../lib/components/icons/MicrosoftIcon.svelte | 8 +++- .../src/lib/components/icons/MiroIcon.svelte | 2 +- .../lib/components/icons/MistralIcon.svelte | 8 +++- .../lib/components/icons/MixpanelIcon.svelte | 2 +- .../lib/components/icons/MollieIcon.svelte | 2 +- .../lib/components/icons/MondayIcon.svelte | 2 +- .../lib/components/icons/MongodbIcon.svelte | 2 +- .../lib/components/icons/MotimateIcon.svelte | 2 +- .../src/lib/components/icons/MqttIcon.svelte | 2 +- .../src/lib/components/icons/NatsIcon.svelte | 2 +- .../lib/components/icons/NeonDbIcon.svelte | 2 +- .../lib/components/icons/NetBoxIcon.svelte | 2 +- .../lib/components/icons/NetlifyIcon.svelte | 2 +- .../lib/components/icons/NetsuiteIcon.svelte | 8 +++- .../lib/components/icons/NewsApiIcon.svelte | 2 +- .../lib/components/icons/NextcloudIcon.svelte | 2 +- .../lib/components/icons/NocoDbIcon.svelte | 8 +++- .../lib/components/icons/NotionIcon.svelte | 2 +- .../src/lib/components/icons/NuIcon.svelte | 2 +- .../src/lib/components/icons/OauthIcon.svelte | 2 +- .../src/lib/components/icons/OdkIcon.svelte | 2 +- .../src/lib/components/icons/OktaIcon.svelte | 2 +- .../lib/components/icons/OneSignalIcon.svelte | 2 +- .../components/icons/OpenRouterIcon.svelte | 2 +- .../components/icons/OpenWeatherIcon.svelte | 8 +++- .../lib/components/icons/OpenaiIcon.svelte | 2 +- .../lib/components/icons/OracleDBIcon.svelte | 2 +- .../lib/components/icons/OutreachIcon.svelte | 2 +- .../src/lib/components/icons/PHPIcon.svelte | 2 +- .../lib/components/icons/PagerDutyIcon.svelte | 2 +- .../lib/components/icons/PaintbrushOff.svelte | 8 +++- .../lib/components/icons/PandaDocIcon.svelte | 2 +- .../lib/components/icons/PaychexIcon.svelte | 2 +- .../lib/components/icons/PaylocityIcon.svelte | 2 +- .../lib/components/icons/PaypalIcon.svelte | 2 +- .../lib/components/icons/PersonaIcon.svelte | 2 +- .../lib/components/icons/PersonioIcon.svelte | 2 +- .../lib/components/icons/PhraseIcon.svelte | 2 +- .../lib/components/icons/PineconeIcon.svelte | 2 +- .../lib/components/icons/PinterestIcon.svelte | 8 +++- .../lib/components/icons/PipedriveIcon.svelte | 8 +++- .../components/icons/PlanetScaleIcon.svelte | 2 +- .../lib/components/icons/PocketIdIcon.svelte | 2 +- .../lib/components/icons/PostgresIcon.svelte | 8 +++- .../lib/components/icons/PostmarkIcon.svelte | 8 +++- .../components/icons/PowershellIcon.svelte | 2 +- .../lib/components/icons/PusherIcon.svelte | 2 +- .../lib/components/icons/PushoverIcon.svelte | 7 +++- .../lib/components/icons/QoveryIcon.svelte | 8 +++- .../components/icons/QuestionInputIcon.svelte | 2 +- .../components/icons/QuickbooksIcon.svelte | 2 +- .../src/lib/components/icons/RIcon.svelte | 2 +- .../lib/components/icons/RaindropIcon.svelte | 2 +- .../src/lib/components/icons/ReactIcon.svelte | 2 +- .../lib/components/icons/ReadmeIcon.svelte | 2 +- .../lib/components/icons/ReadwiseIcon.svelte | 2 +- .../lib/components/icons/RecordIcon.svelte | 2 +- .../lib/components/icons/RecraftIcon.svelte | 7 +++- .../lib/components/icons/RedditIcon.svelte | 8 +++- .../lib/components/icons/RenderIcon.svelte | 2 +- .../lib/components/icons/ReplicateIcon.svelte | 2 +- .../lib/components/icons/ResendIcon.svelte | 2 +- .../src/lib/components/icons/RestIcon.svelte | 2 +- .../components/icons/RingCentralIcon.svelte | 2 +- .../components/icons/RocketChatIcon.svelte | 8 +++- .../src/lib/components/icons/RssIcon.svelte | 2 +- .../src/lib/components/icons/RubyIcon.svelte | 22 +--------- .../lib/components/icons/RunPodIcon.svelte | 2 +- .../src/lib/components/icons/RustIcon.svelte | 2 +- .../src/lib/components/icons/S3Icon.svelte | 2 +- .../src/lib/components/icons/SageIcon.svelte | 2 +- .../components/icons/SalesflareIcon.svelte | 2 +- .../components/icons/SalesforceIcon.svelte | 2 +- .../src/lib/components/icons/SassIcon.svelte | 2 +- .../components/icons/SchedulePollIcon.svelte | 2 +- .../lib/components/icons/SegmentIcon.svelte | 2 +- .../lib/components/icons/SendflakeIcon.svelte | 2 +- .../lib/components/icons/SendgridIcon.svelte | 2 +- .../components/icons/SensorTowerIcon.svelte | 2 +- .../lib/components/icons/SentryIcon.svelte | 2 +- .../components/icons/ServiceNowIcon.svelte | 2 +- .../lib/components/icons/ShopifyIcon.svelte | 2 +- .../lib/components/icons/ShortcutIcon.svelte | 2 +- .../components/icons/ShutterstockIcon.svelte | 2 +- .../lib/components/icons/SigNozIcon.svelte | 8 +++- .../components/icons/SignatureAuthIcon.svelte | 2 +- .../src/lib/components/icons/Slack.svelte | 8 +++- .../components/icons/SmartsheetIcon.svelte | 2 +- .../lib/components/icons/SnowflakeIcon.svelte | 2 +- .../components/icons/SparklesOffIcon.svelte | 2 +- .../lib/components/icons/SpeechifyIcon.svelte | 2 +- .../lib/components/icons/SplitwiseIcon.svelte | 8 +++- .../lib/components/icons/SpotifyIcon.svelte | 2 +- .../lib/components/icons/SquareIcon.svelte | 2 +- .../lib/components/icons/StraleIcon.svelte | 2 +- .../lib/components/icons/StravaIcon.svelte | 8 +++- .../lib/components/icons/StripeIcon.svelte | 2 +- .../lib/components/icons/SupabaseIcon.svelte | 2 +- .../lib/components/icons/SurrealdbIcon.svelte | 2 +- .../lib/components/icons/SvelteIcon.svelte | 2 +- .../src/lib/components/icons/TallyIcon.svelte | 2 +- .../lib/components/icons/TaskadeIcon.svelte | 2 +- .../lib/components/icons/TelegramIcon.svelte | 7 +++- .../lib/components/icons/TelnyxIcon.svelte | 2 +- .../src/lib/components/icons/TerraIcon.svelte | 2 +- .../components/icons/TheirStackIcon.svelte | 2 +- .../lib/components/icons/ThreadsIcon.svelte | 2 +- .../lib/components/icons/TodoistIcon.svelte | 8 +++- .../components/icons/TogetherAiIcon.svelte | 2 +- .../src/lib/components/icons/TogglIcon.svelte | 10 ++++- .../components/icons/TomorrowIoIcon.svelte | 2 +- .../lib/components/icons/TrelloIcon.svelte | 2 +- .../components/icons/TripadvisorIcon.svelte | 8 +++- .../src/lib/components/icons/TursoIcon.svelte | 2 +- .../lib/components/icons/TwilioIcon.svelte | 2 +- .../lib/components/icons/TwitchIcon.svelte | 8 +++- .../lib/components/icons/TwitterIcon.svelte | 2 +- .../lib/components/icons/TypeformIcon.svelte | 2 +- .../lib/components/icons/UltravoxIcon.svelte | 8 +++- .../lib/components/icons/VectaraIcon.svelte | 2 +- .../lib/components/icons/VercelIcon.svelte | 2 +- .../src/lib/components/icons/VismaIcon.svelte | 2 +- .../src/lib/components/icons/VueIcon.svelte | 2 +- .../lib/components/icons/WebdavIcon.svelte | 8 +++- .../lib/components/icons/WebflowIcon.svelte | 8 +++- .../icons/WhatsappBusinessIcon.svelte | 8 +++- .../components/icons/WindmillAiIcon.svelte | 2 +- .../lib/components/icons/WindmillIcon.svelte | 2 +- .../lib/components/icons/WindmillIcon2.svelte | 2 +- .../src/lib/components/icons/WizIcon.svelte | 2 +- .../components/icons/WooCommerceIcon.svelte | 2 +- .../lib/components/icons/WordpressIcon.svelte | 2 +- .../src/lib/components/icons/XataIcon.svelte | 8 +++- .../src/lib/components/icons/XeroIcon.svelte | 2 +- .../src/lib/components/icons/YamlIcon.svelte | 2 +- .../src/lib/components/icons/YelpIcon.svelte | 8 +++- .../src/lib/components/icons/YnabIcon.svelte | 2 +- .../lib/components/icons/YoutubeIcon.svelte | 8 +++- .../lib/components/icons/ZammadIcon.svelte | 8 +++- .../lib/components/icons/ZendeskIcon.svelte | 2 +- .../lib/components/icons/ZeroTierIcon.svelte | 8 +++- .../lib/components/icons/ZitadelIcon.svelte | 2 +- .../lib/components/icons/ZixflowIcon.svelte | 2 +- .../src/lib/components/icons/ZohoIcon.svelte | 2 +- .../src/lib/components/icons/ZoomIcon.svelte | 2 +- .../src/lib/components/icons/ZuploIcon.svelte | 2 +- .../lib/components/icons/brands/Auth0.svelte | 2 +- .../components/icons/brands/Discord.svelte | 2 +- .../lib/components/icons/brands/Github.svelte | 2 +- .../lib/components/icons/brands/Gitlab.svelte | 2 +- .../lib/components/icons/brands/Google.svelte | 2 +- .../components/icons/brands/Microsoft.svelte | 2 +- .../lib/components/icons/brands/Okta.svelte | 2 +- .../components/icons/triggers/AmqpIcon.svelte | 2 +- .../components/icons/triggers/AwsIcon.svelte | 2 +- .../icons/triggers/AzureIcon.svelte | 2 +- .../icons/triggers/GithubIcon.svelte | 2 +- .../icons/triggers/GoogleCloudIcon.svelte | 2 +- .../icons/triggers/GoogleIcon.svelte | 2 +- .../icons/triggers/KafkaIcon.svelte | 2 +- .../components/icons/triggers/MqttIcon.svelte | 2 +- .../components/icons/triggers/NatsIcon.svelte | 2 +- .../icons/triggers/NextcloudIcon.svelte | 2 +- .../routes/kitchen_sink/icons/+page.svelte | 21 +++++++++- 363 files changed, 842 insertions(+), 404 deletions(-) diff --git a/frontend/src/lib/components/icons/AblyIcon.svelte b/frontend/src/lib/components/icons/AblyIcon.svelte index ce4c360152..63212ba3b4 100644 --- a/frontend/src/lib/components/icons/AblyIcon.svelte +++ b/frontend/src/lib/components/icons/AblyIcon.svelte @@ -11,7 +11,7 @@
    + +
    + {:else} + + {/if} +{/snippet} +
    @@ -144,6 +160,7 @@
    +
    @@ -169,14 +186,14 @@ class="flex-1 flex items-center justify-center py-5 bg-surface text-secondary min-h-[72px]" style={lightVars} > - + {@render mark(Icon)}
    {#if showDark}
    - + {@render mark(Icon)}
    {/if}
From 1a7682891d5800b70d83833a61c75d21fedf67e8 Mon Sep 17 00:00:00 2001 From: Guilhem Date: Fri, 21 Aug 2026 10:20:02 +0200 Subject: [PATCH 16/48] collapse conditionally hidden fields in schema forms (#10791) A field whose `showExpr` evaluates false skipped its `ArgInput` but still rendered the padded row that wraps it, so the enclosing ResizeTransitionWrapper measured 8px (16px with `largeGap`) of leftover padding per hidden field. Simulating a `oneOf` with a selector and one `showExpr` branch per variant stacked one such gap per unselected branch. Move the `!hidden[argName]` check onto the row itself so nothing is rendered for a hidden field and the wrapper collapses to 0px. Claude-Session: https://claude.ai/code/session_01Lx7KEPQC4SXjVjYkkks7Za Co-authored-by: Claude Opus 5 (1M context) --- frontend/src/lib/components/SchemaForm.svelte | 231 +++++++++--------- 1 file changed, 113 insertions(+), 118 deletions(-) diff --git a/frontend/src/lib/components/SchemaForm.svelte b/frontend/src/lib/components/SchemaForm.svelte index 4df7b4dc0b..0bd2f23e61 100644 --- a/frontend/src/lib/components/SchemaForm.svelte +++ b/frontend/src/lib/components/SchemaForm.svelte @@ -363,127 +363,122 @@ /> {/if} - -
{ - dispatch('click', argName) - }} - > - {#if args && typeof args == 'object' && prop} - - {#if !hidden[argName]} - { - dispatch('change') - }} - on:nestedChange={() => { - dispatch('nestedChange') - }} - on:acceptChange={(e) => dispatch('acceptChange', e.detail)} - on:rejectChange={(e) => dispatch('rejectChange', e.detail)} - on:keydownCmdEnter={() => dispatch('keydownCmdEnter')} - {disablePortal} - {resourceTypes} - {prettifyHeader} - autofocus={i == 0 && autofocus ? true : null} - label={argName} - description={prop?.description} - bind:value={args[argName]} - type={prop?.type} - oneOf={prop?.oneOf} - required={schema?.required?.includes(argName)} - pattern={prop?.pattern} - bind:valid={inputCheck[argName]} - defaultValue={defaultValues?.[argName] ?? - structuredClone($state.snapshot(prop?.default))} - enum_={dynamicEnums?.[argName] ?? prop?.enum} - format={prop?.format} - contentEncoding={prop?.contentEncoding} - customErrorMessage={prop?.customErrorMessage} - bind:properties={ - () => prop?.properties, - (v) => { - if (prop) prop.properties = v - } + + {#if args && typeof args == 'object' && prop && !hidden[argName]} + +
{ + dispatch('click', argName) + }} + > + { + dispatch('change') + }} + on:nestedChange={() => { + dispatch('nestedChange') + }} + on:acceptChange={(e) => dispatch('acceptChange', e.detail)} + on:rejectChange={(e) => dispatch('rejectChange', e.detail)} + on:keydownCmdEnter={() => dispatch('keydownCmdEnter')} + {disablePortal} + {resourceTypes} + {prettifyHeader} + autofocus={i == 0 && autofocus ? true : null} + label={argName} + description={prop?.description} + bind:value={args[argName]} + type={prop?.type} + oneOf={prop?.oneOf} + required={schema?.required?.includes(argName)} + pattern={prop?.pattern} + bind:valid={inputCheck[argName]} + defaultValue={defaultValues?.[argName] ?? + structuredClone($state.snapshot(prop?.default))} + enum_={dynamicEnums?.[argName] ?? prop?.enum} + format={prop?.format} + contentEncoding={prop?.contentEncoding} + customErrorMessage={prop?.customErrorMessage} + bind:properties={ + () => prop?.properties, + (v) => { + if (prop) prop.properties = v } - bind:order={ - () => prop?.order, - (v) => { - if (prop) prop.order = v - } + } + bind:order={ + () => prop?.order, + (v) => { + if (prop) prop.order = v } - nestedRequired={prop?.required} - itemsType={prop?.items} - disabled={disabledArgs.includes(argName) || disabled || prop?.disabled} - {compact} - {variableEditor} - {itemPicker} - bind:pickForField - password={linkedSecrets.includes(argName)} - extra={prop} - {showSchemaExplorer} - simpleTooltip={schemaFieldTooltip[argName]} - {onlyMaskPassword} - nullable={prop?.nullable} - title={prop?.title} - placeholder={prop?.placeholder} - orderEditable={dndConfig != undefined} - otherArgs={{ ...args, [argName]: undefined }} - {helperScript} - {lightHeader} - diffStatus={diff[argName] ?? undefined} - {nestedParent} - {shouldDispatchChanges} - {nestedClasses} - {appPath} - {computeS3ForceViewerPolicies} - {workspace} - {css} - {displayType} - > - {#snippet actions()} - {@render actions_render?.({ item })} - {#if linkedSecretCandidates?.includes(argName)} -
- { - if (e.detail === 'secret') { - if (!linkedSecrets.includes(argName)) { - linkedSecrets = [...linkedSecrets, argName] - } - } else { - linkedSecrets = linkedSecrets.filter((s) => s !== argName) + } + nestedRequired={prop?.required} + itemsType={prop?.items} + disabled={disabledArgs.includes(argName) || disabled || prop?.disabled} + {compact} + {variableEditor} + {itemPicker} + bind:pickForField + password={linkedSecrets.includes(argName)} + extra={prop} + {showSchemaExplorer} + simpleTooltip={schemaFieldTooltip[argName]} + {onlyMaskPassword} + nullable={prop?.nullable} + title={prop?.title} + placeholder={prop?.placeholder} + orderEditable={dndConfig != undefined} + otherArgs={{ ...args, [argName]: undefined }} + {helperScript} + {lightHeader} + diffStatus={diff[argName] ?? undefined} + {nestedParent} + {shouldDispatchChanges} + {nestedClasses} + {appPath} + {computeS3ForceViewerPolicies} + {workspace} + {css} + {displayType} + > + {#snippet actions()} + {@render actions_render?.({ item })} + {#if linkedSecretCandidates?.includes(argName)} +
+ { + if (e.detail === 'secret') { + if (!linkedSecrets.includes(argName)) { + linkedSecrets = [...linkedSecrets, argName] } - }} - > - {#snippet children({ item })} - - - {/snippet} - -
{/if} - {/snippet} - - {/if} - - - {/if} -
+ } else { + linkedSecrets = linkedSecrets.filter((s) => s !== argName) + } + }} + > + {#snippet children({ item })} + + + {/snippet} + +
{/if} + {/snippet} + +
+ {/if} {/if} {/each} From 92a454b7a81cb1ecb98954387cbb8a361932775d Mon Sep 17 00:00:00 2001 From: hugocasa Date: Fri, 21 Aug 2026 10:39:46 +0200 Subject: [PATCH 17/48] fix: split the MCP script tools into createScript and updateScript (#10783) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit * fix: let the MCP createScript tool deploy without a parent hash The tool advertised creating a new script with `parent_hash` left unset, but `parent_hash` was one of its declared arguments — and a client that requires every declared argument to be filled has no way to leave it unset. The values such a caller invents (`""`, `"0"`, a zero hash) are all rejected by `/scripts/create`, so no script was ever created. `parent_hash` is now gone from the tool, and the MCP layer sends `auto_parent` in its place: the server resolves the lineage from the path, creating the script when the path is free and deploying a new version of it when it is not. That is what the tool already claimed to do, and it no longer asks the caller to track a hash to do it. `x-mcp-tool-fixed-fields` is the general mechanism behind this — body fields the MCP layer fills in itself, absent from the tool schema. A null argument is also dropped from the assembled body now, for the same reason the placeholder hashes were a problem: it is how a caller with no value to give says so, and the API rejects it rather than falling back to the field's default. Fixes GIT-973 Co-Authored-By: Claude Opus 5 * fix: hash a script version once auto_parent has resolved its parent `create_script` hashed the incoming script before the `auto_parent` block filled in `parent_hash`, and the version hash covers that field. A deploy that let the server resolve the parent was therefore hashed as if the path had no history, so redeploying content the path had held before collided with that archived version and returned "A script with same hash ... already exists!" instead of becoming a new version of the lineage. Reverting a script to an earlier state was impossible for any caller relying on auto_parent alone, which is now every MCP caller. The hash and the duplicate-hash check move below the resolution, so an auto_parent deploy hashes the lineage it will actually be attached to. Callers passing an explicit `parent_hash` are unaffected: the resolution block leaves their `ns` untouched, so they hash exactly as before. The CLI masked this by sending `parent_hash` and `auto_parent` together, using auto_parent only as a stale-hash fallback. Co-Authored-By: Claude Opus 5 * docs: state the constraint that pins the script hash site Co-Authored-By: Claude Opus 5 * fix: reject a fixed-fields spec the MCP layer would not honour `validate_fixed_fields` ran only for an operation that declares a request body, and passed any body whose properties it could not see. Two shapes reached the generated tool with fixed fields that are dropped at call time: an operation with no `requestBody`, where the body builder returns before reading them, and a pass-through body, which carries the runnable's own arguments and never receives a key of ours. Both are now generation-time errors, so the only specs that get the extension are the ones where it means something. Also name the folder-derived `on_behalf_of` alongside `parent_hash` at the hash site: both are written to `ns` before it, and a reader who knows about only one could reintroduce the early hash. Co-Authored-By: Claude Opus 5 * fix: keep fixed fields internal and catch a misspelled one `EndpointTool` is what `list_tools` publishes as the tool catalogue, so deriving `body_fixed_fields` into it put a field in the caller's view that is by definition not the caller's to set, and that the OpenAPI schema does not declare. It is no longer serialized. The generator also only checked a fixed key against the exposed subset of the body properties, which cannot tell a field deliberately left out of `x-mcp-tool-include-fields` from a misspelling of one. A key the API does not declare is now a generation-time error rather than one serde discards in silence, and the extension must be a non-empty mapping — an empty list previously slipped through the type check on its way to being ignored. Narrow the hash-site comment to the ordering it actually constrains. Co-Authored-By: Claude Opus 5 * feat: split the MCP script tools into createScript and updateScript Scripts were the only entity in the MCP surface without the create/update pair every other one has, because the REST API has no update route for them: a script is immutably versioned, so `POST /scripts/create` is also its update, and one tool had to infer which the caller meant from the state of the path. That inference is what GIT-973 is. `parent_hash` told the two apart, and an MCP client that requires every declared argument to be filled has no way to leave it unset, so no script could be created: `""` is a 422, `"0"` is a 422, and `"0000000000000000"` is a 400. Naming the intent removes the field instead of the guard. `createScript` means the path should be free and keeps refusing an occupied one; `updateScript` names the version it supersedes in its URL, so the body carries no hash either. Picking the wrong one now fails loudly rather than succeeding on the wrong script. - New `POST /w/{workspace}/scripts/update/{path}`, deploying a new version of the script the URL names. Its body `path` is the destination, defaulting to the URL's, so setting a different one moves the script and keeps its history — which no MCP client could ask for while `createScript` was the only tool. - New `x-mcp-tool-optional-fields`, dropping a body field from the tool's `required` where the handler defaults it. `updateScript` uses it for that destination path: required, an agent has to restate the path on every edit, and a value that drifts from the URL's silently moves the script. - `assemble_request_body` drops null-valued arguments, matching what the pass-through branch already did. A client that must fill in every argument says "no value" with `null`, and the API rejects that for a bare `String` field rather than falling back to its default. Fixes GIT-973 Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01CEWFHpmTBauDBi93MnsT7s * fix: confine updateScript to the token's script paths `endpoint_path_policy` is what applies an `mcp:scripts:` token's path patterns to an endpoint tool, and a tool it does not name is not confined at all. `updateScript` was not named, so a path-scoped token could deploy over, and move, any script in the workspace: the proxy mints a bare `scripts:write` for a caller whose only scopes are `mcp:`-prefixed, and nothing downstream held a pattern. The destination path has to bind only when supplied — omitting it is how a caller updates in place — so `PathArgs` grows `optional_fields`, checked when present and never required. Empty reads as absent, matching the handler, which now takes an empty body `path` for "leave it where it is" rather than moving the script to the empty path: a caller obliged to fill in every field sends `""` as readily as null. That shape also fixes `updateFlow`, whose entry named `path__path` for the URL argument. The generator gives the URL path the plain name, so the lookup never matched and every confined call failed closed on a missing argument. Both sides now have a drift guard: a script/flow tool the URL addresses by path must have a policy. The backend one lives in windmill-api, where the generated catalogue is, since the policy is in windmill-mcp and neither crate sees both. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01CEWFHpmTBauDBi93MnsT7s * fix: address the review round on the script tool split Four findings, three of them one bug: a destination path the caller left empty. `update_script` read it as "leave it where it is", the confinement check skipped it on the strength of that, and `update_flow` did neither — it takes the empty string literally and moves the flow there, so the skipped check was the only thing standing in front of that move. A database constraint refuses the empty path, so nothing was reachable through it, but the confinement was relying on a property of one handler that its sibling did not have. The MCP layer now strips an empty optional destination from the arguments, so no handler receives one and there is nothing left for the check to skip. Neither tool depends on the other's reading of it any more. `update_script` also resolved the head before opening the deploying transaction. A version landing in between is caught — it leaves a child behind, and the linear-lineage check refuses that — but an archive leaves none, and the hash of an archived version still exists, so the deploy would have chained onto it and revived the script the archive had just retired. The resolution moves into the transaction. The scope check on the URL path moves ahead of that resolution, so a path outside the token's scope answers the same whether or not a script is there, rather than telling the two apart through 404 against 403. `x-mcp-tool-optional-fields` goes: the generator already strips a body field that collides with a same-named path parameter from `required`, so the extension regenerated byte-for-byte identical output. The test that pinned the destination as optional stays — it pins the behavior, which is now the collision handling's. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01CEWFHpmTBauDBi93MnsT7s * fix: lock the head an updateScript supersedes Moving the resolution into the deploying transaction narrowed the archive race without closing it. The plain SELECT took no row lock, so an archive could still land between it and the parent-existence check below, which finds the parent by hash and never looks at `archived` — the deploy then chained onto the archived version and inserted a live child, reviving the script the archive had retired. `FOR UPDATE` on the resolution is what makes the row the head rather than a head it once was: the archive either waits for the deploy, or wins and leaves the row failing the `archived` qualifier on re-check, so no version resolves at all. The regression test stages that interleaving rather than approximating it. It holds the head row from a second connection so the deploy parks on it, waits for a backend to actually be blocked before archiving — without that wait the request loses to a local UPDATE and never reaches its resolution, which is the sequential case the neighbouring test already covers — then asserts the update is refused. It returns 201 and revives the script with the lock removed. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01CEWFHpmTBauDBi93MnsT7s * fix: have the MCP layer name the path an update keeps The tool lets a caller omit the destination, and the endpoint was absorbing that by accepting a body without a `path` and defaulting it from the URL. The OpenAPI schema says `path` is required, so the two disagreed and a generated REST client could not follow the contract the description promised. The MCP layer fills the destination in instead, from the path the item is already at, since that is what omitting it means. The endpoint then always receives a body naming its own path and matches its schema, `update_script` takes a `NewScript` rather than picking a JSON object apart to inject a default, and the empty string stops being a value any handler has to interpret — `update_flow` reads one as the empty path, which is why it was stripped a commit ago. The alternative, an `EditScript` schema differing from `NewScript` only in whether `path` is required, was measured and rejected: openapi-ts drops the `required` of an `allOf` branch, so `NewScript` came out with every field optional and broke 15 frontend types. Loosening a schema every API consumer shares, to make one field optional on one route, is the worse trade. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01CEWFHpmTBauDBi93MnsT7s * fix: tell a superseded update apart from a missing script Locking the head made the loser of two concurrent deploys answer 404 "Script not found" for a path the caller can see holds a script: its lock re-check finds the row archived and filtered, and nothing looked further. It now looks — a live version at the path means this deploy lost to one that superseded the version it set out to supersede, which is a conflict to retry, not a script to go find. The regression test stages that interleaving the way the archive one does, with the winner leaving a live head behind rather than an archived path. It answers 404 with the branch removed. The rationale for the lock also sat in two places; it stays at the query, which is where dropping it would do the damage. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01CEWFHpmTBauDBi93MnsT7s * docs: drop the path default update_script no longer applies The handler stopped defaulting the body's path when the MCP layer took the job over; its doc comment still described the old contract. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01CEWFHpmTBauDBi93MnsT7s * docs: sync the deref YAML with the update route's path contract The dereferenced bundle rewraps prose at its own width, so the edit that updated the canonical spec and the JSON bundle matched nothing here and left the served YAML still offering a default the endpoint no longer applies. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01CEWFHpmTBauDBi93MnsT7s * docs: stop the script tools describing a parent_hash they cannot take `description` is read by two audiences: it documents the route, and it opens the MCP tool's text. Written for the first, it told an agent that createScript "does it too when given that version's `parent_hash`" — a field neither tool exposes, and inviting exactly the call this branch exists to make impossible. updateScript's told the agent to repeat the URL's path while its own instructions say to omit it; both work, since the MCP layer fills it in, but only one of them can be the advice. Both now describe what the operation does and leave the mechanics to the text that belongs to each caller: the request body's own description for REST, the tool instructions for an agent. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01CEWFHpmTBauDBi93MnsT7s * docs: give the create route's two audiences their own description Removing the `parent_hash` sentence took a true fact out of the REST documentation: the create route does still deploy a new version, and still rename, when the body names the version it supersedes. Nothing replaced the explanation, and the field carried no description of its own. `description` cannot serve both readers — it documents an endpoint whose schema has `parent_hash`, and it opens a tool whose filtered schema deliberately does not. `x-mcp-tool-description` stands in for it on the tool, the way `x-mcp-tool-name` already does for the name, so the route keeps its full contract and the agent is not told to send a field it has no way to send. What `parent_hash` does now sits on the field, where a REST caller looks for it and where `x-mcp-tool-include-fields` drops it before an agent sees it. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01CEWFHpmTBauDBi93MnsT7s * docs: tell an agent a new version is not runnable the instant it deploys A deploy returns before its lockfile exists, so a script run straight after one can still execute the previous version. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01CEWFHpmTBauDBi93MnsT7s * docs: say why a new version is not runnable the instant it deploys Its lock is generated asynchronously, so a script run straight after a deploy can still execute the previous version. On both script tools: a freshly created script is no more immediately runnable than a freshly updated one. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01CEWFHpmTBauDBi93MnsT7s * docs: bound the wait after a deploy instead of naming a signal for it `getScriptByPath` reports the new hash the instant the version exists, while its lock is still null, so the previous version is what a run by path executes. There is no signal that fixes this: the deploy evicts DEPLOYED_SCRIPT_HASH_CACHE, but anything resolving the path before the lock lands re-populates it with the old hash, and the lock landing evicts nothing. Waiting for a non-null lock is necessary and not sufficient, so pointing at one would have been a second wrong answer. Measured: a run right after the lock lands still gets the previous version, and the same run 65s later gets the new one, which is the cache's 60s TTL. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01CEWFHpmTBauDBi93MnsT7s --------- Co-authored-by: Claude Opus 5 Co-authored-by: Ruben Fiszel --- .../generate_mcp_tools.py | 19 +- .../tests/scripts.rs | 369 ++++++++++++++++++ backend/windmill-api-scripts/src/scripts.rs | 146 ++++++- backend/windmill-api/openapi-deref.json | 62 ++- backend/windmill-api/openapi-deref.yaml | 85 +++- backend/windmill-api/openapi.yaml | 53 ++- .../src/mcp/auto_generated_endpoints.rs | 71 +++- backend/windmill-api/src/mcp/utils.rs | 102 ++++- backend/windmill-mcp/src/server/mod.rs | 2 +- backend/windmill-mcp/src/server/runner.rs | 244 +++++++++--- .../mcp/endpointScopePolicy.test.ts | 13 + .../lib/components/mcp/endpointScopePolicy.ts | 1 + frontend/src/lib/mcpEndpointTools.ts | 68 +++- 13 files changed, 1134 insertions(+), 101 deletions(-) diff --git a/backend/generate_mcp_endpoints_tools/generate_mcp_tools.py b/backend/generate_mcp_endpoints_tools/generate_mcp_tools.py index b2fbc18109..72feb5392e 100644 --- a/backend/generate_mcp_endpoints_tools/generate_mcp_tools.py +++ b/backend/generate_mcp_endpoints_tools/generate_mcp_tools.py @@ -411,9 +411,18 @@ def schema_to_rust_value(schema: Optional[Dict[str, Any]]) -> str: return f"Some(serde_json::json!({json.dumps(schema, indent=8, ensure_ascii=False)}))" def build_tool_description(operation: Dict[str, Any], method: str, path: str) -> str: - """Build the MCP tool description from OpenAPI summary and description.""" + """Build the MCP tool description from OpenAPI summary and description. + + `x-mcp-tool-description` stands in for the operation's own description, for a route + whose two audiences need different text: the description documents the endpoint, + including fields like `parent_hash` that `x-mcp-tool-include-fields` deliberately + keeps out of the tool, and naming one there tells an agent to send what its schema + does not offer. + """ summary = operation.get('summary', '').strip() - description = operation.get('description', '').strip() + description = ( + operation.get('x-mcp-tool-description') or operation.get('description', '') + ).strip() if summary and description: return f"{summary}: {description}".rstrip('.!? ') @@ -463,7 +472,11 @@ def find_mcp_tools(spec: Dict[str, Any]) -> List[Dict[str, Any]]: return tools def generate_typescript_code(tools: List[Dict[str, Any]], spec: Dict[str, Any], base_path: str = "") -> str: - """Generate TypeScript code with MCP endpoint tools.""" + """Generate TypeScript code with MCP endpoint tools. + + This catalogue only feeds the frontend's MCP scope picker, which reads names and + methods. + """ if not tools: return """// Auto-generated MCP tools from OpenAPI specification // This file is generated by generate_mcp_tools.py - DO NOT EDIT MANUALLY diff --git a/backend/windmill-api-integration-tests/tests/scripts.rs b/backend/windmill-api-integration-tests/tests/scripts.rs index e5f6cb8aa8..8c88c53454 100644 --- a/backend/windmill-api-integration-tests/tests/scripts.rs +++ b/backend/windmill-api-integration-tests/tests/scripts.rs @@ -455,6 +455,375 @@ async fn test_auto_parent_resolves_parent_hash(db: Pool) -> anyhow::Re "v3 parent_hashes should contain v2 hash {v2_hash}, got: {parent_hashes:?}" ); + // Redeploy v1's exact body. The version hash covers the parent, so this is a + // distinct version of the lineage rather than a repeat of the archived v1 — + // which it is not if the hash is taken before auto_parent resolves the parent. + let mut revert = new_script( + "u/test-user/auto_parent_test", + "v1", + "export async function main() { return 1; }", + ); + revert["auto_parent"] = json!(true); + let resp = authed(client().post(format!("{base}/create"))) + .json(&revert) + .send() + .await + .unwrap(); + assert_eq!( + resp.status(), + 201, + "reverting to v1's content with auto_parent: {}", + resp.text().await? + ); + + Ok(()) +} + +/// The update route carries the version being superseded in its URL, so a caller that +/// cannot read a `parent_hash` still chains onto the history instead of forking it. +#[sqlx::test(migrations = "../migrations", fixtures("base"))] +async fn test_update_script_chains_moves_and_refuses_a_free_path( + db: Pool, +) -> anyhow::Result<()> { + initialize_tracing().await; + let server = ApiServer::start(db.clone()).await?; + let port = server.addr.port(); + let base = format!("http://localhost:{port}/api/w/test-workspace/scripts"); + let path = "u/test-user/update_test"; + + // Nothing deployed there yet: an update has no version to supersede. + let resp = authed(client().post(format!("{base}/update/{path}"))) + .json(&new_script( + path, + "v1", + "export async function main() { return 1; }", + )) + .send() + .await + .unwrap(); + assert_eq!(resp.status(), 404, "update of a free path must be refused"); + + let resp = authed(client().post(format!("{base}/create"))) + .json(&new_script( + path, + "v1", + "export async function main() { return 1; }", + )) + .send() + .await + .unwrap(); + assert_eq!(resp.status(), 201, "create v1: {}", resp.text().await?); + let v1_hash = authed_get(port, "get/p", path) + .await + .json::() + .await?["hash"] + .as_str() + .unwrap() + .to_string(); + + // The body repeats the path, so the script stays where it is, chained onto v1. + let resp = authed(client().post(format!("{base}/update/{path}"))) + .json(&new_script( + path, + "v2", + "export async function main() { return 2; }", + )) + .send() + .await + .unwrap(); + assert_eq!( + resp.status(), + 201, + "update in place: {}", + resp.text().await? + ); + + let body = authed_get(port, "get/p", path) + .await + .json::() + .await?; + assert_eq!(body["summary"], "v2"); + let v2_hash = body["hash"].as_str().unwrap().to_string(); + let parent_hashes = body["parent_hashes"].as_array().unwrap(); + assert!( + parent_hashes.iter().any(|h| h.as_str() == Some(&v1_hash)), + "v2 must descend from v1 {v1_hash}, got: {parent_hashes:?}" + ); + + // A body path that differs moves the script, taking the history with it. + let moved_path = "u/test-user/update_test_moved"; + let resp = authed(client().post(format!("{base}/update/{path}"))) + .json(&new_script( + moved_path, + "v3", + "export async function main() { return 3; }", + )) + .send() + .await + .unwrap(); + assert_eq!(resp.status(), 201, "move: {}", resp.text().await?); + + let body = authed_get(port, "get/p", moved_path) + .await + .json::() + .await?; + assert_eq!(body["summary"], "v3"); + let parent_hashes = body["parent_hashes"].as_array().unwrap(); + assert!( + parent_hashes.iter().any(|h| h.as_str() == Some(&v2_hash)), + "the moved script must descend from v2 {v2_hash}, got: {parent_hashes:?}" + ); + assert_eq!( + authed_get(port, "get/p", path) + .await + .json::() + .await?["archived"], + json!(true), + "the vacated path must be left archived" + ); + + Ok(()) +} + +/// An archived path holds no version to supersede, so an update must not revive it. The +/// resolution that decides this runs in the deploying transaction rather than ahead of +/// it, which is what also covers an archive landing mid-deploy — a race this sequential +/// test cannot stage, so it pins the reachable half. +#[sqlx::test(migrations = "../migrations", fixtures("base"))] +async fn test_update_script_does_not_revive_an_archived_path( + db: Pool, +) -> anyhow::Result<()> { + initialize_tracing().await; + let server = ApiServer::start(db.clone()).await?; + let port = server.addr.port(); + let base = format!("http://localhost:{port}/api/w/test-workspace/scripts"); + let path = "u/test-user/archived_update_test"; + + let resp = authed(client().post(format!("{base}/create"))) + .json(&new_script( + path, + "v1", + "export async function main() { return 1; }", + )) + .send() + .await + .unwrap(); + assert_eq!(resp.status(), 201, "create v1: {}", resp.text().await?); + + let resp = authed(client().post(format!("{base}/archive/p/{path}"))) + .send() + .await + .unwrap(); + assert_eq!(resp.status(), 200, "archive: {}", resp.text().await?); + + let resp = authed(client().post(format!("{base}/update/{path}"))) + .json(&new_script( + path, + "v2", + "export async function main() { return 2; }", + )) + .send() + .await + .unwrap(); + assert_eq!( + resp.status(), + 404, + "updating an archived path must not revive it" + ); + + assert_eq!( + authed_get(port, "get/p", path) + .await + .json::() + .await?["archived"], + json!(true), + "the path must still be archived" + ); + + Ok(()) +} + +/// Stages the interleaving the sequential test above cannot: the update resolves its +/// parent while an archive is mid-flight. Holding the head row locked from another +/// connection parks the update on that row, so the archive lands first by construction +/// — the ordering that, unlocked, hands the deploy an archived hash to chain onto. +#[sqlx::test(migrations = "../migrations", fixtures("base"))] +async fn test_update_script_loses_a_race_with_archive(db: Pool) -> anyhow::Result<()> { + initialize_tracing().await; + let server = ApiServer::start(db.clone()).await?; + let port = server.addr.port(); + let base = format!("http://localhost:{port}/api/w/test-workspace/scripts"); + let path = "u/test-user/raced_update_test"; + + let resp = authed(client().post(format!("{base}/create"))) + .json(&new_script( + path, + "v1", + "export async function main() { return 1; }", + )) + .send() + .await + .unwrap(); + assert_eq!(resp.status(), 201, "create v1: {}", resp.text().await?); + + // Take the head row before the update can, so it blocks where it resolves. + let mut blocker = db.begin().await?; + let head: i64 = sqlx::query_scalar( + "SELECT hash FROM script WHERE path = $1 AND archived = false AND workspace_id = $2 \ + FOR UPDATE", + ) + .bind(path) + .bind("test-workspace") + .fetch_one(&mut *blocker) + .await?; + + let update = tokio::spawn({ + let base = base.clone(); + let body = new_script(path, "v2", "export async function main() { return 2; }"); + async move { + authed(client().post(format!("{base}/update/{path}"))) + .json(&body) + .send() + .await + .unwrap() + } + }); + + // Order the archive after whatever the update has already read: wait until it is + // parked on the row lock. Without this the update can lose to a local UPDATE and + // never reach its resolution, which is the sequential case the test above covers. + let mut parked = false; + for _ in 0..400 { + let waiting: i64 = sqlx::query_scalar( + "SELECT count(*) FROM pg_stat_activity WHERE wait_event_type = 'Lock' \ + AND datname = current_database() AND pid <> pg_backend_pid()", + ) + .fetch_one(&db) + .await?; + if waiting > 0 { + parked = true; + break; + } + tokio::time::sleep(std::time::Duration::from_millis(25)).await; + } + assert!(parked, "the update never parked on the head row lock"); + + // Archive under the lock the update is waiting on, then release it. + sqlx::query("UPDATE script SET archived = true WHERE hash = $1 AND workspace_id = $2") + .bind(head) + .bind("test-workspace") + .execute(&mut *blocker) + .await?; + blocker.commit().await?; + + let resp = tokio::time::timeout(std::time::Duration::from_secs(20), update).await??; + assert_eq!( + resp.status(), + 404, + "an update that lost the race must not revive the archived script" + ); + assert_eq!( + authed_get(port, "get/p", path) + .await + .json::() + .await?["archived"], + json!(true), + "the path must be left archived" + ); + + Ok(()) +} + +/// The same interleaving, except the winner leaves a live head behind. The loser must +/// say so rather than "not found" of a path the caller can see holds a script. +#[sqlx::test(migrations = "../migrations", fixtures("base"))] +async fn test_update_script_reports_losing_to_a_concurrent_deploy( + db: Pool, +) -> anyhow::Result<()> { + initialize_tracing().await; + let server = ApiServer::start(db.clone()).await?; + let port = server.addr.port(); + let base = format!("http://localhost:{port}/api/w/test-workspace/scripts"); + let path = "u/test-user/superseded_update_test"; + + let resp = authed(client().post(format!("{base}/create"))) + .json(&new_script( + path, + "v1", + "export async function main() { return 1; }", + )) + .send() + .await + .unwrap(); + assert_eq!(resp.status(), 201, "create v1: {}", resp.text().await?); + + let mut winner = db.begin().await?; + let head: i64 = sqlx::query_scalar( + "SELECT hash FROM script WHERE path = $1 AND archived = false AND workspace_id = $2 \ + FOR UPDATE", + ) + .bind(path) + .bind("test-workspace") + .fetch_one(&mut *winner) + .await?; + + let update = tokio::spawn({ + let base = base.clone(); + let body = new_script(path, "v2", "export async function main() { return 2; }"); + async move { + authed(client().post(format!("{base}/update/{path}"))) + .json(&body) + .send() + .await + .unwrap() + } + }); + + let mut parked = false; + for _ in 0..400 { + let waiting: i64 = sqlx::query_scalar( + "SELECT count(*) FROM pg_stat_activity WHERE wait_event_type = 'Lock' \ + AND datname = current_database() AND pid <> pg_backend_pid()", + ) + .fetch_one(&db) + .await?; + if waiting > 0 { + parked = true; + break; + } + tokio::time::sleep(std::time::Duration::from_millis(25)).await; + } + assert!(parked, "the update never parked on the head row lock"); + + // What a deploy leaves behind: the old head archived, a new one live at the path. + // Copied through a temp table so this does not have to restate every column. + sqlx::query("CREATE TEMP TABLE superseding ON COMMIT DROP AS SELECT * FROM script WHERE hash = $1") + .bind(head) + .execute(&mut *winner) + .await?; + sqlx::query("UPDATE superseding SET hash = $1, archived = false, parent_hashes = ARRAY[$2]") + .bind(head + 1) + .bind(head) + .execute(&mut *winner) + .await?; + sqlx::query("UPDATE script SET archived = true WHERE hash = $1") + .bind(head) + .execute(&mut *winner) + .await?; + sqlx::query("INSERT INTO script SELECT * FROM superseding") + .execute(&mut *winner) + .await?; + winner.commit().await?; + + let resp = tokio::time::timeout(std::time::Duration::from_secs(20), update).await??; + let status = resp.status(); + let body = resp.text().await?; + assert_eq!(status, 400, "losing the race should not read as success: {body}"); + assert!( + body.contains("deployed to concurrently"), + "the loser must say it was superseded, not that the script is missing: {body}" + ); + Ok(()) } diff --git a/backend/windmill-api-scripts/src/scripts.rs b/backend/windmill-api-scripts/src/scripts.rs index 3b66f7317b..df82a7a928 100644 --- a/backend/windmill-api-scripts/src/scripts.rs +++ b/backend/windmill-api-scripts/src/scripts.rs @@ -107,6 +107,7 @@ pub fn workspaced_service() -> Router { .route("/list", get(list_scripts)) .route("/list_search", get(list_search_scripts)) .route("/create", post(create_script)) + .route("/update/{*path}", post(update_script)) .route("/create_snapshot", post(create_snapshot_script)) .route("/archive/p/{*path}", post(archive_script_by_path)) .route("/get/p/{*path}", get(get_script_by_path)) @@ -561,6 +562,7 @@ async fn create_snapshot_script( user_db.clone(), webhook.clone(), query.skip_if_noop, + None, ) .await?; let mut nh = new_hash.to_string(); @@ -647,6 +649,70 @@ async fn create_script( Path(w_id): Path, Query(query): Query, Json(ns): Json, +) -> Result<(StatusCode, String)> { + deploy_script( + authed, + user_db, + webhook, + db, + w_id, + query.skip_if_noop, + ns, + None, + ) + .await +} + +/// Deploy a new version of the script at `path`, which must already hold one. +/// +/// The URL names the version being superseded, so the body needs no `parent_hash`: a +/// caller that cannot read one (an MCP client, whose tool schema has no hash field) +/// still gets a version chained onto the history rather than a fork of it. The body's +/// own `path` is where the script should end up: the URL's again to leave it there, +/// another to move it. +async fn update_script( + authed: ApiAuthed, + Extension(user_db): Extension, + Extension(webhook): Extension, + Extension(db): Extension, + Path((w_id, path)): Path<(String, StripPath)>, + Query(query): Query, + Json(mut ns): Json, +) -> Result<(StatusCode, String)> { + let path = path.to_path(); + // Superseding the version at this path is a write to it, checked before anything + // reads the row so a path outside the token's scope answers the same whether or + // not a script is there. + check_scopes(&authed, || format!("scripts:write:{}", path))?; + + // Lineage comes from the URL, resolved in the deploying transaction; letting a body + // field name a parent, or a body flag re-derive one from the destination, would fork + // the history or turn a move into a copy. + ns.parent_hash = None; + ns.auto_parent = None; + + deploy_script( + authed, + user_db, + webhook, + db, + w_id, + query.skip_if_noop, + ns, + Some(path.to_string()), + ) + .await +} + +async fn deploy_script( + authed: ApiAuthed, + user_db: UserDB, + webhook: WebhookShared, + db: DB, + w_id: String, + skip_if_noop: bool, + ns: NewScript, + supersede_head_at: Option, ) -> Result<(StatusCode, String)> { if let RuleCheckResult::Blocked(msg) = check_deploy_rules( &w_id, @@ -670,7 +736,8 @@ async fn create_script( db.clone(), user_db, webhook, - query.skip_if_noop, + skip_if_noop, + supersede_head_at, ) .await?; tx.commit().await?; @@ -997,6 +1064,9 @@ async fn create_script_internal<'c>( user_db: UserDB, webhook: WebhookShared, skip_if_noop: bool, + // When set, the parent is the live head at this path rather than anything the body + // named, resolved against the deploying transaction. + supersede_head_at: Option, ) -> Result<( ScriptHash, Transaction<'c, Postgres>, @@ -1054,7 +1124,6 @@ async fn create_script_internal<'c>( // Caller-intent: CLI / git-sync deploys ask us to preserve any existing // user draft at this path instead of wiping it as part of the deploy. let skip_draft_deletion = ns.skip_draft_deletion.unwrap_or(false); - let hash = ScriptHash(hash_script(&ns)); let authed = maybe_refresh_folders(&ns.path, &w_id, authed, &db).await; let mut tx: Transaction<'_, Postgres> = user_db.begin(&authed).await?; @@ -1101,21 +1170,6 @@ async fn create_script_internal<'c>( let legacy_on_behalf_of_email = windmill_common::legacy_on_behalf_of_email(resolved_on_behalf_of.as_deref(), &w_id, &db) .await?; - if sqlx::query_scalar!( - "SELECT 1 FROM script WHERE hash = $1 AND workspace_id = $2", - hash.0, - &w_id - ) - .fetch_optional(&mut *tx) - .await? - .is_some() - { - return Err(Error::BadRequest( - "A script with same hash (hence same path, description, summary, content) already \ - exists!" - .to_owned(), - )); - }; // When auto_parent is set, serialize concurrent creates for the same (workspace, path) // so the clashing_script query always sees the latest committed head. if ns.auto_parent.unwrap_or(false) { @@ -1127,6 +1181,43 @@ async fn create_script_internal<'c>( .fetch_one(&mut *tx) .await?; } + if let Some(source) = supersede_head_at.as_deref() { + // Locked, or this is a head it once was: nothing below re-checks `archived`, so + // an archive slipping in leaves the deploy chaining onto the version it retired + // and reviving it. + let head = sqlx::query_scalar::<_, i64>( + "SELECT hash FROM script WHERE path = $1 AND archived = false AND workspace_id = $2 \ + FOR UPDATE", + ) + .bind(source) + .bind(&w_id) + .fetch_optional(&mut *tx) + .await?; + let Some(head) = head else { + // A live version here now means this deploy lost the lock to one that + // superseded the version it set out to supersede. Calling that "not found", + // of a path the caller can see holds a script, sends them after the wrong + // problem — the answer is to read it again and redeploy. + let superseded = sqlx::query_scalar::<_, i64>( + "SELECT hash FROM script WHERE path = $1 AND archived = false \ + AND workspace_id = $2", + ) + .bind(source) + .bind(&w_id) + .fetch_optional(&mut *tx) + .await? + .is_some(); + return Err(if superseded { + Error::BadRequest(format!( + "The script at {source} was deployed to concurrently; read it again \ + and redeploy" + )) + } else { + Error::NotFound(format!("Script not found at path {source}")) + }); + }; + ns.parent_hash = Some(ScriptHash(head)); + } let clashing_script = sqlx::query_as::<_, Script>(&format!( "SELECT {} FROM script WHERE path = $1 AND archived = false AND workspace_id = $2", windmill_common::scripts::SCRIPT_COLUMNS, @@ -1151,6 +1242,27 @@ async fn create_script_internal<'c>( } } + // Must stay below the parent resolution above: an auto_parent deploy hashed before + // it carries a first deploy's lineage, so redeploying content the path has held + // before collides with that archived version instead of superseding it. The + // folder-derived `on_behalf_of` set further up is likewise covered by the hash. + let hash = ScriptHash(hash_script(&ns)); + if sqlx::query_scalar!( + "SELECT 1 FROM script WHERE hash = $1 AND workspace_id = $2", + hash.0, + &w_id + ) + .fetch_optional(&mut *tx) + .await? + .is_some() + { + return Err(Error::BadRequest( + "A script with same hash (hence same path, description, summary, content) already \ + exists!" + .to_owned(), + )); + }; + let parent_hashes_and_perms: Option = match (&ns.parent_hash, clashing_script) { (None, None) => Ok(None), (None, Some(s)) => Err(Error::BadRequest(format!( diff --git a/backend/windmill-api/openapi-deref.json b/backend/windmill-api/openapi-deref.json index fd40b6fa5e..3688bdfe94 100644 --- a/backend/windmill-api/openapi-deref.json +++ b/backend/windmill-api/openapi-deref.json @@ -12768,13 +12768,12 @@ "/w/{workspace}/scripts/create": { "post": { "summary": "create script", - "description": "Creates a new script when the path does not already exist.\nCreates a new version of an existing script when called with the same path and the current `parent_hash`.\n", + "description": "Creates a new script at a path that does not already hold one.\nSupplying `parent_hash` instead deploys a new version of the script that hash names, and a `path` differing from that version's moves the script there. `POST /w/{workspace}/scripts/update/{path}` does the same, naming the superseded version in its URL.\n", "operationId": "createScript", "x-mcp-tool": true, - "x-mcp-instructions": "To create a NEW script, specify the path (e.g., 'f/my_folder/my_script'), the content (source code), and the language, and leave parent_hash unset. For TypeScript, use 'bun' unless deno-specific APIs are needed. To UPDATE an existing script, do NOT delete and recreate it: call this tool with the same path and set parent_hash to the script's current hash, which you can read from the `hash` field returned by getScriptByPath. This creates a new version while preserving the script's history.", + "x-mcp-instructions": "Specify the path (e.g., 'f/my_folder/my_script'), the content (source code), and the language. For TypeScript, use 'bun' unless deno-specific APIs are needed. A path that already holds a script is refused: use updateScript to deploy a new version of it, and do NOT delete and recreate a script to change it. A new version generates its lock async and can take up to a minute before a run by path uses it rather than the previous one.", "x-mcp-tool-include-fields": [ "path", - "parent_hash", "content", "language", "summary", @@ -12813,6 +12812,60 @@ } } } + }, + "x-mcp-tool-description": "Creates a script at a path that does not already hold one." + } + }, + "/w/{workspace}/scripts/update/{path}": { + "post": { + "summary": "update script", + "description": "Deploys a new version of the script at `path`, which must already hold one.\nThe body's `path` is the destination: the same path leaves the script where it\nis, a different one moves it there and archives the old path.\n", + "operationId": "updateScript", + "x-mcp-tool": true, + "x-mcp-instructions": "Deploys a new version of an existing script, preserving its history, so do NOT delete and recreate a script to change it. Send the whole script, not a patch: read the current one with getScriptByPath first, unless you wrote its content yourself. Set path__body only to move the script to a different path; omit it to leave the script where it is. A path that holds no script is refused: use createScript to create one. A new version generates its lock async and can take up to a minute before a run by path uses it rather than the previous one.", + "x-mcp-tool-include-fields": [ + "path", + "content", + "language", + "summary", + "description", + "kind", + "tag", + "deployment_message" + ], + "tags": [ + "script" + ], + "parameters": [ + { + "$ref": "#/components/parameters/WorkspaceId" + }, + { + "$ref": "#/components/parameters/ScriptPath" + } + ], + "requestBody": { + "description": "The new version of the script, whose `path` is where it should end up.", + "required": true, + "content": { + "application/json": { + "schema": { + "$ref": "#/components/schemas/NewScript" + } + } + } + }, + "responses": { + "201": { + "description": "new script version created", + "content": { + "text/plain": { + "schema": { + "type": "string" + } + } + } + } } } }, @@ -37588,7 +37641,8 @@ "type": "string" }, "parent_hash": { - "type": "string" + "type": "string", + "description": "The hash of the version this one supersedes: deploying with it archives that version and chains the new one onto its history, and a `path` differing from the superseded version's moves the script there." }, "auto_parent": { "type": "boolean", diff --git a/backend/windmill-api/openapi-deref.yaml b/backend/windmill-api/openapi-deref.yaml index 7efc30a509..fbeeb2579d 100644 --- a/backend/windmill-api/openapi-deref.yaml +++ b/backend/windmill-api/openapi-deref.yaml @@ -13300,24 +13300,25 @@ paths: post: summary: create script description: > - Creates a new script when the path does not already exist. + Creates a new script at a path that does not already hold one. - Creates a new version of an existing script when called with the same - path and the current `parent_hash`. + Supplying `parent_hash` instead deploys a new version of the script that + hash names, and a `path` differing from that version's moves the script + there. `POST /w/{workspace}/scripts/update/{path}` does the same, naming + the superseded version in its URL. operationId: createScript x-mcp-tool: true + x-mcp-tool-description: Creates a script at a path that does not already hold one. x-mcp-instructions: >- - To create a NEW script, specify the path (e.g., - 'f/my_folder/my_script'), the content (source code), and the language, - and leave parent_hash unset. For TypeScript, use 'bun' unless - deno-specific APIs are needed. To UPDATE an existing script, do NOT - delete and recreate it: call this tool with the same path and set - parent_hash to the script's current hash, which you can read from the - `hash` field returned by getScriptByPath. This creates a new version - while preserving the script's history. + Specify the path (e.g., 'f/my_folder/my_script'), the content (source + code), and the language. + For TypeScript, use 'bun' unless deno-specific APIs are needed. A path + that already holds a script is refused: use updateScript to deploy a + new version of it, and do NOT delete and recreate a script to change + it. A new version generates its lock async and can take up to a minute + before a run by path uses it rather than the previous one. x-mcp-tool-include-fields: - path - - parent_hash - content - language - summary @@ -13344,6 +13345,11 @@ paths: type: string parent_hash: type: string + description: >- + The hash of the version this one supersedes: deploying with it + archives that version and chains the new one onto its history, + and a `path` differing from the superseded version's moves the + script there. auto_parent: type: boolean description: >- @@ -13496,6 +13502,61 @@ paths: text/plain: schema: type: string + /w/{workspace}/scripts/update/{path}: + post: + summary: update script + description: > + Deploys a new version of the script at `path`, which must already hold + one. The body's `path` is the destination: the same path leaves the + script where it is, a different one moves it there and archives the old + path. + operationId: updateScript + x-mcp-tool: true + x-mcp-instructions: >- + Deploys a new version of an existing script, preserving its history, so + do NOT delete and recreate a script to change it. Send the whole + script, not a patch: read the current one with getScriptByPath first, + unless you wrote its content yourself. Set path__body only to move the + script to a different path; omit it to leave the script where it is. A + path that holds no script is refused: use createScript to create one. A + new version generates its lock async and can take up to a minute before + a run by path uses it rather than the previous one. + x-mcp-tool-include-fields: + - path + - content + - language + - summary + - description + - kind + - tag + - deployment_message + tags: + - script + parameters: + - name: workspace + in: path + required: true + schema: *ref_4 + - name: path + in: path + required: true + schema: *ref_98 + requestBody: + description: The new version of the script, whose `path` is where it should end up. + required: true + content: + application/json: + schema: + type: object + properties: *ref_394 + required: *ref_395 + responses: + '201': + description: new script version created + content: + text/plain: + schema: + type: string /w/{workspace}/scripts/toggle_workspace_error_handler/p/{path}: post: summary: Toggle ON and OFF the workspace error handler for a given script diff --git a/backend/windmill-api/openapi.yaml b/backend/windmill-api/openapi.yaml index 375c15f2fb..5ed18d1a18 100644 --- a/backend/windmill-api/openapi.yaml +++ b/backend/windmill-api/openapi.yaml @@ -9448,14 +9448,15 @@ paths: post: summary: create script description: | - Creates a new script when the path does not already exist. - Creates a new version of an existing script when called with the same path and the current `parent_hash`. + Creates a new script at a path that does not already hold one. + Supplying `parent_hash` instead deploys a new version of the script that hash names, and a `path` differing from that version's moves the script there. `POST /w/{workspace}/scripts/update/{path}` does the same, naming the superseded version in its URL. operationId: createScript x-mcp-tool: true - x-mcp-instructions: "To create a NEW script, specify the path (e.g., 'f/my_folder/my_script'), the content (source code), and the language, and leave parent_hash unset. For TypeScript, use 'bun' unless deno-specific APIs are needed. To UPDATE an existing script, do NOT delete and recreate it: call this tool with the same path and set parent_hash to the script's current hash, which you can read from the `hash` field returned by getScriptByPath. This creates a new version while preserving the script's history." + # The description above names `parent_hash`, which this tool does not expose. + x-mcp-tool-description: Creates a script at a path that does not already hold one. + x-mcp-instructions: "Specify the path (e.g., 'f/my_folder/my_script'), the content (source code), and the language. For TypeScript, use 'bun' unless deno-specific APIs are needed. A path that already holds a script is refused: use updateScript to deploy a new version of it, and do NOT delete and recreate a script to change it. A new version generates its lock async and can take up to a minute before a run by path uses it rather than the previous one." x-mcp-tool-include-fields: - path - - parent_hash - content - language - summary @@ -9483,6 +9484,46 @@ paths: schema: type: string + /w/{workspace}/scripts/update/{path}: + post: + summary: update script + description: | + Deploys a new version of the script at `path`, which must already hold one. + The body's `path` is the destination: the same path leaves the script where it + is, a different one moves it there and archives the old path. + operationId: updateScript + x-mcp-tool: true + x-mcp-instructions: "Deploys a new version of an existing script, preserving its history, so do NOT delete and recreate a script to change it. Send the whole script, not a patch: read the current one with getScriptByPath first, unless you wrote its content yourself. Set path__body only to move the script to a different path; omit it to leave the script where it is. A path that holds no script is refused: use createScript to create one. A new version generates its lock async and can take up to a minute before a run by path uses it rather than the previous one." + x-mcp-tool-include-fields: + - path + - content + - language + - summary + - description + - kind + - tag + - deployment_message + tags: + - script + parameters: + - $ref: "#/components/parameters/WorkspaceId" + - $ref: "#/components/parameters/ScriptPath" + requestBody: + description: The new version of the script, whose `path` is where it should end up. + required: true + content: + application/json: + schema: + $ref: "#/components/schemas/NewScript" + + responses: + "201": + description: new script version created + content: + text/plain: + schema: + type: string + /w/{workspace}/scripts/toggle_workspace_error_handler/p/{path}: post: summary: Toggle ON and OFF the workspace error handler for a given script @@ -26521,6 +26562,10 @@ components: type: string parent_hash: type: string + description: >- + The hash of the version this one supersedes: deploying with it archives that + version and chains the new one onto its history, and a `path` differing from + the superseded version's moves the script there. auto_parent: type: boolean description: >- diff --git a/backend/windmill-api/src/mcp/auto_generated_endpoints.rs b/backend/windmill-api/src/mcp/auto_generated_endpoints.rs index 86b9fdfb11..6b9d0f14fc 100644 --- a/backend/windmill-api/src/mcp/auto_generated_endpoints.rs +++ b/backend/windmill-api/src/mcp/auto_generated_endpoints.rs @@ -678,9 +678,8 @@ pub fn all_tools() -> Vec { }, EndpointTool { name: Cow::Borrowed("createScript"), - description: Cow::Borrowed("create script: Creates a new script when the path does not already exist. -Creates a new version of an existing script when called with the same path and the current `parent_hash`"), - instructions: Cow::Borrowed("To create a NEW script, specify the path (e.g., 'f/my_folder/my_script'), the content (source code), and the language, and leave parent_hash unset. For TypeScript, use 'bun' unless deno-specific APIs are needed. To UPDATE an existing script, do NOT delete and recreate it: call this tool with the same path and set parent_hash to the script's current hash, which you can read from the `hash` field returned by getScriptByPath. This creates a new version while preserving the script's history."), + description: Cow::Borrowed("create script: Creates a script at a path that does not already hold one"), + instructions: Cow::Borrowed("Specify the path (e.g., 'f/my_folder/my_script'), the content (source code), and the language. For TypeScript, use 'bun' unless deno-specific APIs are needed. A path that already holds a script is refused: use updateScript to deploy a new version of it, and do NOT delete and recreate a script to change it. A new version generates its lock async and can take up to a minute before a run by path uses it rather than the previous one."), path: Cow::Borrowed("/w/{workspace}/scripts/create"), method: Cow::Borrowed("POST"), path_params_schema: None, @@ -691,9 +690,6 @@ Creates a new version of an existing script when called with the same path and t "path": { "type": "string" }, - "parent_hash": { - "type": "string" - }, "summary": { "type": "string" }, @@ -729,6 +725,69 @@ Creates a new version of an existing script when called with the same path and t query_field_renames: None, body_field_renames: None, }, + EndpointTool { + name: Cow::Borrowed("updateScript"), + description: Cow::Borrowed("update script: Deploys a new version of the script at `path`, which must already hold one. +The body's `path` is the destination: the same path leaves the script where it +is, a different one moves it there and archives the old path"), + instructions: Cow::Borrowed("Deploys a new version of an existing script, preserving its history, so do NOT delete and recreate a script to change it. Send the whole script, not a patch: read the current one with getScriptByPath first, unless you wrote its content yourself. Set path__body only to move the script to a different path; omit it to leave the script where it is. A path that holds no script is refused: use createScript to create one. A new version generates its lock async and can take up to a minute before a run by path uses it rather than the previous one."), + path: Cow::Borrowed("/w/{workspace}/scripts/update/{path}"), + method: Cow::Borrowed("POST"), + path_params_schema: Some(serde_json::json!({ + "type": "object", + "properties": { + "path": { + "type": "string" + } + }, + "required": [ + "path" + ] +})), + query_params_schema: None, + body_schema: Some(serde_json::json!({ + "type": "object", + "properties": { + "summary": { + "type": "string" + }, + "description": { + "type": "string" + }, + "content": { + "type": "string" + }, + "language": { + "type": "string", + "description": "Possible values: python3, deno, go, bash, powershell, postgresql, mysql, bigquery, snowflake, mssql, oracledb, graphql, nativets, bun, php, rust, ansible, csharp, nu, java, ruby, rlang, duckdb, bunnative, dbt" + }, + "kind": { + "type": "string", + "description": "Possible values: script, failure, trigger, command, approval, preprocessor" + }, + "tag": { + "type": "string" + }, + "deployment_message": { + "type": "string" + }, + "path__body": { + "type": "string", + "description": "(body parameter). Defaults to `path` when omitted; set it only to change the path." + } + }, + "required": [ + "summary", + "content", + "language" + ], + "minProperties": 1 +})), + query_field_renames: None, + body_field_renames: Some(serde_json::json!({ + "path__body": "path" +})), + }, EndpointTool { name: Cow::Borrowed("deleteScriptByHash"), description: Cow::Borrowed("delete script by hash (erase content but keep hash, require admin)"), diff --git a/backend/windmill-api/src/mcp/utils.rs b/backend/windmill-api/src/mcp/utils.rs index 9802d7acab..94271bb662 100644 --- a/backend/windmill-api/src/mcp/utils.rs +++ b/backend/windmill-api/src/mcp/utils.rs @@ -531,14 +531,20 @@ fn assemble_request_body( let props = schema.get("properties")?.as_object()?; + // A null argument is how a client that must fill in every declared argument says + // "no value". Forwarding it would reach a field the API declares as a bare + // `String`, which rejects null outright rather than falling back to its default. let body_map: serde_json::Map = props .keys() .filter_map(|param_name| { - args_map.get(param_name).map(|value| { - // Use the original name as the key in the request body - let original_name = get_original_name(param_name, body_field_renames); - (original_name, value.clone()) - }) + args_map + .get(param_name) + .filter(|value| !value.is_null()) + .map(|value| { + // Use the original name as the key in the request body + let original_name = get_original_name(param_name, body_field_renames); + (original_name, value.clone()) + }) }) .collect(); @@ -875,6 +881,28 @@ mod tests { .unwrap_or_else(|| panic!("{name} must be a generated endpoint tool")) } + /// A script/flow tool the URL addresses by path, but `endpoint_path_policy` does + /// not name, falls through to no policy — which is no path confinement at all, so + /// a token scoped to `mcp:scripts:f/team/*` reaches every script through it. The + /// catalogue lives here and the policy in windmill-mcp, so neither crate notices + /// a tool added on one side and forgotten on the other; this is where they meet. + #[test] + fn every_path_addressed_script_or_flow_tool_is_path_confined() { + let unpoliced: Vec = crate::mcp::auto_generated_endpoints::all_tools() + .into_iter() + .filter(|t| { + (t.path.contains("/scripts/") || t.path.contains("/flows/")) + && t.path.contains("{path}") + && !windmill_mcp::server::has_endpoint_path_policy(&t.name) + }) + .map(|t| t.name.to_string()) + .collect(); + assert!( + unpoliced.is_empty(), + "path-addressed script/flow tools with no path policy: {unpoliced:?}" + ); + } + #[test] fn build_request_body_passthrough_forwards_script_args_minus_path() { // runScriptByPath-shaped body: additionalProperties, no declared props. @@ -1018,6 +1046,70 @@ mod tests { ); } + /// Neither script tool asks for a hash: `createScript` never has a parent, and + /// `updateScript` names the version it supersedes in its URL. A caller that must + /// fill in every declared argument has none to invent a value for, and the one it + /// invents anyway is not a field either tool declares, so it never reaches the API. + #[test] + fn script_tools_take_no_parent_hash() { + for name in ["createScript", "updateScript"] { + let tool = generated_tool(name); + let props = tool.body_schema.as_ref().unwrap()["properties"] + .as_object() + .unwrap(); + assert!( + !props.contains_key("parent_hash"), + "{name} must not ask for a parent_hash" + ); + } + + let body = build_request_body( + &generated_tool("createScript"), + &args_of(json!({ + "path": "u/admin/s", + "summary": "s", + "content": "export async function main() {}", + "language": "bun", + "parent_hash": "0000000000000000", + // A client that must fill in every argument says "no value" with null; + // forwarded, it would reach a field the API declares as a bare String. + "description": null, + })), + ) + .unwrap() + .expect("createScript body should be built"); + let obj = body.as_object().unwrap(); + + assert!(!obj.contains_key("parent_hash")); + assert!(!obj.contains_key("description")); + } + + /// The path an `updateScript` call omits is the one its URL already carries, so the + /// tool must not oblige an agent to restate it — a value that drifts from the URL's + /// moves the script instead of editing it. + #[test] + fn update_script_does_not_require_the_destination_path() { + let tool = generated_tool("updateScript"); + let body_schema = tool.body_schema.as_ref().unwrap(); + // Renamed off the URL's own `path` parameter, and mapped back on the way out. + assert!( + body_schema["properties"] + .as_object() + .unwrap() + .contains_key("path__body"), + "updateScript must still offer a destination path, which is what moves a script" + ); + assert!( + !body_schema["required"] + .as_array() + .unwrap() + .iter() + .any(|r| r == "path__body" || r == "path"), + "updateScript must not require the destination path, got: {}", + body_schema["required"] + ); + } + #[test] fn validate_path_param_value_accepts_legitimate_windmill_paths() { for ok in [ diff --git a/backend/windmill-mcp/src/server/mod.rs b/backend/windmill-mcp/src/server/mod.rs index fcc0a3618d..b6fb7a5b0a 100644 --- a/backend/windmill-mcp/src/server/mod.rs +++ b/backend/windmill-mcp/src/server/mod.rs @@ -17,7 +17,7 @@ pub use endpoints::{ endpoint_tool_to_mcp_tool, endpoint_tool_to_mcp_tool_multi, is_endpoint_read_only, list_workspaces_tool, non_empty_body_fields, EndpointTool, }; -pub use runner::Runner; +pub use runner::{has_endpoint_path_policy, Runner}; pub use tools::create_tool_from_item; // Re-export rmcp types for convenience diff --git a/backend/windmill-mcp/src/server/runner.rs b/backend/windmill-mcp/src/server/runner.rs index d891b737a7..f407504c4b 100644 --- a/backend/windmill-mcp/src/server/runner.rs +++ b/backend/windmill-mcp/src/server/runner.rs @@ -165,27 +165,58 @@ enum EndpointPathPolicy { /// Reads/writes the script/flow named by the listed path arguments. The /// endpoint scope grants the capability; when the token also carries path /// patterns for `kind`, every listed argument must match them. - PathArgs { kind: &'static str, fields: &'static [&'static str] }, + /// + /// `optional_fields` is for a destination path the tool lets the caller omit, where + /// absent means "leave it where it is": requiring it would refuse every call that + /// does not move anything. Absent or empty, it is filled in from the argument it + /// pairs with before the request is built, so the item stays where it is without + /// the endpoint having to accept a body that omits its own path. Names are the ones + /// the tool exposes, so a body path colliding with the URL's is `path__body` — the + /// URL path keeps the plain name (see `generate_mcp_tools.py`). + PathArgs { + kind: &'static str, + fields: &'static [&'static str], + /// `(destination, source)` pairs: a destination path the tool lets the caller + /// omit, and the argument it means when omitted. + optional_fields: &'static [(&'static str, &'static str)], + }, /// Affects scripts without taking a checkable path (delete-by-hash) or /// executes arbitrary code (preview). Unavailable to path-confined tokens — /// allowing these would bypass the path patterns entirely. Unconfinable(&'static str), } +/// Whether `endpoint_path_policy` confines this tool's paths. Public so the crate that +/// owns the generated endpoint catalogue can assert every path-addressed script/flow +/// tool is covered: one that is not falls through to no confinement at all. +pub fn has_endpoint_path_policy(endpoint_name: &str) -> bool { + endpoint_path_policy(endpoint_name).is_some() +} + fn endpoint_path_policy(endpoint_name: &str) -> Option { use EndpointPathPolicy::*; match endpoint_name { "runScriptByPath" => Some(RunByPath("script")), "runFlowByPath" => Some(RunByPath("flow")), "getScriptByPath" | "deleteScriptByPath" | "createScript" => { - Some(PathArgs { kind: "script", fields: &["path"] }) + Some(PathArgs { kind: "script", fields: &["path"], optional_fields: &[] }) } "getFlowByPath" | "deleteFlowByPath" | "createFlow" => { - Some(PathArgs { kind: "flow", fields: &["path"] }) + Some(PathArgs { kind: "flow", fields: &["path"], optional_fields: &[] }) } - // updateFlow addresses the flow via the URL path and can move it to the - // path given in the body — both must stay within scope. - "updateFlow" => Some(PathArgs { kind: "flow", fields: &["path__path", "path__body"] }), + // These address the item via the URL path and can move it to the path given + // in the body — both must stay within scope, and the body one only binds + // when supplied, since omitting it is how a caller updates in place. + "updateScript" => Some(PathArgs { + kind: "script", + fields: &["path"], + optional_fields: &[("path__body", "path")], + }), + "updateFlow" => Some(PathArgs { + kind: "flow", + fields: &["path"], + optional_fields: &[("path__body", "path")], + }), "deleteScriptByHash" | "runScriptPreviewAndWaitResult" => Some(Unconfinable("script")), _ => None, } @@ -257,9 +288,37 @@ fn endpoint_tool_in_scope( fn authorize_endpoint_call( scope_config: &crate::common::scope::McpScopeConfig, endpoint_tool: &EndpointTool, - args: &Value, + args: &mut Value, read_only: bool, ) -> Result<(), ErrorData> { + // A destination the caller omitted, or left empty because it had to fill in every + // argument, means "leave the item where it is". Resolved to the path the item is + // already at, so the endpoint receives a body naming its own path rather than one + // that omits it, and so `update_flow` cannot read the empty string as the empty + // path and move the flow there. The confinement below then sees a destination + // equal to the source, which it already has to allow. + if let Some(EndpointPathPolicy::PathArgs { optional_fields, .. }) = + endpoint_path_policy(&endpoint_tool.name) + { + for (destination, source) in optional_fields { + let unset = args + .get(destination) + .is_none_or(|v| !v.as_str().is_some_and(|s| !s.is_empty())); + let Some(current) = args + .get(source) + .and_then(|v| v.as_str()) + .map(str::to_string) + else { + continue; + }; + if unset { + if let Some(obj) = args.as_object_mut() { + obj.insert(destination.to_string(), Value::String(current)); + } + } + } + } + match endpoint_path_policy(&endpoint_tool.name) { Some(EndpointPathPolicy::RunByPath(kind)) => { let path = require_path_arg(endpoint_tool, args, "path")?; @@ -284,11 +343,19 @@ fn authorize_endpoint_call( )); } match policy { - Some(EndpointPathPolicy::PathArgs { kind, fields }) + Some(EndpointPathPolicy::PathArgs { kind, fields, optional_fields }) if path_confined(scope_config, kind) => { - for field in fields { - let path = require_path_arg(endpoint_tool, args, field)?; + // Filled in above when omitted, so each one names a real path. + let supplied = optional_fields + .iter() + .filter_map(|(field, _)| args.get(field).and_then(|v| v.as_str()).map(Ok)); + for path in fields + .iter() + .map(|field| require_path_arg(endpoint_tool, args, field)) + .chain(supplied) + { + let path = path?; if !scope_config.is_allowed(kind, path) { return Err(ErrorData::internal_error( format!("Access denied: {} '{}' not in token scope", kind, path), @@ -605,7 +672,8 @@ impl Runner { if endpoint_tool.name.as_ref() == name.as_ref() { // Authorize against the token's MCP scopes and read-only flag, // including the run-by-path path check (shared with multi mode). - authorize_endpoint_call(scope_config, endpoint_tool, &args, read_only)?; + let mut args = args; + authorize_endpoint_call(scope_config, endpoint_tool, &mut args, read_only)?; // This is an endpoint tool, call via backend. The backend's own error // code is kept: a client that retries an internal error would loop on @@ -788,7 +856,7 @@ impl Runner { scope_config: &crate::common::scope::McpScopeConfig, read_only: bool, name: std::borrow::Cow<'static, str>, - args: Value, + mut args: Value, ) -> Result { if name.as_ref() == "list_workspaces" { let workspaces = self @@ -823,7 +891,7 @@ impl Runner { // checked against the script/flow scope for that path — the endpoint // scope alone would let a granular token run items outside its allowed // paths. - authorize_endpoint_call(scope_config, endpoint_tool, &args, read_only)?; + authorize_endpoint_call(scope_config, endpoint_tool, &mut args, read_only)?; // Workspace-scoped endpoints need an explicit target workspace and a // per-workspace auth; global endpoints (e.g. docs) use the base identity. @@ -915,20 +983,26 @@ mod tests { ]); let tool = ep("runScriptByPath", "POST"); - assert!( - authorize_endpoint_call(&config, &tool, &json!({"path": "f/team/deploy"}), false) - .is_ok() - ); - assert!( - authorize_endpoint_call(&config, &tool, &json!({"path": "f/secret/admin"}), false) - .is_err() - ); + assert!(authorize_endpoint_call( + &config, + &tool, + &mut json!({"path": "f/team/deploy"}), + false + ) + .is_ok()); + assert!(authorize_endpoint_call( + &config, + &tool, + &mut json!({"path": "f/secret/admin"}), + false + ) + .is_err()); } #[test] fn run_by_path_call_requires_path_arg() { let tool = ep("runFlowByPath", "POST"); - assert!(authorize_endpoint_call(&cfg(&["mcp:all"]), &tool, &json!({}), false).is_err()); + assert!(authorize_endpoint_call(&cfg(&["mcp:all"]), &tool, &mut json!({}), false).is_err()); } #[test] @@ -938,14 +1012,14 @@ mod tests { assert!(authorize_endpoint_call( &config, &ep("runFlowByPath", "POST"), - &json!({"path": "f/team/x"}), + &mut json!({"path": "f/team/x"}), false ) .is_ok()); assert!(authorize_endpoint_call( &config, &ep("runScriptByPath", "POST"), - &json!({"path": "f/team/x"}), + &mut json!({"path": "f/team/x"}), false ) .is_err()); @@ -957,7 +1031,7 @@ mod tests { assert!(authorize_endpoint_call( &cfg(&["mcp:endpoints:getVariable"]), &get_var, - &json!({"path": "u/a/b"}), + &mut json!({"path": "u/a/b"}), false ) .is_ok()); @@ -965,12 +1039,14 @@ mod tests { assert!(authorize_endpoint_call( &cfg(&["mcp:scripts:f/team/*"]), &get_var, - &json!({"path": "u/a/b"}), + &mut json!({"path": "u/a/b"}), false ) .is_err()); // mcp:all (non-granular) allows any endpoint. - assert!(authorize_endpoint_call(&cfg(&["mcp:all"]), &get_var, &json!({}), false).is_ok()); + assert!( + authorize_endpoint_call(&cfg(&["mcp:all"]), &get_var, &mut json!({}), false).is_ok() + ); } #[test] @@ -979,14 +1055,14 @@ mod tests { assert!(authorize_endpoint_call( &cfg(&["mcp:all"]), &ep("createResource", "POST"), - &json!({}), + &mut json!({}), true ) .is_err()); assert!(authorize_endpoint_call( &cfg(&["mcp:all"]), &ep("getVariable", "GET"), - &json!({"path": "u/a/b"}), + &mut json!({"path": "u/a/b"}), true ) .is_ok()); @@ -1027,30 +1103,30 @@ mod tests { for name in ["getScriptByPath", "deleteScriptByPath", "createScript"] { let tool = ep(name, "POST"); assert!( - authorize_endpoint_call(&config, &tool, &json!({"path": "f/team/x"}), false) + authorize_endpoint_call(&config, &tool, &mut json!({"path": "f/team/x"}), false) .is_ok(), "{name} should allow in-scope path" ); assert!( - authorize_endpoint_call(&config, &tool, &json!({"path": "f/secret/x"}), false) + authorize_endpoint_call(&config, &tool, &mut json!({"path": "f/secret/x"}), false) .is_err(), "{name} should deny out-of-scope path" ); // Confinement can't be verified without the path argument. assert!( - authorize_endpoint_call(&config, &tool, &json!({}), false).is_err(), + authorize_endpoint_call(&config, &tool, &mut json!({}), false).is_err(), "{name} should require the path argument when confined" ); } for name in ["getFlowByPath", "deleteFlowByPath", "createFlow"] { let tool = ep(name, "POST"); assert!( - authorize_endpoint_call(&config, &tool, &json!({"path": "f/team/x"}), false) + authorize_endpoint_call(&config, &tool, &mut json!({"path": "f/team/x"}), false) .is_ok(), "{name} should allow in-scope path" ); assert!( - authorize_endpoint_call(&config, &tool, &json!({"path": "f/secret/x"}), false) + authorize_endpoint_call(&config, &tool, &mut json!({"path": "f/secret/x"}), false) .is_err(), "{name} should deny out-of-scope path" ); @@ -1066,7 +1142,7 @@ mod tests { assert!(authorize_endpoint_call( &config, &ep(name, "POST"), - &json!({"path": "f/anywhere/x"}), + &mut json!({"path": "f/anywhere/x"}), false ) .is_ok()); @@ -1076,7 +1152,7 @@ mod tests { assert!(authorize_endpoint_call( &star, &ep("createScript", "POST"), - &json!({"path": "f/anywhere/x"}), + &mut json!({"path": "f/anywhere/x"}), false ) .is_ok()); @@ -1091,7 +1167,7 @@ mod tests { assert!(authorize_endpoint_call( &config, &tool, - &json!({"path__path": "f/team/a", "path__body": "f/team/b"}), + &mut json!({"path": "f/team/a", "path__body": "f/team/b"}), false ) .is_ok()); @@ -1099,7 +1175,7 @@ mod tests { assert!(authorize_endpoint_call( &config, &tool, - &json!({"path__path": "f/team/a", "path__body": "f/secret/a"}), + &mut json!({"path": "f/team/a", "path__body": "f/secret/a"}), false ) .is_err()); @@ -1107,12 +1183,87 @@ mod tests { assert!(authorize_endpoint_call( &config, &tool, - &json!({"path__path": "f/secret/a", "path__body": "f/team/a"}), + &mut json!({"path": "f/secret/a", "path__body": "f/team/a"}), false ) .is_err()); } + // updateScript has updateFlow's shape, and the destination it can move a script + // to is optional: omitting it updates in place, so requiring it would refuse + // every call that moves nothing. + #[test] + fn update_script_confines_target_and_optional_destination() { + let config = cfg(&["mcp:scripts:f/team/*", "mcp:endpoints:*"]); + let tool = ep("updateScript", "POST"); + for mut args in [ + json!({"path": "f/team/a"}), + json!({"path": "f/team/a", "path__body": "f/team/b"}), + // Empty destination reads as absent, the same way the handler reads it. + json!({"path": "f/team/a", "path__body": ""}), + ] { + assert!( + authorize_endpoint_call(&config, &tool, &mut args, false).is_ok(), + "in-scope update should be allowed: {args}" + ); + } + for mut args in [ + // Deploying over a script outside the allowed folder. + json!({"path": "f/secret/a"}), + // Moving one out of it. + json!({"path": "f/team/a", "path__body": "f/secret/a"}), + ] { + assert!( + authorize_endpoint_call(&config, &tool, &mut args, false).is_err(), + "out-of-scope update should be denied: {args}" + ); + } + } + + /// A destination the caller omitted, or left empty because it had to fill in every + /// argument, is filled in with the path the item is already at. The endpoint then + /// never receives a body that omits its own path, and `update_flow` never receives + /// the empty string, which it would read as the empty path and move the flow there. + #[test] + fn an_unset_optional_destination_becomes_the_current_path() { + for tool in ["updateScript", "updateFlow"] { + let kind_scope = if tool == "updateScript" { + "mcp:scripts:f/team/*" + } else { + "mcp:flows:f/team/*" + }; + for mut args in [ + json!({"path": "f/team/a", "summary": "s"}), + json!({"path": "f/team/a", "path__body": "", "summary": "s"}), + json!({"path": "f/team/a", "path__body": null, "summary": "s"}), + ] { + authorize_endpoint_call( + &cfg(&[kind_scope, "mcp:endpoints:*"]), + &ep(tool, "POST"), + &mut args, + false, + ) + .unwrap_or_else(|e| panic!("{tool} should allow an unset destination: {e:?}")); + assert_eq!( + args, + json!({"path": "f/team/a", "path__body": "f/team/a", "summary": "s"}), + "{tool} should fill the destination in with the current path" + ); + } + } + + // A destination that names somewhere else is left for the confinement to check. + let mut args = json!({"path": "f/team/a", "path__body": "f/team/b"}); + authorize_endpoint_call( + &cfg(&["mcp:scripts:f/team/*", "mcp:endpoints:*"]), + &ep("updateScript", "POST"), + &mut args, + false, + ) + .unwrap(); + assert_eq!(args["path__body"], "f/team/b"); + } + // Tools that can't be path-checked (delete-by-hash) or execute arbitrary // code (preview) would bypass path confinement, so a path-confined token is // denied them entirely — and doesn't see them listed. @@ -1121,14 +1272,19 @@ mod tests { let confined = cfg(&["mcp:scripts:f/team/*", "mcp:endpoints:*"]); for name in ["deleteScriptByHash", "runScriptPreviewAndWaitResult"] { let tool = ep(name, "POST"); - assert!(authorize_endpoint_call(&confined, &tool, &json!({}), false).is_err()); + assert!(authorize_endpoint_call(&confined, &tool, &mut json!({}), false).is_err()); assert!(!endpoint_tool_in_scope(&confined, &tool)); // Without script path patterns the tools stay available. + assert!(authorize_endpoint_call( + &cfg(&["mcp:endpoints:*"]), + &tool, + &mut json!({}), + false + ) + .is_ok()); assert!( - authorize_endpoint_call(&cfg(&["mcp:endpoints:*"]), &tool, &json!({}), false) - .is_ok() + authorize_endpoint_call(&cfg(&["mcp:all"]), &tool, &mut json!({}), false).is_ok() ); - assert!(authorize_endpoint_call(&cfg(&["mcp:all"]), &tool, &json!({}), false).is_ok()); assert!(endpoint_tool_in_scope(&cfg(&["mcp:endpoints:*"]), &tool)); } // Flow-only confinement doesn't affect script-kind unconfinable tools. @@ -1136,7 +1292,7 @@ mod tests { assert!(authorize_endpoint_call( &flow_confined, &ep("deleteScriptByHash", "POST"), - &json!({}), + &mut json!({}), false ) .is_ok()); diff --git a/frontend/src/lib/components/mcp/endpointScopePolicy.test.ts b/frontend/src/lib/components/mcp/endpointScopePolicy.test.ts index e224279f4c..86832f6c44 100644 --- a/frontend/src/lib/components/mcp/endpointScopePolicy.test.ts +++ b/frontend/src/lib/components/mcp/endpointScopePolicy.test.ts @@ -1,9 +1,11 @@ import { describe, expect, it } from 'vitest' import { + endpointPathPolicy, isEndpointExposed, parseMcpScopeState, pruneEndpointSelection } from './endpointScopePolicy' +import { mcpEndpointTools } from '$lib/mcpEndpointTools' const state = (...scopes: string[]) => parseMcpScopeState(scopes) @@ -23,6 +25,17 @@ describe('isEndpointExposed', () => { expect(isEndpointExposed(state('mcp:all'), 'runScriptByPath')).toBe(true) }) + // A script/flow tool the URL addresses by path but the policy does not name falls + // through to "no policy", which is "not path-confined at all" — a scoped token then + // reaches every path of that kind. Catch the omission here rather than in a review. + it('gives every path-addressed script/flow tool a policy', () => { + const unpoliced = mcpEndpointTools + .filter((e) => /\/(scripts|flows)\//.test(e.path) && e.path.includes('{path}')) + .map((e) => e.name) + .filter((name) => endpointPathPolicy(name) === undefined) + expect(unpoliced).toEqual([]) + }) + it('withholds unconfinable tools from path-confined tokens', () => { const confined = state('mcp:scripts:f/team/*', 'mcp:endpoints:*') expect(isEndpointExposed(confined, 'runScriptPreviewAndWaitResult')).toBe(false) diff --git a/frontend/src/lib/components/mcp/endpointScopePolicy.ts b/frontend/src/lib/components/mcp/endpointScopePolicy.ts index 3c571a30d5..2b9fecd136 100644 --- a/frontend/src/lib/components/mcp/endpointScopePolicy.ts +++ b/frontend/src/lib/components/mcp/endpointScopePolicy.ts @@ -30,6 +30,7 @@ export function endpointPathPolicy(name: string): EndpointPathPolicy | undefined case 'getScriptByPath': case 'deleteScriptByPath': case 'createScript': + case 'updateScript': return { kind: 'pathArgs', resource: 'script' } case 'getFlowByPath': case 'deleteFlowByPath': diff --git a/frontend/src/lib/mcpEndpointTools.ts b/frontend/src/lib/mcpEndpointTools.ts index 2326e46f22..5896e9e1ce 100644 --- a/frontend/src/lib/mcpEndpointTools.ts +++ b/frontend/src/lib/mcpEndpointTools.ts @@ -687,8 +687,8 @@ export const mcpEndpointTools: EndpointTool[] = [ }, { name: "createScript", - description: "create script: Creates a new script when the path does not already exist.\nCreates a new version of an existing script when called with the same path and the current `parent_hash`", - instructions: "To create a NEW script, specify the path (e.g., 'f/my_folder/my_script'), the content (source code), and the language, and leave parent_hash unset. For TypeScript, use 'bun' unless deno-specific APIs are needed. To UPDATE an existing script, do NOT delete and recreate it: call this tool with the same path and set parent_hash to the script's current hash, which you can read from the `hash` field returned by getScriptByPath. This creates a new version while preserving the script's history.", + description: "create script: Creates a script at a path that does not already hold one", + instructions: "Specify the path (e.g., 'f/my_folder/my_script'), the content (source code), and the language. For TypeScript, use 'bun' unless deno-specific APIs are needed. A path that already holds a script is refused: use updateScript to deploy a new version of it, and do NOT delete and recreate a script to change it. A new version generates its lock async and can take up to a minute before a run by path uses it rather than the previous one.", path: "/w/{workspace}/scripts/create", method: "POST", pathParamsSchema: undefined, @@ -699,9 +699,6 @@ export const mcpEndpointTools: EndpointTool[] = [ "path": { "type": "string" }, - "parent_hash": { - "type": "string" - }, "summary": { "type": "string" }, @@ -737,6 +734,67 @@ export const mcpEndpointTools: EndpointTool[] = [ queryFieldRenames: undefined, bodyFieldRenames: undefined }, + { + name: "updateScript", + description: "update script: Deploys a new version of the script at `path`, which must already hold one.\nThe body's `path` is the destination: the same path leaves the script where it\nis, a different one moves it there and archives the old path", + instructions: "Deploys a new version of an existing script, preserving its history, so do NOT delete and recreate a script to change it. Send the whole script, not a patch: read the current one with getScriptByPath first, unless you wrote its content yourself. Set path__body only to move the script to a different path; omit it to leave the script where it is. A path that holds no script is refused: use createScript to create one. A new version generates its lock async and can take up to a minute before a run by path uses it rather than the previous one.", + path: "/w/{workspace}/scripts/update/{path}", + method: "POST", + pathParamsSchema: { + "type": "object", + "properties": { + "path": { + "type": "string" + } + }, + "required": [ + "path" + ] +}, + queryParamsSchema: undefined, + bodySchema: { + "type": "object", + "properties": { + "summary": { + "type": "string" + }, + "description": { + "type": "string" + }, + "content": { + "type": "string" + }, + "language": { + "type": "string", + "description": "Possible values: python3, deno, go, bash, powershell, postgresql, mysql, bigquery, snowflake, mssql, oracledb, graphql, nativets, bun, php, rust, ansible, csharp, nu, java, ruby, rlang, duckdb, bunnative, dbt" + }, + "kind": { + "type": "string", + "description": "Possible values: script, failure, trigger, command, approval, preprocessor" + }, + "tag": { + "type": "string" + }, + "deployment_message": { + "type": "string" + }, + "path__body": { + "type": "string", + "description": "(body parameter). Defaults to `path` when omitted; set it only to change the path." + } + }, + "required": [ + "summary", + "content", + "language" + ], + "minProperties": 1 +}, + queryFieldRenames: undefined, + bodyFieldRenames: { + "path__body": "path" +} + }, { name: "deleteScriptByHash", description: "delete script by hash (erase content but keep hash, require admin)", From 28b2ca63672c916e07bd028eab07728d1aa4f0fe Mon Sep 17 00:00:00 2001 From: Guilhem Date: Fri, 21 Aug 2026 10:40:34 +0200 Subject: [PATCH 18/48] feat: inline login errors and a narrower single-column login card (#10777) * feat: inline login errors and a narrower single-column login card Co-Authored-By: Claude Opus 5 (1M context) * fix: address login review findings (overflow, error leak, a11y, dev gate) Co-Authored-By: Claude Opus 5 (1M context) * fix: scope login form ids per instance Co-Authored-By: Claude Opus 5 (1M context) * fix: drop the duplicate dark mode toggle and tighten the login heading gap Co-Authored-By: Claude Opus 5 (1M context) * fix: replay the login shake on every retry, not just the first Co-Authored-By: Claude Opus 5 (1M context) * fix: address standards and spec review findings on the login page Co-Authored-By: Claude Opus 5 (1M context) * fix: put the login error under the field it is about Co-Authored-By: Claude Opus 5 (1M context) * fix: attribute a login failure to the credentials it was sent with Co-Authored-By: Claude Opus 5 (1M context) * fix: hide the third-party toggle once the password form is open Co-Authored-By: Claude Opus 5 (1M context) * feat: brand the logged-out pages from one top header instead of a centered logo Co-Authored-By: Claude Opus 5 (1M context) * feat: remember the login method that last worked on this browser Co-Authored-By: Claude Opus 5 (1M context) * feat: lead the login card with the last used method and anchor its layout Co-Authored-By: Claude Opus 5 (1M context) * fix: address review round findings on the login card Co-Authored-By: Claude Opus 5 (1M context) * fix: key third-party buttons by method kind and drop a history comment Co-Authored-By: Claude Opus 5 (1M context) --------- Co-authored-by: Claude Opus 5 (1M context) Co-authored-by: Ruben Fiszel --- .../src/lib/components/CenteredModal.svelte | 24 +- frontend/src/lib/components/Login.svelte | 411 ++++++++++++++---- .../src/lib/components/LoginHeading.svelte | 28 ++ .../src/lib/components/LoginPageHeader.svelte | 29 +- frontend/src/lib/components/Password.svelte | 15 +- .../lib/components/icons/GitlabIcon.svelte | 20 +- .../lib/components/icons/MicrosoftIcon.svelte | 20 +- .../lib/components/icons/brands/Auth0.svelte | 24 - .../lib/components/icons/brands/Github.svelte | 23 - .../lib/components/icons/brands/Gitlab.svelte | 23 - .../lib/components/icons/brands/Google.svelte | 23 - .../components/icons/brands/Microsoft.svelte | 23 - .../lib/components/icons/brands/Okta.svelte | 26 -- frontend/src/lib/components/icons/index.ts | 6 + frontend/src/lib/lastLoginMethod.test.ts | 47 ++ frontend/src/lib/lastLoginMethod.ts | 61 +++ frontend/src/lib/loginError.test.ts | 40 ++ frontend/src/lib/loginError.ts | 24 + .../src/routes/(root)/(logged)/+layout.svelte | 4 + .../(logged)/user/(user)/login/+page.svelte | 53 ++- .../approve/[workspace]/[job]/+page.svelte | 1 - .../[job]/[resume]/[hmac]/+page.svelte | 2 +- .../routes/kitchen_sink/login/+page.svelte | 210 +++++++++ .../routes/user/forgot-password/+page.svelte | 21 +- .../routes/user/reset-password/+page.svelte | 21 +- frontend/tailwind.config.cjs | 7 +- 26 files changed, 866 insertions(+), 320 deletions(-) create mode 100644 frontend/src/lib/components/LoginHeading.svelte delete mode 100644 frontend/src/lib/components/icons/brands/Auth0.svelte delete mode 100644 frontend/src/lib/components/icons/brands/Github.svelte delete mode 100644 frontend/src/lib/components/icons/brands/Gitlab.svelte delete mode 100644 frontend/src/lib/components/icons/brands/Google.svelte delete mode 100644 frontend/src/lib/components/icons/brands/Microsoft.svelte delete mode 100644 frontend/src/lib/components/icons/brands/Okta.svelte create mode 100644 frontend/src/lib/lastLoginMethod.test.ts create mode 100644 frontend/src/lib/lastLoginMethod.ts create mode 100644 frontend/src/lib/loginError.test.ts create mode 100644 frontend/src/lib/loginError.ts create mode 100644 frontend/src/routes/kitchen_sink/login/+page.svelte diff --git a/frontend/src/lib/components/CenteredModal.svelte b/frontend/src/lib/components/CenteredModal.svelte index a72df74002..e3f8ea1604 100644 --- a/frontend/src/lib/components/CenteredModal.svelte +++ b/frontend/src/lib/components/CenteredModal.svelte @@ -1,14 +1,12 @@ + -
- {#if autoRedirecting} -

Signing you in…

- {/if} -
+ +{#snippet errorMessage()} + +{/snippet} + + +{#snippet lastUsedBadge()} + +
+ + Last used + +
+{/snippet} + +{#snippet providerButtons()} +
{#if !logins} {#each Array(4) as _} {/each} {:else} - {#each providers as { type, icon }} - {#if logins?.some((login) => login.type === type)} + {#each orderedThirdParty as entry (entry.method.kind === 'saml' ? 'saml:' : `oauth:${entry.method.provider}`)} +
+ {#if sameLoginMethod(lastUsed, entry.method)} + {@render lastUsedBadge()} + {/if} - {/if} +
{/each} - {#each logins.filter((login) => !providersType?.includes(login.type)) as login} - - {/each} - {/if} - {#if saml} - {/if}
- {#if !autoRedirecting && !disablePasswordLogin && (saml || (logins && logins.length > 0))} -
0 ? 'mt-6' : '')}> - -
+{/snippet} + +{#snippet orDivider()} +
+
+ or +
+
+{/snippet} + +
+ {#if autoRedirecting} +

Signing you in…

+ {/if} + + {#if !passwordFirst} + {@render providerButtons()} + {#if !autoRedirecting && !disablePasswordLogin && (saml || (logins && logins.length > 0))} + {@render orDivider()} + + {#if !showPassword} +
+ +
+ {/if} + {/if} {/if} {#if !autoRedirecting && showPassword && !disablePasswordLogin} @@ -525,51 +725,73 @@ Welcome! Default credentials admin@windmill.dev / changeme have been prefilled.

{/if} -
- {#if isCloudHosted()} +
+ {#if cloudHosted}

To get credentials without the OAuth providers above, send an email at contact@windmill.dev

{/if} -
- -
- { - // Only move on once the field holds something: while the browser's - // credential dropdown is open, Enter belongs to the dropdown - if (e.key === 'Enter' && !e.isComposing && !e.repeat && e.currentTarget.value) { - e.preventDefault() - passwordField?.focus() +
+
+ +
+ { + // Only move on once the field holds something: while the browser's + // credential dropdown is open, Enter belongs to the dropdown + if (e.key === 'Enter' && !e.isComposing && !e.repeat && e.currentTarget.value) { + e.preventDefault() + passwordField?.focus() + } } - } - }} - /> + }} + /> +
+ +
+ +
+ +
+ +
+ {@render errorMessage()}
-
- -
- -
+ {/if} + + {#if passwordFirst && (saml || (logins && logins.length > 0))} + {@render orDivider()} + {@render providerButtons()} + {/if}
diff --git a/frontend/src/lib/components/LoginHeading.svelte b/frontend/src/lib/components/LoginHeading.svelte new file mode 100644 index 0000000000..478a5e3a53 --- /dev/null +++ b/frontend/src/lib/components/LoginHeading.svelte @@ -0,0 +1,28 @@ + + + +
+ {#if hasThirdParty !== undefined} +

+ {hasThirdParty ? `Log in or sign up to ${instanceName}` : `Log in to ${instanceName}`} +

+

+ {hasThirdParty + ? 'Log in or sign up with any of the methods below' + : 'Log in with your email and password'} +

+ {/if} +
diff --git a/frontend/src/lib/components/LoginPageHeader.svelte b/frontend/src/lib/components/LoginPageHeader.svelte index d841f48aa1..7b49c97e34 100644 --- a/frontend/src/lib/components/LoginPageHeader.svelte +++ b/frontend/src/lib/components/LoginPageHeader.svelte @@ -1,11 +1,34 @@ - -
-
+
+ +
+ {#if showBrand} + {#if $whitelabelNameStore} + {capitalize($whitelabelNameStore)} + {:else} + + Windmill + {/if} + {/if} +
+ +
diff --git a/frontend/src/lib/components/Password.svelte b/frontend/src/lib/components/Password.svelte index 72e7e43024..569150f7b4 100644 --- a/frontend/src/lib/components/Password.svelte +++ b/frontend/src/lib/components/Password.svelte @@ -18,6 +18,10 @@ autocomplete?: HTMLInputAttributes['autocomplete'] /** Off for login-style fields: keeps Enter free to submit. Overrides `minRows`. */ allowMultiline?: boolean + /** Renders the field in its error state; the message itself is the caller's to display. */ + error?: boolean + /** id of the element holding that message, wired up as aria-describedby. */ + describedBy?: string onKeyDown?: (event: KeyboardEvent) => void onBlur?: (event: FocusEvent) => void } @@ -32,11 +36,14 @@ id, autocomplete = 'new-password', allowMultiline = true, + error = false, + describedBy = undefined, onKeyDown, onBlur }: Props = $props() let red = $derived(required && (password == '' || password == undefined)) + let hasError = $derived(red || error) let hideValue = $state(true) let forceMultiline = $state(false) let isMultiline = $derived( @@ -76,7 +83,7 @@ onBlur?.(e), onkeydown: (e) => { onKeyDown?.(e) @@ -99,13 +108,15 @@ onBlur?.(e), onkeydown: (e) => { if (allowMultiline && e.key === 'Enter') { diff --git a/frontend/src/lib/components/icons/GitlabIcon.svelte b/frontend/src/lib/components/icons/GitlabIcon.svelte index 88fa898a3d..d8aaf12e0e 100644 --- a/frontend/src/lib/components/icons/GitlabIcon.svelte +++ b/frontend/src/lib/components/icons/GitlabIcon.svelte @@ -1,16 +1,30 @@ + import { twMerge } from 'tailwind-merge' + interface Props { - height?: string - width?: string + size?: number + height?: number + width?: number + class?: string } - let { height = '24px', width = '24px' }: Props = $props() + let { + size = undefined, + height: heightProp = 24, + width: widthProp = 24, + class: clazz = '' + }: Props = $props() + + const { width, height } = $derived( + size ? { width: size, height: size } : { width: widthProp, height: heightProp } + ) - interface Props { - size?: number - style?: string - class?: string - } - - let { size = 16, style = 'fill: white;', class: clazz = '' }: Props = $props() - - - - auth0-svg - - diff --git a/frontend/src/lib/components/icons/brands/Github.svelte b/frontend/src/lib/components/icons/brands/Github.svelte deleted file mode 100644 index 35d8319f7f..0000000000 --- a/frontend/src/lib/components/icons/brands/Github.svelte +++ /dev/null @@ -1,23 +0,0 @@ - - - - - - diff --git a/frontend/src/lib/components/icons/brands/Gitlab.svelte b/frontend/src/lib/components/icons/brands/Gitlab.svelte deleted file mode 100644 index 8b4a1e828f..0000000000 --- a/frontend/src/lib/components/icons/brands/Gitlab.svelte +++ /dev/null @@ -1,23 +0,0 @@ - - - - - - diff --git a/frontend/src/lib/components/icons/brands/Google.svelte b/frontend/src/lib/components/icons/brands/Google.svelte deleted file mode 100644 index 9798da0c36..0000000000 --- a/frontend/src/lib/components/icons/brands/Google.svelte +++ /dev/null @@ -1,23 +0,0 @@ - - - - - - diff --git a/frontend/src/lib/components/icons/brands/Microsoft.svelte b/frontend/src/lib/components/icons/brands/Microsoft.svelte deleted file mode 100644 index 3c4acbb9ec..0000000000 --- a/frontend/src/lib/components/icons/brands/Microsoft.svelte +++ /dev/null @@ -1,23 +0,0 @@ - - - - > - - diff --git a/frontend/src/lib/components/icons/brands/Okta.svelte b/frontend/src/lib/components/icons/brands/Okta.svelte deleted file mode 100644 index 469b60d76c..0000000000 --- a/frontend/src/lib/components/icons/brands/Okta.svelte +++ /dev/null @@ -1,26 +0,0 @@ - - - - oktaddd-svg - - diff --git a/frontend/src/lib/components/icons/index.ts b/frontend/src/lib/components/icons/index.ts index d5a7f1df4a..825f671e43 100644 --- a/frontend/src/lib/components/icons/index.ts +++ b/frontend/src/lib/components/icons/index.ts @@ -337,6 +337,12 @@ import type { Component } from 'svelte' * Most variants are the pre-audit artwork painted with currentColor. GoogleCloudIcon is * greyscale instead — four grey tones rather than one flat colour — because Google's * cloud loses its internal shape when flattened. If the artwork changes, change both. + * + * Brand marks carry a viewBox that centres the artwork in a box 24/22 of its bounding + * size, so the mark occupies the same safe area a lucide glyph does on its 24 grid. + * Vendor SVGs come with whatever padding the vendor chose — none for Google, 10% for + * GitHub — so a raw viewBox makes them render at visibly different sizes from each other + * and from the lucide icons beside them. Re-derive the viewBox when replacing artwork. */ export const APP_TO_ICON_COMPONENT = { diff --git a/frontend/src/lib/lastLoginMethod.test.ts b/frontend/src/lib/lastLoginMethod.test.ts new file mode 100644 index 0000000000..ffe1714f93 --- /dev/null +++ b/frontend/src/lib/lastLoginMethod.test.ts @@ -0,0 +1,47 @@ +import { describe, it, expect, beforeEach } from 'vitest' +import { + clearPendingLoginMethod, + confirmPendingLoginMethod, + getLastLoginMethod, + markLoginMethodPending, + rememberLoginMethod +} from './lastLoginMethod' + +describe('lastLoginMethod', () => { + beforeEach(() => localStorage.clear()) + + it('only promotes a pending method when told a session exists', () => { + markLoginMethodPending({ kind: 'oauth', provider: 'gitlab' }) + expect(getLastLoginMethod()).toBeUndefined() + + confirmPendingLoginMethod() + expect(getLastLoginMethod()).toEqual({ kind: 'oauth', provider: 'gitlab' }) + + // the pending slot is spent, so a later confirm cannot re-promote it + localStorage.removeItem('lastLoginMethod') + confirmPendingLoginMethod() + expect(getLastLoginMethod()).toBeUndefined() + }) + + it('forgets a pending method that never became a login', () => { + rememberLoginMethod({ kind: 'password' }) + markLoginMethodPending({ kind: 'oauth', provider: 'github' }) + clearPendingLoginMethod() + + confirmPendingLoginMethod() + expect(getLastLoginMethod()).toEqual({ kind: 'password' }) + }) + + it('ignores stored values it does not recognise', () => { + for (const stored of [ + 'not json', + '{}', + '"password"', + '{"kind":"oauth"}', + '{"kind":"carrier"}' + ]) { + localStorage.setItem('lastLoginMethod', stored) + expect(getLastLoginMethod()).toBeUndefined() + } + }) +}) diff --git a/frontend/src/lib/lastLoginMethod.ts b/frontend/src/lib/lastLoginMethod.ts new file mode 100644 index 0000000000..db1010911b --- /dev/null +++ b/frontend/src/lib/lastLoginMethod.ts @@ -0,0 +1,61 @@ +// The login method that last worked on this browser, so the card can put it first and badge it. +// Purely a hint: it is never read for anything but ordering and a label. +export type LastLoginMethod = + | { kind: 'password' } + | { kind: 'oauth'; provider: string } + | { kind: 'saml' } + +const CONFIRMED_KEY = 'lastLoginMethod' +// OAuth and SAML leave the page before the outcome is known, so the method is parked here and +// only promoted once a session exists — otherwise an abandoned provider would claim the badge. +const PENDING_KEY = 'lastLoginMethodPending' + +function read(key: string): LastLoginMethod | undefined { + try { + const raw = localStorage.getItem(key) + if (!raw) return undefined + const parsed = JSON.parse(raw) + if (parsed?.kind === 'password' || parsed?.kind === 'saml') return parsed + if (parsed?.kind === 'oauth' && typeof parsed.provider === 'string') return parsed + return undefined + } catch { + return undefined + } +} + +function write(key: string, method: LastLoginMethod | undefined) { + try { + if (method) localStorage.setItem(key, JSON.stringify(method)) + else localStorage.removeItem(key) + } catch (e) { + console.error('Could not record the last login method', e) + } +} + +export function getLastLoginMethod(): LastLoginMethod | undefined { + return read(CONFIRMED_KEY) +} + +export function rememberLoginMethod(method: LastLoginMethod) { + write(CONFIRMED_KEY, method) + write(PENDING_KEY, undefined) +} + +export function markLoginMethodPending(method: LastLoginMethod) { + write(PENDING_KEY, method) +} + +export function clearPendingLoginMethod() { + write(PENDING_KEY, undefined) +} + +/** Call only where a session is proven: whatever redirect was in flight is what worked. */ +export function confirmPendingLoginMethod() { + const pending = read(PENDING_KEY) + if (pending) rememberLoginMethod(pending) +} + +export function sameLoginMethod(a: LastLoginMethod | undefined, b: LastLoginMethod): boolean { + if (!a || a.kind !== b.kind) return false + return a.kind !== 'oauth' || a.provider === (b as { provider: string }).provider +} diff --git a/frontend/src/lib/loginError.test.ts b/frontend/src/lib/loginError.test.ts new file mode 100644 index 0000000000..cf76dca5ec --- /dev/null +++ b/frontend/src/lib/loginError.test.ts @@ -0,0 +1,40 @@ +import { describe, it, expect } from 'vitest' +import { loginErrorMessage } from './loginError' + +describe('loginErrorMessage', () => { + it('maps the backend rejection to one message for a wrong email and a wrong password', () => { + expect(loginErrorMessage({ status: 400, body: 'Bad request: Invalid login' })).toBe( + 'Invalid email or password.' + ) + }) + + it('surfaces the messages the endpoint is known to produce', () => { + expect( + loginErrorMessage({ + status: 400, + body: 'Bad request: Password login is disabled on this instance' + }) + ).toBe('Password login is disabled on this instance') + // The rate limiter's own wording is not echoed back: a 429 always gets this sentence. + expect(loginErrorMessage({ status: 429, body: 'Bad request: slow down' })).toBe( + 'Too many login attempts. Please try again later.' + ) + }) + + it('never surfaces server text it does not recognise, to an unauthenticated visitor', () => { + const sqlError = { + status: 400, + body: 'Bad request: SqlErr: error returned from database: relation "password" does not exist @backend/windmill-api-users/src/users.rs:123' + } + expect(loginErrorMessage(sqlError)).toBe('Could not sign you in. Please try again.') + expect( + loginErrorMessage({ status: 502, body: '502 Bad Gateway' }) + ).toBe('Could not sign you in. Please try again.') + expect(loginErrorMessage(new TypeError('Failed to fetch'))).toBe( + 'Could not sign you in. Please try again.' + ) + expect(loginErrorMessage({ status: 400, body: { error: { message: { nested: true } } } })).toBe( + 'Could not sign you in. Please try again.' + ) + }) +}) diff --git a/frontend/src/lib/loginError.ts b/frontend/src/lib/loginError.ts new file mode 100644 index 0000000000..d26f44abd6 --- /dev/null +++ b/frontend/src/lib/loginError.ts @@ -0,0 +1,24 @@ +// Only messages the login endpoint is known to produce are shown. Anything else — a SQL error +// (which the API also returns as a 400, with the query and a source location), a proxy's HTML +// error page — would otherwise be printed verbatim to an unauthenticated visitor. +const KNOWN_LOGIN_ERRORS = ['Password login is disabled on this instance'] + +const GENERIC_LOGIN_ERROR = 'Could not sign you in. Please try again.' + +export function loginErrorMessage(err: any): string { + // The API returns errors as plain text, prefixed by their class (e.g. "Bad request: Invalid + // login"); ApiError.message is only the HTTP status text. + const raw = typeof err?.body === 'string' ? err.body : err?.body?.error?.message + const body = typeof raw === 'string' ? raw : '' + const detail = body.replace(/^(Bad request|Internal|Error): /, '').trim() + if (detail === 'Invalid login') { + return 'Invalid email or password.' + } + if (err?.status === 429) { + return 'Too many login attempts. Please try again later.' + } + if (KNOWN_LOGIN_ERRORS.includes(detail)) { + return detail + } + return GENERIC_LOGIN_ERROR +} diff --git a/frontend/src/routes/(root)/(logged)/+layout.svelte b/frontend/src/routes/(root)/(logged)/+layout.svelte index e55cfd54a9..257af0fbbd 100644 --- a/frontend/src/routes/(root)/(logged)/+layout.svelte +++ b/frontend/src/routes/(root)/(logged)/+layout.svelte @@ -63,6 +63,7 @@ import { syncTutorialsTodos } from '$lib/tutorialUtils' import { PanelLeftClose, PanelLeftOpen, Home, Play, Search, WandSparkles } from 'lucide-svelte' import { getUserExt } from '$lib/user' + import { confirmPendingLoginMethod } from '$lib/lastLoginMethod' import { deepEqual } from 'fast-equals' import { twMerge } from 'tailwind-merge' import OperatorMenu from '$lib/components/sidebar/OperatorMenu.svelte' @@ -315,6 +316,9 @@ } } const user = await getUserExt(workspace) + // getUserExt resolves to undefined on failure, so a user is the only proof of a + // session: without it a cancelled SSO round trip would claim the "Last used" badge. + if (user) confirmPendingLoginMethod() // Every workspace change starts a fetch without cancelling the one before it, // so a slow response can land after a faster one for the workspace the user // has since moved to. The store must describe the active workspace: letting a diff --git a/frontend/src/routes/(root)/(logged)/user/(user)/login/+page.svelte b/frontend/src/routes/(root)/(logged)/user/(user)/login/+page.svelte index 2ea7ab1c89..d7520bd873 100644 --- a/frontend/src/routes/(root)/(logged)/user/(user)/login/+page.svelte +++ b/frontend/src/routes/(root)/(logged)/user/(user)/login/+page.svelte @@ -10,17 +10,18 @@ enterpriseLicense, whitelabelNameStore } from '$lib/stores' - import { classNames, emptyString, parseQueryParams } from '$lib/utils' + import { emptyString, parseQueryParams } from '$lib/utils' import { getUserExt } from '$lib/user' - import { WindmillIcon } from '$lib/components/icons' import LoginPageHeader from '$lib/components/LoginPageHeader.svelte' - import DarkModeToggle from '$lib/components/sidebar/DarkModeToggle.svelte' + import { WindmillIcon } from '$lib/components/icons' import { clearStores } from '$lib/storeUtils' import { setLicense } from '$lib/enterpriseUtils' import Login from '$lib/components/Login.svelte' + import LoginHeading from '$lib/components/LoginHeading.svelte' import { onMount } from 'svelte' import { refreshSuperadmin } from '$lib/refreshUser' import { isValidLogoutRedirect, toSameOriginRelativePath } from '$lib/logoutRedirect' + import { confirmPendingLoginMethod } from '$lib/lastLoginMethod' const email = page.url.searchParams.get('email') ?? '' const password = page.url.searchParams.get('password') ?? '' @@ -37,8 +38,10 @@ const sameOriginRd = toSameOriginRelativePath(rawRd) const rd = sameOriginRd ?? rawRd - let showPassword = false let firstTime = $state(false) + // A third-party login creates the account on first use, so the page only offers sign-up + // once the instance has one configured. undefined until the card reports what it loaded. + let hasThirdParty = $state(undefined) function clearWindmillCloudCookies() { const domain = window.location.hostname @@ -118,6 +121,8 @@ async function redirectIfNecessary() { await UserService.getCurrentEmail() + // Reached only with a session: an SSO round trip that landed back here worked. + confirmPendingLoginMethod() redirectUser() } @@ -134,30 +139,32 @@ } -
- -
+ +
+ + +
{#if !$enterpriseLicense || !$whitelabelNameStore} - + {/if}
-

- Log in or sign up -

-

- Log in or sign up with any of the methods below -

+
+ +
-
-
- -
- +
+ (hasThirdParty = options.hasThirdParty)} + />
diff --git a/frontend/src/routes/approve/[workspace]/[job]/+page.svelte b/frontend/src/routes/approve/[workspace]/[job]/+page.svelte index 6b88204c08..34e704d6fa 100644 --- a/frontend/src/routes/approve/[workspace]/[job]/+page.svelte +++ b/frontend/src/routes/approve/[workspace]/[job]/+page.svelte @@ -212,7 +212,6 @@ {#if error} diff --git a/frontend/src/routes/approve/[workspace]/[job]/[resume]/[hmac]/+page.svelte b/frontend/src/routes/approve/[workspace]/[job]/[resume]/[hmac]/+page.svelte index 5a2f0b4746..18fe489a9a 100644 --- a/frontend/src/routes/approve/[workspace]/[job]/[resume]/[hmac]/+page.svelte +++ b/frontend/src/routes/approve/[workspace]/[job]/[resume]/[hmac]/+page.svelte @@ -173,7 +173,7 @@ - + {#if error}
{#if error.startsWith('Not authorized:')} diff --git a/frontend/src/routes/kitchen_sink/login/+page.svelte b/frontend/src/routes/kitchen_sink/login/+page.svelte new file mode 100644 index 0000000000..0c51f65bec --- /dev/null +++ b/frontend/src/routes/kitchen_sink/login/+page.svelte @@ -0,0 +1,210 @@ + + +{#if enabled} +
+
+
+

Login page states

+

+ Real cards, fixed instance config, no API calls. Sign in always fails so the error state + is one click away (click twice for the shake). +

+
+
+
Card width
+ +
+ + {#snippet children({ item })} + + {#if scope !== 'self'} + + {/if} + + {/snippet} + +
+ {#if !usage.loading && !usage.error && rows.length > 0} +
+ + {priced.total === 0 && priced.hasUnpriced + ? '—' + : `${totalIsEstimated ? '~' : ''}${formatUsd(priced.total)}`} + + + {usage.current?.truncated ? 'across the rows below' : 'total'}{priced.hasUnpriced + ? ' (partial)' + : ''} + +
+ {/if} +
+ + {#if usage.loading} +

Loading…

+ {:else if usage.error} +

{usageError(usage.error)}

+ {:else if rows.length === 0} +

No AI usage recorded in this period.

+ {:else} + {#if usage.current?.truncated} +

+ More rows matched than are shown; the highest-volume ones are listed. Narrow the range + or group differently to see the rest. +

+ {/if} + + + + {groupBy} + In + Out + Requests + + + Cost + + {#snippet text()} + A cost marked ~ is estimated from this workspace's model rates. A cost + without one was returned by the provider's API for those requests, and is used + as is. "no rate" means the model has no price set, so its spend stays out of + the total. + {/snippet} + + + + + + + {#each rows as row (row.key)} + + {row.key} + {formatTokenCount(row.tokensIn)} + {formatTokenCount(row.tokensOut)} + {row.requests} + + {row.cost === undefined + ? 'no rate' + : `${row.reported ? '' : '~'}${formatUsd(row.cost)}`} + + + {/each} + + + {/if} +
+ diff --git a/frontend/src/lib/components/workspaceSettings/ModelPricing.svelte b/frontend/src/lib/components/workspaceSettings/ModelPricing.svelte new file mode 100644 index 0000000000..85824c1c95 --- /dev/null +++ b/frontend/src/lib/components/workspaceSettings/ModelPricing.svelte @@ -0,0 +1,252 @@ + + +{#if Object.keys(aiProviders).length > 0} + +
+ {#each Object.entries(modelsByProvider).filter(([_, models]) => models.length > 0) as [provider, models]} + {@const isExpanded = !collapsedProviders[provider]} +
+ + + {#if isExpanded} +
+
+ {#each models as { model }} + {@const key = modelKey(provider as AIProvider, model)} + {@const rates = currentRates(provider as AIProvider, model)} + {@const overridden = isOverridden(provider as AIProvider, model)} +
+
+ +
+ {model} +
+
+ {#each RATE_FIELDS as field} +
+ + {field.replace('_', ' ')} + +
+ { + if (e.currentTarget.value === '') { + if (field === 'cache_read' || field === 'cache_write') { + clearCacheRate(provider as AIProvider, model, field) + } + return + } + const value = parseFloat(e.currentTarget.value) + if (!isNaN(value)) { + updateRate(provider as AIProvider, model, field, value) + } + }, + onblur: (e: Event & { currentTarget: HTMLInputElement }) => { + // Resync a field the state refused, so what is shown is what is stored. + const stored = currentRates(provider as AIProvider, model)?.[ + field + ] + e.currentTarget.value = + stored === undefined ? '' : String(stored) + errors[key] = '' + } + }} + /> +
+
+ {/each} + $ / 1M +
+
+ {#if overridden} +
+ Overriding the built-in price + +
+ {/if} + {#if errors[key]} +
{errors[key]}
+ {/if} +
+ {/each} +
+
+ {/if} +
+ {/each} +
+ +{/if} diff --git a/frontend/src/lib/utils/aiUsageReporter.ts b/frontend/src/lib/utils/aiUsageReporter.ts new file mode 100644 index 0000000000..b0457541d9 --- /dev/null +++ b/frontend/src/lib/utils/aiUsageReporter.ts @@ -0,0 +1,154 @@ +import { get } from 'svelte/store' +import { OpenAPI } from '$lib/gen' +import { workspaceStore } from '$lib/stores' + +// Per-workspace AI token spend, batched into the backend `ai_token_usage` +// accumulator that powers the workspace and per-user usage views. +// +// Deliberately separate from `featureUsage.ts`: that buffer carries anonymous +// product telemetry that leaves the instance, and its events must not identify a +// user. These events are attributed to the caller (server-side, from the session) +// and never leave the instance, so the two must not share a transport. +// +// Only token counts are sent. Money is derived when the usage is read, from the +// price table plus the workspace's overrides, so correcting a rate also corrects +// history. The one exception is a cost the provider itself billed back. + +export interface AiUsageEvent { + provider: string + model: string + /** Empty for chats not attached to an AI session. */ + sessionId?: string + inputTokens: number + cacheReadTokens: number + cacheWriteTokens: number + outputTokens: number + /** Set only where the provider reports what it actually charged, in USD. */ + costUsd?: number + /** Workspace whose API route carries the batch; defaults to the active workspace. */ + workspace?: string +} + +interface AiUsageEventPayload { + provider: string + model: string + session_id: string + input_tokens: number + cache_read_tokens: number + cache_write_tokens: number + output_tokens: number + reported_cost_nano_usd?: number + requests: number +} + +const FLUSH_INTERVAL_MS = 15_000 +// Backend caps a batch at 50 events; chunk larger flushes. +const MAX_EVENTS_PER_REQUEST = 50 + +const NANO_USD_PER_USD = 1_000_000_000 + +// One accumulator per (workspace, provider, model, session): a chat that sends +// several turns before a flush produces one upsert instead of one per turn. +const pending = new Map() +let timer: ReturnType | undefined + +/** + * Record AI token spend. Fire-and-forget: events are summed locally and flushed + * in batches. + */ +export function logAiUsage(event: AiUsageEvent): void { + const workspace = event.workspace ?? get(workspaceStore) ?? undefined + if (!workspace) return + const sessionId = event.sessionId ?? '' + const mapKey = JSON.stringify([workspace, event.provider, event.model, sessionId]) + const existing = pending.get(mapKey)?.event + const target: AiUsageEventPayload = existing ?? { + provider: event.provider, + model: event.model, + session_id: sessionId, + input_tokens: 0, + cache_read_tokens: 0, + cache_write_tokens: 0, + output_tokens: 0, + requests: 0 + } + target.input_tokens += Math.max(0, Math.round(event.inputTokens)) + target.cache_read_tokens += Math.max(0, Math.round(event.cacheReadTokens)) + target.cache_write_tokens += Math.max(0, Math.round(event.cacheWriteTokens)) + target.output_tokens += Math.max(0, Math.round(event.outputTokens)) + target.requests += 1 + if (event.costUsd !== undefined) { + target.reported_cost_nano_usd = + (target.reported_cost_nano_usd ?? 0) + + Math.max(0, Math.round(event.costUsd * NANO_USD_PER_USD)) + } + pending.set(mapKey, { workspace, event: target }) + + if (timer === undefined) { + timer = setTimeout(() => { + timer = undefined + void flushAiUsage() + }, FLUSH_INTERVAL_MS) + } +} + +export async function flushAiUsage(): Promise { + if (timer !== undefined) { + clearTimeout(timer) + timer = undefined + } + if (pending.size === 0) return + + const byWorkspace = new Map() + for (const { workspace, event } of pending.values()) { + let events = byWorkspace.get(workspace) + if (!events) { + events = [] + byWorkspace.set(workspace, events) + } + events.push(event) + } + pending.clear() + + // Start every chunk request synchronously before awaiting: the pagehide flush + // only protects requests that were already issued (keepalive can't help a fetch + // that never started). + const inflight: Promise[] = [] + for (const [workspace, events] of byWorkspace) { + for (let i = 0; i < events.length; i += MAX_EVENTS_PER_REQUEST) { + inflight.push(send(workspace, events.slice(i, i + MAX_EVENTS_PER_REQUEST))) + } + } + await Promise.all(inflight) +} + +async function send(workspace: string, events: AiUsageEventPayload[]): Promise { + try { + // Raw fetch instead of the generated client: `keepalive` lets the request + // finish after tab close/navigation, which is when the final flush runs. + // Auth rides on the token cookie (WITH_CREDENTIALS app setup). + await fetch(`${OpenAPI.BASE}/w/${encodeURIComponent(workspace)}/ai/usage`, { + method: 'POST', + credentials: 'include', + keepalive: true, + headers: { 'Content-Type': 'application/json' }, + body: JSON.stringify({ events }) + }) + } catch { + // Accounting is best-effort: a dropped batch under-reports spend, which is + // better than surfacing a network error in the middle of a chat. + } +} + +if (typeof document !== 'undefined') { + // Flush what's buffered before the tab goes away. pagehide covers + // close/navigation paths where visibilitychange is not delivered. + document.addEventListener('visibilitychange', () => { + if (document.visibilityState === 'hidden') { + void flushAiUsage() + } + }) + window.addEventListener('pagehide', () => { + void flushAiUsage() + }) +} From 9c557859c5ffede921690cd3d224239b9305c9b8 Mon Sep 17 00:00:00 2001 From: hugocasa Date: Mon, 24 Aug 2026 11:08:56 +0200 Subject: [PATCH 40/48] feat: AI agent evals: datasets, scored runs and comparison (#10633) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit * feat: eval datasets and standalone runs for reusable AI agents Co-Authored-By: Claude Opus 5 (1M context) * feat: agent eval drawer with case editor, runs and capture entry points Co-Authored-By: Claude Opus 5 (1M context) * docs: document AI agent eval datasets and standalone runs Co-Authored-By: Claude Opus 5 (1M context) * fix: say how many eval cases the list is not showing Co-Authored-By: Claude Opus 5 (1M context) * fix: address review findings on eval datasets - keep an edited case's conversation and tool inputs: serde(flatten) silently drops Box fields, so the update payload is spelled out - remount the case editor per case so one case's turns cannot leak into another - require jobs:read / flow_conversations:read on the capture endpoints, which UserDB does not gate by token scope - take the dataset lock in create and update so a delete cannot be undone by a concurrent metadata write, and delete cases before metadata - load more cases beyond the first page, and stop capping the agent picker - record that the version stamp is taken at enqueue, not at resolution Co-Authored-By: Claude Opus 5 (1M context) * fix: address round-2 review findings on eval datasets - block operators from dataset and case writes - pass the editor's operating workspace through the drawer and the capture request, instead of assuming the navigation workspace - discard superseded case-list responses so switching datasets cannot land the previous dataset's cases - reject a dataset without a case_id (or vice versa) rather than running an inline case under a dangling association - run unsaved edits inline instead of silently running the stored case - surface the API error body on a failed run - fetch dataset metadata concurrently when listing - $bindable() without a default on the optional open prop - correct the permission and enqueue-time-version wording in the docs Co-Authored-By: Claude Opus 5 (1M context) * fix: run an untouched saved case by reference again The editor writes back keys the stored case omits, so comparing the raw objects reported every unedited case as edited: the run went inline and lost the dataset/case stamp its history depends on. Compare a normalized form, and pin it with a test. Also scope the history query to the drawer's workspace and drop superseded responses. Co-Authored-By: Claude Opus 5 (1M context) * feat: show a dataset's cases as a table, and fix round-4 review findings The case list showed one case at a time with no overview. It is now a table with the case, where it was captured from, and its last run — the last-run column is a single jobs query on the path stamp rather than a request per row. Review fixes in the same file: - keep the edit baseline on the selected case rather than looking it up in the loaded page, so a case beyond page 1 is not treated as unedited and run stale - release the loading state when a superseded case load returns early - reload every loaded page after a write instead of collapsing to page 1 - last remaining 'resolved to' wording in the version tooltip Co-Authored-By: Claude Opus 5 (1M context) * feat: run a dataset as an experiment, with scorers as runnables An experiment runs every case of a dataset against one subject and records the exact case set it executed, so a result set stays reproducible while the dataset keeps changing. Each case runs as its own small flow — the agent, then a step per scorer — so a case keeps the run stamp, history query and trajectory view a single run already has, and scorers need no orchestration of their own. Results are read back per step by node id rather than by walking a nested loop's status. A scorer is any runnable taking (input, output, expected): a script, a flow, or a reusable agent used as a judge. A judge is prompted with the case and the answer as one JSON message; a script or flow receives them as named arguments. Scores accept a bare number, a boolean or {score}. Co-Authored-By: Claude Opus 5 (1M context) * feat: results table for an experiment, with scorer columns One row per case: status, the agent's answer, and a column per scorer, with the mean per scorer above the table and a link into each case's run for its trajectory. Averages skip cases a scorer produced no number for — counting a missing score as zero would read as a regression. The drawer's left pane becomes Cases / Results, and Results carries the scorer picker and Run dataset. Co-Authored-By: Claude Opus 5 (1M context) * feat: compare an experiment against a baseline Per-scorer deltas on each row and on the mean, and a filter down to the rows that regressed. Rows join by case id, so a case added after the baseline ran has no delta instead of counting as a change. Co-Authored-By: Claude Opus 5 (1M context) * fix: address round-5 review findings on experiments - match scorers by label when diffing two experiments; joining by array position subtracted one scorer from another whenever the scorer sets differed - report a row's status from the case job, not the agent step, so a case whose scorer failed no longer reads as a success - delete a dataset's experiments with it: they hold copies of its cases, and a recreated dataset of the same path would have exposed them - select the experiment that Run dataset just started instead of leaving the table on the previous one - expected is scored now, so stop describing it as having no consumer Co-Authored-By: Claude Opus 5 (1M context) * fix: address round-6 review findings on experiments - hold the dataset lock across an experiment launch, so a delete landing between reading the cases and writing the experiment cannot recreate the deleted dataset's inputs - match scorers between experiments on kind and path, not on label: labels default to a path's last segment, so f/a/quality and f/b/quality compared against each other - average mean deltas over the cases both runs scored; comparing each run's own average reported a regression from a case the baseline never ran, with no regressed row to point at - openapi: the row status is the job's, which is also canceled/skipped; runEval takes scorers; the update-case body no longer advertises source, which the handler deliberately ignores - record why the experiment prefix cannot reach a sibling dataset Co-Authored-By: Claude Opus 5 (1M context) * fix: address round-7 review findings on experiments - release the dataset lock for the push loop and retake it for the write, re-checking the dataset still exists: holding it across the whole launch made every capture and case edit on that dataset 409 until the last job queued - assemble experiment results with bounded concurrency; a 100-case, 3-scorer experiment was 400 sequential lookups, each itself several queries - clear the baseline when it becomes the selected experiment, which was comparing a run against itself and reporting zero deltas - take the header mean over the same cases as its delta while comparing, so the two numbers beside each other describe the same set - a canceled or skipped case is no longer the same grey dot as a running one Co-Authored-By: Claude Opus 5 (1M context) * fix: address round-8 review findings on experiments - verify the dataset's identity, not just its existence, before recording an experiment: the path can be deleted and recreated during the push loop, and the experiment holds copies of the old dataset's cases - give the recording lock a longer budget than a case edit, since its jobs are already queued and giving up strands them, and say so when it fails - keep score lookups sequential within a case: nesting two bounded streams multiplied into 32 in-flight queries against a 50-connection pool - clear a baseline that no longer belongs to the loaded experiments, so switching datasets does not leave comparison mode on with nothing to compare - keep a scorer's own mean when the baseline never ran it, instead of blanking a column full of numbers - EvalCaseDraft.expected no longer claims nothing scores it Co-Authored-By: Claude Opus 5 (1M context) * fix: do not trust an experiment's job ids, and require write to record one Experiment objects live in workspace object storage, which a script can write directly, and results are read on the unrestricted pool — so a forged experiment naming another flow job returned output the jobs API would have refused. Only jobs this server stamped with that experiment's id are read now. Also from round 9: - recording an experiment requires write on the dataset, not read: it persists into the dataset's namespace and its shared list - clear the results table when the selection changes and surface a failed load, instead of labelling the previous experiment's numbers as the new one's - a storage fault is no longer reported as a deleted dataset - the lock-timeout message at the recording site no longer says to retry, which would run the whole dataset again on top of the jobs already queued - ExperimentRow.status documents canceled and skipped Co-Authored-By: Claude Opus 5 (1M context) * fix: bind the experiment trust check to the requested dataset The previous check matched jobs on the experiment id alone, which the stored object supplies — so copying another dataset's experiment JSON under a readable key carried its jobs' output along with it. A job is now only read if it was stamped for this experiment *and* for the dataset the caller's read access was checked against, and an experiment that names a different dataset is not served from this key at all. Also from round 10: - add the .sqlx entry for that query; without it every SQLX_OFFLINE build failed - serve results over GET: as POST the route-scope middleware classified a read as ai_evals:write, locking read-only tokens out of their own results - clear the selected and baseline experiments synchronously when the dataset changes, so the previous dataset's id is not requested under the new one Co-Authored-By: Claude Opus 5 (1M context) * fix: address round-11 review findings on experiments and scorers - give scorers the whole case input, not just the message: an answer that came from attachments or a replayed conversation could not be judged on it - accept a judge's boolean and structured {score} answers, including stringified ones, and pin every documented scorer shape with a test - record an experiment for the cases that did launch when a later push fails, instead of leaving those jobs running with nothing to attribute them to - do not capture a preview parent's synthetic runnable_path as a host flow; the saved case could not be rerun - clear the case table before loading a dataset and surface a failed load, so a failure cannot leave the previous dataset's cases under the new name - keep the results table through a refresh of the same experiment - exclude flow-step jobs from the per-case last-run lookup - drop case sets from the experiment list, which is only used to pick a run - report a database failure at the recording lock as itself, not as contention Co-Authored-By: Claude Opus 5 (1M context) * fix: address round-12 review findings on capture and run history - load flow_node.flow for flownode parents: an agent inside a deployed branch or loop captured without its agent, host flow or tool bindings - decide host_flow_path by whether the path resolves to a flow, not by job kind: excluding previews wholesale also dropped the flow editor's step test, whose path is real - page the per-case last-run lookup by created_before until the loaded cases are covered; one page of 200 reported older cases as never run - do not record an experiment when nothing launched - only attach the case input to a job when a scorer will read it - keep the case table through a save; only a different dataset clears it - drop the superseded duplicate comment on the score parser Co-Authored-By: Claude Opus 5 (1M context) * fix: stop refetching run history on every case write Reading the case list before the first await made the whole job-history query a dependency of it, so every save, delete and Load more refetched up to 1000 job rows and blanked the column. Read untracked instead. - an empty Last run cell now distinguishes never-ran from not-found-within the page bound, which the comment already claimed and the cell did not - reloading a dataset no longer replaces a populated table with a skeleton - keep the score-parser comment that describes every shape it handles Co-Authored-By: Claude Opus 5 (1M context) * refactor: keep eval datasets in Postgres instead of object storage Datasets, cases and experiments become rows (`eval_dataset`, `eval_case`, `eval_experiment`, `eval_experiment_case`) rather than objects under a `wmill_eval_datasets/` prefix. What a run produced is still the job's: only case inputs and an experiment's case snapshot are stored. This removes the machinery the object store needed: - The advisory lock and the read-modify-write of a per-dataset JSONL. A case is a row, so there is nothing to serialize. - The launch-time identity check on the dataset. The foreign key makes a concurrent delete fail the transaction instead. - The trust guard on an experiment's job ids, which existed because a script can write workspace object storage directly and could forge an experiment naming somebody else's job. An experiment now chooses every job id and records itself before pushing anything, so a launch that dies partway leaves a recorded case whose job is missing rather than a running job nothing accounts for; cases that never reached the queue are removed again. Row-level security on `eval_dataset` is the authority on who may read or write a dataset, so `extra_perms` grants work and the rule is not mirrored in Rust. Cases and experiments carry a read policy derived from their dataset and no write policy: they are written on the unrestricted pool after the dataset row itself has been asked, with `SELECT ... FOR UPDATE`, whether the caller may write it. Cases are capped at 256 KiB each and 10 000 per dataset, refused rather than truncated. Attachments are S3 references, not inline bytes, so a case that approaches either cap is a mistake rather than a use case. Evals no longer need the `parquet` feature or a configured workspace object storage. Co-Authored-By: Claude Opus 5 * style: align the eval drawer with the design system - Scorer chips are `Badge`s rather than a hand-rolled bordered span, and the section header is a `Label` with its tooltip, as are the case editor's fields (which also gets the label colour right). - The results table showed status as a coloured bullet, which says nothing to a colour-blind reader. It now carries the same icons the runs table uses, with the status as its accessible name. - Feedback colours move to the `-500` shades the brand guidelines name. - The conversation JSON error uses `TextInput`'s `error` prop for the border and the caption style for the message, as elsewhere. Co-Authored-By: Claude Opus 5 * feat: author an expected answer, tags and attachments on a case Every scorer is handed `(input, output, expected)`, but nothing could produce an `expected` except a conversation capture: the case editor had no field for it and a captured run left it empty. So: - The editor gains Expected, Tags and a read-only list of the attachments a captured case carries. Expected is plain text, or JSON when the answer has structure. - Capturing from an AI agent run keeps what that run answered, which is the only moment a reference answer exists for free. The results table also laid itself out by content, so a long answer pushed the scores — the numbers the table exists for — off the edge of the pane. It is fixed-layout now, with the text columns bounded. Co-Authored-By: Claude Opus 5 * docs: expected is captured from a run and can be authored Co-Authored-By: Claude Opus 5 * feat: link a saved agent when inserting an ai agent step "AI Agent" in the step picker was a leaf that always created a blank step, so reusing a saved agent meant inserting a blank one, opening its step input and linking it there. It is a category now, like Flow and AI Sandbox, listing the workspace's `ai_agent` resources next to a blank option, filtered by the picker's own search. A picked agent produces a step that is already linked rather than one linked afterwards: `agent` set, no tools, and only the flow-local `user_message`/`user_attachments` transforms. Seeding the brain keys there would leave transforms a linked step never reads and that `AgentResourceBar` strips on its next link change. Each `on:new` forwarder rebuilds the insert detail field by field instead of spreading it, so a new field is dropped unless the forwarder names it. `agentPath` is typed on both `GraphEventHandlers.insert` and `FlowGraphV2`'s `onInsert` so the next one to forget it fails the check. Co-Authored-By: Claude Opus 5 * feat: restore the link on cancel and simplify the agent bar Cancel on an agent edit forked the step into a standalone copy, which is the opposite of what the word means and needed a paragraph under the card to explain. It discards the edits and re-links the step now, leaving the agent untouched; diverging from an agent is Unlink's job, on the linked card. This flow's `tool_inputs` survive the round trip as overrides, so Cancel no longer folds them into the tools the way Unlink does. Linking a step to a saved agent happens in the step picker at insert time, so the bar's own resource picker is gone and "Save as agent" is the one action left. Its `+` button was a trap besides: it opened the generic resource form, where an agent would have to be written as raw JSON. The card itself was `surface-secondary`, the sections token, so in dark mode it was darker than the pane and read as a sunken well rather than an elevated card. It uses `surface-tertiary` as the brand table prescribes, its tool chips are `Badge`s, and the editing card no longer overflows the pane and clips its own buttons. The remaining tooltip follows the inline `Label` convention rather than sitting in a flex row whose gap stacked on the trigger's own margin. Co-Authored-By: Claude Opus 5 * feat: rework the AI agent evals surface into one table Evals become a single pane: a dataset of cases, one column per scorer, one row per case, with the run being looked at chosen from the toolbar. Runs are permanent. Running the whole dataset opens one; running a single case records nothing at all — it is a job, and looking at what it did is not a claim that it belongs in the history. Its result and its scores sit over the row until they are saved as a run, which carries the cases that were not rerun and the scoring jobs themselves, so the number that is saved is the number that was looked at. A scorer is a runnable: a judge agent or a script, created in one click and edited in place. Scores carry a reason and per-assertion checks, shown on hover with a rescore button. What ran is always named. A run records the agent version, or — for a configuration that is not deployed — a hash of it, so a table can say that its numbers describe an agent that no longer exists: those rows dim and the table offers to rerun. An agent's draft can be run directly instead of the deployed value, and once those edits are deployed the runs that made them are recognised as that version. A step with no agent of its own is evaluable too, and saving it as an agent moves its history onto it. Co-Authored-By: Claude Opus 5 * feat: keep an agent's in-progress edits on the agent Editing a linked agent forks it into the step, which is what makes the edits runnable there — but the agent is what is being edited, so that is where the unsaved state belongs. The edit is mirrored into the agent's own resource draft as it is made. It then survives leaving the flow, shows the agent as drafted wherever it appears, and is what evals run when asked to run the draft rather than what is deployed. Deploying or cancelling clears it; opening Edit without changing anything does not mark the agent as drafted. Co-Authored-By: Claude Opus 5 * feat: shape the evals surface around a saved agent Evals hang off an `ai_agent` resource, so the surface is now only ever about one: the `draft` subject kind, the standalone-step subject and the move that carried a step's history onto a newly saved agent are gone. - A run is permanent and numbered per agent. Running a single case is a trial: it answers in the panel and never touches the table. - "Run scorers only" opens a run of its own that reuses the answers of the run you are looking at, so a scorer added later measures what already ran without calling the agent again. - A draft run whose configuration is later deployed is stamped, once, to the version it became, so its label stops reading `v23 + edits` forever. - A scorer can carry a pass threshold, read off the scores already recorded. - The table is the case, its answer and one number per scorer; datasets are created and edited in a drawer; a run that executed an earlier state of the current draft says so above the table, in one line. - Which agent a step is, whether it is being edited, and which version it is on is a strip above the step's tabs, because it is true of every tab. - Capturing a case from a step test or a conversation is dropped, and with it the `memory` override on a linked step that nothing set. Co-Authored-By: Claude Opus 5 * feat: run past versions of an agent, and number versions per resource The evals home becomes one table of every run of the agent, whichever dataset each is of, with one badge per scorer. A list spanning datasets cannot hold every dataset's scorers to look a name up, so a score carries its name and kind with its number, and thresholds are joined in per run and column. Run now asks what to run: the latest agent, resolved when the run executes as a flow step does, any past version, or the unsaved edits. Pinning is a subject kind of its own, since a linked step resolves the resource live and inlining is the only way to run a version that is no longer current. Scorers move into the edit-dataset drawer. The column header over a run reports and nothing else: a run is permanent, and a control there that changed the columns would edit the past from the one place that must not. Adding one offers four ways rather than two, writing and reusing being different jobs, and both new kinds open with a summary filled in. Versions are numbered per resource. `resource_version.id` is one identity sequence for the whole table, so an agent saved nine times read v4 ... v24, and the gaps counted writes in workspaces the reader cannot see. The id stays how a version is addressed; the new number is what it is called, in the resource history drawer as well as here. It is assigned on write rather than counted on read because trimming past the cap and clearing a history both take the oldest rows, and counting the survivors would renumber a version a run already names. Co-Authored-By: Claude Opus 5 * fix: read the dataset a remembered selection names Reopening the evals modal restored the last dataset from storage as a bare path, without reading the row it names. Every "is this already the one?" test compared against that selection, so all of them short-circuited and the dataset was never loaded: editing it opened a drawer with no summary, no scorers and no cases. The remembered path is now brought into context the same way any other choice is, and the tests compare against the dataset that is loaded rather than the one that is selected, so a selection can no longer stand for a read that did not happen. Co-Authored-By: Claude Opus 5 * feat: give dialogs a trail in their header A dialog deep enough to navigate had nowhere to say where you were: the header held a fixed title, and the way back was a control each body placed for itself, somewhere in a toolbar that moves with everything else the toolbar holds. The header is the one part of the surface that does not move, which is where the trail belongs. `Modal` takes an optional `trail` of levels below its title, rendered as a breadcrumb whose ancestors are the way back. Declarative on purpose: callers of this depth already hold the state that says where they are, so the dialog reads it rather than owning a stack they would have to push and pop in step with it. Escape follows the trail. Leaving a level is what someone deep in a dialog means by it, and closing the whole surface throws away the navigating they did to get there; at the root it closes as before. That only works if a dialog can tell it is the surface being addressed, so `Disposable` now answers `isTopmost()` and the dialog asks before acting: it keeps Escape for itself, so nothing else was arbitrating between it and a drawer opened from inside it, and both were acting on one key press. Evals is the first caller: its runs list is the root, a run is a level in it, and the back button that used to sit above the table is gone. Co-Authored-By: Claude Opus 5 * fix: portal dialogs out of wherever they were opened from A dialog rendered in place inherits whatever the calling component happens to sit inside. One `transform`, `filter` or `overflow` anywhere above it makes its `fixed` positioning resolve against that ancestor instead of the viewport, and a surface meant to cover the app is then confined to a box it never asked for: the nav rail paints over it and its own edges are clipped. Drawers have always portalled for this reason. Dialogs only did so when an enclosing pane claimed them, and rendered in place otherwise, so the same screen could show a drawer over everything and a dialog trapped behind the nav. They now portal the same way: to the pane when one claims it, to `body` otherwise. Co-Authored-By: Claude Opus 5 * fix: make the dialog's title the first step of its trail The trail listed levels below the title, so a dialog one level deep read "Evals > All runs > Run 20 · v6": three steps for two places, the first two of them the same place under different names. The title is the root, so it is the root's own segment, and the trail a dialog is given is now the whole path with that segment at its head. Its height stopped moving too. A heading carries a line-height of its own, so a header holding only an h3 stood six pixels shorter than one holding segments as well, and the dialog's whole top edge stepped as you navigated. Co-Authored-By: Claude Opus 5 * fix: sharpen the evals controls around where you are standing Each screen now offers what belongs to it. The list starts runs; a run is a record, so it offers only the one thing that acts on the record itself, which is measuring the answers it already stored. Starting a fresh run from inside one asked which agent and which dataset from the screen least about either, and scoring an existing run was offered from the list, where there is no run to score. Which run and what it is read against are one question asked twice, so they sit together rather than at opposite ends of a row. Choosing what to run is now a toggle over the two states worth naming, the draft and the saved agent, with every earlier version one click further: running an old version is deliberate, and a list made all three look alike. The draft is read when the dialog opens rather than taken from the caller's polled copy, which could be seconds behind an agent edited a moment ago and would leave the option out exactly when it is the reason for opening the dialog. The dataset field carries its path under it and its edit button on hover, as a resource picker does, so the closed field says what the open list said. Edits waiting on an agent are a "draft" here as everywhere else in Windmill, rather than "+ edits". The dialog runs an evaluation rather than "the agent", which is what it was already called everywhere it is recorded. An agent being edited keeps its evals button on a line of its own, clear of the decision to save or discard. Co-Authored-By: Claude Opus 5 * fix: settle the evals controls on the patterns Windmill already has The version choice uses ToggleButtonMore, as the AI provider picker does: the two states worth naming stay in the group, the rest are behind the overflow menu, and the one you pick joins the group rather than appearing in a second control below it. The deployed one says which version it resolves to. A run offers nothing to start. Scoring an existing run again was the last thing left there, and it was one button explaining a distinction that the run and the dataset already make between them. The warning that a run executed an earlier draft is about the run on screen, so it goes when the run does rather than following you back to the list, and it sits against the table instead of inside a frame of its own. A dataset just created stays open for its scorers and cases: those are what a dataset is, they can only be added to one that exists, and closing on create sent you to find it again to add them. Scorer settings are a cog rather than a word, now that the row holds three actions. Co-Authored-By: Claude Opus 5 * fix: close the gap in the version toggle and say what naming a dataset does The overflow trigger is not a pill, so the room it reserves showed as a gap between it and the button before it; it is pulled in by that much. The dataset field gets its clear button, which is also the slot the edit button is positioned against, so the two now sit where a resource picker puts them. Naming a new dataset said nothing about what happens next, and the drawer looked like it was missing the rest of itself. It says so instead: a scorer and a case both belong to a dataset, so there is nothing to attach either to until this one exists, and creating it leaves the drawer open on them. Co-Authored-By: Claude Opus 5 * feat: choose a dataset's scorers while naming it A scorer is a reference to a runnable, not a child of the dataset, so it needs the dataset's name but not its row. The list is collected in the drawer while the dataset is being named and sent with the create, which already accepts one, so a dataset arrives holding the columns that were chosen for it rather than being made empty and then edited to hold them. Cases stay where they were: a case *is* a row of the dataset, so there is nothing for it to be a row of until one exists. The drawer says which of the two is which instead of leaving the screen looking like it is missing the rest of itself. Co-Authored-By: Claude Opus 5 * fix: level the version toggle and name the dataset in its own field The overflow trigger stands a row taller than a toggle button, so the group grew to its height and left the sunken background showing under every pill beside it. Every child of the group is the same height now, which is why the AI provider picker never had the band: it sizes them all alike. The dataset field says the summary with the path after it rather than carrying the path on a line below. The list stacks the two, which a one-line field cannot do, so it says both the other way round. Co-Authored-By: Claude Opus 5 * fix: tidy the evals forms and the run's own controls Picking a scorer that exists chooses between two sources rather than showing both: the ones already measuring something, and everything else in the workspace. The first list says what each is called with its path under it and what it already measures on the right, instead of three columns that were the same path truncated three ways whenever a scorer had no name of its own. A dataset's drawer says what it is for on the page rather than under an icon, and its summary is sized like the field beneath it. The run's own row lines up with the table under it, the warning above that table is spaced off the rule rather than sitting on it, and adding a case is gone from a run: a run is a record of cases that were answered, so curating them from it is editing what it measured. Co-Authored-By: Claude Opus 5 * feat: create a dataset holding the cases written for it Creating a dataset takes the cases to create it with, so one can be assembled in a single act instead of made empty and then filled in. The drawer holds them while the dataset is being named, gives them ids of its own to be edited by, and sends them with the create. Every case is checked before the dataset is written. `eval_case` grants users no write, so the rows cannot be inserted in the transaction that creates the dataset under the caller's own policies; validating first is what keeps "created holding these cases" from becoming "created, holding some of them", and the rows that do follow go in one transaction of their own. Co-Authored-By: Claude Opus 5 * fix: name the button for what it opens, and say what each version is Starting an evaluation asks which state of the agent and which dataset, and both cost a provider bill, so a button that read as spending one on the way past was lying about the click. It opens something, and says so. Running one case from the panel keeps its own name and its play icon, because that one does run on click. The version options say what they are rather than what they are not: what a flow step would or would not run is a fact about somewhere else, and someone choosing what to evaluate is not standing in it. Co-Authored-By: Claude Opus 5 * fix: give the editing card two rows and mark evals as beta At the width of a step panel the card's one row wrapped: the line naming the agent, the line saying what saving does, and the two buttons deciding the edits' fate all fought for it. Deciding gets a row of its own, and evals sits against the line it is about, since evals of an agent being edited run the edits. Evals is named wherever it is offered. It read as a word in one state of the card and as an icon in the other, which is two things to recognise for one door. The dialog carries a beta badge against its own name, before any level below it: every way in lands there, so it is said once and stays put as you navigate. The version toggle spells out which is which. Both are the agent at v2 and the difference between them is the whole choice, so it is worth the width. Co-Authored-By: Claude Opus 5 * fix: name a new dataset, and lay the scorer's settings out like a step's inputs A new dataset arrives called "Dataset 1", which the path follows as it follows any summary: a dataset with none was one every table could only call by its path, and the two seeds are what the summary rule already produces. Scorer settings put each field's description between its label and its input, where a step's inputs put theirs, and its inputs are the size the rest of the drawer uses. The runnable behind the column is a link to it with its kind's icon, since it is a resource of its own and the one thing about it these fields cannot change. The line explaining that a pass line re-reads recorded scores went: the threshold is a number to set, and how it is applied is not a decision being made here. Co-Authored-By: Claude Opus 5 * feat: curate a dataset in the drawer and save it in one act The drawer holds the cases while they are edited and writes them when it is saved: added, changed and dropped, whichever it is. Typing no longer writes, so a set is never half saved while someone is still deciding what is in it, and Save means the same thing whether the dataset exists yet or not. A case panel offers reading rather than acting. Running one case now and editing one from a run were the last two ways to change a record from the screen showing it, and the machinery behind the first went with it. The answer is rendered as the prose it is, under what it is: the case's result, whichever run is selected above it. The rest is what the run's table was doing to its own edges: a column name is clipped to its column rather than running into the next, the table squares off against an open panel, and that panel closes with the run it belonged to. Co-Authored-By: Claude Opus 5 * fix: one border above a table, and a link to the run's job The row above the table drew a bottom border and the table draws its own top edge, so every table sat under two lines. The row keeps its spacing and the table keeps its edge. A column header no longer spins while its scores arrive: the cells under it are where the numbers are missing, and they say so themselves. The beta badge is the height of the word beside it rather than of the line it sits on. A run is one flow and therefore one job, so the run says where that job is: what it is doing, what it cost and what it logged are all there rather than reconstructed from the table. Co-Authored-By: Claude Opus 5 * feat: stream scores as each scorer finishes, and show them per case A scorer runs after the agent inside the case's own iteration, so its verdict can be read as soon as its step is done. Waiting for the iteration to end held every column of a case back until the last of them finished, which is why answers arrived one at a time and scores all at once. Reading a job that is still running needs one guard: a module with nothing in it is a step that has not run, not one that produced nothing, and recording the second makes a failure that never goes away. The panel beside the table shows what each column made of the case and why. The reason a judge gave was stored and never shown, which is the half of a score that says anything. It stops repeating the question the header already asks, and a case still running reads as waiting rather than as an answer that says "Running". A run is a number beside a dataset, so the list puts the two together. Co-Authored-By: Claude Opus 5 * feat: score a case with every scorer at once The scorers of a case read the answer and never each other, so they ran one after another for no reason: measuring a case now takes as long as its slowest column rather than as long as all of them. Each is a branch of its own, kept from failing the others, so a judge that errors costs its own column and no more. An iteration is three steps again — answer, payload, scores — rather than one per scorer, and each branch is named for the column it produces, so the graph of a run says which scorer did what instead of spelling out an id. Co-Authored-By: Claude Opus 5 * fix: read a judge's score out of the JSON it nearly wrote A judge quoting the agent inside its own reason writes those quotes unescaped, which is invalid JSON and also the most ordinary sentence for it to produce. The whole verdict was being thrown away over it, so a column that had a number reported having none. The number and the reason are now read straight out of such text. Deliberately not a second JSON parser: it finds the two keys and takes what follows, which is what survives a quote in the middle of a sentence. A case still running says so with a spinner rather than with the word "Running" sitting where its answer goes. Co-Authored-By: Claude Opus 5 * feat: ask a judge for a shape instead of trusting it to write one A new judge carries an output schema, so the provider holds it to `{score, reason}` rather than the prompt asking it to. Windmill already delivers a schema whichever way the model takes it, a tool for Claude and Bedrock and the native parameter elsewhere, so there is no list of models to keep here. An agent with no runs offers its first one where the first row would be, rather than from a toolbar above a table that has nothing in it. Starting a run no longer picks a dataset for you. It fell back to whichever came first, which on an agent that has never run means offering another agent's set as though it were the obvious one; and with no dataset at all it says so and offers the one move there is. Co-Authored-By: Claude Opus 5 * feat: report a column that failed throughout, and hold the run dialog The runs overview dropped any column that produced no number, so a judge that failed on every case of a run vanished from the row and read as a column nobody had asked for. The aggregate now reports every column that has cells, with the count of the ones it failed on, and the badge says "failed" where there is nothing to average. A column with no cells at all is still left out: that one was added after the run and has nothing to say about it. Creating a dataset closes the drawer rather than turning it into an edit of what it just made: scorers and cases already ship with the create, so there is nothing left to stay open for. Reached from the run dialog, it gives the screen back with the new dataset selected, and the dialog keeps the version you had already chosen. Also: - the case panel's job link moves to the panel's own header, where its scope is: the job is the whole iteration, not the answer it sat over - one action in the scorer drawer's header, as its neighbours have. The reuse list picks rather than adds, and says which dataset each column already measures - adding a case is the last row of the list it lands in - the pane shows what it has read rather than an empty state it has not earned yet, and its rows say they open - the linked agent card loses a border it had inside another one * fix: keep the linked agent card's outline The card is a thing inside the step's inputs rather than a section of them, and the outline is what says so. Only the rule inside it goes: the detail it separates is already set apart by being detail. * refactor: fit the eval surface to the shipped design * feat: give a nested dialog a back control and the runs list its own moves Co-Authored-By: Claude Fable 5 * feat: put a dialog's description under its title Co-Authored-By: Claude Fable 5 * refactor: fold a dialog's back control into the crumb it returns to Co-Authored-By: Claude Fable 5 * feat: edit a dataset's cases as a table rather than a list beside a form Co-Authored-By: Claude Fable 5 * feat: edit a dataset's cases in the grid the data tables are edited in Co-Authored-By: Claude Fable 5 * feat: edit a grid cell of prose in place, and cap a dataset at one page Co-Authored-By: Claude Fable 5 * refactor: keep the cell editor's styles beside it, not in the vendored theme Co-Authored-By: Claude Fable 5 * fix: keep an empty cell empty and cap the editor's growth Co-Authored-By: Claude Fable 5 * feat: name the step that assembles a run for the scorers Co-Authored-By: Claude Fable 5 * feat: run the payload step natively, and say so when nothing serves that tag Co-Authored-By: Claude Fable 5 * fix: report an answer as answered while its scorers are still running Co-Authored-By: Claude Fable 5 * feat: let a scorer say a case is not one it measures Co-Authored-By: Claude Fable 5 * fix: score the answer, and leave a case with no expected answer unmeasured Co-Authored-By: Claude Fable 5 * refactor: split the evals backend into modules Co-Authored-By: Claude Fable 5 * fix: record what a run produced so it outlives its jobs * fix: read only the agent step's own tool jobs into the payload * fix: pin a run's configuration and give the judge the attachments * feat: write a dataset's cases in one transaction * chore: refresh the sqlx cache for the eval queries * fix: drop results a newer selection has superseded * fix: keep a draft the agent editor never opened on * feat: let a run record what it produced instead of waiting to be read * fix: serialize the replacements of a dataset's cases * fix: stop the poller from superseding a read slower than its interval * chore: refresh the sqlx cache * fix: keep a failed read from settling a cell as a case with no answer * fix: hold the case grid while its save is in flight * fix: keep a failed collect step from failing the run it recorded * chore: refresh the sqlx cache * fix: commit an open cell into the save that reads it * refactor: size the eval buttons with unifiedSize * docs: describe a run as the one flow it is * fix: show a run's recorded rows when part of it cannot be collected * refactor: size the remaining PR-added buttons with unifiedSize * fix: save the dataset name that was submitted, not the one typed after * fix: force an open cell into the save that was pressed for it * fix: refuse to score a run whose evidence could not be read * fix: hold one lock over a dataset's case count and its writes * fix: keep one unreadable run from costing the whole runs list * refactor: drop the banned bindable-default from the eval props * fix: hold the scorer controls while the dataset is written * fix: read only the caller's own draft of an agent * docs: say in the contract that a run pins its configuration * fix: say a scorer did not run rather than blaming a missing answer * feat: resume the agent draft you already had when you press Edit * refactor: build the trail and dataset controls from Button * fix: clear the open-cell flag when the drawer reopens * chore: refresh the sqlx cache * fix: read a run's configuration and its version from one snapshot * fix: refuse a dataset path or summary the column cannot hold * refactor: handle the agent draft the way the resource editor does * fix: run only a configuration the launch actually read * docs: bound dataset path and summary where they are submitted * fix: surface a stalled agent draft instead of claiming it is kept * fix: stop claiming a draft holds edits a failed write never sent * fix: word a missing score only once the run says whether the case answered * fix: let a breadcrumb crumb shrink so its truncation applies * docs: describe where an agent's unsaved edits live and what drops them * fix: keep harvesting scores when the run cannot yet word a missing one * fix: report a refused draft write the card was reading as a save * fix: drop the refused draft write when the server copy is taken instead * refactor: build the scorer and dataset pickers from the design system * fix: say what removing a scorer column actually does * fix: drop a refused draft write wherever the server copy is read * fix: let a picker row be as tall as the two lines it holds * docs: record what removing a scorer column does to recorded runs * fix: send a queued draft write before reopening, and drop only what it refuses * refactor: write the agent draft at commit points instead of mirroring keystrokes Co-Authored-By: Claude Fable 5 * refactor: run an agent's edits from the step instead of keeping them as a draft Co-Authored-By: Claude Fable 5 * fix: make the diff badge keyboard operable and refuse an edits run without its edits Co-Authored-By: Claude Fable 5 * fix: drop the dataset icon from the scorer picker rows Co-Authored-By: Claude Fable 5 * fix: size the evals buttons like the rest of windmill and call a run of edits edits Co-Authored-By: Claude Fable 5 * fix: count a brain expression as an edit of the linked agent * fix: cap scorers per dataset and report a launched run as launched * fix: harvest scores in one read, refuse duplicate case ids, allow group paths * fix: mint scorer ids server-side, save a dataset edit in one request, check attachments * fix: write a dataset edit and its cases in one transaction * fix: atomic dataset create/edit, reset eval pane per agent, stable pending scorer ids * refactor: govern eval_case writes by RLS so a dataset edit is one transaction * fix: pin launch snapshot, order case locks, cap dataset size, guard stale load * fix: cap dataset bytes on single-case writes, reset run-dialog flag on load failure * feat: migrate eval datasets on username change, settle unspawned cases, drop unused case endpoints * fix: resolve scorer scripts as the caller and pin their hash; migrate scorer paths on rename * fix: bound a failed tool call's error to the payload truncation cap * fix: pin scorer hash as a hex string, reject missing judges, migrate eval authorship * fix: record an out-of-range scorer result as an error, not a score * fix: resolve judges in one caller-scoped read, pin deployed scripts, bound pass_if * fix: settle unspawned cases only when the run completes, and their score cells too * feat: reassign eval datasets and their path references when offboarding a user * fix: use the regex backreference in offboarding eval path rewrites * fix: register eval datasets in offboarding registries, keep resource-version param name * refactor: name the resource-version path param id, since it is the row id not the version * fix: validate dataset paths canonically, clone eval data on fork, surface eval load and launch failures * docs: note MCP tool results are not yet surfaced to eval scorers * fix: show the eval error state on any load failure, not only an empty dataset list * fix: preserve eval case order across a batched save Co-Authored-By: Claude Opus 4.8 Claude-Session: https://claude.ai/code/session_01LVjewUvXFEjNLLw7kxz41h * docs: scope the eval launch delete-safety guarantee to the assembly window Co-Authored-By: Claude Opus 4.8 Claude-Session: https://claude.ai/code/session_01LVjewUvXFEjNLLw7kxz41h * fix: only offer deployed scripts as eval scorers, drop unbuilt rescore claim Co-Authored-By: Claude Opus 4.8 Claude-Session: https://claude.ai/code/session_01LVjewUvXFEjNLLw7kxz41h * fix: enforce 0-1 scorer threshold in the settings drawer and clear stale eval load errors Co-Authored-By: Claude Opus 4.8 Claude-Session: https://claude.ai/code/session_01LVjewUvXFEjNLLw7kxz41h * fix: scope subject version/hash reads to the caller and keep a 0 pass threshold Co-Authored-By: Claude Opus 4.8 Claude-Session: https://claude.ai/code/session_01LVjewUvXFEjNLLw7kxz41h * fix: select the saved dataset when creating or renaming from the Run dialog Co-Authored-By: Claude Opus 4.8 Claude-Session: https://claude.ai/code/session_01LVjewUvXFEjNLLw7kxz41h * fix: gate eval dataset rename on path ownership, not just write access Co-Authored-By: Claude Opus 4.8 Claude-Session: https://claude.ai/code/session_01LVjewUvXFEjNLLw7kxz41h * fix: tolerate a malformed agent config when resolving the deployed label Co-Authored-By: Claude Opus 4.8 Claude-Session: https://claude.ai/code/session_01LVjewUvXFEjNLLw7kxz41h * refactor: trim eval code and comments, fix shared select and modal paths Co-Authored-By: Claude Fable 5 * fix: drop the rename warning when editing an eval dataset path Co-Authored-By: Claude Fable 5 * fix: add eval dataset delete, keep summary on partial edits, settle resultless scorer cells Co-Authored-By: Claude Fable 5 * test: cover parseThreshold and subjectLabel Co-Authored-By: Claude Fable 5 * fix: hold dataset Save during a scorer write, derive draft_hash only from the carried draft Co-Authored-By: Claude Fable 5 --------- Co-authored-by: Claude Opus 5 (1M context) --- ...2df5e7d3ff6a15a93e1c0c95e262f7b3d0ef1.json | 17 + ...c60f3e7c2af0331388c4b358035de865a121a.json | 35 + ...f23abc46d5db95da095bb15726c5a2db7ad2f.json | 24 + ...c72459d45e275bfeafa2952349cae259ac9f0.json | 16 + ...98e474f23b013f642ba29ad6f68e6f047c1e5.json | 24 + ...ae5c79794213cf6278167f8afdbca30b1b15c.json | 16 + ...f52e2730501774a0ca4dc855380e6c6487917.json | 22 + ...b251446d7701a893cb7852b81cf842c0fa228.json | 59 ++ ...7d1bddaba15ef0fbe72bb4f43b45140f184ce.json | 28 + ...af168a4fa2c64e446a824038cf267236fe979.json | 23 + ...4af9aeb181672a2f2330dbe65bfe586376450.json | 15 + ...96604c1562cdfb0bdb54bd3b0c6579d73e46d.json | 28 + ...33235a349731828d500c74a28b577469e2624.json | 23 + ...b3527e750d85d3b8cc0eadef55461de4a2687.json | 23 + ...ac5b5b34c5de851f1c6f832b868101edcb052.json | 23 + ...d6974e02a7c81d91217d3f74e03113edc9b0a.json | 16 + ...858f472be9f1bb8e504cc3dc2eb998e3f7b91.json | 16 + ...08360569aae717eff3fe9cc9253a261ae2fa7.json | 23 + ...f1a35f5dc8f6828c2f3df51520683af48ebbd.json | 16 + ...ec73feba3141f7345ffd1a6554fe44a6a9171.json | 24 + ...3a577bb85b389ab056909a326212565a338bc.json | 15 + ...8b01b5175e14eee47ca39fe63242a78da767e.json | 16 + ...da68e465ce7ce67e80770f22a36a9e8320657.json | 59 ++ ...9fe4424eaed36d5ac9e4cd0a2a9d8b878eeb.json} | 10 +- ...05ded7dfe1c64517289ad67028001417d30a3.json | 47 + ...153c43903f929ae5d62fbba12610f89c36d55.json | 2 +- ...cea189be47e344d11871a90faa9220d509646.json | 16 + ...a6bfb8b08420cc2b2a52e90064d6a12f86113.json | 76 ++ ...eac5838995dce3d2a696769e00fb74c34288f.json | 22 + ...126a87ee5dc4bd6664eaf31c2c0096fea08db.json | 26 + ...0d07f3514628a4970b5f25ddf3920ba5e9c53.json | 23 + ...54469259621218616ef64a4923f3396b183a2.json | 14 + ...f27d2e39781fccc4dd8419f934085d5453fc7.json | 20 + ...63a373b5c427a0b2f48ca775fb8f5a0096e18.json | 26 + ...3d57b0f69b05f96ffafae0325cf3ff04f0f40.json | 60 ++ ...4a216e3887fdc0a9bab0747a447c01a4b37a5.json | 16 + ...92c62d40a4ab97fab0f2dd330bd7629c0a632.json | 64 ++ ...362439f5b12a3dd3290a8dab17ccf9c391f3.json} | 12 +- ...6219919f438e9a3ce0c3e996ea61d9b4f2ef8.json | 16 + ...ae60da7f9ecc12f8f98b926345a70f941845b.json | 24 + ...aff3d53432da9ef57f8a775850efcd50a965c.json | 29 + ...164f1a56e328902dd46fd9b68d801c16f4ffa.json | 16 + ...3fedb64cfafa9fe74cd9446eb6ee526c2d9a0.json | 58 ++ ...1b8ae45e16c11c8003c5f64c9f53d0e7226aa.json | 23 + ...5db87e3101bcbcb0fa359885ca3807c1c9ca2.json | 23 + ...62a53440a5c713b58e6c4a467f6f0d5d7e37c.json | 16 + ...d8ae66f891421b72fc6adfa8a38060ca2f3fa.json | 15 + ...b75ef7de6e0ad74e0cc6c4056b782155b86c7.json | 53 + ...d8f3f0339a2a088243f045a212f9357205407.json | 27 + ...2340328da39ac763bb26b6bcf82027fad48e5.json | 29 + ...53d23ec2af084b2f93da24c920532c1916384.json | 6 +- ...3728f1cabe3774f581401514f522ea7522de5.json | 29 + ...fa74b8b35375dfb0f5d39b398369468ad0774.json | 54 + ...f61c82248f3f2c9a69c458bff3978ffd17379.json | 66 ++ ...9d7271b953dda42e4ba09864da7bda731e752.json | 54 + ...c02ec917b6ecb014f938e68ced3cc7fe9dc86.json | 16 + ...e630bad0715bbbb3597cb1b14e41abe9fec14.json | 23 + ...59f2ec33b0c83b5c4a332a4b6e54d2591767c.json | 18 + ...7c13719ae650e9c788ebf125627b785fb9ee0.json | 23 + ...e292c958e71293695ce18f9905af4dd940495.json | 49 + ...602c405fe620937a4563a17db80219a2930d2.json | 23 + ...8dd950ad0b1f30cde9c0392384c9a490290b9.json | 17 + ...4fcee17400a0ffc172d2a257c3fc15617fbd9.json | 41 + ...aa409e03b6bfb7423929c201f39db4228cc36.json | 23 + ...83204fddf0ccdca6d7c798a4ff6527548e96f.json | 20 + ...064ce566cea6655e3cd11a9d7016cb08482cb.json | 21 + ...ec172af9dd334786275c391e80f1c38bbb45b.json | 18 + ...fda6341e215025dc6d8985891b91dad6d9dcb.json | 26 + ...5dce0711e90f7a4e1786d39a50d52400a3cb9.json | 16 + ...9a6ffebdcadb20e4c6a68768a9ad42fe9328.json} | 16 +- ...ccd7ca7229f9d4e5c3af6c460ee7fe3c946e6.json | 23 + ...d8ada02aa4353feb2f63e7a1f520a4000ed70.json | 36 + ...288e6c04f4a1a76a2038679f8f538320cab1f.json | 24 + .../20260812100659_ai_evals.down.sql | 6 + .../migrations/20260812100659_ai_evals.up.sql | 275 +++++ ...819073729_resource_version_number.down.sql | 17 + ...60819073729_resource_version_number.up.sql | 52 + backend/windmill-api-auth/src/scopes.rs | 3 + backend/windmill-api-users/src/users.rs | 1 + .../windmill-api-workspaces/src/workspaces.rs | 32 + backend/windmill-api/openapi.yaml | 805 +++++++++++++- backend/windmill-api/src/ai_evals/datasets.rs | 473 +++++++++ backend/windmill-api/src/ai_evals/mod.rs | 371 +++++++ backend/windmill-api/src/ai_evals/payload.rs | 410 ++++++++ backend/windmill-api/src/ai_evals/results.rs | 856 +++++++++++++++ backend/windmill-api/src/ai_evals/run.rs | 950 +++++++++++++++++ backend/windmill-api/src/ai_evals/scorers.rs | 285 +++++ backend/windmill-api/src/ai_evals/scoring.rs | 712 +++++++++++++ backend/windmill-api/src/ai_evals/subject.rs | 135 +++ backend/windmill-api/src/ai_evals/template.rs | 132 +++ backend/windmill-api/src/lib.rs | 2 + backend/windmill-api/src/offboarding.rs | 68 ++ backend/windmill-api/src/token.rs | 6 + backend/windmill-api/src/users.rs | 81 ++ backend/windmill-store/src/resources.rs | 31 +- backend/windmill-worker/src/ai_executor.rs | 6 +- docs/ai-agent-evals.md | 482 +++++++++ docs/reusable-ai-agents.md | 8 + frontend/src/lib/components/Path.svelte | 9 +- .../components/ResourceVersionHistory.svelte | 35 +- .../lib/components/aiEvals/AddScorer.svelte | 410 ++++++++ .../components/aiEvals/AgentEvalModal.svelte | 68 ++ .../components/aiEvals/EvalCasesGrid.svelte | 132 +++ .../aiEvals/EvalDatasetDrawer.svelte | 419 ++++++++ .../components/aiEvals/EvalRunDialog.svelte | 279 +++++ .../components/aiEvals/EvalRunsList.svelte | 170 +++ .../lib/components/aiEvals/EvalScorers.svelte | 383 +++++++ .../lib/components/aiEvals/EvalsPane.svelte | 978 ++++++++++++++++++ .../lib/components/aiEvals/evalUtils.test.ts | 46 + .../src/lib/components/aiEvals/evalUtils.ts | 107 ++ .../display/table/multilineCellEditor.css | 26 + .../display/table/multilineCellEditor.ts | 108 ++ .../common/drawer/Disposable.svelte | 7 + .../lib/components/common/modal/Modal.svelte | 135 ++- .../components/flows/agentEditStore.svelte.ts | 49 +- .../components/flows/agentEditStore.test.ts | 19 + .../flows/agentResourceUtils.test.ts | 19 + .../components/flows/agentResourceUtils.ts | 22 + .../flows/content/AgentResourceBar.svelte | 394 +++++-- .../flows/content/ScriptEditorDrawer.svelte | 94 +- .../components/flows/flowStateUtils.svelte.ts | 16 +- .../flows/map/FlowModuleSchemaMap.svelte | 8 +- .../flows/map/InsertModuleInner.svelte | 96 +- .../flows/pickers/TopLevelNode.svelte | 2 +- .../lib/components/graph/FlowGraphV2.svelte | 2 + .../components/graph/graphBuilder.svelte.ts | 2 + .../graph/renderers/edges/BaseEdge.svelte | 3 +- .../components/select/SelectDropdown.svelte | 16 +- .../(root)/(logged)/resources/+page.svelte | 18 + 129 files changed, 11998 insertions(+), 239 deletions(-) create mode 100644 backend/.sqlx/query-01bce88dd622f314d1a09c24cd12df5e7d3ff6a15a93e1c0c95e262f7b3d0ef1.json create mode 100644 backend/.sqlx/query-0276e6030abb2eb00a68c568a9cc60f3e7c2af0331388c4b358035de865a121a.json create mode 100644 backend/.sqlx/query-0335de6713de6678b9bf266121af23abc46d5db95da095bb15726c5a2db7ad2f.json create mode 100644 backend/.sqlx/query-0aae275d9196e742b5783df4e67c72459d45e275bfeafa2952349cae259ac9f0.json create mode 100644 backend/.sqlx/query-0d6700ccffb8179e365bbc1f03398e474f23b013f642ba29ad6f68e6f047c1e5.json create mode 100644 backend/.sqlx/query-1815730982dcaf7239ddcb22f88ae5c79794213cf6278167f8afdbca30b1b15c.json create mode 100644 backend/.sqlx/query-186c663249ffada82abf61ce214f52e2730501774a0ca4dc855380e6c6487917.json create mode 100644 backend/.sqlx/query-196939257a334f7d37aa6d66153b251446d7701a893cb7852b81cf842c0fa228.json create mode 100644 backend/.sqlx/query-1b6e229545f6b877e72d21728257d1bddaba15ef0fbe72bb4f43b45140f184ce.json create mode 100644 backend/.sqlx/query-1db80f3ba2c6c769a98424ebf9aaf168a4fa2c64e446a824038cf267236fe979.json create mode 100644 backend/.sqlx/query-242845c86084e010ab33c2197d44af9aeb181672a2f2330dbe65bfe586376450.json create mode 100644 backend/.sqlx/query-2b41dc4d872af0e230c31bef1a496604c1562cdfb0bdb54bd3b0c6579d73e46d.json create mode 100644 backend/.sqlx/query-307d5b797e51122dbf087e5dc9f33235a349731828d500c74a28b577469e2624.json create mode 100644 backend/.sqlx/query-30b8590939bf3d6770cabca9f4ab3527e750d85d3b8cc0eadef55461de4a2687.json create mode 100644 backend/.sqlx/query-316e7c86082b6ee2864b88674cfac5b5b34c5de851f1c6f832b868101edcb052.json create mode 100644 backend/.sqlx/query-34fbb2b141ad691e0cdc55bf2ebd6974e02a7c81d91217d3f74e03113edc9b0a.json create mode 100644 backend/.sqlx/query-3ca2f72d2917f48644cb79daba9858f472be9f1bb8e504cc3dc2eb998e3f7b91.json create mode 100644 backend/.sqlx/query-44a39475ba202bd5852b666335308360569aae717eff3fe9cc9253a261ae2fa7.json create mode 100644 backend/.sqlx/query-4550ae568abf23045259f95195bf1a35f5dc8f6828c2f3df51520683af48ebbd.json create mode 100644 backend/.sqlx/query-467ed4d282af003dd3b0d9542caec73feba3141f7345ffd1a6554fe44a6a9171.json create mode 100644 backend/.sqlx/query-479427dc09ebacb80cb20d553e93a577bb85b389ab056909a326212565a338bc.json create mode 100644 backend/.sqlx/query-4be42f447e10e420f2a909579398b01b5175e14eee47ca39fe63242a78da767e.json create mode 100644 backend/.sqlx/query-4bf7f1a0fd87e79bb789cee06a5da68e465ce7ce67e80770f22a36a9e8320657.json rename backend/.sqlx/{query-e74e283951aa87627a46aa8286819cf7aa4fecbde17bc7a67ee6f49c427cee9e.json => query-4e4a9c7b6e95f81101a68a6cc6d29fe4424eaed36d5ac9e4cd0a2a9d8b878eeb.json} (59%) create mode 100644 backend/.sqlx/query-5329ce41bbdc36698ea059fec5f05ded7dfe1c64517289ad67028001417d30a3.json create mode 100644 backend/.sqlx/query-5d3560d7a42f86436fec362a790cea189be47e344d11871a90faa9220d509646.json create mode 100644 backend/.sqlx/query-5d5186bb17092425664d8c4f92ca6bfb8b08420cc2b2a52e90064d6a12f86113.json create mode 100644 backend/.sqlx/query-5dcaea907b1ebb2854becc44c00eac5838995dce3d2a696769e00fb74c34288f.json create mode 100644 backend/.sqlx/query-61773dd5d5952607eddfacb1717126a87ee5dc4bd6664eaf31c2c0096fea08db.json create mode 100644 backend/.sqlx/query-6bc9d682aabdf8e79beb693e0090d07f3514628a4970b5f25ddf3920ba5e9c53.json create mode 100644 backend/.sqlx/query-77db9fcef0d3f9c9eb9edba6c0a54469259621218616ef64a4923f3396b183a2.json create mode 100644 backend/.sqlx/query-790e90a3aca284fd060aac049fcf27d2e39781fccc4dd8419f934085d5453fc7.json create mode 100644 backend/.sqlx/query-7a14c6815a7acc912fd8836191263a373b5c427a0b2f48ca775fb8f5a0096e18.json create mode 100644 backend/.sqlx/query-7b040feaa84e85fff1a5ad1ddfc3d57b0f69b05f96ffafae0325cf3ff04f0f40.json create mode 100644 backend/.sqlx/query-7f373cf063907999580d0541a1a4a216e3887fdc0a9bab0747a447c01a4b37a5.json create mode 100644 backend/.sqlx/query-80a5dd06cc5b9f7cb0bcd3b8cdd92c62d40a4ab97fab0f2dd330bd7629c0a632.json rename backend/.sqlx/{query-e27ed86394e6568afdf7a1dd72edd7943c87fdf1d05fba193cbe575cb7281db5.json => query-82b2e8383ae7e345e45fb8a1b2bb362439f5b12a3dd3290a8dab17ccf9c391f3.json} (60%) create mode 100644 backend/.sqlx/query-858b5fe344d79913921d8c05e9e6219919f438e9a3ce0c3e996ea61d9b4f2ef8.json create mode 100644 backend/.sqlx/query-864184467477e73a45935bf9439ae60da7f9ecc12f8f98b926345a70f941845b.json create mode 100644 backend/.sqlx/query-897e8da49e4a5c3efdfb2c36fc7aff3d53432da9ef57f8a775850efcd50a965c.json create mode 100644 backend/.sqlx/query-8c319ac3eb2a289a6709ea1ea1c164f1a56e328902dd46fd9b68d801c16f4ffa.json create mode 100644 backend/.sqlx/query-9071eff54395ca39809a5d55d1e3fedb64cfafa9fe74cd9446eb6ee526c2d9a0.json create mode 100644 backend/.sqlx/query-97a701e16c0ce4b8c6a1394c2a71b8ae45e16c11c8003c5f64c9f53d0e7226aa.json create mode 100644 backend/.sqlx/query-9b1ad1bbf0c2dca3ce1cc9433c35db87e3101bcbcb0fa359885ca3807c1c9ca2.json create mode 100644 backend/.sqlx/query-9d3ca755b323330033eb891ac7162a53440a5c713b58e6c4a467f6f0d5d7e37c.json create mode 100644 backend/.sqlx/query-a4c842e395714346d5178190793d8ae66f891421b72fc6adfa8a38060ca2f3fa.json create mode 100644 backend/.sqlx/query-a5e8cf0e559742330d67d36d9ddb75ef7de6e0ad74e0cc6c4056b782155b86c7.json create mode 100644 backend/.sqlx/query-a7b589b8d5cded97905bc24412ad8f3f0339a2a088243f045a212f9357205407.json create mode 100644 backend/.sqlx/query-b2db4f32c615a99db7af23729682340328da39ac763bb26b6bcf82027fad48e5.json create mode 100644 backend/.sqlx/query-bac36542b16b687a823067c013e3728f1cabe3774f581401514f522ea7522de5.json create mode 100644 backend/.sqlx/query-bbce2221f5724016543a5ac1db7fa74b8b35375dfb0f5d39b398369468ad0774.json create mode 100644 backend/.sqlx/query-bdb7185233941d1472a55c8ade3f61c82248f3f2c9a69c458bff3978ffd17379.json create mode 100644 backend/.sqlx/query-be17a65f144cc21e849c7f8cfbf9d7271b953dda42e4ba09864da7bda731e752.json create mode 100644 backend/.sqlx/query-d10efb37765ac9a7f2e15f71dbbc02ec917b6ecb014f938e68ced3cc7fe9dc86.json create mode 100644 backend/.sqlx/query-d5cdb1121f2a414c0a70f4f7e4ae630bad0715bbbb3597cb1b14e41abe9fec14.json create mode 100644 backend/.sqlx/query-db19932d940b2467147eb2c13b059f2ec33b0c83b5c4a332a4b6e54d2591767c.json create mode 100644 backend/.sqlx/query-db63d41718c50e264a949885d2d7c13719ae650e9c788ebf125627b785fb9ee0.json create mode 100644 backend/.sqlx/query-e3c080b84f50622e0a74524111ae292c958e71293695ce18f9905af4dd940495.json create mode 100644 backend/.sqlx/query-e6d7e9779eaa6e584b613675ae8602c405fe620937a4563a17db80219a2930d2.json create mode 100644 backend/.sqlx/query-e719e98cceef1383f882632c9398dd950ad0b1f30cde9c0392384c9a490290b9.json create mode 100644 backend/.sqlx/query-e72d71852b6c5998b8d46407d164fcee17400a0ffc172d2a257c3fc15617fbd9.json create mode 100644 backend/.sqlx/query-e942a74104771b192c33ea03dbcaa409e03b6bfb7423929c201f39db4228cc36.json create mode 100644 backend/.sqlx/query-eb0f25a10f4f1264e482674c06783204fddf0ccdca6d7c798a4ff6527548e96f.json create mode 100644 backend/.sqlx/query-ee40e48afb5520b7ff84883204f064ce566cea6655e3cd11a9d7016cb08482cb.json create mode 100644 backend/.sqlx/query-f0e943244b125d0c42a9b472701ec172af9dd334786275c391e80f1c38bbb45b.json create mode 100644 backend/.sqlx/query-f59afd524e3f216487ad0a780b1fda6341e215025dc6d8985891b91dad6d9dcb.json create mode 100644 backend/.sqlx/query-f6c4c40b098ba06f4b3b057af1f5dce0711e90f7a4e1786d39a50d52400a3cb9.json rename backend/.sqlx/{query-ef59abddc518f5213827e47a31aee49be917a61c46916c29d79c094438b1ff35.json => query-f907114909b1064a3d5eb603e5929a6ffebdcadb20e4c6a68768a9ad42fe9328.json} (69%) create mode 100644 backend/.sqlx/query-facbf7337d7ffa3f3e6287e2910ccd7ca7229f9d4e5c3af6c460ee7fe3c946e6.json create mode 100644 backend/.sqlx/query-fcf570337b2ceeb0f9dcc311144d8ada02aa4353feb2f63e7a1f520a4000ed70.json create mode 100644 backend/.sqlx/query-fd12bbe0605c80fced218a5a2e1288e6c04f4a1a76a2038679f8f538320cab1f.json create mode 100644 backend/migrations/20260812100659_ai_evals.down.sql create mode 100644 backend/migrations/20260812100659_ai_evals.up.sql create mode 100644 backend/migrations/20260819073729_resource_version_number.down.sql create mode 100644 backend/migrations/20260819073729_resource_version_number.up.sql create mode 100644 backend/windmill-api/src/ai_evals/datasets.rs create mode 100644 backend/windmill-api/src/ai_evals/mod.rs create mode 100644 backend/windmill-api/src/ai_evals/payload.rs create mode 100644 backend/windmill-api/src/ai_evals/results.rs create mode 100644 backend/windmill-api/src/ai_evals/run.rs create mode 100644 backend/windmill-api/src/ai_evals/scorers.rs create mode 100644 backend/windmill-api/src/ai_evals/scoring.rs create mode 100644 backend/windmill-api/src/ai_evals/subject.rs create mode 100644 backend/windmill-api/src/ai_evals/template.rs create mode 100644 docs/ai-agent-evals.md create mode 100644 frontend/src/lib/components/aiEvals/AddScorer.svelte create mode 100644 frontend/src/lib/components/aiEvals/AgentEvalModal.svelte create mode 100644 frontend/src/lib/components/aiEvals/EvalCasesGrid.svelte create mode 100644 frontend/src/lib/components/aiEvals/EvalDatasetDrawer.svelte create mode 100644 frontend/src/lib/components/aiEvals/EvalRunDialog.svelte create mode 100644 frontend/src/lib/components/aiEvals/EvalRunsList.svelte create mode 100644 frontend/src/lib/components/aiEvals/EvalScorers.svelte create mode 100644 frontend/src/lib/components/aiEvals/EvalsPane.svelte create mode 100644 frontend/src/lib/components/aiEvals/evalUtils.test.ts create mode 100644 frontend/src/lib/components/aiEvals/evalUtils.ts create mode 100644 frontend/src/lib/components/apps/components/display/table/multilineCellEditor.css create mode 100644 frontend/src/lib/components/apps/components/display/table/multilineCellEditor.ts diff --git a/backend/.sqlx/query-01bce88dd622f314d1a09c24cd12df5e7d3ff6a15a93e1c0c95e262f7b3d0ef1.json b/backend/.sqlx/query-01bce88dd622f314d1a09c24cd12df5e7d3ff6a15a93e1c0c95e262f7b3d0ef1.json new file mode 100644 index 0000000000..64bf73746f --- /dev/null +++ b/backend/.sqlx/query-01bce88dd622f314d1a09c24cd12df5e7d3ff6a15a93e1c0c95e262f7b3d0ef1.json @@ -0,0 +1,17 @@ +{ + "db_name": "PostgreSQL", + "query": "UPDATE eval_experiment\n SET subject = jsonb_set(\n jsonb_set(subject, '{kind}', '\"agent\"'),\n '{version}', to_jsonb($4::bigint))\n WHERE workspace_id = $1 AND dataset_path = $2 AND id = $3\n AND subject ->> 'kind' = 'agent_draft'", + "describe": { + "columns": [], + "parameters": { + "Left": [ + "Text", + "Text", + "Uuid", + "Int8" + ] + }, + "nullable": [] + }, + "hash": "01bce88dd622f314d1a09c24cd12df5e7d3ff6a15a93e1c0c95e262f7b3d0ef1" +} diff --git a/backend/.sqlx/query-0276e6030abb2eb00a68c568a9cc60f3e7c2af0331388c4b358035de865a121a.json b/backend/.sqlx/query-0276e6030abb2eb00a68c568a9cc60f3e7c2af0331388c4b358035de865a121a.json new file mode 100644 index 0000000000..a7452b0602 --- /dev/null +++ b/backend/.sqlx/query-0276e6030abb2eb00a68c568a9cc60f3e7c2af0331388c4b358035de865a121a.json @@ -0,0 +1,35 @@ +{ + "db_name": "PostgreSQL", + "query": "SELECT path, value, version FROM resource_version WHERE workspace_id = $1 AND id = $2", + "describe": { + "columns": [ + { + "ordinal": 0, + "name": "path", + "type_info": "Varchar" + }, + { + "ordinal": 1, + "name": "value", + "type_info": "Jsonb" + }, + { + "ordinal": 2, + "name": "version", + "type_info": "Int8" + } + ], + "parameters": { + "Left": [ + "Text", + "Int8" + ] + }, + "nullable": [ + false, + true, + false + ] + }, + "hash": "0276e6030abb2eb00a68c568a9cc60f3e7c2af0331388c4b358035de865a121a" +} diff --git a/backend/.sqlx/query-0335de6713de6678b9bf266121af23abc46d5db95da095bb15726c5a2db7ad2f.json b/backend/.sqlx/query-0335de6713de6678b9bf266121af23abc46d5db95da095bb15726c5a2db7ad2f.json new file mode 100644 index 0000000000..f56ef7ead0 --- /dev/null +++ b/backend/.sqlx/query-0335de6713de6678b9bf266121af23abc46d5db95da095bb15726c5a2db7ad2f.json @@ -0,0 +1,24 @@ +{ + "db_name": "PostgreSQL", + "query": "SELECT coalesce(max(run_number), 0) + 1 FROM eval_experiment\n WHERE workspace_id = $1 AND dataset_path = $2 AND subject ->> 'path' = $3", + "describe": { + "columns": [ + { + "ordinal": 0, + "name": "?column?", + "type_info": "Int4" + } + ], + "parameters": { + "Left": [ + "Text", + "Text", + "Text" + ] + }, + "nullable": [ + null + ] + }, + "hash": "0335de6713de6678b9bf266121af23abc46d5db95da095bb15726c5a2db7ad2f" +} diff --git a/backend/.sqlx/query-0aae275d9196e742b5783df4e67c72459d45e275bfeafa2952349cae259ac9f0.json b/backend/.sqlx/query-0aae275d9196e742b5783df4e67c72459d45e275bfeafa2952349cae259ac9f0.json new file mode 100644 index 0000000000..25e389fd30 --- /dev/null +++ b/backend/.sqlx/query-0aae275d9196e742b5783df4e67c72459d45e275bfeafa2952349cae259ac9f0.json @@ -0,0 +1,16 @@ +{ + "db_name": "PostgreSQL", + "query": "UPDATE eval_dataset SET scorers = COALESCE((\n SELECT jsonb_agg(\n CASE WHEN elem->>'path' LIKE ('u/' || $2 || '/%')\n THEN jsonb_set(elem, '{path}', to_jsonb(REGEXP_REPLACE(elem->>'path', 'u/' || $2 || '/(.*)', $1 || '/\\1')))\n ELSE elem END)\n FROM jsonb_array_elements(scorers) elem), '[]'::jsonb)\n WHERE workspace_id = $3\n AND EXISTS (SELECT 1 FROM jsonb_array_elements(scorers) e WHERE e->>'path' LIKE ('u/' || $2 || '/%'))", + "describe": { + "columns": [], + "parameters": { + "Left": [ + "Text", + "Text", + "Text" + ] + }, + "nullable": [] + }, + "hash": "0aae275d9196e742b5783df4e67c72459d45e275bfeafa2952349cae259ac9f0" +} diff --git a/backend/.sqlx/query-0d6700ccffb8179e365bbc1f03398e474f23b013f642ba29ad6f68e6f047c1e5.json b/backend/.sqlx/query-0d6700ccffb8179e365bbc1f03398e474f23b013f642ba29ad6f68e6f047c1e5.json new file mode 100644 index 0000000000..2ec3036fb0 --- /dev/null +++ b/backend/.sqlx/query-0d6700ccffb8179e365bbc1f03398e474f23b013f642ba29ad6f68e6f047c1e5.json @@ -0,0 +1,24 @@ +{ + "db_name": "PostgreSQL", + "query": "SELECT pg_advisory_xact_lock(hashtext('ai_eval_open:' || $1 || '/' || $2 || '/' || $3))", + "describe": { + "columns": [ + { + "ordinal": 0, + "name": "pg_advisory_xact_lock", + "type_info": "Void" + } + ], + "parameters": { + "Left": [ + "Text", + "Text", + "Text" + ] + }, + "nullable": [ + null + ] + }, + "hash": "0d6700ccffb8179e365bbc1f03398e474f23b013f642ba29ad6f68e6f047c1e5" +} diff --git a/backend/.sqlx/query-1815730982dcaf7239ddcb22f88ae5c79794213cf6278167f8afdbca30b1b15c.json b/backend/.sqlx/query-1815730982dcaf7239ddcb22f88ae5c79794213cf6278167f8afdbca30b1b15c.json new file mode 100644 index 0000000000..a8134921e5 --- /dev/null +++ b/backend/.sqlx/query-1815730982dcaf7239ddcb22f88ae5c79794213cf6278167f8afdbca30b1b15c.json @@ -0,0 +1,16 @@ +{ + "db_name": "PostgreSQL", + "query": "UPDATE eval_experiment SET subject = jsonb_set(subject, '{path}', to_jsonb(REGEXP_REPLACE(subject->>'path', 'u/' || $2 || '/(.*)', $1 || '/\\1'))) WHERE subject->>'path' LIKE ('u/' || $2 || '/%') AND workspace_id = $3", + "describe": { + "columns": [], + "parameters": { + "Left": [ + "Text", + "Text", + "Text" + ] + }, + "nullable": [] + }, + "hash": "1815730982dcaf7239ddcb22f88ae5c79794213cf6278167f8afdbca30b1b15c" +} diff --git a/backend/.sqlx/query-186c663249ffada82abf61ce214f52e2730501774a0ca4dc855380e6c6487917.json b/backend/.sqlx/query-186c663249ffada82abf61ce214f52e2730501774a0ca4dc855380e6c6487917.json new file mode 100644 index 0000000000..5cead4df77 --- /dev/null +++ b/backend/.sqlx/query-186c663249ffada82abf61ce214f52e2730501774a0ca4dc855380e6c6487917.json @@ -0,0 +1,22 @@ +{ + "db_name": "PostgreSQL", + "query": "SELECT count(*) AS \"count!\" FROM eval_experiment_case\n WHERE experiment_id = $1 AND status IS NOT NULL", + "describe": { + "columns": [ + { + "ordinal": 0, + "name": "count!", + "type_info": "Int8" + } + ], + "parameters": { + "Left": [ + "Uuid" + ] + }, + "nullable": [ + null + ] + }, + "hash": "186c663249ffada82abf61ce214f52e2730501774a0ca4dc855380e6c6487917" +} diff --git a/backend/.sqlx/query-196939257a334f7d37aa6d66153b251446d7701a893cb7852b81cf842c0fa228.json b/backend/.sqlx/query-196939257a334f7d37aa6d66153b251446d7701a893cb7852b81cf842c0fa228.json new file mode 100644 index 0000000000..d9643875cf --- /dev/null +++ b/backend/.sqlx/query-196939257a334f7d37aa6d66153b251446d7701a893cb7852b81cf842c0fa228.json @@ -0,0 +1,59 @@ +{ + "db_name": "PostgreSQL", + "query": "SELECT path, summary, scorers, created_at, created_by,\n edited_at, edited_by\n FROM eval_dataset WHERE workspace_id = $1 AND path = $2", + "describe": { + "columns": [ + { + "ordinal": 0, + "name": "path", + "type_info": "Varchar" + }, + { + "ordinal": 1, + "name": "summary", + "type_info": "Varchar" + }, + { + "ordinal": 2, + "name": "scorers", + "type_info": "Jsonb" + }, + { + "ordinal": 3, + "name": "created_at", + "type_info": "Timestamptz" + }, + { + "ordinal": 4, + "name": "created_by", + "type_info": "Varchar" + }, + { + "ordinal": 5, + "name": "edited_at", + "type_info": "Timestamptz" + }, + { + "ordinal": 6, + "name": "edited_by", + "type_info": "Varchar" + } + ], + "parameters": { + "Left": [ + "Text", + "Text" + ] + }, + "nullable": [ + false, + true, + false, + false, + false, + false, + false + ] + }, + "hash": "196939257a334f7d37aa6d66153b251446d7701a893cb7852b81cf842c0fa228" +} diff --git a/backend/.sqlx/query-1b6e229545f6b877e72d21728257d1bddaba15ef0fbe72bb4f43b45140f184ce.json b/backend/.sqlx/query-1b6e229545f6b877e72d21728257d1bddaba15ef0fbe72bb4f43b45140f184ce.json new file mode 100644 index 0000000000..54e381be1a --- /dev/null +++ b/backend/.sqlx/query-1b6e229545f6b877e72d21728257d1bddaba15ef0fbe72bb4f43b45140f184ce.json @@ -0,0 +1,28 @@ +{ + "db_name": "PostgreSQL", + "query": "SELECT case_id, ordinal FROM eval_experiment_case WHERE experiment_id = $1", + "describe": { + "columns": [ + { + "ordinal": 0, + "name": "case_id", + "type_info": "Uuid" + }, + { + "ordinal": 1, + "name": "ordinal", + "type_info": "Int4" + } + ], + "parameters": { + "Left": [ + "Uuid" + ] + }, + "nullable": [ + false, + false + ] + }, + "hash": "1b6e229545f6b877e72d21728257d1bddaba15ef0fbe72bb4f43b45140f184ce" +} diff --git a/backend/.sqlx/query-1db80f3ba2c6c769a98424ebf9aaf168a4fa2c64e446a824038cf267236fe979.json b/backend/.sqlx/query-1db80f3ba2c6c769a98424ebf9aaf168a4fa2c64e446a824038cf267236fe979.json new file mode 100644 index 0000000000..01bae0f126 --- /dev/null +++ b/backend/.sqlx/query-1db80f3ba2c6c769a98424ebf9aaf168a4fa2c64e446a824038cf267236fe979.json @@ -0,0 +1,23 @@ +{ + "db_name": "PostgreSQL", + "query": "SELECT DISTINCT path FROM script\n WHERE workspace_id = $1 AND path = ANY($2)\n AND deleted = false AND lock IS NOT NULL AND lock_error_logs IS NULL", + "describe": { + "columns": [ + { + "ordinal": 0, + "name": "path", + "type_info": "Varchar" + } + ], + "parameters": { + "Left": [ + "Text", + "TextArray" + ] + }, + "nullable": [ + false + ] + }, + "hash": "1db80f3ba2c6c769a98424ebf9aaf168a4fa2c64e446a824038cf267236fe979" +} diff --git a/backend/.sqlx/query-242845c86084e010ab33c2197d44af9aeb181672a2f2330dbe65bfe586376450.json b/backend/.sqlx/query-242845c86084e010ab33c2197d44af9aeb181672a2f2330dbe65bfe586376450.json new file mode 100644 index 0000000000..38dcbadfb0 --- /dev/null +++ b/backend/.sqlx/query-242845c86084e010ab33c2197d44af9aeb181672a2f2330dbe65bfe586376450.json @@ -0,0 +1,15 @@ +{ + "db_name": "PostgreSQL", + "query": "UPDATE eval_score SET error = 'The case did not run'\n WHERE experiment_id = $1 AND ordinal = ANY($2)\n AND score IS NULL AND error IS NULL AND NOT not_applicable", + "describe": { + "columns": [], + "parameters": { + "Left": [ + "Uuid", + "Int4Array" + ] + }, + "nullable": [] + }, + "hash": "242845c86084e010ab33c2197d44af9aeb181672a2f2330dbe65bfe586376450" +} diff --git a/backend/.sqlx/query-2b41dc4d872af0e230c31bef1a496604c1562cdfb0bdb54bd3b0c6579d73e46d.json b/backend/.sqlx/query-2b41dc4d872af0e230c31bef1a496604c1562cdfb0bdb54bd3b0c6579d73e46d.json new file mode 100644 index 0000000000..4506a038fc --- /dev/null +++ b/backend/.sqlx/query-2b41dc4d872af0e230c31bef1a496604c1562cdfb0bdb54bd3b0c6579d73e46d.json @@ -0,0 +1,28 @@ +{ + "db_name": "PostgreSQL", + "query": "SELECT path, scorers FROM eval_dataset\n WHERE workspace_id = $1 ORDER BY edited_at DESC LIMIT 100", + "describe": { + "columns": [ + { + "ordinal": 0, + "name": "path", + "type_info": "Varchar" + }, + { + "ordinal": 1, + "name": "scorers", + "type_info": "Jsonb" + } + ], + "parameters": { + "Left": [ + "Text" + ] + }, + "nullable": [ + false, + false + ] + }, + "hash": "2b41dc4d872af0e230c31bef1a496604c1562cdfb0bdb54bd3b0c6579d73e46d" +} diff --git a/backend/.sqlx/query-307d5b797e51122dbf087e5dc9f33235a349731828d500c74a28b577469e2624.json b/backend/.sqlx/query-307d5b797e51122dbf087e5dc9f33235a349731828d500c74a28b577469e2624.json new file mode 100644 index 0000000000..2cfec01e9b --- /dev/null +++ b/backend/.sqlx/query-307d5b797e51122dbf087e5dc9f33235a349731828d500c74a28b577469e2624.json @@ -0,0 +1,23 @@ +{ + "db_name": "PostgreSQL", + "query": "SELECT path FROM resource WHERE workspace_id = $1 AND path = ANY($2) AND resource_type = 'ai_agent'", + "describe": { + "columns": [ + { + "ordinal": 0, + "name": "path", + "type_info": "Varchar" + } + ], + "parameters": { + "Left": [ + "Text", + "TextArray" + ] + }, + "nullable": [ + false + ] + }, + "hash": "307d5b797e51122dbf087e5dc9f33235a349731828d500c74a28b577469e2624" +} diff --git a/backend/.sqlx/query-30b8590939bf3d6770cabca9f4ab3527e750d85d3b8cc0eadef55461de4a2687.json b/backend/.sqlx/query-30b8590939bf3d6770cabca9f4ab3527e750d85d3b8cc0eadef55461de4a2687.json new file mode 100644 index 0000000000..d3a443a0bf --- /dev/null +++ b/backend/.sqlx/query-30b8590939bf3d6770cabca9f4ab3527e750d85d3b8cc0eadef55461de4a2687.json @@ -0,0 +1,23 @@ +{ + "db_name": "PostgreSQL", + "query": "SELECT path FROM eval_dataset WHERE path LIKE $1 AND workspace_id = $2", + "describe": { + "columns": [ + { + "ordinal": 0, + "name": "path", + "type_info": "Varchar" + } + ], + "parameters": { + "Left": [ + "Text", + "Text" + ] + }, + "nullable": [ + false + ] + }, + "hash": "30b8590939bf3d6770cabca9f4ab3527e750d85d3b8cc0eadef55461de4a2687" +} diff --git a/backend/.sqlx/query-316e7c86082b6ee2864b88674cfac5b5b34c5de851f1c6f832b868101edcb052.json b/backend/.sqlx/query-316e7c86082b6ee2864b88674cfac5b5b34c5de851f1c6f832b868101edcb052.json new file mode 100644 index 0000000000..a3443236ea --- /dev/null +++ b/backend/.sqlx/query-316e7c86082b6ee2864b88674cfac5b5b34c5de851f1c6f832b868101edcb052.json @@ -0,0 +1,23 @@ +{ + "db_name": "PostgreSQL", + "query": "SELECT version FROM resource_version WHERE workspace_id = $1 AND path = $2\n ORDER BY version DESC LIMIT 1", + "describe": { + "columns": [ + { + "ordinal": 0, + "name": "version", + "type_info": "Int8" + } + ], + "parameters": { + "Left": [ + "Text", + "Text" + ] + }, + "nullable": [ + false + ] + }, + "hash": "316e7c86082b6ee2864b88674cfac5b5b34c5de851f1c6f832b868101edcb052" +} diff --git a/backend/.sqlx/query-34fbb2b141ad691e0cdc55bf2ebd6974e02a7c81d91217d3f74e03113edc9b0a.json b/backend/.sqlx/query-34fbb2b141ad691e0cdc55bf2ebd6974e02a7c81d91217d3f74e03113edc9b0a.json new file mode 100644 index 0000000000..c79145adaf --- /dev/null +++ b/backend/.sqlx/query-34fbb2b141ad691e0cdc55bf2ebd6974e02a7c81d91217d3f74e03113edc9b0a.json @@ -0,0 +1,16 @@ +{ + "db_name": "PostgreSQL", + "query": "DELETE FROM eval_case\n WHERE workspace_id = $1 AND dataset_path = $2 AND NOT (id = ANY($3))", + "describe": { + "columns": [], + "parameters": { + "Left": [ + "Text", + "Text", + "UuidArray" + ] + }, + "nullable": [] + }, + "hash": "34fbb2b141ad691e0cdc55bf2ebd6974e02a7c81d91217d3f74e03113edc9b0a" +} diff --git a/backend/.sqlx/query-3ca2f72d2917f48644cb79daba9858f472be9f1bb8e504cc3dc2eb998e3f7b91.json b/backend/.sqlx/query-3ca2f72d2917f48644cb79daba9858f472be9f1bb8e504cc3dc2eb998e3f7b91.json new file mode 100644 index 0000000000..fd5e4325a0 --- /dev/null +++ b/backend/.sqlx/query-3ca2f72d2917f48644cb79daba9858f472be9f1bb8e504cc3dc2eb998e3f7b91.json @@ -0,0 +1,16 @@ +{ + "db_name": "PostgreSQL", + "query": "UPDATE eval_experiment SET created_by = $1 WHERE created_by = $2 AND workspace_id = $3", + "describe": { + "columns": [], + "parameters": { + "Left": [ + "Varchar", + "Text", + "Text" + ] + }, + "nullable": [] + }, + "hash": "3ca2f72d2917f48644cb79daba9858f472be9f1bb8e504cc3dc2eb998e3f7b91" +} diff --git a/backend/.sqlx/query-44a39475ba202bd5852b666335308360569aae717eff3fe9cc9253a261ae2fa7.json b/backend/.sqlx/query-44a39475ba202bd5852b666335308360569aae717eff3fe9cc9253a261ae2fa7.json new file mode 100644 index 0000000000..cf0806757d --- /dev/null +++ b/backend/.sqlx/query-44a39475ba202bd5852b666335308360569aae717eff3fe9cc9253a261ae2fa7.json @@ -0,0 +1,23 @@ +{ + "db_name": "PostgreSQL", + "query": "SELECT status::text AS \"status!\" FROM v2_job_completed WHERE id = $1 AND workspace_id = $2", + "describe": { + "columns": [ + { + "ordinal": 0, + "name": "status!", + "type_info": "Text" + } + ], + "parameters": { + "Left": [ + "Uuid", + "Text" + ] + }, + "nullable": [ + null + ] + }, + "hash": "44a39475ba202bd5852b666335308360569aae717eff3fe9cc9253a261ae2fa7" +} diff --git a/backend/.sqlx/query-4550ae568abf23045259f95195bf1a35f5dc8f6828c2f3df51520683af48ebbd.json b/backend/.sqlx/query-4550ae568abf23045259f95195bf1a35f5dc8f6828c2f3df51520683af48ebbd.json new file mode 100644 index 0000000000..fd2ab8fbee --- /dev/null +++ b/backend/.sqlx/query-4550ae568abf23045259f95195bf1a35f5dc8f6828c2f3df51520683af48ebbd.json @@ -0,0 +1,16 @@ +{ + "db_name": "PostgreSQL", + "query": "UPDATE eval_dataset SET path = REGEXP_REPLACE(path,'u/' || $2 || '/(.*)','u/' || $1 || '/\\1') WHERE path LIKE ('u/' || $2 || '/%') AND workspace_id = $3", + "describe": { + "columns": [], + "parameters": { + "Left": [ + "Text", + "Text", + "Text" + ] + }, + "nullable": [] + }, + "hash": "4550ae568abf23045259f95195bf1a35f5dc8f6828c2f3df51520683af48ebbd" +} diff --git a/backend/.sqlx/query-467ed4d282af003dd3b0d9542caec73feba3141f7345ffd1a6554fe44a6a9171.json b/backend/.sqlx/query-467ed4d282af003dd3b0d9542caec73feba3141f7345ffd1a6554fe44a6a9171.json new file mode 100644 index 0000000000..5399d071a9 --- /dev/null +++ b/backend/.sqlx/query-467ed4d282af003dd3b0d9542caec73feba3141f7345ffd1a6554fe44a6a9171.json @@ -0,0 +1,24 @@ +{ + "db_name": "PostgreSQL", + "query": "WITH updated AS (\n UPDATE eval_dataset SET path = REGEXP_REPLACE(path, 'u/' || $2 || '/(.*)', $1 || '/\\1')\n WHERE path LIKE ('u/' || $2 || '/%') AND workspace_id = $3\n RETURNING 1\n ) SELECT COUNT(*) FROM updated", + "describe": { + "columns": [ + { + "ordinal": 0, + "name": "count", + "type_info": "Int8" + } + ], + "parameters": { + "Left": [ + "Text", + "Text", + "Text" + ] + }, + "nullable": [ + null + ] + }, + "hash": "467ed4d282af003dd3b0d9542caec73feba3141f7345ffd1a6554fe44a6a9171" +} diff --git a/backend/.sqlx/query-479427dc09ebacb80cb20d553e93a577bb85b389ab056909a326212565a338bc.json b/backend/.sqlx/query-479427dc09ebacb80cb20d553e93a577bb85b389ab056909a326212565a338bc.json new file mode 100644 index 0000000000..8dd3b1ec7d --- /dev/null +++ b/backend/.sqlx/query-479427dc09ebacb80cb20d553e93a577bb85b389ab056909a326212565a338bc.json @@ -0,0 +1,15 @@ +{ + "db_name": "PostgreSQL", + "query": "INSERT INTO eval_dataset (workspace_id, path, summary, scorers, extra_perms, created_at, created_by, edited_at, edited_by)\n SELECT $2, path, summary, scorers, extra_perms, created_at, created_by, edited_at, edited_by\n FROM eval_dataset WHERE workspace_id = $1", + "describe": { + "columns": [], + "parameters": { + "Left": [ + "Text", + "Varchar" + ] + }, + "nullable": [] + }, + "hash": "479427dc09ebacb80cb20d553e93a577bb85b389ab056909a326212565a338bc" +} diff --git a/backend/.sqlx/query-4be42f447e10e420f2a909579398b01b5175e14eee47ca39fe63242a78da767e.json b/backend/.sqlx/query-4be42f447e10e420f2a909579398b01b5175e14eee47ca39fe63242a78da767e.json new file mode 100644 index 0000000000..2107f75f72 --- /dev/null +++ b/backend/.sqlx/query-4be42f447e10e420f2a909579398b01b5175e14eee47ca39fe63242a78da767e.json @@ -0,0 +1,16 @@ +{ + "db_name": "PostgreSQL", + "query": "UPDATE eval_case SET created_by = $1 WHERE created_by = $2 AND workspace_id = $3", + "describe": { + "columns": [], + "parameters": { + "Left": [ + "Varchar", + "Text", + "Text" + ] + }, + "nullable": [] + }, + "hash": "4be42f447e10e420f2a909579398b01b5175e14eee47ca39fe63242a78da767e" +} diff --git a/backend/.sqlx/query-4bf7f1a0fd87e79bb789cee06a5da68e465ce7ce67e80770f22a36a9e8320657.json b/backend/.sqlx/query-4bf7f1a0fd87e79bb789cee06a5da68e465ce7ce67e80770f22a36a9e8320657.json new file mode 100644 index 0000000000..a4ddcb531a --- /dev/null +++ b/backend/.sqlx/query-4bf7f1a0fd87e79bb789cee06a5da68e465ce7ce67e80770f22a36a9e8320657.json @@ -0,0 +1,59 @@ +{ + "db_name": "PostgreSQL", + "query": "SELECT path, summary, scorers, created_at, created_by, edited_at, edited_by\n FROM eval_dataset WHERE workspace_id = $1 AND path = $2 FOR UPDATE", + "describe": { + "columns": [ + { + "ordinal": 0, + "name": "path", + "type_info": "Varchar" + }, + { + "ordinal": 1, + "name": "summary", + "type_info": "Varchar" + }, + { + "ordinal": 2, + "name": "scorers", + "type_info": "Jsonb" + }, + { + "ordinal": 3, + "name": "created_at", + "type_info": "Timestamptz" + }, + { + "ordinal": 4, + "name": "created_by", + "type_info": "Varchar" + }, + { + "ordinal": 5, + "name": "edited_at", + "type_info": "Timestamptz" + }, + { + "ordinal": 6, + "name": "edited_by", + "type_info": "Varchar" + } + ], + "parameters": { + "Left": [ + "Text", + "Text" + ] + }, + "nullable": [ + false, + true, + false, + false, + false, + false, + false + ] + }, + "hash": "4bf7f1a0fd87e79bb789cee06a5da68e465ce7ce67e80770f22a36a9e8320657" +} diff --git a/backend/.sqlx/query-e74e283951aa87627a46aa8286819cf7aa4fecbde17bc7a67ee6f49c427cee9e.json b/backend/.sqlx/query-4e4a9c7b6e95f81101a68a6cc6d29fe4424eaed36d5ac9e4cd0a2a9d8b878eeb.json similarity index 59% rename from backend/.sqlx/query-e74e283951aa87627a46aa8286819cf7aa4fecbde17bc7a67ee6f49c427cee9e.json rename to backend/.sqlx/query-4e4a9c7b6e95f81101a68a6cc6d29fe4424eaed36d5ac9e4cd0a2a9d8b878eeb.json index 77399cdf62..0796577849 100644 --- a/backend/.sqlx/query-e74e283951aa87627a46aa8286819cf7aa4fecbde17bc7a67ee6f49c427cee9e.json +++ b/backend/.sqlx/query-4e4a9c7b6e95f81101a68a6cc6d29fe4424eaed36d5ac9e4cd0a2a9d8b878eeb.json @@ -1,6 +1,6 @@ { "db_name": "PostgreSQL", - "query": "SELECT path, value FROM resource_version WHERE workspace_id = $1 AND id = $2", + "query": "SELECT path, scorers FROM eval_dataset WHERE workspace_id = $1 AND path = ANY($2)", "describe": { "columns": [ { @@ -10,20 +10,20 @@ }, { "ordinal": 1, - "name": "value", + "name": "scorers", "type_info": "Jsonb" } ], "parameters": { "Left": [ "Text", - "Int8" + "TextArray" ] }, "nullable": [ false, - true + false ] }, - "hash": "e74e283951aa87627a46aa8286819cf7aa4fecbde17bc7a67ee6f49c427cee9e" + "hash": "4e4a9c7b6e95f81101a68a6cc6d29fe4424eaed36d5ac9e4cd0a2a9d8b878eeb" } diff --git a/backend/.sqlx/query-5329ce41bbdc36698ea059fec5f05ded7dfe1c64517289ad67028001417d30a3.json b/backend/.sqlx/query-5329ce41bbdc36698ea059fec5f05ded7dfe1c64517289ad67028001417d30a3.json new file mode 100644 index 0000000000..54019fe5bc --- /dev/null +++ b/backend/.sqlx/query-5329ce41bbdc36698ea059fec5f05ded7dfe1c64517289ad67028001417d30a3.json @@ -0,0 +1,47 @@ +{ + "db_name": "PostgreSQL", + "query": "SELECT id, input, expected, created_at, created_by\n FROM eval_case\n WHERE workspace_id = $1 AND dataset_path = $2\n ORDER BY created_at, id", + "describe": { + "columns": [ + { + "ordinal": 0, + "name": "id", + "type_info": "Uuid" + }, + { + "ordinal": 1, + "name": "input", + "type_info": "Jsonb" + }, + { + "ordinal": 2, + "name": "expected", + "type_info": "Jsonb" + }, + { + "ordinal": 3, + "name": "created_at", + "type_info": "Timestamptz" + }, + { + "ordinal": 4, + "name": "created_by", + "type_info": "Varchar" + } + ], + "parameters": { + "Left": [ + "Text", + "Text" + ] + }, + "nullable": [ + false, + false, + true, + false, + false + ] + }, + "hash": "5329ce41bbdc36698ea059fec5f05ded7dfe1c64517289ad67028001417d30a3" +} diff --git a/backend/.sqlx/query-5a219a2532517869578c4504ff3153c43903f929ae5d62fbba12610f89c36d55.json b/backend/.sqlx/query-5a219a2532517869578c4504ff3153c43903f929ae5d62fbba12610f89c36d55.json index 36ddb8ab9f..713ccb9dd3 100644 --- a/backend/.sqlx/query-5a219a2532517869578c4504ff3153c43903f929ae5d62fbba12610f89c36d55.json +++ b/backend/.sqlx/query-5a219a2532517869578c4504ff3153c43903f929ae5d62fbba12610f89c36d55.json @@ -15,7 +15,7 @@ ] }, "nullable": [ - true + null ] }, "hash": "5a219a2532517869578c4504ff3153c43903f929ae5d62fbba12610f89c36d55" diff --git a/backend/.sqlx/query-5d3560d7a42f86436fec362a790cea189be47e344d11871a90faa9220d509646.json b/backend/.sqlx/query-5d3560d7a42f86436fec362a790cea189be47e344d11871a90faa9220d509646.json new file mode 100644 index 0000000000..548516f69c --- /dev/null +++ b/backend/.sqlx/query-5d3560d7a42f86436fec362a790cea189be47e344d11871a90faa9220d509646.json @@ -0,0 +1,16 @@ +{ + "db_name": "PostgreSQL", + "query": "UPDATE eval_experiment_case c SET job_id = j.id\n FROM v2_job j\n WHERE j.parent_job = $3 AND j.workspace_id = $2\n AND (j.args -> 'iter' -> 'value' ->> 'case_id')::uuid = c.case_id\n AND c.experiment_id = $1 AND c.job_id IS NULL", + "describe": { + "columns": [], + "parameters": { + "Left": [ + "Uuid", + "Text", + "Uuid" + ] + }, + "nullable": [] + }, + "hash": "5d3560d7a42f86436fec362a790cea189be47e344d11871a90faa9220d509646" +} diff --git a/backend/.sqlx/query-5d5186bb17092425664d8c4f92ca6bfb8b08420cc2b2a52e90064d6a12f86113.json b/backend/.sqlx/query-5d5186bb17092425664d8c4f92ca6bfb8b08420cc2b2a52e90064d6a12f86113.json new file mode 100644 index 0000000000..8bf4956d79 --- /dev/null +++ b/backend/.sqlx/query-5d5186bb17092425664d8c4f92ca6bfb8b08420cc2b2a52e90064d6a12f86113.json @@ -0,0 +1,76 @@ +{ + "db_name": "PostgreSQL", + "query": "SELECT ordinal, case_id, input, expected, job_id, subject_version,\n subject_draft_hash, output, answered, status\n FROM eval_experiment_case\n WHERE experiment_id = $1 ORDER BY ordinal", + "describe": { + "columns": [ + { + "ordinal": 0, + "name": "ordinal", + "type_info": "Int4" + }, + { + "ordinal": 1, + "name": "case_id", + "type_info": "Uuid" + }, + { + "ordinal": 2, + "name": "input", + "type_info": "Jsonb" + }, + { + "ordinal": 3, + "name": "expected", + "type_info": "Jsonb" + }, + { + "ordinal": 4, + "name": "job_id", + "type_info": "Uuid" + }, + { + "ordinal": 5, + "name": "subject_version", + "type_info": "Int8" + }, + { + "ordinal": 6, + "name": "subject_draft_hash", + "type_info": "Varchar" + }, + { + "ordinal": 7, + "name": "output", + "type_info": "Text" + }, + { + "ordinal": 8, + "name": "answered", + "type_info": "Bool" + }, + { + "ordinal": 9, + "name": "status", + "type_info": "Varchar" + } + ], + "parameters": { + "Left": [ + "Uuid" + ] + }, + "nullable": [ + false, + false, + false, + true, + true, + true, + true, + true, + true, + true + ] + }, + "hash": "5d5186bb17092425664d8c4f92ca6bfb8b08420cc2b2a52e90064d6a12f86113" +} diff --git a/backend/.sqlx/query-5dcaea907b1ebb2854becc44c00eac5838995dce3d2a696769e00fb74c34288f.json b/backend/.sqlx/query-5dcaea907b1ebb2854becc44c00eac5838995dce3d2a696769e00fb74c34288f.json new file mode 100644 index 0000000000..5c3ee8da92 --- /dev/null +++ b/backend/.sqlx/query-5dcaea907b1ebb2854becc44c00eac5838995dce3d2a696769e00fb74c34288f.json @@ -0,0 +1,22 @@ +{ + "db_name": "PostgreSQL", + "query": "SELECT DISTINCT experiment_id FROM eval_score\n WHERE experiment_id = ANY($1) AND score IS NULL AND error IS NULL\n AND NOT not_applicable", + "describe": { + "columns": [ + { + "ordinal": 0, + "name": "experiment_id", + "type_info": "Uuid" + } + ], + "parameters": { + "Left": [ + "UuidArray" + ] + }, + "nullable": [ + false + ] + }, + "hash": "5dcaea907b1ebb2854becc44c00eac5838995dce3d2a696769e00fb74c34288f" +} diff --git a/backend/.sqlx/query-61773dd5d5952607eddfacb1717126a87ee5dc4bd6664eaf31c2c0096fea08db.json b/backend/.sqlx/query-61773dd5d5952607eddfacb1717126a87ee5dc4bd6664eaf31c2c0096fea08db.json new file mode 100644 index 0000000000..278ccbe48e --- /dev/null +++ b/backend/.sqlx/query-61773dd5d5952607eddfacb1717126a87ee5dc4bd6664eaf31c2c0096fea08db.json @@ -0,0 +1,26 @@ +{ + "db_name": "PostgreSQL", + "query": "UPDATE eval_case SET input = $4, expected = $5\n WHERE workspace_id = $1 AND dataset_path = $2 AND id = $3\n RETURNING id", + "describe": { + "columns": [ + { + "ordinal": 0, + "name": "id", + "type_info": "Uuid" + } + ], + "parameters": { + "Left": [ + "Text", + "Text", + "Uuid", + "Jsonb", + "Jsonb" + ] + }, + "nullable": [ + false + ] + }, + "hash": "61773dd5d5952607eddfacb1717126a87ee5dc4bd6664eaf31c2c0096fea08db" +} diff --git a/backend/.sqlx/query-6bc9d682aabdf8e79beb693e0090d07f3514628a4970b5f25ddf3920ba5e9c53.json b/backend/.sqlx/query-6bc9d682aabdf8e79beb693e0090d07f3514628a4970b5f25ddf3920ba5e9c53.json new file mode 100644 index 0000000000..e2a97d9c11 --- /dev/null +++ b/backend/.sqlx/query-6bc9d682aabdf8e79beb693e0090d07f3514628a4970b5f25ddf3920ba5e9c53.json @@ -0,0 +1,23 @@ +{ + "db_name": "PostgreSQL", + "query": "SELECT scorers FROM eval_dataset WHERE workspace_id = $1 AND path = $2 FOR UPDATE", + "describe": { + "columns": [ + { + "ordinal": 0, + "name": "scorers", + "type_info": "Jsonb" + } + ], + "parameters": { + "Left": [ + "Text", + "Text" + ] + }, + "nullable": [ + false + ] + }, + "hash": "6bc9d682aabdf8e79beb693e0090d07f3514628a4970b5f25ddf3920ba5e9c53" +} diff --git a/backend/.sqlx/query-77db9fcef0d3f9c9eb9edba6c0a54469259621218616ef64a4923f3396b183a2.json b/backend/.sqlx/query-77db9fcef0d3f9c9eb9edba6c0a54469259621218616ef64a4923f3396b183a2.json new file mode 100644 index 0000000000..84bd3cc921 --- /dev/null +++ b/backend/.sqlx/query-77db9fcef0d3f9c9eb9edba6c0a54469259621218616ef64a4923f3396b183a2.json @@ -0,0 +1,14 @@ +{ + "db_name": "PostgreSQL", + "query": "DELETE FROM eval_experiment WHERE id = $1", + "describe": { + "columns": [], + "parameters": { + "Left": [ + "Uuid" + ] + }, + "nullable": [] + }, + "hash": "77db9fcef0d3f9c9eb9edba6c0a54469259621218616ef64a4923f3396b183a2" +} diff --git a/backend/.sqlx/query-790e90a3aca284fd060aac049fcf27d2e39781fccc4dd8419f934085d5453fc7.json b/backend/.sqlx/query-790e90a3aca284fd060aac049fcf27d2e39781fccc4dd8419f934085d5453fc7.json new file mode 100644 index 0000000000..101f3e9703 --- /dev/null +++ b/backend/.sqlx/query-790e90a3aca284fd060aac049fcf27d2e39781fccc4dd8419f934085d5453fc7.json @@ -0,0 +1,20 @@ +{ + "db_name": "PostgreSQL", + "query": "INSERT INTO eval_experiment_case\n (experiment_id, ordinal, case_id, input, expected, subject_version,\n subject_draft_hash)\n SELECT $1, ordinal, case_id, input, expected, subject_version, subject_draft_hash\n FROM UNNEST($2::int[], $3::uuid[], $4::jsonb[], $5::jsonb[], $6::bigint[], $7::text[])\n AS t(ordinal, case_id, input, expected, subject_version, subject_draft_hash)", + "describe": { + "columns": [], + "parameters": { + "Left": [ + "Uuid", + "Int4Array", + "UuidArray", + "JsonbArray", + "JsonbArray", + "Int8Array", + "TextArray" + ] + }, + "nullable": [] + }, + "hash": "790e90a3aca284fd060aac049fcf27d2e39781fccc4dd8419f934085d5453fc7" +} diff --git a/backend/.sqlx/query-7a14c6815a7acc912fd8836191263a373b5c427a0b2f48ca775fb8f5a0096e18.json b/backend/.sqlx/query-7a14c6815a7acc912fd8836191263a373b5c427a0b2f48ca775fb8f5a0096e18.json new file mode 100644 index 0000000000..ddcdb3a030 --- /dev/null +++ b/backend/.sqlx/query-7a14c6815a7acc912fd8836191263a373b5c427a0b2f48ca775fb8f5a0096e18.json @@ -0,0 +1,26 @@ +{ + "db_name": "PostgreSQL", + "query": "INSERT INTO eval_case\n (workspace_id, dataset_path, input, expected, created_by, created_at)\n VALUES ($1, $2, $3, $4, $5, clock_timestamp())\n RETURNING id", + "describe": { + "columns": [ + { + "ordinal": 0, + "name": "id", + "type_info": "Uuid" + } + ], + "parameters": { + "Left": [ + "Varchar", + "Varchar", + "Jsonb", + "Jsonb", + "Varchar" + ] + }, + "nullable": [ + false + ] + }, + "hash": "7a14c6815a7acc912fd8836191263a373b5c427a0b2f48ca775fb8f5a0096e18" +} diff --git a/backend/.sqlx/query-7b040feaa84e85fff1a5ad1ddfc3d57b0f69b05f96ffafae0325cf3ff04f0f40.json b/backend/.sqlx/query-7b040feaa84e85fff1a5ad1ddfc3d57b0f69b05f96ffafae0325cf3ff04f0f40.json new file mode 100644 index 0000000000..4665140ce9 --- /dev/null +++ b/backend/.sqlx/query-7b040feaa84e85fff1a5ad1ddfc3d57b0f69b05f96ffafae0325cf3ff04f0f40.json @@ -0,0 +1,60 @@ +{ + "db_name": "PostgreSQL", + "query": "SELECT s.experiment_id AS \"experiment_id!\", s.scorer_id AS \"scorer_id!\",\n avg(s.score) AS mean,\n count(s.score) AS \"scored!\",\n count(*) FILTER (WHERE s.error IS NOT NULL) AS \"failed!\",\n count(*) FILTER (WHERE t.pass_if IS NOT NULL AND s.score >= t.pass_if)\n AS \"passed!\",\n bool_or(t.pass_if IS NOT NULL) AS \"has_threshold!\"\n FROM eval_score s\n JOIN unnest($1::uuid[], $2::text[], $3::float8[])\n AS t(experiment_id, scorer_id, pass_if)\n ON t.experiment_id = s.experiment_id AND t.scorer_id = s.scorer_id\n GROUP BY s.experiment_id, s.scorer_id", + "describe": { + "columns": [ + { + "ordinal": 0, + "name": "experiment_id!", + "type_info": "Uuid" + }, + { + "ordinal": 1, + "name": "scorer_id!", + "type_info": "Varchar" + }, + { + "ordinal": 2, + "name": "mean", + "type_info": "Float8" + }, + { + "ordinal": 3, + "name": "scored!", + "type_info": "Int8" + }, + { + "ordinal": 4, + "name": "failed!", + "type_info": "Int8" + }, + { + "ordinal": 5, + "name": "passed!", + "type_info": "Int8" + }, + { + "ordinal": 6, + "name": "has_threshold!", + "type_info": "Bool" + } + ], + "parameters": { + "Left": [ + "UuidArray", + "TextArray", + "Float8Array" + ] + }, + "nullable": [ + false, + false, + null, + null, + null, + null, + null + ] + }, + "hash": "7b040feaa84e85fff1a5ad1ddfc3d57b0f69b05f96ffafae0325cf3ff04f0f40" +} diff --git a/backend/.sqlx/query-7f373cf063907999580d0541a1a4a216e3887fdc0a9bab0747a447c01a4b37a5.json b/backend/.sqlx/query-7f373cf063907999580d0541a1a4a216e3887fdc0a9bab0747a447c01a4b37a5.json new file mode 100644 index 0000000000..d73778feec --- /dev/null +++ b/backend/.sqlx/query-7f373cf063907999580d0541a1a4a216e3887fdc0a9bab0747a447c01a4b37a5.json @@ -0,0 +1,16 @@ +{ + "db_name": "PostgreSQL", + "query": "UPDATE eval_experiment_case\n SET subject_version = $3, subject_draft_hash = NULL\n WHERE experiment_id = $1 AND subject_draft_hash = $2", + "describe": { + "columns": [], + "parameters": { + "Left": [ + "Uuid", + "Text", + "Int8" + ] + }, + "nullable": [] + }, + "hash": "7f373cf063907999580d0541a1a4a216e3887fdc0a9bab0747a447c01a4b37a5" +} diff --git a/backend/.sqlx/query-80a5dd06cc5b9f7cb0bcd3b8cdd92c62d40a4ab97fab0f2dd330bd7629c0a632.json b/backend/.sqlx/query-80a5dd06cc5b9f7cb0bcd3b8cdd92c62d40a4ab97fab0f2dd330bd7629c0a632.json new file mode 100644 index 0000000000..5b3af727e9 --- /dev/null +++ b/backend/.sqlx/query-80a5dd06cc5b9f7cb0bcd3b8cdd92c62d40a4ab97fab0f2dd330bd7629c0a632.json @@ -0,0 +1,64 @@ +{ + "db_name": "PostgreSQL", + "query": "SELECT ordinal, scorer_id, score, reason, checks, error, not_applicable, definition\n FROM eval_score WHERE experiment_id = $1", + "describe": { + "columns": [ + { + "ordinal": 0, + "name": "ordinal", + "type_info": "Int4" + }, + { + "ordinal": 1, + "name": "scorer_id", + "type_info": "Varchar" + }, + { + "ordinal": 2, + "name": "score", + "type_info": "Float8" + }, + { + "ordinal": 3, + "name": "reason", + "type_info": "Text" + }, + { + "ordinal": 4, + "name": "checks", + "type_info": "Jsonb" + }, + { + "ordinal": 5, + "name": "error", + "type_info": "Text" + }, + { + "ordinal": 6, + "name": "not_applicable", + "type_info": "Bool" + }, + { + "ordinal": 7, + "name": "definition", + "type_info": "Varchar" + } + ], + "parameters": { + "Left": [ + "Uuid" + ] + }, + "nullable": [ + false, + false, + true, + true, + true, + true, + false, + false + ] + }, + "hash": "80a5dd06cc5b9f7cb0bcd3b8cdd92c62d40a4ab97fab0f2dd330bd7629c0a632" +} diff --git a/backend/.sqlx/query-e27ed86394e6568afdf7a1dd72edd7943c87fdf1d05fba193cbe575cb7281db5.json b/backend/.sqlx/query-82b2e8383ae7e345e45fb8a1b2bb362439f5b12a3dd3290a8dab17ccf9c391f3.json similarity index 60% rename from backend/.sqlx/query-e27ed86394e6568afdf7a1dd72edd7943c87fdf1d05fba193cbe575cb7281db5.json rename to backend/.sqlx/query-82b2e8383ae7e345e45fb8a1b2bb362439f5b12a3dd3290a8dab17ccf9c391f3.json index 68825a19c7..a50a31419e 100644 --- a/backend/.sqlx/query-e27ed86394e6568afdf7a1dd72edd7943c87fdf1d05fba193cbe575cb7281db5.json +++ b/backend/.sqlx/query-82b2e8383ae7e345e45fb8a1b2bb362439f5b12a3dd3290a8dab17ccf9c391f3.json @@ -1,6 +1,6 @@ { "db_name": "PostgreSQL", - "query": "SELECT id, created_at, created_by FROM resource_version\n WHERE workspace_id = $1 AND path = $2 ORDER BY id DESC LIMIT $3", + "query": "SELECT id, version, created_at, created_by FROM resource_version\n WHERE workspace_id = $1 AND path = $2 ORDER BY id DESC LIMIT $3", "describe": { "columns": [ { @@ -10,11 +10,16 @@ }, { "ordinal": 1, + "name": "version", + "type_info": "Int8" + }, + { + "ordinal": 2, "name": "created_at", "type_info": "Timestamptz" }, { - "ordinal": 2, + "ordinal": 3, "name": "created_by", "type_info": "Varchar" } @@ -27,10 +32,11 @@ ] }, "nullable": [ + false, false, false, true ] }, - "hash": "e27ed86394e6568afdf7a1dd72edd7943c87fdf1d05fba193cbe575cb7281db5" + "hash": "82b2e8383ae7e345e45fb8a1b2bb362439f5b12a3dd3290a8dab17ccf9c391f3" } diff --git a/backend/.sqlx/query-858b5fe344d79913921d8c05e9e6219919f438e9a3ce0c3e996ea61d9b4f2ef8.json b/backend/.sqlx/query-858b5fe344d79913921d8c05e9e6219919f438e9a3ce0c3e996ea61d9b4f2ef8.json new file mode 100644 index 0000000000..0b8900d23a --- /dev/null +++ b/backend/.sqlx/query-858b5fe344d79913921d8c05e9e6219919f438e9a3ce0c3e996ea61d9b4f2ef8.json @@ -0,0 +1,16 @@ +{ + "db_name": "PostgreSQL", + "query": "UPDATE eval_dataset SET edited_by = $1 WHERE edited_by = $2 AND workspace_id = $3", + "describe": { + "columns": [], + "parameters": { + "Left": [ + "Varchar", + "Text", + "Text" + ] + }, + "nullable": [] + }, + "hash": "858b5fe344d79913921d8c05e9e6219919f438e9a3ce0c3e996ea61d9b4f2ef8" +} diff --git a/backend/.sqlx/query-864184467477e73a45935bf9439ae60da7f9ecc12f8f98b926345a70f941845b.json b/backend/.sqlx/query-864184467477e73a45935bf9439ae60da7f9ecc12f8f98b926345a70f941845b.json new file mode 100644 index 0000000000..7da7df136e --- /dev/null +++ b/backend/.sqlx/query-864184467477e73a45935bf9439ae60da7f9ecc12f8f98b926345a70f941845b.json @@ -0,0 +1,24 @@ +{ + "db_name": "PostgreSQL", + "query": "SELECT value FROM resource_version\n WHERE version = $1 AND workspace_id = $2 AND path = $3", + "describe": { + "columns": [ + { + "ordinal": 0, + "name": "value", + "type_info": "Jsonb" + } + ], + "parameters": { + "Left": [ + "Int8", + "Text", + "Text" + ] + }, + "nullable": [ + true + ] + }, + "hash": "864184467477e73a45935bf9439ae60da7f9ecc12f8f98b926345a70f941845b" +} diff --git a/backend/.sqlx/query-897e8da49e4a5c3efdfb2c36fc7aff3d53432da9ef57f8a775850efcd50a965c.json b/backend/.sqlx/query-897e8da49e4a5c3efdfb2c36fc7aff3d53432da9ef57f8a775850efcd50a965c.json new file mode 100644 index 0000000000..8610e3372b --- /dev/null +++ b/backend/.sqlx/query-897e8da49e4a5c3efdfb2c36fc7aff3d53432da9ef57f8a775850efcd50a965c.json @@ -0,0 +1,29 @@ +{ + "db_name": "PostgreSQL", + "query": "SELECT r.value AS \"value: sqlx::types::Json\",\n (SELECT version FROM resource_version v\n WHERE v.workspace_id = r.workspace_id AND v.path = r.path\n ORDER BY v.version DESC LIMIT 1) AS version\n FROM resource r\n WHERE r.workspace_id = $1 AND r.path = $2 AND r.resource_type = 'ai_agent'", + "describe": { + "columns": [ + { + "ordinal": 0, + "name": "value: sqlx::types::Json", + "type_info": "Jsonb" + }, + { + "ordinal": 1, + "name": "version", + "type_info": "Int8" + } + ], + "parameters": { + "Left": [ + "Text", + "Text" + ] + }, + "nullable": [ + true, + null + ] + }, + "hash": "897e8da49e4a5c3efdfb2c36fc7aff3d53432da9ef57f8a775850efcd50a965c" +} diff --git a/backend/.sqlx/query-8c319ac3eb2a289a6709ea1ea1c164f1a56e328902dd46fd9b68d801c16f4ffa.json b/backend/.sqlx/query-8c319ac3eb2a289a6709ea1ea1c164f1a56e328902dd46fd9b68d801c16f4ffa.json new file mode 100644 index 0000000000..f60f76b002 --- /dev/null +++ b/backend/.sqlx/query-8c319ac3eb2a289a6709ea1ea1c164f1a56e328902dd46fd9b68d801c16f4ffa.json @@ -0,0 +1,16 @@ +{ + "db_name": "PostgreSQL", + "query": "UPDATE eval_dataset SET extra_perms = extra_perms - ('u/' || $2) || jsonb_build_object(('u/' || $1), extra_perms->('u/' || $2)) WHERE extra_perms ? ('u/' || $2) AND workspace_id = $3", + "describe": { + "columns": [], + "parameters": { + "Left": [ + "Text", + "Text", + "Text" + ] + }, + "nullable": [] + }, + "hash": "8c319ac3eb2a289a6709ea1ea1c164f1a56e328902dd46fd9b68d801c16f4ffa" +} diff --git a/backend/.sqlx/query-9071eff54395ca39809a5d55d1e3fedb64cfafa9fe74cd9446eb6ee526c2d9a0.json b/backend/.sqlx/query-9071eff54395ca39809a5d55d1e3fedb64cfafa9fe74cd9446eb6ee526c2d9a0.json new file mode 100644 index 0000000000..dd4a781db0 --- /dev/null +++ b/backend/.sqlx/query-9071eff54395ca39809a5d55d1e3fedb64cfafa9fe74cd9446eb6ee526c2d9a0.json @@ -0,0 +1,58 @@ +{ + "db_name": "PostgreSQL", + "query": "SELECT path, summary, scorers, created_at, created_by,\n edited_at, edited_by\n FROM eval_dataset WHERE workspace_id = $1 ORDER BY path", + "describe": { + "columns": [ + { + "ordinal": 0, + "name": "path", + "type_info": "Varchar" + }, + { + "ordinal": 1, + "name": "summary", + "type_info": "Varchar" + }, + { + "ordinal": 2, + "name": "scorers", + "type_info": "Jsonb" + }, + { + "ordinal": 3, + "name": "created_at", + "type_info": "Timestamptz" + }, + { + "ordinal": 4, + "name": "created_by", + "type_info": "Varchar" + }, + { + "ordinal": 5, + "name": "edited_at", + "type_info": "Timestamptz" + }, + { + "ordinal": 6, + "name": "edited_by", + "type_info": "Varchar" + } + ], + "parameters": { + "Left": [ + "Text" + ] + }, + "nullable": [ + false, + true, + false, + false, + false, + false, + false + ] + }, + "hash": "9071eff54395ca39809a5d55d1e3fedb64cfafa9fe74cd9446eb6ee526c2d9a0" +} diff --git a/backend/.sqlx/query-97a701e16c0ce4b8c6a1394c2a71b8ae45e16c11c8003c5f64c9f53d0e7226aa.json b/backend/.sqlx/query-97a701e16c0ce4b8c6a1394c2a71b8ae45e16c11c8003c5f64c9f53d0e7226aa.json new file mode 100644 index 0000000000..d068fac4f4 --- /dev/null +++ b/backend/.sqlx/query-97a701e16c0ce4b8c6a1394c2a71b8ae45e16c11c8003c5f64c9f53d0e7226aa.json @@ -0,0 +1,23 @@ +{ + "db_name": "PostgreSQL", + "query": "SELECT COALESCE(s.flow_status, c.flow_status) -> 'modules' AS modules\n FROM v2_job j\n LEFT JOIN v2_job_status s ON s.id = j.id\n LEFT JOIN v2_job_completed c ON c.id = j.id\n WHERE j.id = $1 AND j.workspace_id = $2", + "describe": { + "columns": [ + { + "ordinal": 0, + "name": "modules", + "type_info": "Jsonb" + } + ], + "parameters": { + "Left": [ + "Uuid", + "Text" + ] + }, + "nullable": [ + null + ] + }, + "hash": "97a701e16c0ce4b8c6a1394c2a71b8ae45e16c11c8003c5f64c9f53d0e7226aa" +} diff --git a/backend/.sqlx/query-9b1ad1bbf0c2dca3ce1cc9433c35db87e3101bcbcb0fa359885ca3807c1c9ca2.json b/backend/.sqlx/query-9b1ad1bbf0c2dca3ce1cc9433c35db87e3101bcbcb0fa359885ca3807c1c9ca2.json new file mode 100644 index 0000000000..bc6e2eb971 --- /dev/null +++ b/backend/.sqlx/query-9b1ad1bbf0c2dca3ce1cc9433c35db87e3101bcbcb0fa359885ca3807c1c9ca2.json @@ -0,0 +1,23 @@ +{ + "db_name": "PostgreSQL", + "query": "SELECT j.id AS \"id!\" FROM v2_job j\n LEFT JOIN v2_job_completed c ON c.id = j.id AND c.workspace_id = $2\n WHERE j.id = ANY($1) AND j.workspace_id = $2 AND c.id IS NULL", + "describe": { + "columns": [ + { + "ordinal": 0, + "name": "id!", + "type_info": "Uuid" + } + ], + "parameters": { + "Left": [ + "UuidArray", + "Text" + ] + }, + "nullable": [ + false + ] + }, + "hash": "9b1ad1bbf0c2dca3ce1cc9433c35db87e3101bcbcb0fa359885ca3807c1c9ca2" +} diff --git a/backend/.sqlx/query-9d3ca755b323330033eb891ac7162a53440a5c713b58e6c4a467f6f0d5d7e37c.json b/backend/.sqlx/query-9d3ca755b323330033eb891ac7162a53440a5c713b58e6c4a467f6f0d5d7e37c.json new file mode 100644 index 0000000000..1c6b02aa9f --- /dev/null +++ b/backend/.sqlx/query-9d3ca755b323330033eb891ac7162a53440a5c713b58e6c4a467f6f0d5d7e37c.json @@ -0,0 +1,16 @@ +{ + "db_name": "PostgreSQL", + "query": "UPDATE eval_dataset SET scorers = COALESCE((\n SELECT jsonb_agg(\n CASE WHEN elem->>'path' LIKE ('u/' || $2 || '/%')\n THEN jsonb_set(elem, '{path}', to_jsonb(REGEXP_REPLACE(elem->>'path','u/' || $2 || '/(.*)','u/' || $1 || '/\\1')))\n ELSE elem END)\n FROM jsonb_array_elements(scorers) elem), '[]'::jsonb)\n WHERE workspace_id = $3\n AND EXISTS (SELECT 1 FROM jsonb_array_elements(scorers) e WHERE e->>'path' LIKE ('u/' || $2 || '/%'))", + "describe": { + "columns": [], + "parameters": { + "Left": [ + "Text", + "Text", + "Text" + ] + }, + "nullable": [] + }, + "hash": "9d3ca755b323330033eb891ac7162a53440a5c713b58e6c4a467f6f0d5d7e37c" +} diff --git a/backend/.sqlx/query-a4c842e395714346d5178190793d8ae66f891421b72fc6adfa8a38060ca2f3fa.json b/backend/.sqlx/query-a4c842e395714346d5178190793d8ae66f891421b72fc6adfa8a38060ca2f3fa.json new file mode 100644 index 0000000000..23e4e95b2f --- /dev/null +++ b/backend/.sqlx/query-a4c842e395714346d5178190793d8ae66f891421b72fc6adfa8a38060ca2f3fa.json @@ -0,0 +1,15 @@ +{ + "db_name": "PostgreSQL", + "query": "INSERT INTO eval_case (workspace_id, dataset_path, input, expected, created_at, created_by)\n SELECT $2, dataset_path, input, expected, created_at, created_by\n FROM eval_case WHERE workspace_id = $1", + "describe": { + "columns": [], + "parameters": { + "Left": [ + "Text", + "Varchar" + ] + }, + "nullable": [] + }, + "hash": "a4c842e395714346d5178190793d8ae66f891421b72fc6adfa8a38060ca2f3fa" +} diff --git a/backend/.sqlx/query-a5e8cf0e559742330d67d36d9ddb75ef7de6e0ad74e0cc6c4056b782155b86c7.json b/backend/.sqlx/query-a5e8cf0e559742330d67d36d9ddb75ef7de6e0ad74e0cc6c4056b782155b86c7.json new file mode 100644 index 0000000000..d4b1e174da --- /dev/null +++ b/backend/.sqlx/query-a5e8cf0e559742330d67d36d9ddb75ef7de6e0ad74e0cc6c4056b782155b86c7.json @@ -0,0 +1,53 @@ +{ + "db_name": "PostgreSQL", + "query": "SELECT s.ordinal, s.scorer_id, c.job_id AS \"job_id!\", d.status::text AS status,\n c.answered, (j.id IS NOT NULL) AS \"job_exists!\"\n FROM eval_score s\n JOIN eval_experiment_case c\n ON c.experiment_id = s.experiment_id AND c.ordinal = s.ordinal\n LEFT JOIN v2_job j ON j.id = c.job_id AND j.workspace_id = $2\n LEFT JOIN v2_job_completed d ON d.id = c.job_id AND d.workspace_id = $2\n WHERE s.experiment_id = $1 AND s.score IS NULL AND s.error IS NULL\n AND NOT s.not_applicable AND c.job_id IS NOT NULL", + "describe": { + "columns": [ + { + "ordinal": 0, + "name": "ordinal", + "type_info": "Int4" + }, + { + "ordinal": 1, + "name": "scorer_id", + "type_info": "Varchar" + }, + { + "ordinal": 2, + "name": "job_id!", + "type_info": "Uuid" + }, + { + "ordinal": 3, + "name": "status", + "type_info": "Text" + }, + { + "ordinal": 4, + "name": "answered", + "type_info": "Bool" + }, + { + "ordinal": 5, + "name": "job_exists!", + "type_info": "Bool" + } + ], + "parameters": { + "Left": [ + "Uuid", + "Text" + ] + }, + "nullable": [ + false, + false, + true, + null, + true, + null + ] + }, + "hash": "a5e8cf0e559742330d67d36d9ddb75ef7de6e0ad74e0cc6c4056b782155b86c7" +} diff --git a/backend/.sqlx/query-a7b589b8d5cded97905bc24412ad8f3f0339a2a088243f045a212f9357205407.json b/backend/.sqlx/query-a7b589b8d5cded97905bc24412ad8f3f0339a2a088243f045a212f9357205407.json new file mode 100644 index 0000000000..fb8f4593ab --- /dev/null +++ b/backend/.sqlx/query-a7b589b8d5cded97905bc24412ad8f3f0339a2a088243f045a212f9357205407.json @@ -0,0 +1,27 @@ +{ + "db_name": "PostgreSQL", + "query": "UPDATE eval_dataset\n SET path = COALESCE($6, path), summary = COALESCE($3, summary),\n scorers = COALESCE($4, scorers), edited_at = now(), edited_by = $5\n WHERE workspace_id = $1 AND path = $2\n RETURNING path", + "describe": { + "columns": [ + { + "ordinal": 0, + "name": "path", + "type_info": "Varchar" + } + ], + "parameters": { + "Left": [ + "Text", + "Text", + "Varchar", + "Jsonb", + "Varchar", + "Varchar" + ] + }, + "nullable": [ + false + ] + }, + "hash": "a7b589b8d5cded97905bc24412ad8f3f0339a2a088243f045a212f9357205407" +} diff --git a/backend/.sqlx/query-b2db4f32c615a99db7af23729682340328da39ac763bb26b6bcf82027fad48e5.json b/backend/.sqlx/query-b2db4f32c615a99db7af23729682340328da39ac763bb26b6bcf82027fad48e5.json new file mode 100644 index 0000000000..e0f8e62e11 --- /dev/null +++ b/backend/.sqlx/query-b2db4f32c615a99db7af23729682340328da39ac763bb26b6bcf82027fad48e5.json @@ -0,0 +1,29 @@ +{ + "db_name": "PostgreSQL", + "query": "SELECT id, run_job_id FROM eval_experiment WHERE workspace_id = $1 AND id = $2", + "describe": { + "columns": [ + { + "ordinal": 0, + "name": "id", + "type_info": "Uuid" + }, + { + "ordinal": 1, + "name": "run_job_id", + "type_info": "Uuid" + } + ], + "parameters": { + "Left": [ + "Text", + "Uuid" + ] + }, + "nullable": [ + false, + false + ] + }, + "hash": "b2db4f32c615a99db7af23729682340328da39ac763bb26b6bcf82027fad48e5" +} diff --git a/backend/.sqlx/query-b8e732a03969666444f73397ac153d23ec2af084b2f93da24c920532c1916384.json b/backend/.sqlx/query-b8e732a03969666444f73397ac153d23ec2af084b2f93da24c920532c1916384.json index b892061f56..b336210daf 100644 --- a/backend/.sqlx/query-b8e732a03969666444f73397ac153d23ec2af084b2f93da24c920532c1916384.json +++ b/backend/.sqlx/query-b8e732a03969666444f73397ac153d23ec2af084b2f93da24c920532c1916384.json @@ -98,12 +98,12 @@ null, null, null, - false, + true, null, null, null, - false, - false + true, + true ] }, "hash": "b8e732a03969666444f73397ac153d23ec2af084b2f93da24c920532c1916384" diff --git a/backend/.sqlx/query-bac36542b16b687a823067c013e3728f1cabe3774f581401514f522ea7522de5.json b/backend/.sqlx/query-bac36542b16b687a823067c013e3728f1cabe3774f581401514f522ea7522de5.json new file mode 100644 index 0000000000..b680e889c6 --- /dev/null +++ b/backend/.sqlx/query-bac36542b16b687a823067c013e3728f1cabe3774f581401514f522ea7522de5.json @@ -0,0 +1,29 @@ +{ + "db_name": "PostgreSQL", + "query": "SELECT status::text AS \"status!\", duration_ms FROM v2_job_completed\n WHERE id = $1 AND workspace_id = $2", + "describe": { + "columns": [ + { + "ordinal": 0, + "name": "status!", + "type_info": "Text" + }, + { + "ordinal": 1, + "name": "duration_ms", + "type_info": "Int8" + } + ], + "parameters": { + "Left": [ + "Uuid", + "Text" + ] + }, + "nullable": [ + null, + false + ] + }, + "hash": "bac36542b16b687a823067c013e3728f1cabe3774f581401514f522ea7522de5" +} diff --git a/backend/.sqlx/query-bbce2221f5724016543a5ac1db7fa74b8b35375dfb0f5d39b398369468ad0774.json b/backend/.sqlx/query-bbce2221f5724016543a5ac1db7fa74b8b35375dfb0f5d39b398369468ad0774.json new file mode 100644 index 0000000000..24ea423095 --- /dev/null +++ b/backend/.sqlx/query-bbce2221f5724016543a5ac1db7fa74b8b35375dfb0f5d39b398369468ad0774.json @@ -0,0 +1,54 @@ +{ + "db_name": "PostgreSQL", + "query": "SELECT e.subject, e.run_number, e.run_job_id, e.created_at,\n e.created_by,\n (SELECT count(*) FROM eval_experiment_case c WHERE c.experiment_id = e.id)\n AS \"case_count!\"\n FROM eval_experiment e\n WHERE e.workspace_id = $1 AND e.dataset_path = $2 AND e.id = $3", + "describe": { + "columns": [ + { + "ordinal": 0, + "name": "subject", + "type_info": "Jsonb" + }, + { + "ordinal": 1, + "name": "run_number", + "type_info": "Int4" + }, + { + "ordinal": 2, + "name": "run_job_id", + "type_info": "Uuid" + }, + { + "ordinal": 3, + "name": "created_at", + "type_info": "Timestamptz" + }, + { + "ordinal": 4, + "name": "created_by", + "type_info": "Varchar" + }, + { + "ordinal": 5, + "name": "case_count!", + "type_info": "Int8" + } + ], + "parameters": { + "Left": [ + "Text", + "Text", + "Uuid" + ] + }, + "nullable": [ + false, + false, + false, + false, + false, + null + ] + }, + "hash": "bbce2221f5724016543a5ac1db7fa74b8b35375dfb0f5d39b398369468ad0774" +} diff --git a/backend/.sqlx/query-bdb7185233941d1472a55c8ade3f61c82248f3f2c9a69c458bff3978ffd17379.json b/backend/.sqlx/query-bdb7185233941d1472a55c8ade3f61c82248f3f2c9a69c458bff3978ffd17379.json new file mode 100644 index 0000000000..0e31540e7c --- /dev/null +++ b/backend/.sqlx/query-bdb7185233941d1472a55c8ade3f61c82248f3f2c9a69c458bff3978ffd17379.json @@ -0,0 +1,66 @@ +{ + "db_name": "PostgreSQL", + "query": "SELECT e.id, e.dataset_path, e.subject, e.run_number, e.run_job_id, e.created_at,\n e.created_by,\n (SELECT count(*) FROM eval_experiment_case c WHERE c.experiment_id = e.id)\n AS \"case_count!\"\n FROM eval_experiment e\n JOIN eval_dataset d ON d.workspace_id = e.workspace_id AND d.path = e.dataset_path\n WHERE e.workspace_id = $1\n AND ($3::text IS NULL OR e.subject ->> 'path' = $3)\n ORDER BY e.created_at DESC\n LIMIT $2", + "describe": { + "columns": [ + { + "ordinal": 0, + "name": "id", + "type_info": "Uuid" + }, + { + "ordinal": 1, + "name": "dataset_path", + "type_info": "Varchar" + }, + { + "ordinal": 2, + "name": "subject", + "type_info": "Jsonb" + }, + { + "ordinal": 3, + "name": "run_number", + "type_info": "Int4" + }, + { + "ordinal": 4, + "name": "run_job_id", + "type_info": "Uuid" + }, + { + "ordinal": 5, + "name": "created_at", + "type_info": "Timestamptz" + }, + { + "ordinal": 6, + "name": "created_by", + "type_info": "Varchar" + }, + { + "ordinal": 7, + "name": "case_count!", + "type_info": "Int8" + } + ], + "parameters": { + "Left": [ + "Text", + "Int8", + "Text" + ] + }, + "nullable": [ + false, + false, + false, + false, + false, + false, + false, + null + ] + }, + "hash": "bdb7185233941d1472a55c8ade3f61c82248f3f2c9a69c458bff3978ffd17379" +} diff --git a/backend/.sqlx/query-be17a65f144cc21e849c7f8cfbf9d7271b953dda42e4ba09864da7bda731e752.json b/backend/.sqlx/query-be17a65f144cc21e849c7f8cfbf9d7271b953dda42e4ba09864da7bda731e752.json new file mode 100644 index 0000000000..f28916165f --- /dev/null +++ b/backend/.sqlx/query-be17a65f144cc21e849c7f8cfbf9d7271b953dda42e4ba09864da7bda731e752.json @@ -0,0 +1,54 @@ +{ + "db_name": "PostgreSQL", + "query": "SELECT j.id, j.args AS \"args: sqlx::types::Json>\",\n c.result AS \"result: sqlx::types::Json>\",\n c.status::text AS status, c.duration_ms,\n s.schema AS \"schema: sqlx::types::Json>\"\n FROM v2_job j\n LEFT JOIN v2_job_completed c ON c.id = j.id\n LEFT JOIN script s ON s.workspace_id = j.workspace_id AND s.hash = j.runnable_id\n WHERE j.id = ANY($1) AND j.workspace_id = $2 AND j.parent_job = $3", + "describe": { + "columns": [ + { + "ordinal": 0, + "name": "id", + "type_info": "Uuid" + }, + { + "ordinal": 1, + "name": "args: sqlx::types::Json>", + "type_info": "Jsonb" + }, + { + "ordinal": 2, + "name": "result: sqlx::types::Json>", + "type_info": "Jsonb" + }, + { + "ordinal": 3, + "name": "status", + "type_info": "Text" + }, + { + "ordinal": 4, + "name": "duration_ms", + "type_info": "Int8" + }, + { + "ordinal": 5, + "name": "schema: sqlx::types::Json>", + "type_info": "Json" + } + ], + "parameters": { + "Left": [ + "UuidArray", + "Text", + "Uuid" + ] + }, + "nullable": [ + false, + true, + true, + null, + false, + true + ] + }, + "hash": "be17a65f144cc21e849c7f8cfbf9d7271b953dda42e4ba09864da7bda731e752" +} diff --git a/backend/.sqlx/query-d10efb37765ac9a7f2e15f71dbbc02ec917b6ecb014f938e68ced3cc7fe9dc86.json b/backend/.sqlx/query-d10efb37765ac9a7f2e15f71dbbc02ec917b6ecb014f938e68ced3cc7fe9dc86.json new file mode 100644 index 0000000000..c124b8c3cb --- /dev/null +++ b/backend/.sqlx/query-d10efb37765ac9a7f2e15f71dbbc02ec917b6ecb014f938e68ced3cc7fe9dc86.json @@ -0,0 +1,16 @@ +{ + "db_name": "PostgreSQL", + "query": "UPDATE eval_experiment SET subject = jsonb_set(subject, '{path}', to_jsonb(REGEXP_REPLACE(subject->>'path','u/' || $2 || '/(.*)','u/' || $1 || '/\\1'))) WHERE subject->>'path' LIKE ('u/' || $2 || '/%') AND workspace_id = $3", + "describe": { + "columns": [], + "parameters": { + "Left": [ + "Text", + "Text", + "Text" + ] + }, + "nullable": [] + }, + "hash": "d10efb37765ac9a7f2e15f71dbbc02ec917b6ecb014f938e68ced3cc7fe9dc86" +} diff --git a/backend/.sqlx/query-d5cdb1121f2a414c0a70f4f7e4ae630bad0715bbbb3597cb1b14e41abe9fec14.json b/backend/.sqlx/query-d5cdb1121f2a414c0a70f4f7e4ae630bad0715bbbb3597cb1b14e41abe9fec14.json new file mode 100644 index 0000000000..ad292f64cd --- /dev/null +++ b/backend/.sqlx/query-d5cdb1121f2a414c0a70f4f7e4ae630bad0715bbbb3597cb1b14e41abe9fec14.json @@ -0,0 +1,23 @@ +{ + "db_name": "PostgreSQL", + "query": "UPDATE eval_experiment_case SET status = $2, answered = false\n WHERE experiment_id = $1 AND job_id IS NULL AND status IS NULL\n RETURNING ordinal", + "describe": { + "columns": [ + { + "ordinal": 0, + "name": "ordinal", + "type_info": "Int4" + } + ], + "parameters": { + "Left": [ + "Uuid", + "Varchar" + ] + }, + "nullable": [ + false + ] + }, + "hash": "d5cdb1121f2a414c0a70f4f7e4ae630bad0715bbbb3597cb1b14e41abe9fec14" +} diff --git a/backend/.sqlx/query-db19932d940b2467147eb2c13b059f2ec33b0c83b5c4a332a4b6e54d2591767c.json b/backend/.sqlx/query-db19932d940b2467147eb2c13b059f2ec33b0c83b5c4a332a4b6e54d2591767c.json new file mode 100644 index 0000000000..1595cec076 --- /dev/null +++ b/backend/.sqlx/query-db19932d940b2467147eb2c13b059f2ec33b0c83b5c4a332a4b6e54d2591767c.json @@ -0,0 +1,18 @@ +{ + "db_name": "PostgreSQL", + "query": "INSERT INTO eval_case\n (workspace_id, dataset_path, input, expected, created_by, created_at)\n VALUES ($1, $2, $3, $4, $5, clock_timestamp())", + "describe": { + "columns": [], + "parameters": { + "Left": [ + "Varchar", + "Varchar", + "Jsonb", + "Jsonb", + "Varchar" + ] + }, + "nullable": [] + }, + "hash": "db19932d940b2467147eb2c13b059f2ec33b0c83b5c4a332a4b6e54d2591767c" +} diff --git a/backend/.sqlx/query-db63d41718c50e264a949885d2d7c13719ae650e9c788ebf125627b785fb9ee0.json b/backend/.sqlx/query-db63d41718c50e264a949885d2d7c13719ae650e9c788ebf125627b785fb9ee0.json new file mode 100644 index 0000000000..4602a03f61 --- /dev/null +++ b/backend/.sqlx/query-db63d41718c50e264a949885d2d7c13719ae650e9c788ebf125627b785fb9ee0.json @@ -0,0 +1,23 @@ +{ + "db_name": "PostgreSQL", + "query": "SELECT path FROM eval_dataset WHERE workspace_id = $1 AND path = $2", + "describe": { + "columns": [ + { + "ordinal": 0, + "name": "path", + "type_info": "Varchar" + } + ], + "parameters": { + "Left": [ + "Text", + "Text" + ] + }, + "nullable": [ + false + ] + }, + "hash": "db63d41718c50e264a949885d2d7c13719ae650e9c788ebf125627b785fb9ee0" +} diff --git a/backend/.sqlx/query-e3c080b84f50622e0a74524111ae292c958e71293695ce18f9905af4dd940495.json b/backend/.sqlx/query-e3c080b84f50622e0a74524111ae292c958e71293695ce18f9905af4dd940495.json new file mode 100644 index 0000000000..ace8edc731 --- /dev/null +++ b/backend/.sqlx/query-e3c080b84f50622e0a74524111ae292c958e71293695ce18f9905af4dd940495.json @@ -0,0 +1,49 @@ +{ + "db_name": "PostgreSQL", + "query": "SELECT id, input, expected, created_at, created_by\n FROM eval_case\n WHERE workspace_id = $1 AND dataset_path = $2\n ORDER BY created_at, id\n LIMIT $3 OFFSET $4", + "describe": { + "columns": [ + { + "ordinal": 0, + "name": "id", + "type_info": "Uuid" + }, + { + "ordinal": 1, + "name": "input", + "type_info": "Jsonb" + }, + { + "ordinal": 2, + "name": "expected", + "type_info": "Jsonb" + }, + { + "ordinal": 3, + "name": "created_at", + "type_info": "Timestamptz" + }, + { + "ordinal": 4, + "name": "created_by", + "type_info": "Varchar" + } + ], + "parameters": { + "Left": [ + "Text", + "Text", + "Int8", + "Int8" + ] + }, + "nullable": [ + false, + false, + true, + false, + false + ] + }, + "hash": "e3c080b84f50622e0a74524111ae292c958e71293695ce18f9905af4dd940495" +} diff --git a/backend/.sqlx/query-e6d7e9779eaa6e584b613675ae8602c405fe620937a4563a17db80219a2930d2.json b/backend/.sqlx/query-e6d7e9779eaa6e584b613675ae8602c405fe620937a4563a17db80219a2930d2.json new file mode 100644 index 0000000000..d0e2c29c2d --- /dev/null +++ b/backend/.sqlx/query-e6d7e9779eaa6e584b613675ae8602c405fe620937a4563a17db80219a2930d2.json @@ -0,0 +1,23 @@ +{ + "db_name": "PostgreSQL", + "query": "SELECT args AS \"args: sqlx::types::Json>\" FROM v2_job\n WHERE id = $1 AND workspace_id = $2", + "describe": { + "columns": [ + { + "ordinal": 0, + "name": "args: sqlx::types::Json>", + "type_info": "Jsonb" + } + ], + "parameters": { + "Left": [ + "Uuid", + "Text" + ] + }, + "nullable": [ + true + ] + }, + "hash": "e6d7e9779eaa6e584b613675ae8602c405fe620937a4563a17db80219a2930d2" +} diff --git a/backend/.sqlx/query-e719e98cceef1383f882632c9398dd950ad0b1f30cde9c0392384c9a490290b9.json b/backend/.sqlx/query-e719e98cceef1383f882632c9398dd950ad0b1f30cde9c0392384c9a490290b9.json new file mode 100644 index 0000000000..b4f0820065 --- /dev/null +++ b/backend/.sqlx/query-e719e98cceef1383f882632c9398dd950ad0b1f30cde9c0392384c9a490290b9.json @@ -0,0 +1,17 @@ +{ + "db_name": "PostgreSQL", + "query": "INSERT INTO eval_score (experiment_id, ordinal, scorer_id, definition)\n SELECT $1, ordinal, scorer_id, definition\n FROM UNNEST($2::int[], $3::text[], $4::text[]) AS t(ordinal, scorer_id, definition)\n ON CONFLICT (experiment_id, ordinal, scorer_id)\n DO UPDATE SET definition = EXCLUDED.definition, score = NULL, reason = NULL,\n checks = NULL, error = NULL, not_applicable = false", + "describe": { + "columns": [], + "parameters": { + "Left": [ + "Uuid", + "Int4Array", + "TextArray", + "TextArray" + ] + }, + "nullable": [] + }, + "hash": "e719e98cceef1383f882632c9398dd950ad0b1f30cde9c0392384c9a490290b9" +} diff --git a/backend/.sqlx/query-e72d71852b6c5998b8d46407d164fcee17400a0ffc172d2a257c3fc15617fbd9.json b/backend/.sqlx/query-e72d71852b6c5998b8d46407d164fcee17400a0ffc172d2a257c3fc15617fbd9.json new file mode 100644 index 0000000000..5ce8f9f4c6 --- /dev/null +++ b/backend/.sqlx/query-e72d71852b6c5998b8d46407d164fcee17400a0ffc172d2a257c3fc15617fbd9.json @@ -0,0 +1,41 @@ +{ + "db_name": "PostgreSQL", + "query": "SELECT c.ordinal, c.job_id AS \"job_id!\", d.status::text AS status,\n (j.id IS NOT NULL) AS \"job_exists!\"\n FROM eval_experiment_case c\n LEFT JOIN v2_job j ON j.id = c.job_id AND j.workspace_id = $2\n LEFT JOIN v2_job_completed d ON d.id = c.job_id AND d.workspace_id = $2\n WHERE c.experiment_id = $1 AND c.job_id IS NOT NULL AND c.status IS NULL", + "describe": { + "columns": [ + { + "ordinal": 0, + "name": "ordinal", + "type_info": "Int4" + }, + { + "ordinal": 1, + "name": "job_id!", + "type_info": "Uuid" + }, + { + "ordinal": 2, + "name": "status", + "type_info": "Text" + }, + { + "ordinal": 3, + "name": "job_exists!", + "type_info": "Bool" + } + ], + "parameters": { + "Left": [ + "Uuid", + "Text" + ] + }, + "nullable": [ + false, + true, + null, + null + ] + }, + "hash": "e72d71852b6c5998b8d46407d164fcee17400a0ffc172d2a257c3fc15617fbd9" +} diff --git a/backend/.sqlx/query-e942a74104771b192c33ea03dbcaa409e03b6bfb7423929c201f39db4228cc36.json b/backend/.sqlx/query-e942a74104771b192c33ea03dbcaa409e03b6bfb7423929c201f39db4228cc36.json new file mode 100644 index 0000000000..9daa8a1851 --- /dev/null +++ b/backend/.sqlx/query-e942a74104771b192c33ea03dbcaa409e03b6bfb7423929c201f39db4228cc36.json @@ -0,0 +1,23 @@ +{ + "db_name": "PostgreSQL", + "query": "DELETE FROM eval_dataset WHERE workspace_id = $1 AND path = $2 RETURNING path", + "describe": { + "columns": [ + { + "ordinal": 0, + "name": "path", + "type_info": "Varchar" + } + ], + "parameters": { + "Left": [ + "Text", + "Text" + ] + }, + "nullable": [ + false + ] + }, + "hash": "e942a74104771b192c33ea03dbcaa409e03b6bfb7423929c201f39db4228cc36" +} diff --git a/backend/.sqlx/query-eb0f25a10f4f1264e482674c06783204fddf0ccdca6d7c798a4ff6527548e96f.json b/backend/.sqlx/query-eb0f25a10f4f1264e482674c06783204fddf0ccdca6d7c798a4ff6527548e96f.json new file mode 100644 index 0000000000..9ed995c409 --- /dev/null +++ b/backend/.sqlx/query-eb0f25a10f4f1264e482674c06783204fddf0ccdca6d7c798a4ff6527548e96f.json @@ -0,0 +1,20 @@ +{ + "db_name": "PostgreSQL", + "query": "INSERT INTO eval_experiment\n (id, workspace_id, dataset_path, subject, run_number, created_by, run_job_id)\n VALUES ($1, $2, $3, $4, $5, $6, $7)", + "describe": { + "columns": [], + "parameters": { + "Left": [ + "Uuid", + "Varchar", + "Varchar", + "Jsonb", + "Int4", + "Varchar", + "Uuid" + ] + }, + "nullable": [] + }, + "hash": "eb0f25a10f4f1264e482674c06783204fddf0ccdca6d7c798a4ff6527548e96f" +} diff --git a/backend/.sqlx/query-ee40e48afb5520b7ff84883204f064ce566cea6655e3cd11a9d7016cb08482cb.json b/backend/.sqlx/query-ee40e48afb5520b7ff84883204f064ce566cea6655e3cd11a9d7016cb08482cb.json new file mode 100644 index 0000000000..2280a368ea --- /dev/null +++ b/backend/.sqlx/query-ee40e48afb5520b7ff84883204f064ce566cea6655e3cd11a9d7016cb08482cb.json @@ -0,0 +1,21 @@ +{ + "db_name": "PostgreSQL", + "query": "UPDATE eval_score s\n SET score = t.score, reason = t.reason, checks = t.checks, error = t.error,\n not_applicable = t.not_applicable\n FROM UNNEST($2::int[], $3::text[], $4::double precision[], $5::text[], $6::jsonb[],\n $7::text[], $8::bool[])\n AS t(ordinal, scorer_id, score, reason, checks, error, not_applicable)\n WHERE s.experiment_id = $1 AND s.ordinal = t.ordinal AND s.scorer_id = t.scorer_id", + "describe": { + "columns": [], + "parameters": { + "Left": [ + "Uuid", + "Int4Array", + "TextArray", + "Float8Array", + "TextArray", + "JsonbArray", + "TextArray", + "BoolArray" + ] + }, + "nullable": [] + }, + "hash": "ee40e48afb5520b7ff84883204f064ce566cea6655e3cd11a9d7016cb08482cb" +} diff --git a/backend/.sqlx/query-f0e943244b125d0c42a9b472701ec172af9dd334786275c391e80f1c38bbb45b.json b/backend/.sqlx/query-f0e943244b125d0c42a9b472701ec172af9dd334786275c391e80f1c38bbb45b.json new file mode 100644 index 0000000000..5e0fef8520 --- /dev/null +++ b/backend/.sqlx/query-f0e943244b125d0c42a9b472701ec172af9dd334786275c391e80f1c38bbb45b.json @@ -0,0 +1,18 @@ +{ + "db_name": "PostgreSQL", + "query": "UPDATE eval_experiment_case c\n SET output = COALESCE(c.output, t.output), answered = COALESCE(c.answered, t.answered),\n status = COALESCE(c.status, t.status)\n FROM UNNEST($2::int[], $3::text[], $4::bool[], $5::text[])\n AS t(ordinal, output, answered, status)\n WHERE c.experiment_id = $1 AND c.ordinal = t.ordinal", + "describe": { + "columns": [], + "parameters": { + "Left": [ + "Uuid", + "Int4Array", + "TextArray", + "BoolArray", + "TextArray" + ] + }, + "nullable": [] + }, + "hash": "f0e943244b125d0c42a9b472701ec172af9dd334786275c391e80f1c38bbb45b" +} diff --git a/backend/.sqlx/query-f59afd524e3f216487ad0a780b1fda6341e215025dc6d8985891b91dad6d9dcb.json b/backend/.sqlx/query-f59afd524e3f216487ad0a780b1fda6341e215025dc6d8985891b91dad6d9dcb.json new file mode 100644 index 0000000000..2d8a83ba48 --- /dev/null +++ b/backend/.sqlx/query-f59afd524e3f216487ad0a780b1fda6341e215025dc6d8985891b91dad6d9dcb.json @@ -0,0 +1,26 @@ +{ + "db_name": "PostgreSQL", + "query": "INSERT INTO eval_dataset\n (workspace_id, path, summary, scorers, created_by, edited_by)\n VALUES ($1, $2, $3, $4, $5, $5)\n ON CONFLICT (workspace_id, path) DO NOTHING\n RETURNING path", + "describe": { + "columns": [ + { + "ordinal": 0, + "name": "path", + "type_info": "Varchar" + } + ], + "parameters": { + "Left": [ + "Varchar", + "Varchar", + "Varchar", + "Jsonb", + "Varchar" + ] + }, + "nullable": [ + false + ] + }, + "hash": "f59afd524e3f216487ad0a780b1fda6341e215025dc6d8985891b91dad6d9dcb" +} diff --git a/backend/.sqlx/query-f6c4c40b098ba06f4b3b057af1f5dce0711e90f7a4e1786d39a50d52400a3cb9.json b/backend/.sqlx/query-f6c4c40b098ba06f4b3b057af1f5dce0711e90f7a4e1786d39a50d52400a3cb9.json new file mode 100644 index 0000000000..035dc31c85 --- /dev/null +++ b/backend/.sqlx/query-f6c4c40b098ba06f4b3b057af1f5dce0711e90f7a4e1786d39a50d52400a3cb9.json @@ -0,0 +1,16 @@ +{ + "db_name": "PostgreSQL", + "query": "UPDATE eval_dataset SET created_by = $1 WHERE created_by = $2 AND workspace_id = $3", + "describe": { + "columns": [], + "parameters": { + "Left": [ + "Varchar", + "Text", + "Text" + ] + }, + "nullable": [] + }, + "hash": "f6c4c40b098ba06f4b3b057af1f5dce0711e90f7a4e1786d39a50d52400a3cb9" +} diff --git a/backend/.sqlx/query-ef59abddc518f5213827e47a31aee49be917a61c46916c29d79c094438b1ff35.json b/backend/.sqlx/query-f907114909b1064a3d5eb603e5929a6ffebdcadb20e4c6a68768a9ad42fe9328.json similarity index 69% rename from backend/.sqlx/query-ef59abddc518f5213827e47a31aee49be917a61c46916c29d79c094438b1ff35.json rename to backend/.sqlx/query-f907114909b1064a3d5eb603e5929a6ffebdcadb20e4c6a68768a9ad42fe9328.json index 2a9c7cac4f..a62f4f95a1 100644 --- a/backend/.sqlx/query-ef59abddc518f5213827e47a31aee49be917a61c46916c29d79c094438b1ff35.json +++ b/backend/.sqlx/query-f907114909b1064a3d5eb603e5929a6ffebdcadb20e4c6a68768a9ad42fe9328.json @@ -1,6 +1,6 @@ { "db_name": "PostgreSQL", - "query": "SELECT id, path, created_at, created_by, value FROM resource_version\n WHERE workspace_id = $1 AND id = $2", + "query": "SELECT id, version, path, created_at, created_by, value FROM resource_version\n WHERE workspace_id = $1 AND id = $2", "describe": { "columns": [ { @@ -10,21 +10,26 @@ }, { "ordinal": 1, + "name": "version", + "type_info": "Int8" + }, + { + "ordinal": 2, "name": "path", "type_info": "Varchar" }, { - "ordinal": 2, + "ordinal": 3, "name": "created_at", "type_info": "Timestamptz" }, { - "ordinal": 3, + "ordinal": 4, "name": "created_by", "type_info": "Varchar" }, { - "ordinal": 4, + "ordinal": 5, "name": "value", "type_info": "Jsonb" } @@ -39,9 +44,10 @@ false, false, false, + false, true, true ] }, - "hash": "ef59abddc518f5213827e47a31aee49be917a61c46916c29d79c094438b1ff35" + "hash": "f907114909b1064a3d5eb603e5929a6ffebdcadb20e4c6a68768a9ad42fe9328" } diff --git a/backend/.sqlx/query-facbf7337d7ffa3f3e6287e2910ccd7ca7229f9d4e5c3af6c460ee7fe3c946e6.json b/backend/.sqlx/query-facbf7337d7ffa3f3e6287e2910ccd7ca7229f9d4e5c3af6c460ee7fe3c946e6.json new file mode 100644 index 0000000000..5f6f815e2a --- /dev/null +++ b/backend/.sqlx/query-facbf7337d7ffa3f3e6287e2910ccd7ca7229f9d4e5c3af6c460ee7fe3c946e6.json @@ -0,0 +1,23 @@ +{ + "db_name": "PostgreSQL", + "query": "SELECT path FROM eval_dataset WHERE workspace_id = $1 AND path = $2 FOR UPDATE", + "describe": { + "columns": [ + { + "ordinal": 0, + "name": "path", + "type_info": "Varchar" + } + ], + "parameters": { + "Left": [ + "Text", + "Text" + ] + }, + "nullable": [ + false + ] + }, + "hash": "facbf7337d7ffa3f3e6287e2910ccd7ca7229f9d4e5c3af6c460ee7fe3c946e6" +} diff --git a/backend/.sqlx/query-fcf570337b2ceeb0f9dcc311144d8ada02aa4353feb2f63e7a1f520a4000ed70.json b/backend/.sqlx/query-fcf570337b2ceeb0f9dcc311144d8ada02aa4353feb2f63e7a1f520a4000ed70.json new file mode 100644 index 0000000000..7dc41fa462 --- /dev/null +++ b/backend/.sqlx/query-fcf570337b2ceeb0f9dcc311144d8ada02aa4353feb2f63e7a1f520a4000ed70.json @@ -0,0 +1,36 @@ +{ + "db_name": "PostgreSQL", + "query": "SELECT branch.parent_job AS \"case_job!\", scorer.flow_step_id AS \"module!\",\n done.result AS \"result: sqlx::types::Json>\"\n FROM v2_job branch\n JOIN v2_job scorer ON scorer.parent_job = branch.id\n JOIN v2_job_completed done ON done.id = scorer.id\n WHERE branch.parent_job = ANY($1) AND branch.workspace_id = $2\n AND scorer.flow_step_id = ANY($3)", + "describe": { + "columns": [ + { + "ordinal": 0, + "name": "case_job!", + "type_info": "Uuid" + }, + { + "ordinal": 1, + "name": "module!", + "type_info": "Varchar" + }, + { + "ordinal": 2, + "name": "result: sqlx::types::Json>", + "type_info": "Jsonb" + } + ], + "parameters": { + "Left": [ + "UuidArray", + "Text", + "TextArray" + ] + }, + "nullable": [ + true, + true, + true + ] + }, + "hash": "fcf570337b2ceeb0f9dcc311144d8ada02aa4353feb2f63e7a1f520a4000ed70" +} diff --git a/backend/.sqlx/query-fd12bbe0605c80fced218a5a2e1288e6c04f4a1a76a2038679f8f538320cab1f.json b/backend/.sqlx/query-fd12bbe0605c80fced218a5a2e1288e6c04f4a1a76a2038679f8f538320cab1f.json new file mode 100644 index 0000000000..923a4b5f98 --- /dev/null +++ b/backend/.sqlx/query-fd12bbe0605c80fced218a5a2e1288e6c04f4a1a76a2038679f8f538320cab1f.json @@ -0,0 +1,24 @@ +{ + "db_name": "PostgreSQL", + "query": "SELECT path FROM eval_dataset\n WHERE scorers::text LIKE $1 AND NOT path LIKE $2 AND workspace_id = $3", + "describe": { + "columns": [ + { + "ordinal": 0, + "name": "path", + "type_info": "Varchar" + } + ], + "parameters": { + "Left": [ + "Text", + "Text", + "Text" + ] + }, + "nullable": [ + false + ] + }, + "hash": "fd12bbe0605c80fced218a5a2e1288e6c04f4a1a76a2038679f8f538320cab1f" +} diff --git a/backend/migrations/20260812100659_ai_evals.down.sql b/backend/migrations/20260812100659_ai_evals.down.sql new file mode 100644 index 0000000000..2e9383c761 --- /dev/null +++ b/backend/migrations/20260812100659_ai_evals.down.sql @@ -0,0 +1,6 @@ +DROP TABLE IF EXISTS eval_score; +DROP TABLE IF EXISTS eval_experiment_case; +DROP TABLE IF EXISTS eval_experiment; +DROP TABLE IF EXISTS eval_case; +DROP TABLE IF EXISTS eval_dataset; +DROP FUNCTION IF EXISTS eval_dataset_writable(varchar, varchar); diff --git a/backend/migrations/20260812100659_ai_evals.up.sql b/backend/migrations/20260812100659_ai_evals.up.sql new file mode 100644 index 0000000000..7ef82d5f7e --- /dev/null +++ b/backend/migrations/20260812100659_ai_evals.up.sql @@ -0,0 +1,275 @@ +-- Eval datasets and the cases they hold. Path-addressed like every other Windmill object, so the +-- folder a dataset is named by is what grants access to it. +CREATE TABLE eval_dataset ( + workspace_id VARCHAR(50) NOT NULL, + path VARCHAR(255) NOT NULL, + summary VARCHAR(1000) NULL, + -- The scorers a dataset is scored by. One entry per column of the results table: + -- {id, name, kind, ...kind-specific config}. `id` is assigned once and never reused, so a + -- column stays the same column across experiments when it is renamed or its definition is + -- edited — which is what makes a delta between two experiments meaningful. + scorers JSONB NOT NULL DEFAULT '[]', + extra_perms JSONB NOT NULL DEFAULT '{}', + created_at TIMESTAMPTZ NOT NULL DEFAULT now(), + created_by VARCHAR(50) NOT NULL, + edited_at TIMESTAMPTZ NOT NULL DEFAULT now(), + edited_by VARCHAR(50) NOT NULL, + PRIMARY KEY (workspace_id, path), + FOREIGN KEY (workspace_id) REFERENCES workspace(id) ON DELETE CASCADE +); + +-- A case is the input half of one evaluation: what the agent is fed, and what it was expected to +-- answer. The generated output, the trajectory and every scorer's return value are the job's, not +-- this table's. +-- +-- ON UPDATE CASCADE so renaming a dataset carries its cases instead of stranding them. +CREATE TABLE eval_case ( + id UUID PRIMARY KEY DEFAULT gen_random_uuid(), + workspace_id VARCHAR(50) NOT NULL, + dataset_path VARCHAR(255) NOT NULL, + -- {user_message, user_attachments} + input JSONB NOT NULL DEFAULT '{}', + expected JSONB NULL, + created_at TIMESTAMPTZ NOT NULL DEFAULT now(), + created_by VARCHAR(50) NOT NULL, + FOREIGN KEY (workspace_id, dataset_path) REFERENCES eval_dataset (workspace_id, path) + ON DELETE CASCADE ON UPDATE CASCADE +); + +-- Serves the paginated case list, which is ordered oldest-first so a case keeps its position as +-- the dataset grows. +CREATE INDEX index_eval_case_dataset ON eval_case (workspace_id, dataset_path, created_at, id); + +GRANT ALL ON eval_dataset TO windmill_user; +GRANT ALL ON eval_dataset TO windmill_admin; +GRANT ALL ON eval_case TO windmill_user; +GRANT ALL ON eval_case TO windmill_admin; + +ALTER TABLE eval_dataset ENABLE ROW LEVEL SECURITY; +ALTER TABLE eval_case ENABLE ROW LEVEL SECURITY; + +CREATE POLICY admin_policy ON eval_dataset FOR ALL TO windmill_admin USING (true); +CREATE POLICY admin_policy ON eval_case FOR ALL TO windmill_admin USING (true); + +CREATE POLICY see_folder_extra_perms_user_select ON eval_dataset FOR SELECT TO windmill_user +USING (SPLIT_PART(eval_dataset.path, '/', 1) = 'f' AND SPLIT_PART(eval_dataset.path, '/', 2) = any((select regexp_split_to_array(current_setting('session.folders_read'), ','))::text[])); +CREATE POLICY see_folder_extra_perms_user_insert ON eval_dataset FOR INSERT TO windmill_user +WITH CHECK (SPLIT_PART(eval_dataset.path, '/', 1) = 'f' AND SPLIT_PART(eval_dataset.path, '/', 2) = any((select regexp_split_to_array(current_setting('session.folders_write'), ','))::text[])); +CREATE POLICY see_folder_extra_perms_user_update ON eval_dataset FOR UPDATE TO windmill_user +USING (SPLIT_PART(eval_dataset.path, '/', 1) = 'f' AND SPLIT_PART(eval_dataset.path, '/', 2) = any((select regexp_split_to_array(current_setting('session.folders_write'), ','))::text[])); +CREATE POLICY see_folder_extra_perms_user_delete ON eval_dataset FOR DELETE TO windmill_user +USING (SPLIT_PART(eval_dataset.path, '/', 1) = 'f' AND SPLIT_PART(eval_dataset.path, '/', 2) = any((select regexp_split_to_array(current_setting('session.folders_write'), ','))::text[])); + +CREATE POLICY see_own ON eval_dataset FOR ALL TO windmill_user +USING (SPLIT_PART(eval_dataset.path, '/', 1) = 'u' AND SPLIT_PART(eval_dataset.path, '/', 2) = (select current_setting('session.user'))); +CREATE POLICY see_member ON eval_dataset FOR ALL TO windmill_user +USING (SPLIT_PART(eval_dataset.path, '/', 1) = 'g' AND SPLIT_PART(eval_dataset.path, '/', 2) = any((select regexp_split_to_array(current_setting('session.groups'), ','))::text[])); + +CREATE POLICY see_extra_perms_user_select ON eval_dataset FOR SELECT TO windmill_user +USING (extra_perms ? (select concat('u/', current_setting('session.user')))); +CREATE POLICY see_extra_perms_user_insert ON eval_dataset FOR INSERT TO windmill_user +WITH CHECK ((extra_perms ->> (select concat('u/', current_setting('session.user'))))::boolean); +CREATE POLICY see_extra_perms_user_update ON eval_dataset FOR UPDATE TO windmill_user +USING ((extra_perms ->> (select concat('u/', current_setting('session.user'))))::boolean); +CREATE POLICY see_extra_perms_user_delete ON eval_dataset FOR DELETE TO windmill_user +USING ((extra_perms ->> (select concat('u/', current_setting('session.user'))))::boolean); + +CREATE POLICY see_extra_perms_groups_select ON eval_dataset FOR SELECT TO windmill_user +USING (extra_perms ?| (select regexp_split_to_array(current_setting('session.pgroups'), ','))::text[]); +CREATE POLICY see_extra_perms_groups_insert ON eval_dataset FOR INSERT TO windmill_user +WITH CHECK (exists( + SELECT key, value FROM jsonb_each_text(extra_perms) + WHERE SPLIT_PART(key, '/', 1) = 'g' AND key = ANY((select regexp_split_to_array(current_setting('session.pgroups'), ','))::text[]) + AND value::boolean)); +CREATE POLICY see_extra_perms_groups_update ON eval_dataset FOR UPDATE TO windmill_user +USING (exists( + SELECT key, value FROM jsonb_each_text(extra_perms) + WHERE SPLIT_PART(key, '/', 1) = 'g' AND key = ANY((select regexp_split_to_array(current_setting('session.pgroups'), ','))::text[]) + AND value::boolean)); +CREATE POLICY see_extra_perms_groups_delete ON eval_dataset FOR DELETE TO windmill_user +USING (exists( + SELECT key, value FROM jsonb_each_text(extra_perms) + WHERE SPLIT_PART(key, '/', 1) = 'g' AND key = ANY((select regexp_split_to_array(current_setting('session.pgroups'), ','))::text[]) + AND value::boolean)); + +-- Whether the session may *write* the dataset at (_workspace_id, _path): the same disjunction the +-- dataset's own write policies use, in one place so the cases that hang off a dataset are governed +-- by exactly the rule the dataset is. A read grant is not enough — writing a case is writing the +-- dataset's contents — so this checks write, not merely visibility. +CREATE OR REPLACE FUNCTION eval_dataset_writable(_workspace_id varchar, _path varchar) + RETURNS boolean LANGUAGE sql STABLE AS $$ + SELECT EXISTS ( + SELECT 1 FROM eval_dataset d + WHERE d.workspace_id = _workspace_id AND d.path = _path + AND ( + (SPLIT_PART(d.path, '/', 1) = 'f' AND SPLIT_PART(d.path, '/', 2) = any((select regexp_split_to_array(current_setting('session.folders_write'), ','))::text[])) + OR (SPLIT_PART(d.path, '/', 1) = 'u' AND SPLIT_PART(d.path, '/', 2) = (select current_setting('session.user'))) + OR (SPLIT_PART(d.path, '/', 1) = 'g' AND SPLIT_PART(d.path, '/', 2) = any((select regexp_split_to_array(current_setting('session.groups'), ','))::text[])) + OR ((d.extra_perms ->> (select concat('u/', current_setting('session.user'))))::boolean) + OR EXISTS ( + SELECT 1 FROM jsonb_each_text(d.extra_perms) ep + WHERE SPLIT_PART(ep.key, '/', 1) = 'g' + AND ep.key = ANY((select regexp_split_to_array(current_setting('session.pgroups'), ','))::text[]) + AND ep.value::boolean) + ) + ); +$$; + +-- Cases are the *contents* of a dataset, not independently addressable objects, so both their +-- visibility and who may change them are the parent's, stated once here instead of mirrored in the +-- API and left to drift. Read is the dataset's read (the subquery is itself subject to +-- eval_dataset's SELECT policies above); write is the dataset's write, which `eval_dataset_writable` +-- checks — so a read-only grant on a dataset can list its cases but not edit them. The whole edit +-- of a dataset and its cases therefore runs as one `user_db` transaction, governed by these +-- policies, rather than being split across the unrestricted pool after a hand-written check. +CREATE POLICY see_parent_dataset ON eval_case FOR SELECT TO windmill_user +USING ( + EXISTS ( + SELECT 1 FROM eval_dataset d + WHERE d.workspace_id = eval_case.workspace_id AND d.path = eval_case.dataset_path + ) +); +CREATE POLICY write_parent_dataset_insert ON eval_case FOR INSERT TO windmill_user +WITH CHECK (eval_dataset_writable(eval_case.workspace_id, eval_case.dataset_path)); +CREATE POLICY write_parent_dataset_update ON eval_case FOR UPDATE TO windmill_user +USING (eval_dataset_writable(eval_case.workspace_id, eval_case.dataset_path)) +WITH CHECK (eval_dataset_writable(eval_case.workspace_id, eval_case.dataset_path)); +CREATE POLICY write_parent_dataset_delete ON eval_case FOR DELETE TO windmill_user +USING (eval_dataset_writable(eval_case.workspace_id, eval_case.dataset_path)); +-- One run of a dataset: written once when the dataset is run, and only ever read afterwards, +-- which is what makes it worth comparing against. +CREATE TABLE eval_experiment ( + id UUID PRIMARY KEY, + workspace_id VARCHAR(50) NOT NULL, + dataset_path VARCHAR(255) NOT NULL, + -- {kind, path, version}: what was run, at the version it was at when the run was enqueued. + subject JSONB NOT NULL, + -- A run is named by the number it is: "Run 7" is stable, sorts, and survives history being + -- pruned, which a position computed at read time would not. Allocated per (dataset, subject + -- path) when the run is opened. + run_number INTEGER NOT NULL, + -- A run is one flow: a loop over the cases, each iteration answering and then scoring. This + -- is the job holding it, so the run can be watched, cancelled and rerun as the single thing + -- it is. Assigned before the flow is pushed, so a launch that dies partway leaves an + -- experiment naming a job that never started rather than a flow nothing accounts for. + run_job_id UUID NOT NULL, + created_at TIMESTAMPTZ NOT NULL DEFAULT now(), + created_by VARCHAR(50) NOT NULL, + FOREIGN KEY (workspace_id, dataset_path) REFERENCES eval_dataset (workspace_id, path) + ON DELETE CASCADE ON UPDATE CASCADE +); + +CREATE INDEX index_eval_experiment_dataset ON eval_experiment (workspace_id, dataset_path, created_at DESC); + +-- Serves the per-agent run list, which spans datasets: an agent's history is one list whichever +-- dataset each run was of. +CREATE INDEX index_eval_experiment_subject ON eval_experiment + (workspace_id, (subject ->> 'path'), created_at DESC); + +-- The exact case set an experiment ran, by value: a dataset keeps changing, and a result set that +-- cannot say which inputs produced it is not reproducible. `case_id` is therefore deliberately not +-- a foreign key — deleting a case must not rewrite the history of the runs that used it. +CREATE TABLE eval_experiment_case ( + experiment_id UUID NOT NULL REFERENCES eval_experiment (id) ON DELETE CASCADE, + ordinal INT NOT NULL, + case_id UUID NOT NULL, + input JSONB NOT NULL DEFAULT '{}', + expected JSONB NULL, + -- The iteration of the run's flow that answered this case. Minted by the flow engine, so the + -- case is recorded before it has one and the id is filled in once the iterations exist. + job_id UUID NULL, + -- What the run produced, copied out of the jobs once they have produced it. Jobs have their + -- own retention, and a recorded run has to still read as the run it was once they are gone. + -- `answered` is the agent step's own outcome, which is settled while the iteration around it + -- is still scoring; `status` is the iteration's, once it has one. + output TEXT NULL, + answered BOOLEAN NULL, + status VARCHAR(30) NULL, + -- The resource version the agent was at for this cell, and — for a draft, which has no + -- version to move — the hash of the configuration that actually ran: the only thing that can + -- say a row describes an agent that has since been edited. + subject_version BIGINT NULL, + subject_draft_hash VARCHAR(64) NULL, + started_at TIMESTAMPTZ NOT NULL DEFAULT now(), + PRIMARY KEY (experiment_id, ordinal), + -- A run holds each case once: the pair is what identifies a cell. + CONSTRAINT eval_experiment_case_unique_case UNIQUE (experiment_id, case_id) +); + +-- One scorer's verdict on one run. Separate from the run because scoring is separate from running: +-- a scorer's verdict is stored per run and scorer, independent of the agent execution that +-- produced the answers. +CREATE TABLE eval_score ( + experiment_id UUID NOT NULL, + ordinal INT NOT NULL, + scorer_id VARCHAR(64) NOT NULL, + -- NULL until the verdict has been read out of the run's flow, and when scoring failed. + score DOUBLE PRECISION NULL, + reason TEXT NULL, + -- [{name, passed, detail}], for scorers that report per-assertion results. + checks JSONB NULL, + error TEXT NULL, + -- The scorer read the run and said it had nothing to measure on this case. A verdict, not a + -- failure: the cell is left out of the column's mean and pass rate rather than counted as a + -- zero or reported as a scorer that produced nothing. + not_applicable BOOLEAN NOT NULL DEFAULT false, + -- Hash of the scorer configuration that produced this score, including the script hash or flow + -- version actually executed. Two scores of the same scorer whose definitions differ are still + -- compared, but the column says the scorer changed rather than letting it read as a change of + -- agent. + definition VARCHAR(64) NOT NULL, + created_at TIMESTAMPTZ NOT NULL DEFAULT now(), + PRIMARY KEY (experiment_id, ordinal, scorer_id), + FOREIGN KEY (experiment_id, ordinal) REFERENCES eval_experiment_case (experiment_id, ordinal) + ON DELETE CASCADE +); + +GRANT ALL ON eval_experiment TO windmill_user; +GRANT ALL ON eval_experiment TO windmill_admin; +GRANT ALL ON eval_experiment_case TO windmill_user; +GRANT ALL ON eval_experiment_case TO windmill_admin; +GRANT ALL ON eval_score TO windmill_user; +GRANT ALL ON eval_score TO windmill_admin; + +ALTER TABLE eval_experiment ENABLE ROW LEVEL SECURITY; +ALTER TABLE eval_experiment_case ENABLE ROW LEVEL SECURITY; +ALTER TABLE eval_score ENABLE ROW LEVEL SECURITY; + +CREATE POLICY admin_policy ON eval_experiment FOR ALL TO windmill_admin USING (true); +CREATE POLICY admin_policy ON eval_experiment_case FOR ALL TO windmill_admin USING (true); +CREATE POLICY admin_policy ON eval_score FOR ALL TO windmill_admin USING (true); + +-- Experiments are the *contents* of a dataset, not independently addressable objects, so their +-- visibility is the parent's: the subquery is itself subject to eval_dataset's policies, which +-- stay stated once instead of being mirrored here and left to drift. +-- +-- SELECT only, deliberately. A `FOR ALL ... USING` would be reused as the INSERT/UPDATE/DELETE +-- check expression, and since the subquery is a SELECT it applies the dataset's *read* policies — +-- which would let someone with read-only access to a dataset forge an experiment row naming a job +-- they cannot otherwise read. Writes are done on the unrestricted pool after the API has checked +-- write access to the parent, and a stray `user_db` write to these tables is meant to fail rather +-- than silently succeed. +CREATE POLICY see_parent_dataset ON eval_experiment FOR SELECT TO windmill_user +USING ( + EXISTS ( + SELECT 1 FROM eval_dataset d + WHERE d.workspace_id = eval_experiment.workspace_id AND d.path = eval_experiment.dataset_path + ) +); + +CREATE POLICY see_parent_experiment ON eval_experiment_case FOR SELECT TO windmill_user +USING ( + EXISTS ( + SELECT 1 FROM eval_experiment e + WHERE e.id = eval_experiment_case.experiment_id + ) +); + +-- Visibility is the experiment's, which is the dataset's. +CREATE POLICY see_parent_experiment ON eval_score FOR SELECT TO windmill_user +USING ( + EXISTS ( + SELECT 1 FROM eval_experiment e + WHERE e.id = eval_score.experiment_id + ) +); diff --git a/backend/migrations/20260819073729_resource_version_number.down.sql b/backend/migrations/20260819073729_resource_version_number.down.sql new file mode 100644 index 0000000000..13f95048ee --- /dev/null +++ b/backend/migrations/20260819073729_resource_version_number.down.sql @@ -0,0 +1,17 @@ +-- Back to numbering versions by the table-wide identity sequence, so the function must stop +-- writing a column that is about to go. +CREATE OR REPLACE FUNCTION record_resource_version() RETURNS trigger AS $$ +BEGIN + INSERT INTO resource_version (workspace_id, path, resource_type, value, created_by) + VALUES ( + NEW.workspace_id, NEW.path, NEW.resource_type, NEW.value, + COALESCE(NULLIF(current_setting('session.user', true), ''), NEW.created_by) + ); + + RETURN NEW; +END; +$$ LANGUAGE plpgsql SECURITY DEFINER SET search_path FROM CURRENT; + +DROP INDEX IF EXISTS index_resource_version_number; + +ALTER TABLE resource_version DROP COLUMN version; diff --git a/backend/migrations/20260819073729_resource_version_number.up.sql b/backend/migrations/20260819073729_resource_version_number.up.sql new file mode 100644 index 0000000000..e56dc7a09c --- /dev/null +++ b/backend/migrations/20260819073729_resource_version_number.up.sql @@ -0,0 +1,52 @@ +-- `id` is one identity sequence for the whole table and stays how a version is addressed; +-- `version` is the resource's own count, which is what a version is presented by. +ALTER TABLE resource_version ADD COLUMN version BIGINT; + +UPDATE resource_version rv SET version = ranked.rn +FROM ( + SELECT id, row_number() OVER (PARTITION BY workspace_id, path ORDER BY id) AS rn + FROM resource_version +) ranked +WHERE rv.id = ranked.id; + +ALTER TABLE resource_version ALTER COLUMN version SET NOT NULL; + +-- The number is only meaningful within a path, so the triple is the natural key: it serves the +-- lookup by number and makes a duplicate a hard error rather than two rows claiming v7. +CREATE UNIQUE INDEX index_resource_version_number ON resource_version (workspace_id, path, version); + +-- Numbering is assigned here rather than derived when read because both ways of deleting versions +-- take the oldest ones: the monitor's trim past MAX_RESOURCE_VERSIONS, and clearing a history down +-- to its current value. A number computed by counting the survivors would renumber under either, +-- so a run recorded against v3 would later name a different version. +CREATE OR REPLACE FUNCTION record_resource_version() RETURNS trigger AS $$ +BEGIN + -- `session.user` is set by UserDB::begin for authed requests; worker and system writes fall + -- back to the row's own author. NULLIF because a transaction-local set_config resets the + -- placeholder to the empty string rather than unsetting it, so a pooled connection that + -- previously served an authed request reports '' here, not NULL. + -- + -- MAX + 1 needs no lock of its own: this runs inside the transaction that wrote `resource`, and + -- a concurrent write to the same path blocks on that row's lock — or on the primary key, for an + -- insert — before its own trigger can run, so the maximum cannot be read stale. Deleting + -- versions never lowers it, since both deletions keep the newest row. + INSERT INTO resource_version (workspace_id, path, resource_type, value, created_by, version) + VALUES ( + NEW.workspace_id, NEW.path, NEW.resource_type, NEW.value, + COALESCE(NULLIF(current_setting('session.user', true), ''), NEW.created_by), + (SELECT COALESCE(MAX(version), 0) + 1 FROM resource_version + WHERE workspace_id = NEW.workspace_id AND path = NEW.path) + ); + + -- The per-path cap is enforced by trim_resource_versions in the monitor, not here: trimming + -- on every write would tax a path `setResource` can drive in a loop, to keep a bound that + -- does not need to hold instantaneously. + + RETURN NEW; +END; +-- SECURITY DEFINER so history is written on behalf of every writer without granting anyone direct +-- write access to the table, which users hold SELECT on only. `SET search_path FROM CURRENT` is the +-- injection hardening that goes with it, captured rather than hardcoded so installs running a +-- non-public PG_SCHEMA still resolve (see +-- 20260624103600_repair_folder_labels_search_path.up.sql). +$$ LANGUAGE plpgsql SECURITY DEFINER SET search_path FROM CURRENT; diff --git a/backend/windmill-api-auth/src/scopes.rs b/backend/windmill-api-auth/src/scopes.rs index da24453553..7109a04425 100644 --- a/backend/windmill-api-auth/src/scopes.rs +++ b/backend/windmill-api-auth/src/scopes.rs @@ -289,6 +289,7 @@ pub enum ScopeDomain { OAuth, AI, AiSkills, + AiEvals, // AI agent eval datasets Indexer, Teams, // Microsoft Teams integration @@ -349,6 +350,7 @@ impl ScopeDomain { Self::OAuth => "oauth", Self::AI => "ai", Self::AiSkills => "ai_skills", + Self::AiEvals => "ai_evals", Self::Capture => "capture", Self::Drafts => "drafts", Self::Favorites => "favorites", @@ -404,6 +406,7 @@ impl ScopeDomain { "oauth" => Some(Self::OAuth), "ai" => Some(Self::AI), "ai_skills" => Some(Self::AiSkills), + "ai_evals" => Some(Self::AiEvals), "indexer" | "srch" => Some(Self::Indexer), "teams" => Some(Self::Teams), "native_triggers" => Some(Self::NativeTriggers), diff --git a/backend/windmill-api-users/src/users.rs b/backend/windmill-api-users/src/users.rs index 0c83d5f6ec..8ea3bd0995 100644 --- a/backend/windmill-api-users/src/users.rs +++ b/backend/windmill-api-users/src/users.rs @@ -2399,6 +2399,7 @@ pub async fn delete_workspace_user_internal( "flow", "app", "resource", + "eval_dataset", "variable", "schedule", "group_", diff --git a/backend/windmill-api-workspaces/src/workspaces.rs b/backend/windmill-api-workspaces/src/workspaces.rs index 44018736eb..e04e1c1ab9 100644 --- a/backend/windmill-api-workspaces/src/workspaces.rs +++ b/backend/windmill-api-workspaces/src/workspaces.rs @@ -5471,6 +5471,8 @@ async fn clone_workspace_data( // Clone scripts with new hashes clone_scripts(tx, source_workspace_id, target_workspace_id).await?; + clone_eval_datasets(tx, source_workspace_id, target_workspace_id).await?; + // Clone the dbt graph sidecars. After `clone_scripts`, which keeps each // script's hash: these key on it, and a static descriptor never re-ingests, // so a fork without them shows dbt scripts with no models until someone @@ -6005,6 +6007,36 @@ async fn clone_resources( Ok(()) } +async fn clone_eval_datasets( + tx: &mut Transaction<'_, Postgres>, + source_workspace_id: &str, + target_workspace_id: &str, +) -> Result<()> { + // The authored evaluation data — datasets and their cases — travels with a fork like resources + // and scripts do; the runs (experiments) do not, since they name jobs the fork has no copy of. + sqlx::query!( + "INSERT INTO eval_dataset (workspace_id, path, summary, scorers, extra_perms, created_at, created_by, edited_at, edited_by) + SELECT $2, path, summary, scorers, extra_perms, created_at, created_by, edited_at, edited_by + FROM eval_dataset WHERE workspace_id = $1", + source_workspace_id, + target_workspace_id, + ) + .execute(&mut **tx) + .await?; + // A new id per cloned case: `eval_case`'s primary key is the id alone, unique across the whole + // table, so copying it would collide with the source's own rows. + sqlx::query!( + "INSERT INTO eval_case (workspace_id, dataset_path, input, expected, created_at, created_by) + SELECT $2, dataset_path, input, expected, created_at, created_by + FROM eval_case WHERE workspace_id = $1", + source_workspace_id, + target_workspace_id, + ) + .execute(&mut **tx) + .await?; + Ok(()) +} + async fn clone_variables( tx: &mut Transaction<'_, Postgres>, db: &DB, diff --git a/backend/windmill-api/openapi.yaml b/backend/windmill-api/openapi.yaml index d2f61a5630..477b484926 100644 --- a/backend/windmill-api/openapi.yaml +++ b/backend/windmill-api/openapi.yaml @@ -7819,7 +7819,7 @@ paths: schema: type: string - /w/{workspace}/resources/history/v/{version}: + /w/{workspace}/resources/history/v/{id}: get: summary: get a single resource version, with its value operationId: getResourceVersion @@ -7827,9 +7827,10 @@ paths: - resource parameters: - $ref: "#/components/parameters/WorkspaceId" - - name: version + - name: id in: path required: true + description: The version's id, not its number. schema: type: integer format: int64 @@ -7851,7 +7852,7 @@ paths: required: - missing_references - /w/{workspace}/resources/history/restore/v/{version}: + /w/{workspace}/resources/history/restore/v/{id}: post: summary: restore a resource to a previous version operationId: restoreResourceVersion @@ -7859,9 +7860,10 @@ paths: - resource parameters: - $ref: "#/components/parameters/WorkspaceId" - - name: version + - name: id in: path required: true + description: The version's id, not its number. schema: type: integer format: int64 @@ -11779,6 +11781,453 @@ paths: items: $ref: "#/components/schemas/FlowConversationMessage" + /w/{workspace}/ai_evals/datasets/list: + get: + summary: list eval datasets + operationId: listEvalDatasets + tags: + - ai_evals + parameters: + - $ref: "#/components/parameters/WorkspaceId" + responses: + "200": + description: eval datasets list + content: + application/json: + schema: + type: array + items: + $ref: "#/components/schemas/EvalDataset" + + /w/{workspace}/ai_evals/datasets/create: + post: + summary: create an eval dataset + operationId: createEvalDataset + tags: + - ai_evals + parameters: + - $ref: "#/components/parameters/WorkspaceId" + requestBody: + description: new eval dataset + required: true + content: + application/json: + schema: + type: object + required: [path] + properties: + path: + type: string + maxLength: 255 + summary: + type: string + maxLength: 1000 + scorers: + type: array + maxItems: 20 + items: + $ref: "#/components/schemas/Scorer" + cases: + type: array + maxItems: 1000 + description: The cases to create the dataset holding, so one can be assembled in a single act rather than created empty and filled in afterwards. + items: + $ref: "#/components/schemas/NewEvalCase" + responses: + "200": + description: eval dataset created + content: + text/plain: + schema: + type: string + + /w/{workspace}/ai_evals/datasets/get/{path}: + get: + summary: get an eval dataset + operationId: getEvalDataset + tags: + - ai_evals + parameters: + - $ref: "#/components/parameters/WorkspaceId" + - $ref: "#/components/parameters/Path" + responses: + "200": + description: eval dataset + content: + application/json: + schema: + $ref: "#/components/schemas/EvalDataset" + + /w/{workspace}/ai_evals/datasets/update/{path}: + post: + summary: update an eval dataset + operationId: updateEvalDataset + tags: + - ai_evals + parameters: + - $ref: "#/components/parameters/WorkspaceId" + - $ref: "#/components/parameters/Path" + requestBody: + description: updated eval dataset + required: true + content: + application/json: + schema: + type: object + properties: + path: + type: string + maxLength: 255 + description: > + Renames the dataset. Its cases and experiments follow through the foreign + keys, so a rename keeps the history it already has. + summary: + type: string + maxLength: 1000 + description: Left out to keep the stored summary; sent as "" to clear it. + scorers: + type: array + maxItems: 20 + description: > + Left out to keep the dataset's columns as they are; sent to replace them + wholesale. + items: + $ref: "#/components/schemas/Scorer" + cases: + type: array + maxItems: 1000 + description: > + The cases as they should stand afterwards: all of them, each carrying its id if + the dataset already has it. Sent with the rest of an edit so that a rename the + dataset refuses refuses the case edits with it. + items: + $ref: "#/components/schemas/SaveEvalCase" + responses: + "200": + description: eval dataset updated + content: + text/plain: + schema: + type: string + + /w/{workspace}/ai_evals/datasets/delete/{path}: + post: + summary: delete an eval dataset and all its cases + description: > + The cases, the runs and their recorded case sets go with it through the foreign keys; the + jobs those runs produced are left alone. + operationId: deleteEvalDataset + tags: + - ai_evals + parameters: + - $ref: "#/components/parameters/WorkspaceId" + - $ref: "#/components/parameters/Path" + responses: + "200": + description: eval dataset deleted + content: + text/plain: + schema: + type: string + + /w/{workspace}/ai_evals/cases/list/{path}: + get: + summary: list the cases of an eval dataset + operationId: listEvalCases + tags: + - ai_evals + parameters: + - $ref: "#/components/parameters/WorkspaceId" + - $ref: "#/components/parameters/Path" + - $ref: "#/components/parameters/Page" + - $ref: "#/components/parameters/PerPage" + responses: + "200": + description: eval cases + content: + application/json: + schema: + type: object + required: [cases] + properties: + cases: + type: array + items: + $ref: "#/components/schemas/EvalCase" + /w/{workspace}/ai_evals/subject_state: + get: + summary: what the agent under test is right now + description: > + The version it is deployed at. Small on purpose: the results endpoint reports the same + thing but harvests scores and reads every job to do it, so it is not something to ask for + on its own. + operationId: evalSubjectState + tags: + - ai_evals + parameters: + - $ref: "#/components/parameters/WorkspaceId" + - name: path + in: query + required: true + schema: + type: string + responses: + "200": + description: the subject as it is now + content: + application/json: + schema: + type: object + properties: + version: + type: integer + format: int64 + + /w/{workspace}/ai_evals/run_payload: + get: + summary: the run one iteration of an eval run answered, as its scorers read it + operationId: evalRunPayload + description: > + Called by the step a run's flow places between the agent and its scorers. Every tool call + is enriched with the arguments, result, status and duration of the job that ran it, and + with the schema of the script version it ran, none of which the flow itself can read. + tags: + - ai_evals + parameters: + - $ref: "#/components/parameters/WorkspaceId" + - name: job_id + in: query + required: true + description: The flow job that answered the case. + schema: + type: string + format: uuid + responses: + "200": + description: the run and its rendering + content: + application/json: + schema: + type: object + required: [run, rendered] + properties: + run: + type: object + description: The case, the answer, and every tool call the agent made. + rendered: + type: string + description: The same run as a judge agent is shown it. + + /w/{workspace}/ai_evals/scorer_defaults: + get: + summary: what a new judge agent and a new script scorer are created from + operationId: scorerDefaults + tags: + - ai_evals + parameters: + - $ref: "#/components/parameters/WorkspaceId" + responses: + "200": + description: scorer defaults + content: + application/json: + schema: + type: object + required: [judge_prompt, script_template] + properties: + judge_prompt: + type: string + description: The system prompt a judge agent is created with. + script_template: + type: string + + /w/{workspace}/ai_evals/scorers/recent: + get: + summary: list the scorers already in use in this workspace, most recent first + description: > + Filtered twice, both times by what the caller can read: the datasets they come from, and + the runnables themselves. A scorer they could not run does not appear. + operationId: recentScorers + tags: + - ai_evals + parameters: + - $ref: "#/components/parameters/WorkspaceId" + - name: kind + description: only scorers of this kind + in: query + required: false + schema: + type: string + enum: [script, agent] + responses: + "200": + description: recently used scorers + content: + application/json: + schema: + type: array + items: + allOf: + - $ref: "#/components/schemas/Scorer" + - type: object + required: [dataset] + properties: + dataset: + type: string + description: The dataset it is a column of. + + /w/{workspace}/ai_evals/experiments/run: + post: + summary: run every case of a dataset as one immutable experiment + operationId: runExperiment + tags: + - ai_evals + parameters: + - $ref: "#/components/parameters/WorkspaceId" + requestBody: + description: what to run + required: true + content: + application/json: + schema: + type: object + required: [dataset, subject] + properties: + dataset: + type: string + subject: + $ref: "#/components/schemas/EvalSubject" + responses: + "200": + description: id of the created experiment + content: + text/plain: + schema: + type: string + + /w/{workspace}/ai_evals/experiments/collect: + post: + summary: record what a run produced, so it outlives the jobs that produced it + description: > + Called by a run's own flow as its last step. The answers and scores a run produced live in + its jobs, which have their own retention; this copies them onto the run's rows. Reading a + run does the same, so this is what covers a run nobody opened. + operationId: collectExperiment + tags: + - ai_evals + parameters: + - $ref: "#/components/parameters/WorkspaceId" + - name: id + in: query + required: true + schema: + type: string + format: uuid + responses: + "200": + description: how many of the run's cases are recorded + content: + application/json: + schema: + type: integer + + /w/{workspace}/ai_evals/experiments/list_all: + get: + summary: list every experiment, across datasets + operationId: listAllExperiments + tags: + - ai_evals + parameters: + - $ref: "#/components/parameters/WorkspaceId" + - name: subject_path + description: > + Restrict to one agent's runs, which is what makes the list a history rather than a + log. Runs of what is deployed, of a past version, and of the edits waiting on top are + all that agent's, so this does not discriminate by kind. + in: query + required: false + schema: + type: string + responses: + "200": + description: > + The 100 newest experiments, each naming the dataset it is of. Restricted to datasets + the caller can read. + content: + application/json: + schema: + type: array + items: + $ref: "#/components/schemas/EvalExperiment" + + /w/{workspace}/ai_evals/experiments/results/{path}: + get: + summary: read an experiment's results, one row per case + operationId: experimentResults + tags: + - ai_evals + parameters: + - $ref: "#/components/parameters/WorkspaceId" + - $ref: "#/components/parameters/Path" + - name: id + description: the experiment to read + in: query + required: true + schema: + type: string + format: uuid + - name: baseline + description: > + The experiment every column is compared against. A delta is only computed between two + scores of the same scorer id, and a column the baseline was never scored with reports + it rather than showing a difference. + in: query + required: false + schema: + type: string + format: uuid + responses: + "200": + description: experiment results + content: + application/json: + schema: + type: object + required: [experiment, scorers, rows, means, regressed] + properties: + experiment: + $ref: "#/components/schemas/EvalExperiment" + baseline: + $ref: "#/components/schemas/EvalExperiment" + scorers: + type: array + description: The columns, which belong to the dataset rather than the experiment. + items: + $ref: "#/components/schemas/Scorer" + rows: + type: array + items: + $ref: "#/components/schemas/ExperimentRow" + means: + type: array + items: + $ref: "#/components/schemas/ScorerMean" + regressed: + type: integer + description: Cells scoring lower than the baseline, across every column. + subject_current_version: + type: integer + format: int64 + description: > + The version the subject is on now. A row that ran against an earlier one + describes an agent that no longer exists. + subject_deployed_hash: + type: string + description: > + What the agent hashes to as deployed. A run of unsaved edits carrying this + hash ran exactly what is deployed now — the edits were saved — so it is a run + of that version rather than of edits. + /w/{workspace}/path_autocomplete/list_paths: get: summary: list all paths in a workspace for client-side autocomplete @@ -26178,6 +26627,348 @@ components: additionalProperties: true description: Array of JSON Web Keys for JWT verification + EvalSubject: + type: object + description: What an eval run is executed against. + required: [kind, path] + properties: + kind: + type: string + description: > + `agent` runs the ai_agent resource as it is deployed when the run opens, `agent_draft` + the caller's unsaved edits of it as the editor holds them (carried in `draft`), and + `agent_version` one past version named by `version`. The first and last are read + server-side; all three are inlined into the run, so every case of a run executes one + configuration: a deploy part-way through changes what the next run measures, never + this one. + enum: [agent, agent_draft, agent_version] + path: + type: string + description: Path of the ai_agent resource. + version: + type: integer + format: int64 + nullable: true + description: > + The agent's per-path version number when the run opened: how many times the resource + had been saved, not a resource_version row id. For `agent` and `agent_draft` it names + the configuration the run read and every case executed. For `agent_version` it is the + request's own, says which version to inline, and is required. + draft: + $ref: "#/components/schemas/AgentDraft" + draft_hash: + type: string + description: > + Hash of the configuration a draft run executed, stamped server-side. A draft moves + without the version moving, so this is what dates a run of one. It is also what + recognises a draft run whose configuration was later deployed: when it matches the + agent as deployed, the run's kind and version are rewritten to that version, once, and + the hash is kept as what the resolution rests on. + + AgentDraft: + type: object + description: > + The brain and tools of an agent, as the flow editor holds them. Carried by the request and + present exactly when the subject kind is `agent_draft` — the edits exist only in the editor + — where it is the whole definition of what ran: the run goes through the same unlinked + branch of the agent executor the editor's own test uses. + properties: + input_transforms: + type: object + additionalProperties: true + description: > + The agent's input transforms: provider, system prompt, output type and the rest. The + message and attachments come from the case and override anything named here. + tools: + type: array + items: + type: object + additionalProperties: true + + EvalDataset: + type: object + required: [path, created_at, created_by, edited_at, edited_by] + properties: + path: + type: string + maxLength: 255 + summary: + type: string + maxLength: 1000 + scorers: + type: array + description: The columns of the results table, in display order. + items: + $ref: "#/components/schemas/Scorer" + created_at: + type: string + format: date-time + created_by: + type: string + edited_at: + type: string + format: date-time + edited_by: + type: string + + EvalCaseInput: + type: object + description: The inputs a standalone run feeds the agent. + properties: + user_message: + type: string + user_attachments: + type: array + items: + type: object + + NewEvalCase: + type: object + properties: + input: + $ref: "#/components/schemas/EvalCaseInput" + expected: + description: Reference output a scorer compares a rerun against. + + SaveEvalCase: + allOf: + - type: object + properties: + id: + description: Absent for a case the dataset does not hold yet. + type: string + format: uuid + - $ref: "#/components/schemas/NewEvalCase" + + EvalCase: + allOf: + - type: object + required: [id, created_at, created_by] + properties: + id: + type: string + format: uuid + created_at: + type: string + format: date-time + created_by: + type: string + - $ref: "#/components/schemas/NewEvalCase" + + Scorer: + type: object + description: > + A scorer is a column of the results table, and it is always a runnable: an ai_agent + resource sent the run to grade, or a script handed the run as an argument. `id` is assigned + when the scorer is added to a dataset and never reused: it is what makes a column the same + column across experiments when the scorer is renamed, and a delta is only ever computed + between two scores carrying the same id. A scorer sent without an id is given one. + required: [kind, path] + properties: + id: + type: string + name: + type: string + description: Column header. Defaults to the last segment of the path. + pass_if: + type: number + description: > + A score at or above this counts as a pass, and the column reports a pass rate beside + its mean. Applied when results are read rather than when they are produced, so moving + the line re-reads every score already recorded instead of invalidating them. + kind: + type: string + enum: [script, agent] + path: + type: string + description: The script, or the ai_agent resource used as a judge. + + EvalExperiment: + type: object + description: >- + One run of a dataset: written once when the dataset is run, and only ever read afterwards. + The case set it executed is returned by the results endpoint, not here: a listing would + otherwise send the whole dataset back once per experiment. + required: [id, dataset, subject, run_number, run_job_id, case_count, created_at, created_by] + properties: + id: + type: string + format: uuid + dataset: + type: string + subject: + $ref: "#/components/schemas/EvalSubject" + run_number: + type: integer + description: > + This agent's nth run of this dataset, allocated once and never reused. What a run is + called. Numbered per agent rather than per subject kind: runs of what is deployed and + runs of its draft are the same agent's history. + run_job_id: + type: string + format: uuid + description: > + The flow executing the run: one job holding every case and its scores. + case_count: + type: integer + scores: + type: array + description: > + What the run scored, one entry per scorer that produced a number. Carried on the run + itself so a list of runs can say what each one scored without reading every cell of + every one of them. Empty on a run whose scores have not been read yet. + items: + $ref: "#/components/schemas/ExperimentScore" + running: + type: boolean + description: > + Whether the flow executing this run is still going. What makes a list of runs worth + watching rather than worth reloading. + created_at: + type: string + format: date-time + created_by: + type: string + + ExperimentScore: + type: object + description: >- + One scorer's headline for one run: the two numbers a column reports, over that run's cells. + required: [scorer_id, name, kind, scored, failed] + properties: + scorer_id: + type: string + name: + type: string + description: > + What the column is called in the dataset that ran it, resolved server-side because a + list of runs spanning datasets cannot hold every dataset's scorers to look it up. + kind: + type: string + enum: [agent, script] + mean: + type: number + pass_rate: + type: number + description: > + The share of scored cells at or above the column's threshold, for a column that has + one. Absent where the column has no threshold and the mean is the whole headline. + scored: + type: integer + failed: + type: integer + description: > + How many of the run's cells the column failed on. A column that failed on all of them + has no number to report and is still one of the columns that ran. + + CellScore: + type: object + description: One scorer's verdict on one run, and how it compares with the baseline. + required: [scorer_id, pending, definition_changed] + properties: + scorer_id: + type: string + score: + type: number + reason: + type: string + checks: {} + error: + type: string + not_applicable: + type: boolean + description: > + The scorer read this case and had nothing to measure on it. Left out of the column's + mean and pass rate rather than counted as a zero. + pending: + type: boolean + description: A scoring job is still running for this cell. + passed: + type: boolean + description: > + Which side of the scorer's `pass_if` threshold the score fell on. Absent when the + column has no threshold, or has no score yet. + baseline: + type: number + description: The same scorer's number on the baseline experiment. + definition_changed: + type: boolean + description: > + The baseline's score came from a different definition of this scorer, so the delta is + a change of scorer as much as a change of agent. + + ExperimentRow: + type: object + required: [case_id, input, status, scores] + properties: + case_id: + type: string + format: uuid + input: + $ref: "#/components/schemas/EvalCaseInput" + expected: {} + job_id: + type: string + format: uuid + description: > + The iteration that ran this case. Absent between a run being recorded and its flow + reaching this case, which reads as a case still to run. + status: + type: string + description: > + The case's status; `running` until its iteration completes, and `unavailable` for a + case whose job was retained away before anything read what it produced. + enum: [running, success, failure, canceled, skipped, unavailable] + output: + type: string + description: The agent's answer. The full trajectory stays reachable through job_id. + subject_version: + type: integer + format: int64 + description: > + The agent version this cell ran against. Cells of one experiment can differ, which the + table says rather than averaging two versions silently. + subject_draft_hash: + type: string + description: > + For a run of unsaved edits, the hash of the configuration this cell ran. Edits move + without a version changing, so this is what identifies what ran, and what recognises a + run whose edits were later saved as a run of that version. + scores: + type: array + description: One entry per scorer of the dataset, in column order. + items: + $ref: "#/components/schemas/CellScore" + + ScorerMean: + type: object + description: > + A column's summary. There is no single number for a dataset: averaging a judge with an + exact match would invent one. + required: [scorer_id, scored, missing_in_baseline, definition_changed] + properties: + scorer_id: + type: string + mean: + type: number + baseline_mean: + type: number + pass_rate: + type: number + description: > + The share of scored cells that passed, for a column with a threshold. Reported beside + the mean rather than instead of it: a pass rate says how many cases are good enough, + a mean says by how much, and neither answers the other's question. + baseline_pass_rate: + type: number + scored: + type: integer + missing_in_baseline: + type: integer + description: Cells the baseline has no score for, so a column the baseline never ran shows as unscored rather than as a spurious difference. + definition_changed: + type: boolean + FlowConversation: type: object required: @@ -28593,6 +29384,11 @@ components: id: type: integer format: int64 + description: How this version is addressed. Unique across every resource, so it says nothing about how many times this one has been saved. + version: + type: integer + format: int64 + description: Which version of this resource it is, counted from its first. What a version is called. created_at: type: string format: date-time @@ -28600,6 +29396,7 @@ components: type: string required: - id + - version - created_at ListableResource: diff --git a/backend/windmill-api/src/ai_evals/datasets.rs b/backend/windmill-api/src/ai_evals/datasets.rs new file mode 100644 index 0000000000..da901276b0 --- /dev/null +++ b/backend/windmill-api/src/ai_evals/datasets.rs @@ -0,0 +1,473 @@ +use super::*; + +#[derive(Serialize, Deserialize, Debug, Clone)] +pub struct EvalDataset { + pub path: String, + #[serde(default, skip_serializing_if = "Option::is_none")] + pub summary: Option, + /// The columns of the results table, in display order. + #[serde(default)] + pub scorers: Vec, + pub created_at: DateTime, + pub created_by: String, + pub edited_at: DateTime, + pub edited_by: String, +} + +/// The agent-facing half of a case: exactly the inputs a standalone run feeds the agent. +#[derive(Serialize, Deserialize, Debug, Clone, Default)] +pub struct EvalCaseInput { + #[serde(default, skip_serializing_if = "Option::is_none")] + pub user_message: Option, + #[serde(default, skip_serializing_if = "Option::is_none")] + pub user_attachments: Option>, +} + +#[derive(Serialize, Deserialize, Debug, Clone)] +pub struct EvalCase { + pub id: Uuid, + pub input: EvalCaseInput, + #[serde(default, skip_serializing_if = "Option::is_none")] + pub expected: Option>, + pub created_at: DateTime, + pub created_by: String, +} + +/// The case fields a caller may set. `id`/`created_at`/`created_by` are assigned server-side so +/// a client cannot forge provenance or collide with an existing case. +#[derive(Deserialize, Debug)] +pub struct NewEvalCase { + #[serde(default)] + pub input: EvalCaseInput, + #[serde(default)] + pub expected: Option>, +} + +#[derive(Deserialize)] +pub struct CreateDataset { + pub path: String, + #[serde(default)] + pub summary: Option, + #[serde(default)] + pub scorers: Vec, + /// The cases to create it holding. A case cannot be written before there is a dataset for it + /// to be a row of, so they are sent with it rather than added afterwards. + #[serde(default)] + pub cases: Vec, +} + +#[derive(Deserialize)] +pub struct EditDataset { + /// Renames the dataset. Its cases and experiments follow through the foreign keys. + #[serde(default)] + pub path: Option, + /// Left out to keep the stored summary; sent as `""` to clear it. + #[serde(default)] + pub summary: Option, + /// Left out to keep the dataset's columns as they are; sent to replace them wholesale. + #[serde(default)] + pub scorers: Option>, + /// The cases as they should stand afterwards: all of them, each carrying its `id` if the + /// dataset already has it. Sent with the rest of an edit so a rename the dataset refuses + /// refuses the case edits with it, rather than leaving them written under the old name. + #[serde(default)] + pub cases: Option>, +} + +#[derive(Deserialize)] +pub struct SaveCase { + #[serde(default)] + pub id: Option, + #[serde(default)] + pub input: EvalCaseInput, + #[serde(default)] + pub expected: Option>, +} + +#[derive(Serialize)] +pub struct ListCasesResponse { + pub cases: Vec, +} + +pub async fn list_datasets( + authed: ApiAuthed, + Extension(user_db): Extension, + Path(w_id): Path, +) -> JsonResult> { + let mut tx = user_db.begin(&authed).await?; + let rows = sqlx::query!( + "SELECT path, summary, scorers, created_at, created_by, + edited_at, edited_by + FROM eval_dataset WHERE workspace_id = $1 ORDER BY path", + w_id + ) + .fetch_all(&mut *tx) + .await?; + tx.commit().await?; + Ok(Json( + rows.into_iter() + .map(|row| { + dataset_from_row( + row.path, + row.summary, + row.scorers, + row.created_at, + row.created_by, + row.edited_at, + row.edited_by, + ) + }) + .collect::>>()?, + )) +} + +pub async fn create_dataset( + authed: ApiAuthed, + Extension(user_db): Extension, + Path(w_id): Path, + Json(payload): Json, +) -> Result { + check_proper_path(&payload.path)?; + check_summary(payload.summary.as_deref())?; + if authed.is_operator { + return Err(Error::NotAuthorized( + "Operators cannot create eval datasets".to_string(), + )); + } + check_case_set( + payload + .cases + .iter() + .map(|case| (&case.input, case.expected.as_ref())), + )?; + let mut scorers = payload.scorers; + // A dataset being created has no columns yet, so every id is minted. + assign_scorer_ids(&mut scorers, &std::collections::HashSet::new())?; + let scorers = serde_json::to_value(&scorers)?; + // One `user_db` transaction: the row's insert policy gates the dataset, the cases' insert + // policy gates each case, and the two land together or not at all. + let mut tx = user_db.begin(&authed).await?; + // A path already taken returns no row; a path the caller may not create raises the insert + // policy, which `map_rls_denied` turns into an access error. + let created = sqlx::query_scalar!( + "INSERT INTO eval_dataset + (workspace_id, path, summary, scorers, created_by, edited_by) + VALUES ($1, $2, $3, $4, $5, $5) + ON CONFLICT (workspace_id, path) DO NOTHING + RETURNING path", + w_id, + payload.path, + payload.summary, + scorers, + authed.username, + ) + .fetch_optional(&mut *tx) + .await + .map_err(|e| map_rls_denied(&payload.path, "create", e))?; + if created.is_none() { + return Err(Error::BadRequest(format!( + "Eval dataset {} already exists", + payload.path + ))); + } + for case in &payload.cases { + sqlx::query!( + // clock_timestamp() (not the now() default, which is transaction-stable) so cases + // saved together get strictly increasing created_at and reload in insertion order; + // ORDER BY created_at, id would otherwise tie-break a same-transaction batch on the + // random uuid id. + "INSERT INTO eval_case + (workspace_id, dataset_path, input, expected, created_by, created_at) + VALUES ($1, $2, $3, $4, $5, clock_timestamp())", + w_id, + payload.path, + serde_json::to_value(&case.input)?, + opt_from_raw(case.expected.as_ref())?, + authed.username, + ) + .execute(&mut *tx) + .await?; + } + tx.commit().await?; + + Ok(format!("Created eval dataset {}", payload.path)) +} + +pub async fn get_dataset( + authed: ApiAuthed, + Extension(user_db): Extension, + Path((w_id, path)): Path<(String, String)>, +) -> JsonResult { + Ok(Json(read_dataset(&authed, &user_db, &w_id, &path).await?)) +} + +/// An edit is one transaction: the rename, the summary, the columns and the cases land together +/// or not at all. +pub async fn update_dataset( + authed: ApiAuthed, + Extension(user_db): Extension, + Path((w_id, path)): Path<(String, String)>, + Json(payload): Json, +) -> Result { + if authed.is_operator { + return Err(Error::NotAuthorized( + "Operators cannot modify eval datasets".to_string(), + )); + } + check_summary(payload.summary.as_deref())?; + let new_path = match payload.path.filter(|p| *p != path) { + Some(new_path) => { + check_proper_path(&new_path)?; + // A rename is owner-only, as for every other renamable object. RLS write access is not + // enough: the UPDATE policies carry no explicit WITH CHECK, so Postgres reuses their + // USING, and the row's own extra_perms travels with the rename and would satisfy it + // for any destination. + windmill_api_auth::require_owner_of_path(&authed, &path)?; + Some(new_path) + } + None => None, + }; + if let Some(cases) = &payload.cases { + check_cases(cases)?; + } + // One `user_db` transaction, governed by the row-level policies throughout. The row is read + // `FOR UPDATE` — its UPDATE policy decides who may — which also pins its cases, so a + // concurrent edit cannot restore a removed scorer's id or interleave with the case write. + let mut tx = user_db.clone().begin(&authed).await?; + let current = sqlx::query_scalar!( + "SELECT scorers FROM eval_dataset WHERE workspace_id = $1 AND path = $2 FOR UPDATE", + w_id, + path + ) + .fetch_optional(&mut *tx) + .await?; + let Some(current) = current else { + drop(tx); + return Err(write_refused(&authed, &user_db, &w_id, &path).await); + }; + let existing: std::collections::HashSet = + parse_scorers(current)?.into_iter().map(|s| s.id).collect(); + let scorers = match payload.scorers { + Some(mut scorers) => { + assign_scorer_ids(&mut scorers, &existing)?; + Some(serde_json::to_value(&scorers)?) + } + None => None, + }; + + let updated = sqlx::query_scalar!( + "UPDATE eval_dataset + SET path = COALESCE($6, path), summary = COALESCE($3, summary), + scorers = COALESCE($4, scorers), edited_at = now(), edited_by = $5 + WHERE workspace_id = $1 AND path = $2 + RETURNING path", + w_id, + path, + payload.summary, + scorers, + authed.username, + new_path.as_deref(), + ) + .fetch_optional(&mut *tx) + .await + .map_err(|e| { + if e.as_database_error().and_then(|e| e.code()).as_deref() == Some("23505") { + Error::BadRequest(format!( + "Eval dataset {} already exists", + new_path.as_deref().unwrap_or(&path) + )) + } else { + map_rls_denied(new_path.as_deref().unwrap_or(&path), "rename", e) + } + })?; + // No row updated: the caller cannot write this dataset (its UPDATE policy denied the row) or it + // is gone. A refused rename destination raises 42501 instead, handled just above. + let Some(updated) = updated else { + drop(tx); + return Err(write_refused(&authed, &user_db, &w_id, &path).await); + }; + // Under the name the dataset now has: the cases followed the rename through the foreign key. + if let Some(cases) = &payload.cases { + write_cases(&mut tx, &w_id, &updated, cases, &authed.username).await?; + } + tx.commit().await?; + Ok(format!("Updated eval dataset {}", updated)) +} + +/// The cases, the experiments and their recorded case sets go with the dataset, through the +/// foreign keys. The jobs those experiments produced are not touched: they are jobs, with their +/// own retention. +pub async fn delete_dataset( + authed: ApiAuthed, + Extension(user_db): Extension, + Path((w_id, path)): Path<(String, String)>, +) -> Result { + check_proper_path(&path)?; + if authed.is_operator { + return Err(Error::NotAuthorized( + "Operators cannot delete eval datasets".to_string(), + )); + } + let mut tx = user_db.clone().begin(&authed).await?; + let deleted = sqlx::query_scalar!( + "DELETE FROM eval_dataset WHERE workspace_id = $1 AND path = $2 RETURNING path", + w_id, + path + ) + .fetch_optional(&mut *tx) + .await?; + tx.commit().await?; + if deleted.is_none() { + return Err(write_refused(&authed, &user_db, &w_id, &path).await); + } + Ok(format!("Deleted eval dataset {}", path)) +} + +// ----------------------------------------------------------------------------------------------- +// Cases +// ----------------------------------------------------------------------------------------------- + +async fn read_cases( + tx: &mut sqlx::Transaction<'_, sqlx::Postgres>, + w_id: &str, + dataset: &str, + per_page: usize, + offset: usize, +) -> Result> { + let rows = sqlx::query!( + "SELECT id, input, expected, created_at, created_by + FROM eval_case + WHERE workspace_id = $1 AND dataset_path = $2 + ORDER BY created_at, id + LIMIT $3 OFFSET $4", + w_id, + dataset, + per_page as i64, + offset as i64 + ) + .fetch_all(&mut **tx) + .await?; + rows.into_iter() + .map(|row| { + Ok(EvalCase { + id: row.id, + input: serde_json::from_value(row.input)?, + expected: opt_to_raw(row.expected)?, + created_at: row.created_at, + created_by: row.created_by, + }) + }) + .collect() +} + +pub async fn list_cases( + authed: ApiAuthed, + Extension(user_db): Extension, + Path((w_id, path)): Path<(String, String)>, + Query(pagination): Query, +) -> JsonResult { + check_proper_path(&path)?; + let (per_page, offset) = paginate(pagination); + let mut tx = user_db.begin(&authed).await?; + // The dataset first, so an unknown or unreadable one is a 404 rather than an empty dataset: + // the case rows are invisible in both cases. + let dataset = sqlx::query_scalar!( + "SELECT path FROM eval_dataset WHERE workspace_id = $1 AND path = $2", + w_id, + path + ) + .fetch_optional(&mut *tx) + .await?; + if dataset.is_none() { + return Err(Error::NotFound(format!("Eval dataset {} not found", path))); + } + let cases = read_cases(&mut tx, &w_id, &path, per_page, offset).await?; + tx.commit().await?; + Ok(Json(ListCasesResponse { cases })) +} + +/// What a whole list of cases can be refused for, before any of it is written. +fn check_cases(cases: &[SaveCase]) -> Result<()> { + check_case_set( + cases + .iter() + .map(|case| (&case.input, case.expected.as_ref())), + )?; + // One row per id: the same id twice would write one row twice and return a list longer than + // the dataset it describes, and the save would read as having kept a case it dropped. + let mut ids: Vec = cases.iter().filter_map(|c| c.id).collect(); + ids.sort(); + let submitted = ids.len(); + ids.dedup(); + if ids.len() != submitted { + return Err(Error::BadRequest( + "A case id appears more than once in the dataset".to_string(), + )); + } + Ok(()) +} + +/// Replace a dataset's cases with `cases`, in the caller's transaction: rows not in the list go, +/// rows carrying an id are updated, the rest are added. Returns one id per case, in order. +async fn write_cases( + tx: &mut sqlx::Transaction<'_, sqlx::Postgres>, + w_id: &str, + path: &str, + cases: &[SaveCase], + username: &str, +) -> Result> { + let kept: Vec = cases.iter().filter_map(|c| c.id).collect(); + sqlx::query!( + "DELETE FROM eval_case + WHERE workspace_id = $1 AND dataset_path = $2 AND NOT (id = ANY($3))", + w_id, + path, + &kept + ) + .execute(&mut **tx) + .await?; + + let mut ids = Vec::with_capacity(cases.len()); + for case in cases { + let input = serde_json::to_value(&case.input)?; + let expected = opt_from_raw(case.expected.as_ref())?; + let id = match case.id { + Some(id) => sqlx::query_scalar!( + "UPDATE eval_case SET input = $4, expected = $5 + WHERE workspace_id = $1 AND dataset_path = $2 AND id = $3 + RETURNING id", + w_id, + path, + id, + input, + expected, + ) + .fetch_optional(&mut **tx) + .await? + .ok_or_else(|| Error::NotFound(format!("Eval case {} not found in {}", id, path)))?, + None => sqlx::query_scalar!( + // clock_timestamp() keeps a same-transaction batch in insertion order on reload. + "INSERT INTO eval_case + (workspace_id, dataset_path, input, expected, created_by, created_at) + VALUES ($1, $2, $3, $4, $5, clock_timestamp()) + RETURNING id", + w_id, + path, + input, + expected, + username, + ) + .fetch_one(&mut **tx) + .await + .map_err(|e| { + if is_missing_dataset(&e) { + Error::NotFound(format!("Eval dataset {} not found", path)) + } else { + e.into() + } + })?, + }; + ids.push(id); + } + Ok(ids) +} diff --git a/backend/windmill-api/src/ai_evals/mod.rs b/backend/windmill-api/src/ai_evals/mod.rs new file mode 100644 index 0000000000..b4d6009857 --- /dev/null +++ b/backend/windmill-api/src/ai_evals/mod.rs @@ -0,0 +1,371 @@ +//! Eval datasets for reusable AI agents. +//! +//! Five tables: `eval_dataset` and the `eval_case` rows it holds are the curated inputs; +//! `eval_experiment`, `eval_experiment_case` and `eval_score` are one run of them, written once +//! and only ever read afterwards. +//! +//! Datasets and cases go through `user_db`, so row-level security is the only access authority: +//! `eval_case`'s policies derive from its dataset's (`eval_dataset_writable`, in the migration). +//! The experiment tables carry read policies only and are written on the unrestricted pool after +//! the API has checked access — see `run_experiment` and `collect_experiment`. + +use axum::{ + extract::{Path, Query}, + routing::{get, post}, + Extension, Json, Router, +}; +use chrono::{DateTime, Utc}; +use serde::{Deserialize, Serialize}; +use serde_json::value::RawValue; +use uuid::Uuid; +use windmill_common::{ + db::UserDB, + error::{Error, JsonResult, Result}, + utils::{check_proper_path, paginate, Pagination}, +}; + +use crate::db::{ApiAuthed, DB}; +use windmill_api_auth::check_scopes; + +pub(crate) mod datasets; +pub(crate) mod payload; +pub(crate) mod results; +pub(crate) mod run; +pub(crate) mod scorers; +pub(crate) mod scoring; +pub(crate) mod subject; +pub(crate) mod template; + +pub(crate) use datasets::*; +pub(crate) use payload::*; +pub(crate) use results::*; +pub(crate) use run::*; +pub(crate) use scorers::*; +pub(crate) use scoring::*; +pub(crate) use subject::*; +pub(crate) use template::*; + +pub fn workspaced_service() -> Router { + Router::new() + .route("/datasets/list", get(list_datasets)) + .route("/datasets/create", post(create_dataset)) + .route("/datasets/get/{*path}", get(get_dataset)) + .route("/datasets/update/{*path}", post(update_dataset)) + .route("/datasets/delete/{*path}", post(delete_dataset)) + .route("/cases/list/{*path}", get(list_cases)) + .route("/scorer_defaults", get(scorer_defaults)) + .route("/run_payload", get(run_payload)) + .route("/experiments/run", post(run_experiment)) + .route("/experiments/collect", post(collect_experiment)) + .route("/scorers/recent", get(recent_scorers)) + .route("/subject_state", get(subject_state)) + .route("/experiments/list_all", get(list_all_experiments)) + .route("/experiments/results/{*path}", get(experiment_results)) +} + +/// Checked here rather than left to the column, whose own refusal comes back as an internal +/// database error naming no field. +const MAX_DATASET_SUMMARY_CHARS: usize = 1000; + +fn check_summary(summary: Option<&str>) -> Result<()> { + match summary { + Some(summary) if summary.chars().count() > MAX_DATASET_SUMMARY_CHARS => { + Err(Error::BadRequest(format!( + "This dataset's summary is {} characters, over the {} the column holds.", + summary.chars().count(), + MAX_DATASET_SUMMARY_CHARS + ))) + } + _ => Ok(()), + } +} + +/// A case is text — attachments are S3 references rather than inline bytes. +const MAX_CASE_BYTES: usize = 256 * 1024; +/// The whole case set together, so cases at the per-case cap cannot add up to a dataset a listing +/// or a run must hold hundreds of megabytes of at once. +const MAX_DATASET_BYTES: usize = 16 * 1024 * 1024; +/// Also what a listing returns in one page, so a dataset is always read whole: the editor holds +/// every case at once and writes them together, and half a set on screen is a Save that drops the +/// rest. +const MAX_CASES_PER_DATASET: i64 = 1_000; + +const MAX_EXPERIMENTS_LISTED: i64 = 100; + +const MAX_RECENT_SCORERS: usize = 12; + +/// A run's work is cases × scorers, so this bounds how far one request fans out. +const MAX_SCORERS_PER_DATASET: usize = 20; + +/// The dataset a write was aimed at is gone. Raised from the foreign key rather than from a +/// preceding existence check, so a dataset deleted mid-request cannot slip between the two. +fn is_missing_dataset(e: &sqlx::Error) -> bool { + e.as_database_error().and_then(|d| d.code()).as_deref() == Some("23503") +} + +/// A `user_db` write the row-level policies refused surfaces as SQLSTATE 42501, whose message +/// names the table and the policy. Turn it into one about access. +fn map_rls_denied(path: &str, action: &str, e: sqlx::Error) -> Error { + if e.as_database_error().and_then(|d| d.code()).as_deref() == Some("42501") { + return Error::NotAuthorized(format!("Not allowed to {} eval dataset {}", action, path)); + } + e.into() +} + +/// A write that matched no row is either a dataset that does not exist or one the caller can read +/// but not write. Row-level security cannot distinguish them — both are simply invisible to the +/// statement — so ask again with a plain read. +async fn write_refused(authed: &ApiAuthed, user_db: &UserDB, w_id: &str, path: &str) -> Error { + let visible = async { + let mut tx = user_db.clone().begin(authed).await?; + let found = sqlx::query_scalar!( + "SELECT path FROM eval_dataset WHERE workspace_id = $1 AND path = $2", + w_id, + path + ) + .fetch_optional(&mut *tx) + .await?; + tx.commit().await?; + Ok::<_, Error>(found.is_some()) + } + .await; + match visible { + Ok(true) => Error::NotAuthorized(format!( + "User {} does not have write access to eval dataset {}", + authed.username, path + )), + Ok(false) => Error::NotFound(format!("Eval dataset {} not found", path)), + Err(e) => e, + } +} + +/// One `eval_dataset` row, from the columns every read of the table selects. +fn dataset_from_row( + path: String, + summary: Option, + scorers: serde_json::Value, + created_at: DateTime, + created_by: String, + edited_at: DateTime, + edited_by: String, +) -> Result { + Ok(EvalDataset { + path, + summary, + scorers: parse_scorers(scorers)?, + created_at, + created_by, + edited_at, + edited_by, + }) +} + +/// A dataset's columns. Only this module writes them, through serde, so a value that does not +/// parse is corruption rather than input: defaulting to no columns would let the next save mint +/// fresh scorer ids and orphan every score already recorded. +pub(crate) fn parse_scorers(scorers: serde_json::Value) -> Result> { + serde_json::from_value(scorers) + .map_err(|e| Error::internal_err(format!("eval dataset scorers are not readable: {e}"))) +} + +/// Read the dataset the request names, through `user_db` so that a caller who cannot see it gets +/// the same answer as one asking for a dataset that does not exist. +async fn read_dataset( + authed: &ApiAuthed, + user_db: &UserDB, + w_id: &str, + path: &str, +) -> Result { + check_proper_path(path)?; + let mut tx = user_db.clone().begin(authed).await?; + let row = sqlx::query!( + "SELECT path, summary, scorers, created_at, created_by, + edited_at, edited_by + FROM eval_dataset WHERE workspace_id = $1 AND path = $2", + w_id, + path + ) + .fetch_optional(&mut *tx) + .await?; + tx.commit().await?; + let row = row.ok_or_else(|| Error::NotFound(format!("Eval dataset {} not found", path)))?; + dataset_from_row( + row.path, + row.summary, + row.scorers, + row.created_at, + row.created_by, + row.edited_at, + row.edited_by, + ) +} + +/// The dataset and its cases as one snapshot, so a launch cannot record the cases from before an +/// edit beside the scorers from after it. One transaction is not enough: `user_db` runs at READ +/// COMMITTED, where each statement takes a fresh snapshot, so the row is taken `FOR UPDATE` — +/// which an edit's own `FOR UPDATE` and a case write's foreign-key lock both conflict with. +pub(crate) async fn read_dataset_and_cases( + authed: &ApiAuthed, + user_db: &UserDB, + w_id: &str, + path: &str, +) -> Result<(EvalDataset, Vec)> { + check_proper_path(path)?; + let mut tx = user_db.clone().begin(authed).await?; + let row = sqlx::query!( + "SELECT path, summary, scorers, created_at, created_by, edited_at, edited_by + FROM eval_dataset WHERE workspace_id = $1 AND path = $2 FOR UPDATE", + w_id, + path + ) + .fetch_optional(&mut *tx) + .await?; + let Some(row) = row else { + tx.commit().await?; + return Err(Error::NotFound(format!("Eval dataset {} not found", path))); + }; + let case_rows = sqlx::query!( + "SELECT id, input, expected, created_at, created_by + FROM eval_case + WHERE workspace_id = $1 AND dataset_path = $2 + ORDER BY created_at, id", + w_id, + path + ) + .fetch_all(&mut *tx) + .await?; + tx.commit().await?; + let dataset = dataset_from_row( + row.path, + row.summary, + row.scorers, + row.created_at, + row.created_by, + row.edited_at, + row.edited_by, + )?; + let cases = case_rows + .into_iter() + .map(|row| { + Ok(EvalCase { + id: row.id, + input: serde_json::from_value(row.input)?, + expected: opt_to_raw(row.expected)?, + created_at: row.created_at, + created_by: row.created_by, + }) + }) + .collect::>>()?; + Ok((dataset, cases)) +} + +/// Whether this caller may write a dataset's contents: its cases, and the experiments that run +/// them. +/// +/// `SELECT … FOR UPDATE` applies `eval_dataset`'s UPDATE policies on top of its SELECT policies, +/// so the row itself answers who may write it, and a grant in `extra_perms` is honoured without +/// being mirrored here. +async fn require_dataset_writable( + authed: &ApiAuthed, + user_db: &UserDB, + w_id: &str, + path: &str, +) -> Result<()> { + check_proper_path(path)?; + if authed.is_operator { + return Err(Error::NotAuthorized( + "Operators cannot modify eval datasets".to_string(), + )); + } + let mut tx = user_db.clone().begin(authed).await?; + let writable = sqlx::query_scalar!( + "SELECT path FROM eval_dataset WHERE workspace_id = $1 AND path = $2 FOR UPDATE", + w_id, + path + ) + .fetch_optional(&mut *tx) + .await?; + tx.commit().await?; + if writable.is_some() { + Ok(()) + } else { + Err(write_refused(authed, user_db, w_id, path).await) + } +} + +/// jsonb columns are read as `serde_json::Value` and handed on as `RawValue`: a case's `expected` +/// is arbitrary user JSON that this module never looks inside. +fn opt_to_raw(value: Option) -> Result>> { + value + .map(|v| Ok(serde_json::value::to_raw_value(&v)?)) + .transpose() +} + +fn opt_from_raw(value: Option<&Box>) -> Result> { + value + .map(|v| Ok(serde_json::from_str(v.get())?)) + .transpose() +} + +fn check_case(input: &EvalCaseInput, expected: Option<&Box>) -> Result<()> { + // The shape the agent step reads its attachments in, checked when the case is written rather + // than when a run deserialises the step's arguments, which is after the case was queued. + if let Some(attachments) = &input.user_attachments { + if serde_json::from_str::>(attachments.get()).is_err() { + return Err(Error::BadRequest( + "A case's user_attachments is a list of S3 objects, each with an `s3` key naming \ + the file" + .to_string(), + )); + } + } + check_case_size(input, expected) +} + +/// The bytes one case weighs against its own and the dataset's caps. +fn case_bytes(input: &EvalCaseInput, expected: Option<&Box>) -> Result { + let mut bytes = serde_json::to_vec(input)?.len(); + if let Some(expected) = expected { + bytes += expected.get().len(); + } + Ok(bytes) +} + +/// What a whole case set can be refused for, before any of it is written. +fn check_case_set<'a>( + cases: impl ExactSizeIterator>)>, +) -> Result<()> { + if cases.len() as i64 > MAX_CASES_PER_DATASET { + return Err(Error::BadRequest(format!( + "An eval dataset holds at most {} cases. Split them into several datasets.", + MAX_CASES_PER_DATASET + ))); + } + let mut total = 0usize; + for (input, expected) in cases { + check_case(input, expected)?; + total += case_bytes(input, expected)?; + } + if total > MAX_DATASET_BYTES { + return Err(Error::BadRequest(format!( + "This dataset is {} KiB of cases, over the {} KiB limit. Attachments belong in \ + workspace storage and are referenced by a case, not stored inside it.", + total / 1024, + MAX_DATASET_BYTES / 1024 + ))); + } + Ok(()) +} + +fn check_case_size(input: &EvalCaseInput, expected: Option<&Box>) -> Result<()> { + let bytes = case_bytes(input, expected)?; + if bytes > MAX_CASE_BYTES { + return Err(Error::BadRequest(format!( + "This eval case is {} KiB, over the {} KiB limit. Attachments belong in workspace \ + storage and are referenced by a case, not stored inside it.", + bytes / 1024, + MAX_CASE_BYTES / 1024 + ))); + } + Ok(()) +} diff --git a/backend/windmill-api/src/ai_evals/payload.rs b/backend/windmill-api/src/ai_evals/payload.rs new file mode 100644 index 0000000000..e89948e9b5 --- /dev/null +++ b/backend/windmill-api/src/ai_evals/payload.rs @@ -0,0 +1,410 @@ +use super::*; + +/// What every scorer is handed: the answer, and the calls the agent made to reach it. +/// +/// Built from the job the run already stored, which is what lets a scorer added later score an +/// experiment that has already run. +#[derive(Serialize, Debug, Clone)] +pub struct EvalRunPayload { + pub input: EvalCaseInput, + #[serde(skip_serializing_if = "Option::is_none")] + pub output: Option>, + #[serde(skip_serializing_if = "Option::is_none")] + pub expected: Option>, + pub tool_calls: Vec, + /// The tools that were actually called, with the schema they were called against. A tool + /// whose schema could not be resolved carries `null`, and a scorer validating arguments must + /// treat that as unchecked rather than as a failure. + pub tools: Vec, + pub metrics: EvalMetrics, + pub status: String, + pub job_id: Uuid, +} + +#[derive(Serialize, Debug, Clone)] +pub struct EvalToolCall { + pub name: String, + #[serde(skip_serializing_if = "Option::is_none")] + pub args: Option>, + #[serde(skip_serializing_if = "Option::is_none")] + pub result: Option>, + #[serde(skip_serializing_if = "Option::is_none")] + pub error: Option, + #[serde(skip_serializing_if = "Option::is_none")] + pub duration_ms: Option, + /// Set when the result was too large to carry and was cut down. + #[serde(skip_serializing_if = "std::ops::Not::not")] + pub truncated: bool, +} + +#[derive(Serialize, Debug, Clone)] +pub struct EvalToolDef { + pub name: String, + #[serde(skip_serializing_if = "Option::is_none")] + pub schema: Option>, +} + +#[derive(Serialize, Debug, Clone)] +pub struct EvalMetrics { + pub steps: usize, + #[serde(skip_serializing_if = "Option::is_none")] + pub duration_ms: Option, + /// The provider's token counts, when it reported any. + #[serde(skip_serializing_if = "Option::is_none")] + pub usage: Option>, +} + +/// A tool result large enough to swamp a judge's context is cut here. The scorer is told, so a +/// check reading a truncated result can say so instead of failing on the missing tail. +const MAX_TOOL_RESULT_BYTES: usize = 4 * 1024; + +fn truncate_value(value: Box) -> (Box, bool) { + if value.get().len() <= MAX_TOOL_RESULT_BYTES { + return (value, false); + } + let text = value.get(); + let mut end = MAX_TOOL_RESULT_BYTES; + while !text.is_char_boundary(end) { + end -= 1; + } + match serde_json::value::to_raw_value(&format!("{}… [truncated]", &text[..end])) { + Ok(v) => (v, true), + Err(_) => (value, false), + } +} + +/// Assemble the payload from a completed case job: the agent step's own result carries the answer +/// and the message list, and every message that made a tool call names the job that ran it. +async fn build_run_payload( + db: &DB, + w_id: &str, + job_id: Uuid, + agent_job: Uuid, + input: EvalCaseInput, + expected: Option>, + status: String, + duration_ms: Option, +) -> Result { + // A read that failed is not a run with no answer: handing the scorers an empty payload would + // have them grade the absence of evidence and record that verdict permanently. + let agent_result = agent_result(db, w_id, job_id).await?.map(|(r, _)| r); + + let parsed: Option = agent_result + .as_ref() + .and_then(|r| serde_json::from_str(r.get()).ok()); + let output = parsed + .as_ref() + .and_then(|p| p.get("output")) + .map(|o| serde_json::value::to_raw_value(o)) + .transpose()?; + let usage = parsed + .as_ref() + .and_then(|p| p.get("usage")) + .map(|u| serde_json::value::to_raw_value(u)) + .transpose()?; + + // Walk the messages in order: a tool call is an `agent_action` on the message that made it. + let mut calls: Vec<(String, Option, Option>)> = vec![]; + if let Some(messages) = parsed + .as_ref() + .and_then(|p| p.get("messages")) + .and_then(|m| m.as_array()) + { + for message in messages { + let Some(action) = message.get("agent_action") else { + continue; + }; + match action.get("type").and_then(|t| t.as_str()) { + Some("tool_call") => calls.push(( + action + .get("function_name") + .and_then(|f| f.as_str()) + .unwrap_or("tool") + .to_string(), + action + .get("job_id") + .and_then(|j| j.as_str()) + .and_then(|j| Uuid::parse_str(j).ok()), + None, + )), + // An MCP call runs inside the agent rather than as a job, so its arguments are on + // the action itself. Its result lives in a later `role: "tool"` message rather + // than a child-job row, and is not surfaced to scorers yet. + Some("mcp_tool_call") => calls.push(( + action + .get("function_name") + .and_then(|f| f.as_str()) + .unwrap_or("tool") + .to_string(), + None, + action + .get("arguments") + .map(|a| serde_json::value::to_raw_value(a)) + .transpose()?, + )), + _ => {} + } + } + } + + let call_job_ids: Vec = calls.iter().filter_map(|(_, id, _)| *id).collect(); + let mut jobs = std::collections::HashMap::new(); + if !call_job_ids.is_empty() { + // Constrained to the agent step's own children rather than to the workspace: these ids + // come out of a job result, so a caller who can run a flow can put any id there. A tool + // call is pushed as a child of the agent that made it, which is what makes that the + // boundary. + let rows = sqlx::query!( + "SELECT j.id, j.args AS \"args: sqlx::types::Json>\", + c.result AS \"result: sqlx::types::Json>\", + c.status::text AS status, c.duration_ms, + s.schema AS \"schema: sqlx::types::Json>\" + FROM v2_job j + LEFT JOIN v2_job_completed c ON c.id = j.id + LEFT JOIN script s ON s.workspace_id = j.workspace_id AND s.hash = j.runnable_id + WHERE j.id = ANY($1) AND j.workspace_id = $2 AND j.parent_job = $3", + &call_job_ids, + w_id, + agent_job + ) + .fetch_all(db) + .await?; + for row in rows { + jobs.insert(row.id, row); + } + } + + let mut tool_calls = Vec::with_capacity(calls.len()); + let mut tools: Vec = vec![]; + for (name, call_job_id, inline_args) in calls { + let row = call_job_id.and_then(|id| jobs.get(&id)); + let (result, truncated) = match row.and_then(|r| r.result.as_ref()) { + Some(result) => { + let (value, truncated) = truncate_value(result.0.clone()); + (Some(value), truncated) + } + None => (None, false), + }; + let failed = row + .and_then(|r| r.status.as_deref()) + .map(|s| s != "success") + .unwrap_or(false); + if !tools.iter().any(|t| t.name == name) { + tools.push(EvalToolDef { + name: name.clone(), + schema: row.and_then(|r| r.schema.as_ref()).map(|s| s.0.clone()), + }); + } + // The already truncated result restated. `render_tool_calls` shows `error` and not + // `result` for a failed call, so the judge's context carries the payload once and bounded; + // `result` stays on the raw call for a script scorer. + let error = failed + .then(|| result.as_ref().map(|r| r.get().to_string())) + .flatten(); + tool_calls.push(EvalToolCall { + name, + args: inline_args.or_else(|| row.and_then(|r| r.args.as_ref()).map(|a| a.0.clone())), + result, + error, + duration_ms: row.map(|r| r.duration_ms), + truncated, + }); + } + + Ok(EvalRunPayload { + metrics: EvalMetrics { steps: tool_calls.len(), duration_ms, usage }, + input, + output, + expected, + tool_calls, + tools, + status, + job_id, + }) +} + +#[derive(Deserialize)] +pub struct RunPayloadQuery { + /// The flow job that answered the case: an iteration of a run. + pub job_id: Uuid, +} + +/// What the scorers of one iteration are handed. +#[derive(Serialize)] +pub struct RunPayloadResponse { + pub run: EvalRunPayload, + /// The same run as a judge reads it. Rendered once per case rather than once per judge. + pub rendered: String, +} + +/// Assemble the payload for one answered case, for the step that feeds the scorers. +/// +/// The case is read from the job's arguments rather than from the experiment, so this works for an +/// iteration whose row has not been filled in yet. +pub async fn run_payload( + authed: ApiAuthed, + Extension(db): Extension, + Extension(user_db): Extension, + Path(w_id): Path, + Query(query): Query, +) -> JsonResult { + // `UserDB` enforces row permissions but not a token's scopes, so without this an + // `ai_evals:read` token would read job arguments, results and tool calls that `jobs:read` + // is what actually gates. Job tokens are unscoped, so the run flow's payload step passes. + check_scopes(&authed, || "jobs:read".to_string())?; + // Through `user_db`: the caller is a job token, and it reads what its runner can read. + let mut tx = user_db.begin(&authed).await?; + let args = sqlx::query_scalar!( + "SELECT args AS \"args: sqlx::types::Json>\" FROM v2_job + WHERE id = $1 AND workspace_id = $2", + query.job_id, + w_id + ) + .fetch_optional(&mut *tx) + .await? + .flatten() + .ok_or_else(|| Error::NotFound(format!("Job {} not found", query.job_id)))?; + tx.commit().await?; + + let args: serde_json::Value = serde_json::from_str(args.0.get())?; + // An iteration carries its case; a run recorded one job per case carries the same input under + // the stamp that job was pushed with. + let case = args.get("iter").and_then(|i| i.get("value")); + let input = case + .and_then(|c| c.get("input")) + .or_else(|| args.get("_eval_input")) + .cloned() + .unwrap_or_else(|| serde_json::json!({})); + let expected = case + .and_then(|c| c.get("expected")) + .or_else(|| args.get("expected")) + .cloned(); + + // The agent step's own status and duration, never the iteration's: the iteration goes on to + // assemble this payload and run the scorers, so a scorer reading the iteration's duration + // would be measuring itself. + let agent_job = agent_step_job(&db, &w_id, query.job_id) + .await? + .unwrap_or(query.job_id); + let completed = sqlx::query!( + "SELECT status::text AS \"status!\", duration_ms FROM v2_job_completed + WHERE id = $1 AND workspace_id = $2", + agent_job, + w_id + ) + .fetch_optional(&db) + .await?; + + let run = build_run_payload( + &db, + &w_id, + query.job_id, + agent_job, + serde_json::from_value(input)?, + expected + .map(|e| serde_json::value::to_raw_value(&e)) + .transpose()?, + completed + .as_ref() + .map(|c| c.status.clone()) + // The iteration asking is itself still running: its agent step is what finished. + .unwrap_or_else(|| "success".to_string()), + completed.as_ref().map(|c| c.duration_ms), + ) + .await?; + let rendered = render_run(&run); + Ok(Json(RunPayloadResponse { run, rendered })) +} + +/// The job of the agent step inside a run's flow, from the flow status of either a running or a +/// finished one. +async fn agent_step_job(db: &DB, w_id: &str, flow_job: Uuid) -> Result> { + let modules = sqlx::query_scalar!( + "SELECT COALESCE(s.flow_status, c.flow_status) -> 'modules' AS modules + FROM v2_job j + LEFT JOIN v2_job_status s ON s.id = j.id + LEFT JOIN v2_job_completed c ON c.id = j.id + WHERE j.id = $1 AND j.workspace_id = $2", + flow_job, + w_id + ) + .fetch_optional(db) + .await? + .flatten(); + Ok(modules + .as_ref() + .and_then(|m| m.as_array()) + .and_then(|modules| { + modules + .iter() + .find(|m| m.get("id").and_then(|i| i.as_str()) == Some(AGENT_NODE_ID)) + }) + .and_then(|m| m.get("job")) + .and_then(|j| j.as_str()) + .and_then(|j| Uuid::parse_str(j).ok())) +} + +/// The system prompt a judge agent is created with. It is the agent's own, so editing a judge is +/// editing that resource — there is no second copy of the grading contract on the dataset. +pub const JUDGE_SYSTEM_PROMPT: &str = r#"You are grading one run of an AI agent. + +Score how well the agent handled the request, from 0 to 1. Judge the whole trajectory, not only the +final answer. Penalise asking for information already in the request, calling a tool twice with the +same arguments, and tool errors left unrecovered. + +Reply with JSON only, of the form {"score": , "reason": }."#; + +fn render_json(value: Option<&RawValue>) -> String { + value + .map(|v| v.get().to_string()) + .unwrap_or_else(|| "(none)".to_string()) +} + +/// Tool calls as the judge reads them: numbered, in order, with arguments, result and duration. +fn render_tool_calls(calls: &[EvalToolCall]) -> String { + if calls.is_empty() { + return "(none)".to_string(); + } + calls + .iter() + .enumerate() + .map(|(index, call)| { + let args = call.args.as_ref().map(|a| a.get()).unwrap_or("{}"); + let outcome = match (&call.error, &call.result) { + (Some(error), _) => format!("error: {}", error), + (None, Some(result)) => result.get().to_string(), + (None, None) => "(no result)".to_string(), + }; + let timing = call + .duration_ms + .map(|ms| format!(" ({}ms)", ms)) + .unwrap_or_default(); + format!( + "{}. {}({}) -> {}{}", + index + 1, + call.name, + args, + outcome, + timing + ) + }) + .collect::>() + .join("\n") +} + +/// One run, as a judge is shown it. +fn render_run(run: &EvalRunPayload) -> String { + format!( + "Request: {}\nTool calls, in order:\n{}\nAnswer: {}\nExpected: {}", + run.input.user_message.as_deref().unwrap_or("(none)"), + render_tool_calls(&run.tool_calls), + render_json(run.output.as_deref()), + render_json(run.expected.as_deref()), + ) +} + +/// Module id of a scorer inside a scoring job. `assign_scorer_ids` keeps ids to +/// `[A-Za-z0-9_]`, so this is a valid identifier. +pub(crate) fn scorer_module_id(scorer_id: &str) -> String { + format!("s_{}", scorer_id) +} diff --git a/backend/windmill-api/src/ai_evals/results.rs b/backend/windmill-api/src/ai_evals/results.rs new file mode 100644 index 0000000000..500af88d72 --- /dev/null +++ b/backend/windmill-api/src/ai_evals/results.rs @@ -0,0 +1,856 @@ +use super::*; + +/// One run of a dataset: written once when the dataset is run, and only ever read afterwards. +#[derive(Serialize, Deserialize, Debug, Clone)] +pub struct EvalExperiment { + pub id: Uuid, + pub dataset: String, + pub subject: EvalSubject, + /// This subject's nth run of this dataset, allocated once and never reused: "Run 7" survives + /// history being pruned, which a position computed when the list is read would not. + pub run_number: i32, + /// The flow executing the run: one job holding every case and its scores. + pub run_job_id: Uuid, + pub case_count: i64, + /// What the run scored, one entry per scorer that produced a number. Carried on the run so a + /// list can say what each one scored without reading every cell of every one of them. + #[serde(default, skip_serializing_if = "Vec::is_empty")] + pub scores: Vec, + /// Whether the flow executing this run is still going. What makes a list of runs worth + /// watching rather than worth reloading. + #[serde(default, skip_serializing_if = "std::ops::Not::not")] + pub running: bool, + pub created_at: DateTime, + pub created_by: String, +} + +/// One scorer's headline for one run: the two numbers a column reports, over that run's cells. +#[derive(Serialize, Deserialize, Debug, Clone)] +pub struct ExperimentScore { + pub scorer_id: String, + /// What the column is called in the dataset that ran it, resolved here because a list of runs + /// spanning datasets cannot hold every dataset's scorers to look it up. + pub name: String, + /// `agent` or `script`, for the badge to say which kind of thing produced the number. + pub kind: String, + #[serde(skip_serializing_if = "Option::is_none")] + pub mean: Option, + /// The share of scored cells at or above the column's threshold, for a column that has one. + /// Absent where the column has no threshold and the mean is the whole headline. + #[serde(skip_serializing_if = "Option::is_none")] + pub pass_rate: Option, + pub scored: i64, + /// How many of this run's cells the column failed on. A column that failed on all of them + /// still ran, which is the difference between a headline of nothing and no headline at all. + pub failed: i64, +} + +#[derive(Deserialize)] +pub struct ListExperimentsQuery { + /// Restrict to one agent's runs. Both what was deployed and what was drafted are that agent's + /// history, so this does not discriminate by kind. + #[serde(default)] + pub subject_path: Option, +} + +/// Every run of this agent, across every dataset it has been measured on. +/// +/// Filtered by `user_db`: a run is visible exactly when the dataset it belongs to is. +pub async fn list_all_experiments( + authed: ApiAuthed, + Extension(db): Extension, + Extension(user_db): Extension, + Path(w_id): Path, + Query(query): Query, +) -> JsonResult> { + let mut tx = user_db.clone().begin(&authed).await?; + let rows = sqlx::query!( + "SELECT e.id, e.dataset_path, e.subject, e.run_number, e.run_job_id, e.created_at, + e.created_by, + (SELECT count(*) FROM eval_experiment_case c WHERE c.experiment_id = e.id) + AS \"case_count!\" + FROM eval_experiment e + JOIN eval_dataset d ON d.workspace_id = e.workspace_id AND d.path = e.dataset_path + WHERE e.workspace_id = $1 + AND ($3::text IS NULL OR e.subject ->> 'path' = $3) + ORDER BY e.created_at DESC + LIMIT $2", + w_id, + MAX_EXPERIMENTS_LISTED, + query.subject_path, + ) + .fetch_all(&mut *tx) + .await?; + tx.commit().await?; + let mut experiments = rows + .into_iter() + .map(|row| { + experiment_from_row( + row.id, + row.dataset_path, + row.subject, + row.run_number, + row.run_job_id, + row.case_count, + row.created_at, + row.created_by, + ) + }) + .collect::>>()?; + + resolve_listed_drafts(&authed, &db, &user_db, &w_id, &mut experiments).await?; + let scorers_by_dataset = scorers_of_listed(&authed, &user_db, &w_id, &experiments).await?; + mark_running(&db, &w_id, &mut experiments).await?; + sync_listed_runs(&db, &w_id, &experiments).await?; + let mut scores = experiment_scores(&db, &experiments, &scorers_by_dataset).await?; + for experiment in experiments.iter_mut() { + experiment.scores = scores.remove(&experiment.id).unwrap_or_default(); + } + Ok(Json(experiments)) +} + +/// Which listed runs are still going, read from the flows executing them. A run whose flow is no +/// longer there at all is over: jobs have their own retention, and reading a missing one as +/// unfinished would leave every run older than it spinning. +async fn mark_running(db: &DB, w_id: &str, experiments: &mut [EvalExperiment]) -> Result<()> { + let job_ids: Vec = experiments.iter().map(|e| e.run_job_id).collect(); + if job_ids.is_empty() { + return Ok(()); + } + let unfinished: std::collections::HashSet = sqlx::query_scalar!( + "SELECT j.id AS \"id!\" FROM v2_job j + LEFT JOIN v2_job_completed c ON c.id = j.id AND c.workspace_id = $2 + WHERE j.id = ANY($1) AND j.workspace_id = $2 AND c.id IS NULL", + &job_ids, + w_id + ) + .fetch_all(db) + .await? + .into_iter() + .collect(); + for experiment in experiments.iter_mut() { + experiment.running = unfinished.contains(&experiment.run_job_id); + } + Ok(()) +} + +/// A run of a draft whose edits have since been deployed is a run of that version. Resolved once +/// per subject rather than once per run, because a listing is usually one agent's history. +async fn resolve_listed_drafts( + authed: &ApiAuthed, + db: &DB, + user_db: &UserDB, + w_id: &str, + experiments: &mut [EvalExperiment], +) -> Result<()> { + let drafted: std::collections::HashSet = experiments + .iter() + .filter(|e| e.subject.kind == EvalSubjectKind::AgentDraft) + .map(|e| e.subject.path.clone()) + .collect(); + if drafted.is_empty() { + return Ok(()); + } + // Read each subject as the caller (see experiment_results): an agent the caller cannot read + // yields no hash or version, so its config fingerprint never leaks through the list either. + let mut deployed = std::collections::HashMap::new(); + for path in drafted { + let (hash, version) = match readable_agent_state(authed, user_db, w_id, &path).await? { + Some((config, version)) => (Some(draft_hash(&config)), Some(version)), + None => (None, None), + }; + deployed.insert(path.clone(), (hash, version)); + } + for experiment in experiments.iter_mut() { + let Some((hash, version)) = deployed.get(&experiment.subject.path) else { + continue; + }; + // Each run's own dataset: the list may span them, and the update is keyed on both. + let dataset = experiment.dataset.clone(); + resolve_deployed_draft(db, w_id, &dataset, experiment, hash.as_deref(), *version).await?; + } + Ok(()) +} + +/// How many listed runs one list call reads out of their flows. A run's scores live in its flow +/// until something reads them into `eval_score`, so an unopened run has nothing to report; the cap +/// keeps a long history from turning one list call into a hundred flow reads. +const MAX_RUNS_SYNCED_PER_LIST: usize = 10; + +/// Read the flows of listed runs that still have scores to collect. Runs already collected are +/// skipped, so the steady-state cost of listing is one query rather than one read per run. +async fn sync_listed_runs(db: &DB, w_id: &str, experiments: &[EvalExperiment]) -> Result<()> { + if experiments.is_empty() { + return Ok(()); + } + let ids: Vec = experiments.iter().map(|e| e.id).collect(); + let unread = sqlx::query_scalar!( + "SELECT DISTINCT experiment_id FROM eval_score + WHERE experiment_id = ANY($1) AND score IS NULL AND error IS NULL + AND NOT not_applicable", + &ids + ) + .fetch_all(db) + .await? + .into_iter() + .collect::>(); + for experiment in experiments + .iter() + .filter(|e| unread.contains(&e.id)) + .take(MAX_RUNS_SYNCED_PER_LIST) + { + // Best-effort, for the same reason reading one run is: this is the home screen, and one + // run with an unreadable cell must not cost the list of every other run. + if let Err(e) = sync_run(db, w_id, experiment.id, experiment.run_job_id, false).await { + tracing::warn!("could not collect eval run {}: {e:#}", experiment.id); + } + } + Ok(()) +} + +/// Every listed run's per-scorer headline, in one grouped query. +/// +/// Thresholds come from each run's own dataset as its scorers are *now*, joined per (run, scorer) +/// rather than per scorer: a list spanning datasets is a list of runs whose columns are not the +/// same columns. +async fn experiment_scores( + db: &DB, + experiments: &[EvalExperiment], + scorers_by_dataset: &std::collections::HashMap>, +) -> Result>> { + let mut by_experiment: std::collections::HashMap> = + Default::default(); + // One entry per (run, column) it could have scored, which is what carries the threshold and + // the column's order into the query. + let mut ids: Vec = vec![]; + let mut scorer_ids: Vec = vec![]; + let mut thresholds: Vec> = vec![]; + for experiment in experiments { + for scorer in scorers_by_dataset + .get(&experiment.dataset) + .map(|s| s.as_slice()) + .unwrap_or(&[]) + { + ids.push(experiment.id); + scorer_ids.push(scorer.id.clone()); + thresholds.push(scorer.pass_if); + } + } + if ids.is_empty() { + return Ok(by_experiment); + } + let rows = sqlx::query!( + "SELECT s.experiment_id AS \"experiment_id!\", s.scorer_id AS \"scorer_id!\", + avg(s.score) AS mean, + count(s.score) AS \"scored!\", + count(*) FILTER (WHERE s.error IS NOT NULL) AS \"failed!\", + count(*) FILTER (WHERE t.pass_if IS NOT NULL AND s.score >= t.pass_if) + AS \"passed!\", + bool_or(t.pass_if IS NOT NULL) AS \"has_threshold!\" + FROM eval_score s + JOIN unnest($1::uuid[], $2::text[], $3::float8[]) + AS t(experiment_id, scorer_id, pass_if) + ON t.experiment_id = s.experiment_id AND t.scorer_id = s.scorer_id + GROUP BY s.experiment_id, s.scorer_id", + &ids, + &scorer_ids, + &thresholds as &[Option], + ) + .fetch_all(db) + .await?; + let mut headline: std::collections::HashMap< + (Uuid, String), + (Option, i64, i64, i64, bool), + > = Default::default(); + for row in rows { + headline.insert( + (row.experiment_id, row.scorer_id), + ( + row.mean, + row.scored, + row.failed, + row.passed, + row.has_threshold, + ), + ); + } + // Emitted in the dataset's column order rather than the query's, so the badges on a row read + // left to right the way that dataset's table does. + for experiment in experiments { + for scorer in scorers_by_dataset + .get(&experiment.dataset) + .map(|s| s.as_slice()) + .unwrap_or(&[]) + { + // A column with no cells at all on this run is one added after it. A column that has + // cells is reported even where none produced a number, which is what a column that + // failed throughout looks like. + let Some((mean, scored, failed, passed, has_threshold)) = + headline.get(&(experiment.id, scorer.id.clone())) + else { + continue; + }; + by_experiment + .entry(experiment.id) + .or_default() + .push(ExperimentScore { + scorer_id: scorer.id.clone(), + name: scorer_name(scorer), + kind: scorer.def.kind_str().to_string(), + mean: *mean, + pass_rate: (*has_threshold && *scored > 0) + .then(|| *passed as f64 / *scored as f64), + scored: *scored, + failed: *failed, + }); + } + } + Ok(by_experiment) +} + +/// The scorers of every dataset named by a listed run, read through `user_db` so a run of a +/// dataset the caller cannot read contributes nothing. +async fn scorers_of_listed( + authed: &ApiAuthed, + user_db: &UserDB, + w_id: &str, + experiments: &[EvalExperiment], +) -> Result>> { + let paths: Vec = experiments + .iter() + .map(|e| e.dataset.clone()) + .collect::>() + .into_iter() + .collect(); + if paths.is_empty() { + return Ok(Default::default()); + } + let mut tx = user_db.clone().begin(authed).await?; + let rows = sqlx::query!( + "SELECT path, scorers FROM eval_dataset WHERE workspace_id = $1 AND path = ANY($2)", + w_id, + &paths + ) + .fetch_all(&mut *tx) + .await?; + tx.commit().await?; + rows.into_iter() + .map(|row| Ok((row.path, parse_scorers(row.scorers)?))) + .collect() +} + +#[derive(Deserialize)] +pub struct ExperimentRef { + pub id: Uuid, + /// The experiment every column is compared against. A delta is only ever computed between two + /// scores of the same scorer id. + #[serde(default)] + pub baseline: Option, +} + +/// One scorer's verdict on one run, and how it compares with the baseline. +#[derive(Serialize)] +pub struct CellScore { + pub scorer_id: String, + #[serde(skip_serializing_if = "Option::is_none")] + pub score: Option, + #[serde(skip_serializing_if = "Option::is_none")] + pub reason: Option, + #[serde(skip_serializing_if = "Option::is_none")] + pub checks: Option>, + #[serde(skip_serializing_if = "Option::is_none")] + pub error: Option, + /// The scorer read this case and had nothing to measure on it. Left out of the column's mean + /// and pass rate rather than counted as a zero. + #[serde(skip_serializing_if = "std::ops::Not::not")] + pub not_applicable: bool, + /// A scoring job is still running for this cell. + pub pending: bool, + /// Which side of the scorer's threshold the score fell on, when it has one. + #[serde(skip_serializing_if = "Option::is_none")] + pub passed: Option, + #[serde(skip_serializing_if = "Option::is_none")] + pub baseline: Option, + /// The baseline's score for this scorer was produced by a different definition of it, so the + /// delta is a change of scorer as much as a change of agent. + pub definition_changed: bool, +} + +/// One row per case: what it was asked, what the agent answered, and each scorer's cell. +#[derive(Serialize)] +pub struct ExperimentRow { + pub case_id: Uuid, + pub input: EvalCaseInput, + #[serde(skip_serializing_if = "Option::is_none")] + pub expected: Option>, + /// The iteration that ran this case. Absent between a run being recorded and its flow + /// reaching this case, which reads as a case still to run. + #[serde(skip_serializing_if = "Option::is_none")] + pub job_id: Option, + /// What happened to the answer: the iteration's own `success`/`failure`/`canceled`/`skipped` + /// once it has finished, and until then the agent step's, since the answer is written before + /// the scorers that keep the iteration running have read it. `unavailable` for a case whose + /// job was retained away before anything read what it produced. + pub status: String, + /// The agent's answer, which is what a table cell shows. The whole trajectory stays + /// reachable through `job_id`, so the row carries the text rather than the result object. + #[serde(skip_serializing_if = "Option::is_none")] + pub output: Option, + /// The agent version this cell ran against. Cells of one experiment can differ, which is what + /// the table says instead of averaging two versions silently. + #[serde(skip_serializing_if = "Option::is_none")] + pub subject_version: Option, + /// For a run of unsaved edits, the hash of the configuration this cell ran: edits move without + /// a version changing, and `resolve_deployed_draft` matches this against what is deployed. + #[serde(skip_serializing_if = "Option::is_none")] + pub subject_draft_hash: Option, + /// One entry per scorer of the dataset, in column order. + pub scores: Vec, +} + +/// A column's summary. There is no single number for a dataset: averaging a judge with an exact +/// match would invent one. +#[derive(Serialize)] +pub struct ScorerMean { + pub scorer_id: String, + #[serde(skip_serializing_if = "Option::is_none")] + pub mean: Option, + #[serde(skip_serializing_if = "Option::is_none")] + pub baseline_mean: Option, + /// The share of scored cells that passed, for a column with a threshold. Reported beside the + /// mean rather than instead of it: neither number answers the other's question. + #[serde(skip_serializing_if = "Option::is_none")] + pub pass_rate: Option, + #[serde(skip_serializing_if = "Option::is_none")] + pub baseline_pass_rate: Option, + pub scored: usize, + /// Cells the baseline has no score for, reported so a column the baseline never ran shows as + /// unscored rather than as a spurious difference. + pub missing_in_baseline: usize, + pub definition_changed: bool, +} + +#[derive(Serialize)] +pub struct ExperimentResults { + pub experiment: EvalExperiment, + #[serde(skip_serializing_if = "Option::is_none")] + pub baseline: Option, + /// The columns, which belong to the dataset rather than to the experiment. + pub scorers: Vec, + pub rows: Vec, + pub means: Vec, + /// Cells scoring lower than the baseline, across every column. + pub regressed: usize, + /// The version the subject is on now. A row that ran against an earlier one describes an + /// agent that no longer exists. + #[serde(skip_serializing_if = "Option::is_none")] + pub subject_current_version: Option, + /// What the agent hashes to as deployed. A run of unsaved edits carrying this hash ran exactly + /// what is deployed now — the edits were saved — so it is a run of that version. + #[serde(skip_serializing_if = "Option::is_none")] + pub subject_deployed_hash: Option, +} + +/// The agent's own result is `{output, messages}`; the answer is its `output`. +pub(crate) fn agent_answer(result: &RawValue) -> Option { + let parsed: serde_json::Value = serde_json::from_str(result.get()).ok()?; + match parsed.get("output") { + Some(serde_json::Value::String(s)) => Some(s.clone()), + Some(other) => Some(other.to_string()), + None => None, + } +} + +struct ScoreRow { + score: Option, + reason: Option, + checks: Option, + error: Option, + not_applicable: bool, + definition: String, +} + +/// Every score of one experiment, keyed by the cell and the scorer that produced it. +async fn load_scores( + db: &DB, + experiment_id: Uuid, +) -> Result> { + Ok(sqlx::query!( + "SELECT ordinal, scorer_id, score, reason, checks, error, not_applicable, definition + FROM eval_score WHERE experiment_id = $1", + experiment_id + ) + .fetch_all(db) + .await? + .into_iter() + .map(|r| { + ( + (r.ordinal, r.scorer_id), + ScoreRow { + score: r.score, + reason: r.reason, + checks: r.checks, + error: r.error, + not_applicable: r.not_applicable, + definition: r.definition, + }, + ) + }) + .collect()) +} + +async fn read_experiment(db: &DB, w_id: &str, dataset: &str, id: Uuid) -> Result { + let row = sqlx::query!( + "SELECT e.subject, e.run_number, e.run_job_id, e.created_at, + e.created_by, + (SELECT count(*) FROM eval_experiment_case c WHERE c.experiment_id = e.id) + AS \"case_count!\" + FROM eval_experiment e + WHERE e.workspace_id = $1 AND e.dataset_path = $2 AND e.id = $3", + w_id, + dataset, + id + ) + .fetch_optional(db) + .await? + .ok_or_else(|| { + Error::NotFound(format!( + "Experiment {} not found in eval dataset {}", + id, dataset + )) + })?; + experiment_from_row( + id, + dataset.to_string(), + row.subject, + row.run_number, + row.run_job_id, + row.case_count, + row.created_at, + row.created_by, + ) +} + +/// Recognise a draft run that has since been deployed, and record it as the version it became. +/// +/// Written once rather than derived per read: derived against what is deployed *now*, the next +/// deployment would send a run that already read `v21` back to `v18 + edits`. +async fn resolve_deployed_draft( + db: &DB, + w_id: &str, + dataset: &str, + experiment: &mut EvalExperiment, + deployed_hash: Option<&str>, + deployed_version: Option, +) -> Result<()> { + if experiment.subject.kind != EvalSubjectKind::AgentDraft { + return Ok(()); + } + let (Some(hash), Some(deployed_hash), Some(version)) = ( + experiment.subject.draft_hash.as_deref(), + deployed_hash, + deployed_version, + ) else { + return Ok(()); + }; + if hash != deployed_hash { + return Ok(()); + } + // The hash stays: it is what identifies the configuration, and what this resolution rests on. + experiment.subject.kind = EvalSubjectKind::Agent; + experiment.subject.version = Some(version); + // Both writes in one transaction: a failure between them would leave the experiment promoted + // to a version while its cells stayed a draft's, a split no later read repairs since the + // experiment is no longer a draft. + let mut tx = db.begin().await?; + sqlx::query!( + "UPDATE eval_experiment + SET subject = jsonb_set( + jsonb_set(subject, '{kind}', '\"agent\"'), + '{version}', to_jsonb($4::bigint)) + WHERE workspace_id = $1 AND dataset_path = $2 AND id = $3 + AND subject ->> 'kind' = 'agent_draft'", + w_id, + dataset, + experiment.id, + version, + ) + .execute(&mut *tx) + .await?; + // The cells that ran that configuration are dated by the version too; leaving their hash would + // make the run go on reading as a draft's after the next deployment. + sqlx::query!( + "UPDATE eval_experiment_case + SET subject_version = $3, subject_draft_hash = NULL + WHERE experiment_id = $1 AND subject_draft_hash = $2", + experiment.id, + hash, + version, + ) + .execute(&mut *tx) + .await?; + tx.commit().await?; + Ok(()) +} + +/// Record what a run produced, from inside the run: the last step of a run's own flow calls this. +/// +/// Gated on reading the run rather than on writing its dataset, unlike everything else here: it is +/// the same harvest `experiment_results` performs behind the same check, over the run's own cells, +/// and it reports a count rather than any of what it read. +pub async fn collect_experiment( + authed: ApiAuthed, + Extension(db): Extension, + Extension(user_db): Extension, + Path(w_id): Path, + Query(query): Query, +) -> JsonResult { + // Through `user_db`, so the run is one the caller can see. The row carries the job to read it + // out of, so nothing that is read afterwards is caller-supplied. + let mut tx = user_db.begin(&authed).await?; + let experiment = sqlx::query!( + "SELECT id, run_job_id FROM eval_experiment WHERE workspace_id = $1 AND id = $2", + w_id, + query.id + ) + .fetch_optional(&mut *tx) + .await?; + tx.commit().await?; + let experiment = + experiment.ok_or_else(|| Error::NotFound(format!("Eval run {} not found", query.id)))?; + sync_run(&db, &w_id, experiment.id, experiment.run_job_id, true).await?; + let recorded = sqlx::query_scalar!( + "SELECT count(*) AS \"count!\" FROM eval_experiment_case + WHERE experiment_id = $1 AND status IS NOT NULL", + experiment.id + ) + .fetch_one(&db) + .await?; + Ok(Json(recorded as usize)) +} + +#[derive(Deserialize)] +pub struct ExperimentId { + pub id: Uuid, +} + +/// Collect a run for a reader, without letting the collection decide whether the read succeeds. +/// `collect_experiment` propagates instead: it is the run reporting on itself, and a failure there +/// is worth surfacing to the step that called it. +async fn collect_quietly(db: &DB, w_id: &str, experiment_id: Uuid, run_job_id: Uuid) { + if let Err(e) = sync_run(db, w_id, experiment_id, run_job_id, true).await { + tracing::warn!("could not collect eval run {}: {e:#}", experiment_id); + } +} + +/// The rows a results table is built from. The job ids come out of `eval_experiment_case`, which +/// only this module writes, so they can be read on the unrestricted pool once the dataset read +/// below has established the caller's access. +pub async fn experiment_results( + authed: ApiAuthed, + Extension(db): Extension, + Extension(user_db): Extension, + Path((w_id, dataset)): Path<(String, String)>, + Query(query): Query, +) -> JsonResult { + // The rows carry what the run's jobs produced, which `jobs:read` gates. `UserDB` settles who + // may see the dataset; a token's scopes are a separate question. + check_scopes(&authed, || "jobs:read".to_string())?; + let dataset_row = read_dataset(&authed, &user_db, &w_id, &dataset).await?; + let scorers = dataset_row.scorers; + + let mut experiment = read_experiment(&db, &w_id, &dataset, query.id).await?; + // Best-effort: collecting is what the run's own step is for, and a cell that could not be read + // — a job retained away between the iteration and its children — must not take the whole table + // down with it. The rows already recorded are still the run. + collect_quietly(&db, &w_id, query.id, experiment.run_job_id).await; + let scores = load_scores(&db, query.id).await?; + + let baseline = match query.baseline.filter(|id| *id != query.id) { + Some(id) => { + let baseline = read_experiment(&db, &w_id, &dataset, id).await?; + collect_quietly(&db, &w_id, id, baseline.run_job_id).await; + Some((baseline, load_scores(&db, id).await?)) + } + None => None, + }; + // The baseline is compared case by case, so its cells are keyed by the case they ran. + let baseline_ordinals = match &baseline { + Some((baseline, _)) => sqlx::query!( + "SELECT case_id, ordinal FROM eval_experiment_case WHERE experiment_id = $1", + baseline.id + ) + .fetch_all(&db) + .await? + .into_iter() + .map(|r| (r.case_id, r.ordinal)) + .collect::>(), + None => Default::default(), + }; + + let case_rows = sqlx::query!( + "SELECT ordinal, case_id, input, expected, job_id, subject_version, + subject_draft_hash, output, answered, status + FROM eval_experiment_case + WHERE experiment_id = $1 ORDER BY ordinal", + query.id + ) + .fetch_all(&db) + .await?; + + let mut sums = vec![(0.0f64, 0usize); scorers.len()]; + let mut baseline_sums = vec![(0.0f64, 0usize); scorers.len()]; + let mut passes = vec![0usize; scorers.len()]; + let mut baseline_passes = vec![0usize; scorers.len()]; + let mut missing_in_baseline = vec![0usize; scorers.len()]; + let mut definition_changed = vec![false; scorers.len()]; + let mut regressed = 0usize; + let mut rows = Vec::with_capacity(case_rows.len()); + + for case in case_rows { + let mut cells = Vec::with_capacity(scorers.len()); + for (index, scorer) in scorers.iter().enumerate() { + let current = scores.get(&(case.ordinal, scorer.id.clone())); + let baseline_score = baseline.as_ref().and_then(|(_, baseline_scores)| { + baseline_ordinals + .get(&case.case_id) + .and_then(|ordinal| baseline_scores.get(&(*ordinal, scorer.id.clone()))) + }); + if let Some(score) = current.and_then(|c| c.score) { + sums[index].0 += score; + sums[index].1 += 1; + if scorer.passed(Some(score)) == Some(true) { + passes[index] += 1; + } + } + if let Some(score) = baseline_score.and_then(|b| b.score) { + baseline_sums[index].0 += score; + baseline_sums[index].1 += 1; + if scorer.passed(Some(score)) == Some(true) { + baseline_passes[index] += 1; + } + } else if baseline.is_some() { + missing_in_baseline[index] += 1; + } + let changed = match (current, baseline_score) { + (Some(current), Some(baseline)) => current.definition != baseline.definition, + _ => false, + }; + if changed { + definition_changed[index] = true; + } + if let (Some(score), Some(previous)) = ( + current.and_then(|c| c.score), + baseline_score.and_then(|b| b.score), + ) { + if score < previous { + regressed += 1; + } + } + cells.push(CellScore { + scorer_id: scorer.id.clone(), + score: current.and_then(|c| c.score), + reason: current.and_then(|c| c.reason.clone()), + checks: current + .and_then(|c| c.checks.clone()) + .map(|c| serde_json::value::to_raw_value(&c)) + .transpose()?, + error: current.and_then(|c| c.error.clone()), + not_applicable: current.map(|c| c.not_applicable).unwrap_or(false), + // A row exists because the run was launched with this scorer, so an empty one is a + // score still to come, unless the scorer has already said this case is not one it + // measures. + pending: current + .map(|c| c.score.is_none() && c.error.is_none() && !c.not_applicable) + .unwrap_or(false), + passed: scorer.passed(current.and_then(|c| c.score)), + baseline: baseline_score.and_then(|b| b.score), + definition_changed: changed, + }); + } + rows.push(ExperimentRow { + case_id: case.case_id, + input: serde_json::from_value(case.input)?, + expected: opt_to_raw(case.expected)?, + // The iteration's verdict once it has one. While it is still running, the agent step's: + // the answer is written before the scorers read it, and a spinner beside an answer + // already there reads as an answer still being written. + status: case + .status + .or_else(|| { + case.answered + .map(|ok| if ok { "success" } else { "failure" }.to_string()) + }) + .unwrap_or_else(|| "running".to_string()), + output: case.output, + subject_version: case.subject_version, + subject_draft_hash: case.subject_draft_hash, + job_id: case.job_id, + scores: cells, + }); + } + + let means = scorers + .iter() + .enumerate() + .map(|(index, scorer)| ScorerMean { + scorer_id: scorer.id.clone(), + mean: (sums[index].1 > 0).then(|| sums[index].0 / sums[index].1 as f64), + baseline_mean: (baseline_sums[index].1 > 0) + .then(|| baseline_sums[index].0 / baseline_sums[index].1 as f64), + pass_rate: (scorer.pass_if.is_some() && sums[index].1 > 0) + .then(|| passes[index] as f64 / sums[index].1 as f64), + baseline_pass_rate: (scorer.pass_if.is_some() && baseline_sums[index].1 > 0) + .then(|| baseline_passes[index] as f64 / baseline_sums[index].1 as f64), + scored: sums[index].1, + missing_in_baseline: missing_in_baseline[index], + definition_changed: definition_changed[index], + }) + .collect(); + + // Read as the caller, so a viewer who can see the dataset but not the agent gets neither: the + // agent's version and configuration fingerprint must not leak past its own read permission. + let (subject_deployed_hash, subject_current_version) = + match readable_agent_state(&authed, &user_db, &w_id, &experiment.subject.path).await? { + Some((config, version)) => (Some(draft_hash(&config)), Some(version)), + None => (None, None), + }; + + // A run of unsaved edits whose configuration has since been deployed is a run of that version. + let mut baseline = baseline.map(|(baseline, _)| baseline); + resolve_deployed_draft( + &db, + &w_id, + &dataset, + &mut experiment, + subject_deployed_hash.as_deref(), + subject_current_version, + ) + .await?; + if let Some(baseline) = baseline.as_mut() { + // The compare-to list holds this agent's runs, but the id is the caller's: a run of another + // agent must not be stamped with this one's version. + if baseline.subject.path == experiment.subject.path { + resolve_deployed_draft( + &db, + &w_id, + &dataset, + baseline, + subject_deployed_hash.as_deref(), + subject_current_version, + ) + .await?; + } + } + + Ok(Json(ExperimentResults { + experiment, + baseline, + scorers, + rows, + means, + regressed, + subject_current_version, + subject_deployed_hash, + })) +} diff --git a/backend/windmill-api/src/ai_evals/run.rs b/backend/windmill-api/src/ai_evals/run.rs new file mode 100644 index 0000000000..e102abaf4c --- /dev/null +++ b/backend/windmill-api/src/ai_evals/run.rs @@ -0,0 +1,950 @@ +use super::*; + +/// Node id of the agent step. The answer is read back by this id, so it is part of the stored +/// shape rather than an implementation detail. +pub const AGENT_NODE_ID: &str = "a"; + +/// Node id of the step that assembles what the scorers are handed. +const PAYLOAD_NODE_ID: &str = "p"; +/// Node id of the loop over the dataset's cases. +const CASES_NODE_ID: &str = "cases"; +/// The branch holding every scorer of a case, so they measure it at the same time. +const SCORERS_NODE_ID: &str = "scores"; + +/// In-flight iterations. A dataset is a burst of calls to one provider, so answering every case at +/// once is a run that spends its time being rate-limited. +const RUN_PARALLELISM: u16 = 8; + +/// What each iteration is handed: the case, small enough to sit in every iteration's arguments. +#[derive(Serialize)] +struct CaseIteration { + case_id: Uuid, + ordinal: i32, + input: EvalCaseInput, + #[serde(skip_serializing_if = "Option::is_none")] + expected: Option>, +} + +/// Assembles the payload the scorers read. +/// +/// A step rather than an input transform: every tool call is enriched with the arguments, result, +/// status and duration of the job that ran it, none of which the flow can see. +const PAYLOAD_SCRIPT: &str = r#"//native +// Generated by Windmill: reads the run this iteration answered. +export async function main() { + const id = process.env.WM_FLOW_JOB_ID + const base = process.env.BASE_URL || process.env.BASE_INTERNAL_URL + const res = await fetch( + `${base}/api/w/${process.env.WM_WORKSPACE}/ai_evals/run_payload?job_id=${id}`, + { headers: { Authorization: `Bearer ${process.env.WM_TOKEN}` } } + ) + if (!res.ok) { + throw new Error(`could not read the run of job ${id}: ${res.status} ${await res.text()}`) + } + return await res.json() +} +"#; + +fn payload_module() -> serde_json::Value { + serde_json::json!({ + "id": PAYLOAD_NODE_ID, + "summary": "Assemble the run the scorers read", + "value": { + "type": "rawscript", + // `bunnative` (tag `nativets`), matching the `//native` the script carries. That tag + // belongs to the `native` worker group rather than the default one, so a queued + // iteration never starts when nothing serves it. + "language": "bunnative", + "content": PAYLOAD_SCRIPT, + "lock": EMPTY_BUN_LOCK, + "input_transforms": {} + } + }) +} + +/// Node id of the step that records what the run produced. +const COLLECT_NODE_ID: &str = "collect"; + +/// Copies the run's answers and scores into its own rows, from inside the run. +/// +/// The tables know nothing about the flow, so without this a run started and left is only ever +/// recorded by someone looking at it — after its jobs have been retained away, there is nothing +/// left to record. +const COLLECT_SCRIPT: &str = r#"//native +// Generated by Windmill: records what this run produced, so it outlives the jobs that produced it. +export async function main(experiment_id: string) { + const base = process.env.BASE_URL || process.env.BASE_INTERNAL_URL + const res = await fetch( + `${base}/api/w/${process.env.WM_WORKSPACE}/ai_evals/experiments/collect?id=${experiment_id}`, + { method: 'POST', headers: { Authorization: `Bearer ${process.env.WM_TOKEN}` } } + ) + if (!res.ok) { + throw new Error(`could not record run ${experiment_id}: ${res.status} ${await res.text()}`) + } + return await res.json() +} +"#; + +fn collect_module(experiment_id: Uuid) -> serde_json::Value { + serde_json::json!({ + "id": COLLECT_NODE_ID, + "summary": "Record what the run produced", + // Bookkeeping, so it does not decide whether the run succeeded. What it would have written + // is written again by the first read of the run. + "continue_on_error": true, + "value": { + "type": "rawscript", + "language": "bunnative", + "content": COLLECT_SCRIPT, + "lock": EMPTY_BUN_LOCK, + "input_transforms": { + "experiment_id": { + "type": "static", + "value": experiment_id.to_string(), + }, + } + } + }) +} + +/// The script imports nothing, so its lockfile is the empty one, spelled the way the bun executor +/// splits it. Without a lock a worker running this as bun would resolve dependencies every time. +const EMPTY_BUN_LOCK: &str = "{\n \"dependencies\": {}\n}\n//bun.lock\n"; + +/// What a judge is asked about the case: the run as it reads it, and the case's own attachments, +/// handed to it as they were handed to the agent. +fn judge_case_transforms() -> serde_json::Map { + let mut transforms = serde_json::Map::new(); + transforms.insert( + "user_message".to_string(), + serde_json::json!({ + "type": "javascript", + "expr": format!("results.{}.rendered", PAYLOAD_NODE_ID), + }), + ); + transforms.insert( + "user_attachments".to_string(), + serde_json::json!({ + "type": "javascript", + "expr": format!("results.{}.run.input.user_attachments", PAYLOAD_NODE_ID), + }), + ); + transforms +} + +/// The scorer steps of one iteration, reading the payload the step before them assembled. +/// +/// Each scorer is baked in as `resolve_scorer` resolved it at launch, never linked by path: a +/// linked step resolves the resource when the case reaches it, so a scorer edited mid-run would +/// grade the later cases while every score still names the definition recorded at launch. +fn scorer_modules(scorers: &[(&Scorer, ResolvedScorer)]) -> Vec { + scorers + .iter() + .map(|(scorer, resolved)| { + let value = match resolved { + // A judge is an agent handed the run as its message; its own system prompt is the + // grading contract, which is why editing a judge means editing that agent. + ResolvedScorer::Judge { config } => { + let mut transforms = match &config.input_transforms { + serde_json::Value::Object(map) => map.clone(), + _ => serde_json::Map::new(), + }; + transforms.extend(judge_case_transforms()); + serde_json::json!({ + "type": "aiagent", + "tools": config.tools, + "input_transforms": serde_json::Value::Object(transforms), + }) + } + // `run` is the whole payload; `input`, `output` and `expected` are the same values + // spelled out, so a three-line scorer does not have to reach into it. + ResolvedScorer::Script { hash } => serde_json::json!({ + "type": "script", + "path": scorer.def.path(), + // Serialized as `ScriptHash` (a hex string), which is the only shape a flow + // module's `hash` field deserializes from — a bare number fails in the worker. + "hash": windmill_common::scripts::ScriptHash(*hash), + "input_transforms": { + "run": { + "type": "javascript", + "expr": format!("results.{}.run", PAYLOAD_NODE_ID), + }, + "input": { + "type": "javascript", + "expr": format!("results.{}.run.input", PAYLOAD_NODE_ID), + }, + "output": { + "type": "javascript", + "expr": format!("results.{}.run.output", PAYLOAD_NODE_ID), + }, + "expected": { + "type": "javascript", + "expr": format!("results.{}.run.expected", PAYLOAD_NODE_ID), + }, + } + }), + }; + serde_json::json!({ "id": scorer_module_id(&scorer.id), "value": value }) + }) + .collect() +} + +/// The flow a whole run is: one loop over the dataset's cases, each iteration answering the case +/// and then scoring the answer. +/// +/// One job rather than one per case: a run outlives the tab that started it, and only a worker can +/// notice that the last case finished. The cases live in the flow's value, stored once, rather +/// than in its arguments, which every iteration inherits a copy of. +fn build_run_flow( + config: &AgentDraft, + cases: &[CaseIteration], + scorers: &[(&Scorer, ResolvedScorer)], + experiment_id: Uuid, +) -> Result { + let mut modules: Vec = vec![agent_module(config)?]; + if !scorers.is_empty() { + modules.push(payload_module()); + // One branch each, run together: scorers read the answer and never each other. Each branch + // keeps its own failure, so a judge that errors costs its own column and no other. + modules.push(serde_json::json!({ + "id": SCORERS_NODE_ID, + "value": { + "type": "branchall", + "parallel": true, + "branches": scorers + .iter() + .zip(scorer_modules(scorers)) + .map(|((scorer, _), module)| serde_json::json!({ + // Named for the column it produces: the graph of a run is read to see which + // scorer did what, and a module id is not what a scorer is called. + "summary": scorer_name(scorer), + "skip_failure": true, + "modules": [module], + })) + .collect::>(), + } + })); + } + + Ok(serde_json::from_value(serde_json::json!({ + "modules": [ + { + "id": CASES_NODE_ID, + "value": { + "type": "forloopflow", + "iterator": { "type": "static", "value": cases }, + "parallel": true, + "parallelism": RUN_PARALLELISM, + // One case failing is one cell of the run, not the end of it. + "skip_failures": true, + "modules": modules, + } + }, + // After the loop, so every case has both answered and been scored by the time it runs. + collect_module(experiment_id), + ] + }))?) +} + +/// The agent step, reading its case from the iteration rather than from the flow's arguments. +fn agent_module(config: &AgentDraft) -> Result { + let flow = build_case_flow(config)?; + let mut value = serde_json::to_value(&flow.modules[0].value)?; + if let Some(map) = value.as_object_mut() { + let transforms = map + .entry("input_transforms") + .or_insert_with(|| serde_json::json!({})); + if let Some(transforms) = transforms.as_object_mut() { + for key in ["user_message", "user_attachments"] { + transforms.insert( + key.to_string(), + serde_json::json!({ + "type": "javascript", + "expr": format!("flow_input.iter.value.input.{}", key), + }), + ); + } + } + } + Ok(serde_json::json!({ "id": AGENT_NODE_ID, "value": value })) +} + +/// The agent step as a one-module flow, so the module shape is validated by deserializing +/// through `FlowValue` rather than trusted as raw JSON. +fn build_case_flow(config: &AgentDraft) -> Result { + // The configuration runs exactly as authored: its own brain transforms are the module's, and + // the case supplies the message and the attachments over the top. + let mut input_transforms = match &config.input_transforms { + serde_json::Value::Object(map) => map.clone(), + _ => serde_json::Map::new(), + }; + for key in ["user_message", "user_attachments"] { + input_transforms.insert( + key.to_string(), + serde_json::json!({ "type": "javascript", "expr": format!("flow_input.{}", key) }), + ); + } + + // Always inlined, never a link to the resource: a linked step would resolve the agent when + // each case runs, which is the one thing a run of a named version must not do. + let mut agent_value = serde_json::Map::new(); + agent_value.insert("type".to_string(), serde_json::json!("aiagent")); + agent_value.insert("tools".to_string(), serde_json::json!(config.tools)); + agent_value.insert( + "input_transforms".to_string(), + serde_json::Value::Object(input_transforms), + ); + Ok(serde_json::from_value(serde_json::json!({ + "modules": [{ "id": AGENT_NODE_ID, "value": serde_json::Value::Object(agent_value) }] + }))?) +} + +/// How many times the agent has been saved, not the identity of the row holding that value: runs +/// are named by it and compared by it, so it has to be the resource's own count rather than a +/// sequence the whole instance shares. +pub(crate) async fn current_resource_version( + db: &DB, + w_id: &str, + path: &str, +) -> Result> { + let version = sqlx::query_scalar!( + "SELECT version FROM resource_version WHERE workspace_id = $1 AND path = $2 + ORDER BY version DESC LIMIT 1", + w_id, + path + ) + .fetch_optional(db) + .await?; + Ok(version) +} + +/// Read the agent through `user_db` so a caller who cannot read the resource cannot run it. +pub(crate) async fn require_agent( + authed: &ApiAuthed, + user_db: &UserDB, + w_id: &str, + agent_path: &str, +) -> Result<()> { + let mut tx = user_db.clone().begin(authed).await?; + let resource_type = sqlx::query_scalar!( + "SELECT resource_type FROM resource WHERE workspace_id = $1 AND path = $2", + w_id, + agent_path + ) + .fetch_optional(&mut *tx) + .await?; + tx.commit().await?; + match resource_type.as_deref() { + Some("ai_agent") => Ok(()), + Some(other) => Err(Error::BadRequest(format!( + "Resource {} is a {}, not an ai_agent", + agent_path, other + ))), + None => Err(Error::NotFound(format!("Agent {} not found", agent_path))), + } +} + +/// An `ai_agent` value as the configuration to run it with: its brain becomes the module's input +/// transforms, its tools the module's tools. The same conversion for a draft and for what is +/// deployed, so the two hash comparably — which is what lets a draft run be recognised as the +/// version it became. +fn config_to_draft(value: serde_json::Value) -> Result { + let mut config = match value { + serde_json::Value::Object(map) => map, + _ => return Err(Error::BadRequest("The agent is not an object".to_string())), + }; + let tools = match config.remove("tools") { + Some(serde_json::Value::Array(tools)) => tools, + _ => vec![], + }; + // Every brain key becomes a static transform: `$res:`/`$var:` in them are resolved by the + // same argument machinery a linked step's resource goes through. + let input_transforms = config + .into_iter() + .map(|(key, value)| (key, serde_json::json!({ "type": "static", "value": value }))) + .collect::>(); + Ok(AgentDraft { input_transforms: serde_json::Value::Object(input_transforms), tools }) +} + +/// An agent's deployed value and the version that names it, in the shape a step runs. `None` when +/// the caller cannot see the resource, or it is not a usable agent. +/// +/// Both from one read: a deploy landing between two reads would pair one version's configuration +/// with another's number, and what a run records of its subject is permanent. +pub(crate) async fn readable_agent_state( + authed: &ApiAuthed, + user_db: &UserDB, + w_id: &str, + path: &str, +) -> Result> { + let mut tx = user_db.clone().begin(authed).await?; + let row = sqlx::query!( + "SELECT r.value AS \"value: sqlx::types::Json\", + (SELECT version FROM resource_version v + WHERE v.workspace_id = r.workspace_id AND v.path = r.path + ORDER BY v.version DESC LIMIT 1) AS version + FROM resource r + WHERE r.workspace_id = $1 AND r.path = $2 AND r.resource_type = 'ai_agent'", + w_id, + path + ) + .fetch_optional(&mut *tx) + .await?; + tx.commit().await?; + let Some(row) = row else { + return Ok(None); + }; + let (Some(value), Some(version)) = (row.value, row.version) else { + return Ok(None); + }; + // A resource's value isn't validated against its type on write, so an ai_agent whose value + // isn't a valid config is no usable state rather than an error: one bad row would otherwise + // 400 the whole results page or run list. + match config_to_draft(value.0) { + Ok(config) => Ok(Some((config, version))), + Err(_) => Ok(None), + } +} + +/// Fill in what the client cannot: the version a saved agent is at, or the configuration a past +/// version held. +/// +/// Returns the configuration the run executes, read once here. Every case then executes that one +/// configuration: resolved per case instead, an agent deployed mid-run would be executed by the +/// cases after it while every row still names the version the run started against. +async fn resolve_subject( + authed: &ApiAuthed, + db: &DB, + user_db: &UserDB, + w_id: &str, + subject: &mut EvalSubject, + draft: Option, +) -> Result { + Ok(match subject.kind { + EvalSubjectKind::Agent => { + let Some((config, version)) = + readable_agent_state(authed, user_db, w_id, &subject.path).await? + else { + return Err(Error::BadRequest(format!( + "Agent {} is not a readable ai_agent resource", + subject.path + ))); + }; + subject.version = Some(version); + config + } + EvalSubjectKind::AgentDraft => { + // The edits live nowhere the server can read them, so the request carries them. The + // agent is still read, so a run can only be filed under one the caller can see. + require_agent(authed, user_db, w_id, &subject.path).await?; + // The version the edits are an edit of, as of now: edits record no version of their + // own, so "v15 plus unsaved edits" means the edits and whatever was deployed when the + // run started. + subject.version = current_resource_version(db, w_id, &subject.path).await?; + draft.ok_or_else(|| Error::BadRequest(DRAFT_REQUIRED.to_string()))? + } + EvalSubjectKind::AgentVersion => { + let Some(version) = subject.version else { + return Err(Error::BadRequest( + "A run of a past version must say which version".to_string(), + )); + }; + let config = + agent_version_config(authed, user_db, db, w_id, &subject.path, version).await?; + subject.draft = Some(config.clone()); + config + } + }) +} + +/// One version of an agent out of its history, in the shape a step runs. +/// +/// Read through `user_db` for the agent itself first: a version is the resource as it was, so +/// seeing one is seeing the resource. +async fn agent_version_config( + authed: &ApiAuthed, + user_db: &UserDB, + db: &DB, + w_id: &str, + agent_path: &str, + version: i64, +) -> Result { + require_agent(authed, user_db, w_id, agent_path).await?; + let value = sqlx::query_scalar!( + "SELECT value FROM resource_version + WHERE version = $1 AND workspace_id = $2 AND path = $3", + version, + w_id, + agent_path + ) + .fetch_optional(db) + .await? + .flatten() + .ok_or_else(|| Error::NotFound(format!("Agent {} has no version {}", agent_path, version)))?; + config_to_draft(value).map_err(|_| { + Error::BadRequest(format!( + "Version {} of {} is not an object", + version, agent_path + )) + }) +} + +const DRAFT_REQUIRED: &str = "A run of unsaved edits must carry the configuration being edited"; + +/// The configuration the request may carry, taken out of the subject it belongs to. +/// +/// A saved agent and a past version are read from the workspace by the path they name, so a +/// request carrying a configuration for them would run something other than the agent it claims to +/// be a run of. Unsaved edits are the one kind the request has to carry: they exist only in the +/// editor. +fn validate_subject(subject: &EvalSubject) -> Result> { + if subject.path.trim().is_empty() { + return Err(Error::BadRequest( + "The subject needs a path: it is the agent a run is filed under".to_string(), + )); + } + match (&subject.draft, &subject.kind) { + (Some(draft), EvalSubjectKind::AgentDraft) => Ok(Some(draft.clone())), + (Some(_), _) => Err(Error::BadRequest( + "A saved agent's configuration is read from the workspace; remove it from the request" + .to_string(), + )), + (None, EvalSubjectKind::AgentDraft) => Err(Error::BadRequest(DRAFT_REQUIRED.to_string())), + (None, _) => Ok(None), + } +} + +// ----------------------------------------------------------------------------------------------- +// Experiments +// ----------------------------------------------------------------------------------------------- + +#[derive(Deserialize)] +pub struct RunExperiment { + pub dataset: String, + pub subject: EvalSubject, +} + +/// Open a run of this dataset. +/// +/// Runs are numbered per (dataset, agent) pair, and the deployed agent and its draft share that +/// numbering: they are the same agent, so "Run 7" of a dataset means one thing whether it ran the +/// deployed value or the edits waiting on top of it. +async fn new_run( + tx: &mut sqlx::Transaction<'_, sqlx::Postgres>, + w_id: &str, + dataset: &str, + subject: &EvalSubject, + username: &str, + run_job_id: Uuid, + id: Uuid, +) -> Result { + // Two runs starting together would otherwise read the same run number. Held for the rest of + // this transaction, which pushes no jobs. + sqlx::query!( + "SELECT pg_advisory_xact_lock(hashtext('ai_eval_open:' || $1 || '/' || $2 || '/' || $3))", + w_id, + dataset, + subject.path, + ) + .execute(&mut **tx) + .await?; + let run_number = sqlx::query_scalar!( + "SELECT coalesce(max(run_number), 0) + 1 FROM eval_experiment + WHERE workspace_id = $1 AND dataset_path = $2 AND subject ->> 'path' = $3", + w_id, + dataset, + subject.path, + ) + .fetch_one(&mut **tx) + .await? + .unwrap_or(1); + sqlx::query!( + "INSERT INTO eval_experiment + (id, workspace_id, dataset_path, subject, run_number, created_by, run_job_id) + VALUES ($1, $2, $3, $4, $5, $6, $7)", + id, + w_id, + dataset, + serde_json::to_value(subject.stamp())?, + run_number, + username, + run_job_id, + ) + .execute(&mut **tx) + .await + .map_err(|e| { + if is_missing_dataset(&e) { + Error::NotFound(format!("Eval dataset {} not found", dataset)) + } else { + e.into() + } + })?; + Ok(id) +} + +pub async fn run_experiment( + authed: ApiAuthed, + Extension(db): Extension, + Extension(user_db): Extension, + Path(w_id): Path, + Json(payload): Json, +) -> Result { + check_scopes(&authed, || "jobs:run".to_string())?; + // A write, not a read: it persists an experiment into the dataset. + require_dataset_writable(&authed, &user_db, &w_id, &payload.dataset).await?; + + let mut subject = payload.subject; + let draft = validate_subject(&subject)?; + let config = resolve_subject(&authed, &db, &user_db, &w_id, &mut subject, draft).await?; + + // One snapshot of the dataset: the scorers and the cases a run records must be the same + // revision, or a Save landing between two reads files a run under cases and columns that never + // stood together. + let (dataset, cases) = + read_dataset_and_cases(&authed, &user_db, &w_id, &payload.dataset).await?; + if cases.is_empty() { + return Err(Error::BadRequest(format!( + "Eval dataset {} has no case to run", + payload.dataset + ))); + } + + let case_count = cases.len(); + // Resolved through the caller's own db, so a run executes only runnables the caller may read, + // and what resolving pinned is baked into the flow. Recorded per cell at launch rather than + // when the score comes back, so a scorer edited mid-run reads as the change of scorer it is. + let mut definitions = Vec::with_capacity(dataset.scorers.len()); + let mut scorers: Vec<(&Scorer, ResolvedScorer)> = Vec::with_capacity(dataset.scorers.len()); + for scorer in &dataset.scorers { + let (definition, resolved) = resolve_scorer(&user_db, &authed, &w_id, scorer).await?; + definitions.push(definition); + scorers.push((scorer, resolved)); + } + + let iterations = cases + .iter() + .enumerate() + .map(|(index, case)| CaseIteration { + case_id: case.id, + ordinal: index as i32, + input: case.input.clone(), + expected: case.expected.clone(), + }) + .collect::>(); + // Both ids are chosen here: the run's own collect step is handed the experiment id, and the + // experiment names its job before that job exists. + let experiment_id = Uuid::new_v4(); + let run_job_id = Uuid::new_v4(); + let flow_value = build_run_flow(&config, &iterations, &scorers, experiment_id)?; + + // Recorded before the job is queued, so a launch that dies partway leaves an experiment naming + // a job that never started rather than a flow no experiment accounts for and nothing collects. + let mut tx = db.begin().await?; + let experiment_id = new_run( + &mut tx, + &w_id, + &payload.dataset, + &subject, + &authed.username, + run_job_id, + experiment_id, + ) + .await?; + + let ordinals = (0..case_count as i32).collect::>(); + let case_ids = cases.iter().map(|c| c.id).collect::>(); + let inputs = cases + .iter() + .map(|c| serde_json::to_value(&c.input)) + .collect::, _>>()?; + let expecteds = cases + .iter() + .map(|c| opt_from_raw(c.expected.as_ref())) + .collect::>>()?; + let versions = vec![subject.version; case_count]; + let hashes = vec![subject.draft.as_ref().map(draft_hash); case_count]; + // No job id: the iteration that answers a case is minted by the flow engine, and the case is + // matched back to it once it exists. + sqlx::query!( + "INSERT INTO eval_experiment_case + (experiment_id, ordinal, case_id, input, expected, subject_version, + subject_draft_hash) + SELECT $1, ordinal, case_id, input, expected, subject_version, subject_draft_hash + FROM UNNEST($2::int[], $3::uuid[], $4::jsonb[], $5::jsonb[], $6::bigint[], $7::text[]) + AS t(ordinal, case_id, input, expected, subject_version, subject_draft_hash)", + experiment_id, + &ordinals, + &case_ids, + &inputs, + &expecteds as &[Option], + &versions as &[Option], + &hashes as &[Option], + ) + .execute(&mut *tx) + .await?; + insert_pending_scores(&mut tx, experiment_id, &ordinals, &scorers, &definitions).await?; + // The foreign key makes a delete racing this assembly fail the commit, so nothing is queued. + // A delete landing between this commit and the push below still cascades the experiment away + // while the flow queues; that launch/delete race is a known beta limitation. + tx.commit().await?; + + if let Err(e) = push_run_flow( + &authed, + &db, + &user_db, + &w_id, + &payload.dataset, + &subject, + experiment_id, + run_job_id, + flow_value, + ) + .await + { + // Nothing ran, so there is nothing to keep: one failed push is the whole run. + sqlx::query!("DELETE FROM eval_experiment WHERE id = $1", experiment_id) + .execute(&db) + .await?; + return Err(e); + } + Ok(experiment_id.to_string()) +} + +/// The cells a run will fill in, written at launch. A pending row is what the table reads as a +/// score still being produced, and it is where the definition that produced it is recorded. +async fn insert_pending_scores( + tx: &mut sqlx::Transaction<'_, sqlx::Postgres>, + experiment_id: Uuid, + ordinals: &[i32], + scorers: &[(&Scorer, ResolvedScorer)], + definitions: &[String], +) -> Result<()> { + if scorers.is_empty() || ordinals.is_empty() { + return Ok(()); + } + let mut rows_ordinal = vec![]; + let mut rows_scorer = vec![]; + let mut rows_definition = vec![]; + for ordinal in ordinals { + for ((scorer, _), definition) in scorers.iter().zip(definitions.iter()) { + rows_ordinal.push(*ordinal); + rows_scorer.push(scorer.id.clone()); + rows_definition.push(definition.clone()); + } + } + sqlx::query!( + "INSERT INTO eval_score (experiment_id, ordinal, scorer_id, definition) + SELECT $1, ordinal, scorer_id, definition + FROM UNNEST($2::int[], $3::text[], $4::text[]) AS t(ordinal, scorer_id, definition) + ON CONFLICT (experiment_id, ordinal, scorer_id) + DO UPDATE SET definition = EXCLUDED.definition, score = NULL, reason = NULL, + checks = NULL, error = NULL, not_applicable = false", + experiment_id, + &rows_ordinal, + &rows_scorer, + &rows_definition, + ) + .execute(&mut **tx) + .await?; + Ok(()) +} + +/// Queue the flow a run is. Its id is chosen by the caller, so the experiment can name it before +/// it exists. +async fn push_run_flow( + authed: &ApiAuthed, + db: &DB, + user_db: &UserDB, + w_id: &str, + dataset: &str, + subject: &EvalSubject, + experiment_id: Uuid, + run_job_id: Uuid, + flow_value: windmill_common::flows::FlowValue, +) -> Result { + use windmill_common::{jobs::JobPayload, users::username_to_permissioned_as}; + use windmill_queue::{push, PushArgs, PushIsolationLevel}; + + let mut args = std::collections::HashMap::new(); + // So the job says what it was evaluating when opened cold from the runs page. Every iteration + // inherits these, so they are the stamp and nothing bulkier. + args.insert( + "_eval".to_string(), + serde_json::value::to_raw_value(&serde_json::json!({ + "subject": subject.stamp(), + "dataset": dataset, + "experiment_id": experiment_id, + }))?, + ); + + let path = subject.path.clone(); + let tx = PushIsolationLevel::Isolated(user_db.clone(), authed.clone().into()); + let (uuid, tx) = push( + db, + tx, + w_id, + JobPayload::RawFlow { value: flow_value, path: Some(path), restarted_from: None }, + PushArgs::from(&args), + authed.display_username(), + &authed.email, + username_to_permissioned_as(&authed.username), + authed.token_prefix.as_deref(), + authed.username_override.as_deref(), + None, + None, + None, + None, + None, + Some(run_job_id), + false, + false, + None, + true, + None, + None, + None, + None, + Some(&authed.clone().into()), + false, + None, + authed.trigger_or_fallback(None), + None, + ) + .await?; + tx.commit().await?; + Ok(uuid) +} + +pub(crate) fn experiment_from_row( + id: Uuid, + dataset: String, + subject: serde_json::Value, + run_number: i32, + run_job_id: Uuid, + case_count: i64, + created_at: DateTime, + created_by: String, +) -> Result { + Ok(EvalExperiment { + id, + dataset, + subject: serde_json::from_value(subject)?, + run_number, + run_job_id, + case_count, + // Filled in by the list, which reads every listed run's scores in one query. + scores: vec![], + running: false, + created_at, + created_by, + }) +} + +#[cfg(test)] +mod tests { + use super::*; + + fn agent_config() -> AgentDraft { + AgentDraft { input_transforms: serde_json::json!({}), tools: vec![] } + } + + fn scorer(kind: ScorerDef) -> Scorer { + Scorer { id: "s1".to_string(), name: None, pass_if: None, def: kind } + } + + fn subject(kind: EvalSubjectKind, draft: Option) -> EvalSubject { + EvalSubject { kind, path: "u/me/agent".to_string(), version: None, draft, draft_hash: None } + } + + /// The whole argument for accepting a configuration from the request is that it is accepted + /// for exactly one kind: the edits in progress, which exist nowhere the server can read. A + /// saved agent or a past version carrying one would run something other than what it names. + #[test] + fn a_configuration_is_required_for_edits_and_refused_for_anything_saved() { + assert!( + validate_subject(&subject(EvalSubjectKind::AgentDraft, Some(agent_config()))).is_ok() + ); + assert!(validate_subject(&subject(EvalSubjectKind::Agent, None)).is_ok()); + assert!(validate_subject(&subject(EvalSubjectKind::AgentDraft, None)).is_err()); + assert!(validate_subject(&subject(EvalSubjectKind::Agent, Some(agent_config()))).is_err()); + assert!(validate_subject(&subject( + EvalSubjectKind::AgentVersion, + Some(agent_config()) + )) + .is_err()); + } + + /// Where the collect step sits is load-bearing twice over: inside the loop it would run once + /// per case, and `backfill_case_jobs` matches a case to any child of the run carrying an + /// `iter` argument, which the collect job must therefore never be. + #[test] + fn the_collect_step_runs_once_after_the_loop() { + let experiment = Uuid::new_v4(); + let flow = build_run_flow(&agent_config(), &[], &[], experiment).unwrap(); + let value = serde_json::to_value(&flow).unwrap(); + let modules = value["modules"].as_array().unwrap(); + assert_eq!( + modules + .iter() + .map(|m| m["id"].as_str().unwrap()) + .collect::>(), + vec![CASES_NODE_ID, COLLECT_NODE_ID] + ); + let collect = &modules[1]; + // The run it records is baked in rather than read from the iteration around it, which is + // what makes it a step of the run and not of a case. + assert_eq!( + collect["value"]["input_transforms"]["experiment_id"]["value"] + .as_str() + .unwrap(), + experiment.to_string() + ); + assert!(collect["value"]["input_transforms"]["iter"].is_null()); + // A failed record must not fail a run whose cases all answered. + assert_eq!(collect["continue_on_error"].as_bool(), Some(true)); + } + + /// A code scorer pins the deployed hash it resolved to, and that hash must reach the flow as a + /// `ScriptHash` (a hex string), not the bare number it is in the database. + #[test] + fn a_script_scorer_pins_its_resolved_hash_as_a_hex_string() { + let s = scorer(ScorerDef::Script { path: "f/e/scorer".to_string() }); + let scorers = vec![(&s, ResolvedScorer::Script { hash: 8816320759749465854i64 })]; + let modules = scorer_modules(&scorers); + // A flow module's `hash` deserializes only from a `ScriptHash` (a hex string); emitted as a + // bare number it fails in the worker and every code-scorer column breaks at runtime. + assert!( + modules[0]["value"]["hash"].is_string(), + "the pinned scorer hash must serialize as a hex string, not a number" + ); + } + + /// A judge is pinned by inlining the configuration resolved at launch rather than linked by + /// path, which is the difference between a run that grades against one definition and one that + /// resolves the judge per case. + #[test] + fn a_judge_is_inlined_rather_than_linked() { + let judge = scorer(ScorerDef::Agent { path: "f/e/judge".to_string() }); + let scorers = vec![( + &judge, + ResolvedScorer::Judge { + config: AgentDraft { + input_transforms: serde_json::json!({ + "system_prompt": { "type": "static", "value": "grade it" } + }), + tools: vec![], + }, + }, + )]; + let pinned = scorer_modules(&scorers); + let value = &pinned[0]["value"]; + assert!(value["agent"].is_null()); + assert_eq!( + value["input_transforms"]["system_prompt"]["value"].as_str(), + Some("grade it") + ); + // The case reaches the judge alongside the judge's own transforms. + assert!(value["input_transforms"]["user_message"]["expr"].is_string()); + assert!(value["input_transforms"]["user_attachments"]["expr"].is_string()); + } +} diff --git a/backend/windmill-api/src/ai_evals/scorers.rs b/backend/windmill-api/src/ai_evals/scorers.rs new file mode 100644 index 0000000000..6d6af3096e --- /dev/null +++ b/backend/windmill-api/src/ai_evals/scorers.rs @@ -0,0 +1,285 @@ +use super::*; + +/// A scorer is a column of the results table. +/// +/// `id` is assigned when the scorer is added to a dataset and never reused: it is what makes a +/// column the same column across experiments when the scorer is renamed or its definition is +/// edited, and a delta is only ever computed between two scores carrying the same id. +#[derive(Serialize, Deserialize, Debug, Clone)] +pub struct Scorer { + /// Assigned on write when a new scorer arrives without one, so a client cannot collide two + /// columns onto one id. + #[serde(default)] + pub id: String, + /// The column header. Defaults to the kind, or the last segment of the path. + #[serde(default, skip_serializing_if = "Option::is_none")] + pub name: Option, + /// A score at or above this counts as a pass. Deliberately outside `definition`: where the + /// line sits interprets a score rather than produces it, so moving it re-reads every score + /// already recorded instead of invalidating them. + #[serde(default, skip_serializing_if = "Option::is_none")] + pub pass_if: Option, + #[serde(flatten)] + pub def: ScorerDef, +} + +/// A judge is an `ai_agent` resource sent the run to grade; a script receives the run as an +/// argument. Both are runnables, so every column has a path, a version and code you can open. +#[derive(Serialize, Deserialize, Debug, Clone, PartialEq)] +#[serde(tag = "kind", rename_all = "snake_case")] +pub enum ScorerDef { + Script { path: String }, + Agent { path: String }, +} + +impl ScorerDef { + pub fn path(&self) -> &str { + match self { + ScorerDef::Script { path } | ScorerDef::Agent { path } => path, + } + } + + /// The wire name of the kind, as the client sends it. + pub(crate) fn kind_str(&self) -> &'static str { + match self { + ScorerDef::Script { .. } => "script", + ScorerDef::Agent { .. } => "agent", + } + } + + fn kind_label(&self) -> &'static str { + match self { + ScorerDef::Script { .. } => "Script", + ScorerDef::Agent { .. } => "Judge agent", + } + } +} + +impl Scorer { + /// Whether a score counts as a pass. `None` when the column has no threshold, which keeps a + /// column of plain numbers from being rendered as if it had one. + pub fn passed(&self, score: Option) -> Option { + match (self.pass_if, score) { + (Some(threshold), Some(score)) => Some(score >= threshold), + _ => None, + } + } + + /// What produced a score, recorded with it so a comparison can say the scorer changed instead + /// of letting that read as a difference between two agents. `resolved` is the script hash or + /// resource version that actually ran, which the path alone does not pin. + pub fn definition(&self, resolved: Option<&str>) -> String { + use sha2::{Digest, Sha256}; + let mut hasher = Sha256::new(); + hasher.update(self.def.kind_label().as_bytes()); + hasher.update(b":"); + hasher.update(self.def.path().as_bytes()); + if let Some(resolved) = resolved { + hasher.update(b"@"); + hasher.update(resolved.as_bytes()); + } + hex::encode(hasher.finalize())[..32].to_string() + } +} + +const MAX_SCORER_NAME_CHARS: usize = 120; + +/// Ids are assigned here rather than trusted from the client: an id is kept only when it names a +/// column the dataset already has, so a removed column cannot come back and inherit the scores +/// recorded against it. Anything else is minted as a valid flow module identifier, which the +/// scoring flows it is baked into require (see `scorer_module_id`). +pub(crate) fn assign_scorer_ids( + scorers: &mut Vec, + existing: &std::collections::HashSet, +) -> Result<()> { + if scorers.len() > MAX_SCORERS_PER_DATASET { + return Err(Error::BadRequest(format!( + "An eval dataset holds at most {} scorers", + MAX_SCORERS_PER_DATASET + ))); + } + let mut seen = std::collections::HashSet::new(); + for scorer in scorers.iter_mut() { + if !existing.contains(&scorer.id) || !seen.insert(scorer.id.clone()) { + scorer.id = Uuid::new_v4().simple().to_string(); + seen.insert(scorer.id.clone()); + } + if let Some(name) = &scorer.name { + if name.chars().count() > MAX_SCORER_NAME_CHARS { + return Err(Error::BadRequest(format!( + "Scorer name {} is too long, {} characters at most", + name, MAX_SCORER_NAME_CHARS + ))); + } + } + // A score is 0 to 1, so a threshold outside that range would pass everything or nothing + // regardless of what the scorer measured. + if let Some(pass_if) = scorer.pass_if { + if !(0.0..=1.0).contains(&pass_if) { + return Err(Error::BadRequest(format!( + "Scorer pass threshold {} must be between 0 and 1", + pass_if + ))); + } + } + check_proper_path(scorer.def.path())?; + } + Ok(()) +} + +/// What a column is called: the dataset's own name for it, or the last segment of what it points +/// at. The same fallback the column header uses. +pub(crate) fn scorer_name(scorer: &Scorer) -> String { + scorer + .name + .clone() + .filter(|n| !n.trim().is_empty()) + .unwrap_or_else(|| { + let path = scorer.def.path(); + path.rsplit('/').next().unwrap_or(path).to_string() + }) +} + +#[derive(Serialize)] +pub struct RecentScorer { + #[serde(flatten)] + pub scorer: Scorer, + /// The dataset it is a column of, which is where the user last saw it. + pub dataset: String, +} + +#[derive(Deserialize)] +pub struct RecentScorersQuery { + /// Only scorers of this kind, which is the one the add form was opened for. + #[serde(default)] + pub kind: Option, +} + +/// The scorers already in use in this workspace, most recently edited dataset first. +/// +/// Filtered twice through `user_db`: a scorer appears only if its dataset does, and the runnable +/// is checked the same way, so the list is scorers the caller could actually run. +pub async fn recent_scorers( + authed: ApiAuthed, + Extension(user_db): Extension, + Path(w_id): Path, + Query(query): Query, +) -> JsonResult> { + let mut tx = user_db.begin(&authed).await?; + let datasets = sqlx::query!( + "SELECT path, scorers FROM eval_dataset + WHERE workspace_id = $1 ORDER BY edited_at DESC LIMIT 100", + w_id + ) + .fetch_all(&mut *tx) + .await?; + + let mut seen = std::collections::HashSet::new(); + let mut recent: Vec = vec![]; + for row in datasets { + for scorer in parse_scorers(row.scorers)? { + if query + .kind + .as_deref() + .is_some_and(|kind| kind != scorer.def.kind_str()) + { + continue; + } + let key = (scorer.def.kind_str(), scorer.def.path().to_string()); + if seen.insert(key) { + recent.push(RecentScorer { scorer, dataset: row.path.clone() }); + } + } + } + // Readability is resolved over every candidate, then the list is cut: an unreadable scorer must + // not take a slot a readable one further down would have filled. + let script_paths = recent + .iter() + .filter(|r| matches!(r.scorer.def, ScorerDef::Script { .. })) + .map(|r| r.scorer.def.path().to_string()) + .collect::>(); + let agent_paths = recent + .iter() + .filter(|r| matches!(r.scorer.def, ScorerDef::Agent { .. })) + .map(|r| r.scorer.def.path().to_string()) + .collect::>(); + // Same deployed-version predicate as get_latest_script_hash: a script with no successfully + // locked version can't be resolved at launch, so it must not offer itself as a scorer here. + let readable_scripts = sqlx::query_scalar!( + "SELECT DISTINCT path FROM script + WHERE workspace_id = $1 AND path = ANY($2) + AND deleted = false AND lock IS NOT NULL AND lock_error_logs IS NULL", + w_id, + &script_paths + ) + .fetch_all(&mut *tx) + .await? + .into_iter() + .collect::>(); + let readable_agents = sqlx::query_scalar!( + "SELECT path FROM resource WHERE workspace_id = $1 AND path = ANY($2) AND resource_type = 'ai_agent'", + w_id, + &agent_paths + ) + .fetch_all(&mut *tx) + .await? + .into_iter() + .collect::>(); + tx.commit().await?; + + recent.retain(|r| match &r.scorer.def { + ScorerDef::Script { path } => readable_scripts.contains(path), + ScorerDef::Agent { path } => readable_agents.contains(path), + }); + recent.truncate(MAX_RECENT_SCORERS); + Ok(Json(recent)) +} + +#[cfg(test)] +mod tests { + use super::*; + + /// The definition hash is what tells a comparison that the scorer changed; the path alone + /// would miss an edit to the script itself. + #[test] + fn definition_moves_with_the_runnable_and_not_with_its_name() { + let script = |path: &str, name: Option<&str>| Scorer { + id: "s1".to_string(), + name: name.map(|n| n.to_string()), + pass_if: None, + def: ScorerDef::Script { path: path.to_string() }, + }; + // Renaming a column is not a change of scorer: same runnable, same version. + assert_eq!( + script("f/e/s", None).definition(Some("1234")), + script("f/e/s", Some("Tool discipline")).definition(Some("1234")) + ); + // Same script, newly deployed: the column says the scorer changed. + assert_ne!( + script("f/e/s", None).definition(Some("1234")), + script("f/e/s", None).definition(Some("5678")) + ); + // A judge agent and a script sharing a path are not the same column. + let agent = Scorer { + id: "s1".to_string(), + name: None, + pass_if: None, + def: ScorerDef::Agent { path: "f/e/s".to_string() }, + }; + assert_ne!( + agent.definition(Some("1")), + script("f/e/s", None).definition(Some("1")) + ); + // If the pass line entered the hash, setting a threshold would mark every score already + // recorded as coming from a different scorer. + let mut thresholded = script("f/e/s", None); + thresholded.pass_if = Some(0.7); + assert_eq!( + thresholded.definition(Some("1234")), + script("f/e/s", None).definition(Some("1234")) + ); + assert_eq!(thresholded.passed(Some(0.7)), Some(true)); + assert_eq!(thresholded.passed(Some(0.69)), Some(false)); + assert_eq!(script("f/e/s", None).passed(Some(0.1)), None); + } +} diff --git a/backend/windmill-api/src/ai_evals/scoring.rs b/backend/windmill-api/src/ai_evals/scoring.rs new file mode 100644 index 0000000000..8fff9d040f --- /dev/null +++ b/backend/windmill-api/src/ai_evals/scoring.rs @@ -0,0 +1,712 @@ +use super::*; + +/// What a scorer resolves to, alongside the definition to record: a script by its pinned hash, or +/// a judge by the configuration to inline. +pub(crate) enum ResolvedScorer { + Script { hash: i64 }, + Judge { config: AgentDraft }, +} + +/// The runnable a scorer names, resolved through the caller's *own* database so a run can only +/// execute code the caller may read: a scorer is added with a bare path and nothing checks read +/// access there. +/// +/// Returns the definition to record and what to run: a script by its deployed hash to pin, or a +/// judge by the configuration to inline, so a redeploy midway through a run cannot swap the code +/// out from under a score labelled with the old version. +pub(crate) async fn resolve_scorer( + user_db: &UserDB, + authed: &ApiAuthed, + w_id: &str, + scorer: &Scorer, +) -> Result<(String, ResolvedScorer)> { + match &scorer.def { + ScorerDef::Script { path } => { + // The latest *deployed* hash (no draft, no failed deploy), through the canonical helper + // so the version a scorer pins is the one everything else runs. + let mut tx = user_db.clone().begin(authed).await?; + let hash = windmill_common::get_latest_script_hash(&mut *tx, path, w_id).await?; + tx.commit().await?; + let Some(hash) = hash else { + return Err(Error::BadRequest(format!( + "Scorer script {} is not deployed or not readable", + path + ))); + }; + Ok(( + scorer.definition(Some(&hash.to_string())), + ResolvedScorer::Script { hash }, + )) + } + ScorerDef::Agent { path } => { + let Some((config, version)) = readable_agent_state(authed, user_db, w_id, path).await? + else { + return Err(Error::BadRequest(format!( + "Judge scorer {} is not a readable ai_agent resource", + path + ))); + }; + Ok(( + scorer.definition(Some(&version.to_string())), + ResolvedScorer::Judge { config }, + )) + } + } +} + +/// Bring a run's record up to date with the flow that executed it: which iteration answered which +/// case, what the agent answered, and what its scorers returned. +/// +/// `answers` is what separates the two callers: a listing reports each run's score aggregates and +/// never shows an answer, so harvesting them there reads a column of every case of every listed +/// run to display none of it. +pub(crate) async fn sync_run( + db: &DB, + w_id: &str, + experiment_id: Uuid, + run_job_id: Uuid, + answers: bool, +) -> Result<()> { + backfill_case_jobs(db, w_id, experiment_id, run_job_id).await?; + settle_unspawned_cases(db, w_id, experiment_id, run_job_id).await?; + if answers { + record_case_answers(db, w_id, experiment_id).await?; + } + harvest_flow_scores(db, w_id, experiment_id).await?; + Ok(()) +} + +/// Give a terminal status to cases the run never spawned an iteration for: with no `job_id` there +/// is nothing to read an answer or a score out of, so they would report "running" indefinitely. +async fn settle_unspawned_cases( + db: &DB, + w_id: &str, + experiment_id: Uuid, + run_job_id: Uuid, +) -> Result<()> { + // Only a run that has reached `v2_job_completed` is settled from here. A job absent from the + // tables is as likely mid-launch — the experiment is committed before its job is pushed — as + // aged out, and settling then would cancel the cases of a run about to start. A cancelled run + // lands in `v2_job_completed`, so a cancel before an iteration spawned is still covered. + let Some(terminal_status) = sqlx::query_scalar!( + "SELECT status::text AS \"status!\" FROM v2_job_completed WHERE id = $1 AND workspace_id = $2", + run_job_id, + w_id + ) + .fetch_optional(db) + .await? + else { + return Ok(()); + }; + let settled = sqlx::query_scalar!( + "UPDATE eval_experiment_case SET status = $2, answered = false + WHERE experiment_id = $1 AND job_id IS NULL AND status IS NULL + RETURNING ordinal", + experiment_id, + terminal_status + ) + .fetch_all(db) + .await?; + // The score cells of a case that never ran have no job to read a verdict out of either. + if !settled.is_empty() { + sqlx::query!( + "UPDATE eval_score SET error = 'The case did not run' + WHERE experiment_id = $1 AND ordinal = ANY($2) + AND score IS NULL AND error IS NULL AND NOT not_applicable", + experiment_id, + &settled + ) + .execute(db) + .await?; + } + Ok(()) +} + +/// In-flight reads of what a case's agent step produced. Each is several queries and a run holds +/// up to `MAX_CASES_PER_DATASET` cases, so they go a few at a time. +const HARVEST_CONCURRENCY: usize = 8; + +/// Cases whose scorer results are read in one query: every scorer of every case in the batch, so +/// the batch bounds how much of a run's worth of judge conversations is held at once. +const HARVEST_BATCH_CASES: usize = 100; + +/// Copy what each iteration produced into its row: the agent's answer, whether producing it +/// succeeded, and how the iteration ended. +/// +/// Written once, when it becomes readable, rather than read back out of the jobs whenever the +/// table is displayed — jobs have their own retention, and a run whose rows are kept has to still +/// read as the run it was after they have aged out. +async fn record_case_answers(db: &DB, w_id: &str, experiment_id: Uuid) -> Result<()> { + let unrecorded = sqlx::query!( + "SELECT c.ordinal, c.job_id AS \"job_id!\", d.status::text AS status, + (j.id IS NOT NULL) AS \"job_exists!\" + FROM eval_experiment_case c + LEFT JOIN v2_job j ON j.id = c.job_id AND j.workspace_id = $2 + LEFT JOIN v2_job_completed d ON d.id = c.job_id AND d.workspace_id = $2 + WHERE c.experiment_id = $1 AND c.job_id IS NOT NULL AND c.status IS NULL", + experiment_id, + w_id + ) + .fetch_all(db) + .await?; + + use futures::StreamExt; + let answers = futures::stream::iter(unrecorded.into_iter().map(|row| async move { + // The job was retained away before anything read it: nothing to read, and nothing more + // will ever be there to read. + if !row.job_exists { + return Ok((row.ordinal, None, None, Some("unavailable".to_string()))); + } + // The agent step's own result, never the iteration's: the iteration goes on to score the + // answer, so the answer is settled long before the iteration is. + let agent = agent_result(db, w_id, row.job_id).await?; + // An iteration that ended without an answer — skipped, cancelled, or an agent that failed + // outright — produced none, and saying so is what stops this re-reading it. + let answered = agent + .as_ref() + .map(|(_, success)| *success) + .or_else(|| row.status.is_some().then_some(false)); + let output = agent.as_ref().and_then(|(result, _)| agent_answer(result)); + Ok::<_, Error>((row.ordinal, output, answered, row.status)) + })) + .buffered(HARVEST_CONCURRENCY) + .collect::>() + .await + .into_iter() + .collect::>>()?; + + // One statement for the whole run: the run's own collect step reaches every case at once, and + // a thousand of them one at a time is a thousand round trips. + let mut ordinals = vec![]; + let mut outputs = vec![]; + let mut answered = vec![]; + let mut statuses = vec![]; + for (ordinal, output, was_answered, status) in answers { + // Nothing to record yet, and the iteration may still produce it. + if was_answered.is_none() && status.is_none() { + continue; + } + ordinals.push(ordinal); + outputs.push(output); + answered.push(was_answered); + statuses.push(status); + } + if ordinals.is_empty() { + return Ok(()); + } + sqlx::query!( + "UPDATE eval_experiment_case c + SET output = COALESCE(c.output, t.output), answered = COALESCE(c.answered, t.answered), + status = COALESCE(c.status, t.status) + FROM UNNEST($2::int[], $3::text[], $4::bool[], $5::text[]) + AS t(ordinal, output, answered, status) + WHERE c.experiment_id = $1 AND c.ordinal = t.ordinal", + experiment_id, + &ordinals, + &outputs as &[Option], + &answered as &[Option], + &statuses as &[Option], + ) + .execute(db) + .await?; + Ok(()) +} + +/// The agent step's result, with "there is none" kept apart from "it could not be read": a lookup +/// that failed for any other reason must not be recorded as a case that produced no answer, +/// because nothing reads that row again. +pub(crate) async fn agent_result( + db: &DB, + w_id: &str, + job_id: Uuid, +) -> Result, bool)>> { + match windmill_queue::get_result_and_success_by_id_from_flow( + db, + w_id, + &job_id, + AGENT_NODE_ID, + None, + ) + .await + { + Ok(found) => Ok(Some(found)), + Err(Error::NotFound(_)) => Ok(None), + Err(e) => Err(e), + } +} + +/// Match each case to the iteration that ran it. The flow engine mints those job ids, so the case +/// they belong to is read back from the iteration's own arguments, which survives iterations +/// finishing in any order. +async fn backfill_case_jobs( + db: &DB, + w_id: &str, + experiment_id: Uuid, + run_job_id: Uuid, +) -> Result<()> { + sqlx::query!( + "UPDATE eval_experiment_case c SET job_id = j.id + FROM v2_job j + WHERE j.parent_job = $3 AND j.workspace_id = $2 + AND (j.args -> 'iter' -> 'value' ->> 'case_id')::uuid = c.case_id + AND c.experiment_id = $1 AND c.job_id IS NULL", + experiment_id, + w_id, + run_job_id + ) + .execute(db) + .await?; + Ok(()) +} + +/// Read the scores a run's own flow produced into `eval_score`, so a score outlives the flow +/// that produced it and the retention on its jobs. +async fn harvest_flow_scores(db: &DB, w_id: &str, experiment_id: Uuid) -> Result<()> { + let pending = sqlx::query!( + // Left-joined, so an iteration still running is read too: a scorer runs after the agent + // within that iteration, so its verdict is there to be read as soon as its own step is + // done, and waiting for the iteration to end would hold every column of a case back until + // the last of them finished. + "SELECT s.ordinal, s.scorer_id, c.job_id AS \"job_id!\", d.status::text AS status, + c.answered, (j.id IS NOT NULL) AS \"job_exists!\" + FROM eval_score s + JOIN eval_experiment_case c + ON c.experiment_id = s.experiment_id AND c.ordinal = s.ordinal + LEFT JOIN v2_job j ON j.id = c.job_id AND j.workspace_id = $2 + LEFT JOIN v2_job_completed d ON d.id = c.job_id AND d.workspace_id = $2 + WHERE s.experiment_id = $1 AND s.score IS NULL AND s.error IS NULL + AND NOT s.not_applicable AND c.job_id IS NOT NULL", + experiment_id, + w_id + ) + .fetch_all(db) + .await?; + if pending.is_empty() { + return Ok(()); + } + + // The job tree is walked in SQL rather than once per cell: a live run is read every couple of + // seconds and a full one is up to MAX_CASES_PER_DATASET × MAX_SCORERS_PER_DATASET cells. The + // shape is `build_run_flow`'s: a scorer is the one module of its own branch of the scoring + // step, so its job's parent is that branch and the branch's parent is the case. + let mut case_jobs: Vec = pending.iter().map(|row| row.job_id).collect(); + case_jobs.sort(); + case_jobs.dedup(); + let mut modules: Vec = pending + .iter() + .map(|row| scorer_module_id(&row.scorer_id)) + .collect(); + modules.sort(); + modules.dedup(); + let mut verdicts: Vec<(i32, String, Option<(Verdict, Option)>)> = + Vec::with_capacity(pending.len()); + for batch in case_jobs.chunks(HARVEST_BATCH_CASES) { + let results: std::collections::HashMap<(Uuid, String), Box> = sqlx::query!( + "SELECT branch.parent_job AS \"case_job!\", scorer.flow_step_id AS \"module!\", + done.result AS \"result: sqlx::types::Json>\" + FROM v2_job branch + JOIN v2_job scorer ON scorer.parent_job = branch.id + JOIN v2_job_completed done ON done.id = scorer.id + WHERE branch.parent_job = ANY($1) AND branch.workspace_id = $2 + AND scorer.flow_step_id = ANY($3)", + batch, + w_id, + &modules + ) + .fetch_all(db) + .await? + .into_iter() + .map(|row| { + let result = row + .result + .map(|json| json.0) + .unwrap_or_else(|| RawValue::from_string("null".to_string()).expect("a literal")); + ((row.case_job, row.module), result) + }) + .collect(); + let in_batch: std::collections::HashSet = batch.iter().copied().collect(); + for row in pending.iter().filter(|row| in_batch.contains(&row.job_id)) { + // Nothing left to read the verdict out of. Settled here, since a cell left pending is + // one every later listing would go back to this same absent job for. + if !row.job_exists { + verdicts.push(( + row.ordinal, + row.scorer_id.clone(), + Some(( + Verdict::default(), + Some("The run that produced this score is no longer available".to_string()), + )), + )); + continue; + } + // What to say when the job is over and this scorer left nothing. Only + // `record_case_answers` tells the two states apart and a listing syncs without it, so + // `None` withholds the sentence — not the harvest: a scorer that returned a number is + // read and recorded either way. + let missing = row.answered.map(|answered| { + if answered { + "This scorer did not run for the case" + } else { + "The case produced no answer to score" + } + }); + let result = results + .get(&(row.job_id, scorer_module_id(&row.scorer_id))) + .map(|r| r.as_ref()); + let verdict = settle_verdict(result, row.status.as_deref(), missing); + verdicts.push((row.ordinal, row.scorer_id.clone(), verdict)); + } + } + + // One statement for every cell read, for the same reason the answers are written that way. + let mut ordinals = vec![]; + let mut scorer_ids = vec![]; + let mut scores = vec![]; + let mut reasons = vec![]; + let mut checks = vec![]; + let mut errors = vec![]; + let mut not_applicable = vec![]; + for (ordinal, scorer_id, read) in verdicts { + // Still to come: a scorer whose own step has not run yet. + let Some((verdict, error)) = read else { + continue; + }; + ordinals.push(ordinal); + scorer_ids.push(scorer_id); + scores.push(verdict.score); + reasons.push(verdict.reason); + checks.push(verdict.checks); + errors.push(error); + not_applicable.push(verdict.not_applicable); + } + if ordinals.is_empty() { + return Ok(()); + } + sqlx::query!( + "UPDATE eval_score s + SET score = t.score, reason = t.reason, checks = t.checks, error = t.error, + not_applicable = t.not_applicable + FROM UNNEST($2::int[], $3::text[], $4::double precision[], $5::text[], $6::jsonb[], + $7::text[], $8::bool[]) + AS t(ordinal, scorer_id, score, reason, checks, error, not_applicable) + WHERE s.experiment_id = $1 AND s.ordinal = t.ordinal AND s.scorer_id = t.scorer_id", + experiment_id, + &ordinals, + &scorer_ids, + &scores as &[Option], + &reasons as &[Option], + &checks as &[Option], + &errors as &[Option], + ¬_applicable, + ) + .execute(db) + .await?; + Ok(()) +} + +/// One scorer's verdict, from the result of the step that produced it, inside a job that may +/// still be running: a scorer's own step can be done while the iteration around it is not. `None` +/// while the result is not readable yet, which is a state to wait through rather than to record +/// as a failure; `Some` with an error is a scorer that produced nothing, worded by where it ran. +fn settle_verdict( + result: Option<&RawValue>, + job_status: Option<&str>, + // What to record when the job is over and this scorer produced nothing. A different statement + // depending on where the scorer ran: its own job failed, or the case it was to score never + // produced an answer. `None` when the caller cannot yet tell those apart, which leaves the + // cell pending for a read that can, rather than settling it on the wrong one of the two. + missing_error: Option<&str>, +) -> Option<(Verdict, Option)> { + Some(match result { + Some(value) => { + let verdict = extract_verdict(value); + match verdict { + // A score is a fraction: the mean and the pass rate read it as one, so a number + // outside that range is recorded as an error rather than a value that would + // quietly skew the column. + Verdict { score: Some(score), .. } if !(0.0..=1.0).contains(&score) => ( + Verdict::default(), + Some(format!( + "The scorer returned {}, outside the 0 to 1 range a score must be in", + score + )), + ), + // A number in range, or the scorer saying this case is not one it measures. Both + // are answers, so neither is an error. + Verdict { score: Some(_), .. } | Verdict { not_applicable: true, .. } => { + (verdict, None) + } + // The job around this scorer is still going, so a module with no number in it is + // one that has not run yet. Recording a failure here would make it permanent. + _ if job_status.is_none() => return None, + _ if job_status == Some("success") => ( + verdict, + Some("The scorer returned no number to plot".to_string()), + ), + _ => match missing_error { + Some(missing) => (verdict, Some(missing.to_string())), + None => return None, + }, + } + } + // The iteration is over, so a scorer step with no readable result produced nothing and + // never will; left pending it would be re-read on every listing. + None if job_status == Some("success") => ( + Verdict::default(), + Some("The scorer step produced no result".to_string()), + ), + // The job holding this scorer has not finished, so a module with nothing in it yet is a + // step that has not run rather than one that produced nothing. + None if job_status.is_none() => return None, + None => match missing_error { + Some(missing) => (Verdict::default(), Some(missing.to_string())), + None => return None, + }, + }) +} + +/// The score and reason read straight out of text that failed to parse as JSON. Deliberately not a +/// second JSON parser: it looks for the two keys and takes what follows, which is what survives a +/// model writing an unescaped quote in the middle of a sentence. +fn salvage_verdict(text: &str) -> (Option, Option) { + fn after_key<'a>(text: &'a str, key: &str) -> Option<&'a str> { + let start = text.find(key)? + key.len(); + Some(text[start..].trim_start().strip_prefix(':')?.trim_start()) + } + + let score = after_key(text, "\"score\"").and_then(|rest| { + if rest.starts_with("true") { + return Some(1.0); + } + if rest.starts_with("false") { + return Some(0.0); + } + let end = rest + .find(|c: char| !matches!(c, '0'..='9' | '.' | '-' | '+' | 'e' | 'E')) + .unwrap_or(rest.len()); + rest[..end].parse::().ok() + }); + + // To the last quote of the object, so an unescaped one inside the sentence stays part of it. + let reason = after_key(text, "\"reason\"") + .and_then(|rest| rest.strip_prefix('"')) + .and_then(|rest| { + let body = match rest.rfind('}') { + Some(brace) => &rest[..brace], + None => rest, + }; + let end = body.rfind('"')?; + Some(body[..end].to_string()) + }) + .filter(|reason| !reason.is_empty()); + + (score, reason) +} + +/// A fenced code block as the model wrote it, reduced to what is inside the fence. The opening +/// fence carries a language tag often enough that the first line goes with it. +fn unfence(text: &str) -> &str { + let trimmed = text.trim(); + let Some(rest) = trimmed.strip_prefix("```") else { + return trimmed; + }; + let inner = match rest.split_once('\n') { + Some((_language, body)) => body, + None => rest, + }; + inner.trim_end().trim_end_matches("```").trim() +} + +/// What a scorer said about one run. `not_applicable` is the scorer declining to measure this +/// case: an explicit `{"score": null}`. A bare `null` stays an error, since a scorer that forgot +/// to return is indistinguishable from one that returned nothing on purpose. +#[derive(Default)] +struct Verdict { + score: Option, + reason: Option, + checks: Option, + not_applicable: bool, +} + +impl Verdict { + fn scored(score: f64) -> Self { + Verdict { score: Some(score), ..Default::default() } + } +} + +/// A scorer may return a bare number, a boolean, or `{score, reason, checks}`; an agent wraps its +/// answer in `output`, sometimes as a string holding any of those. Anything with no number in it +/// is left empty rather than guessed at. +fn extract_verdict(value: &RawValue) -> Verdict { + let Ok(parsed) = serde_json::from_str::(value.get()) else { + return Verdict::default(); + }; + fn as_number(value: &serde_json::Value) -> Option { + match value { + serde_json::Value::Number(n) => n.as_f64(), + serde_json::Value::Bool(b) => Some(if *b { 1.0 } else { 0.0 }), + _ => None, + } + } + if let Some(number) = as_number(&parsed) { + return Verdict::scored(number); + } + let serde_json::Value::Object(map) = &parsed else { + // A judge often answers with JSON inside a string, and often fences it as markdown even + // when told to reply with JSON only. + if let serde_json::Value::String(text) = &parsed { + let text = unfence(text); + if let Ok(inner) = serde_json::from_str::(text) { + if let Ok(raw) = serde_json::value::to_raw_value(&inner) { + return extract_verdict(&raw); + } + } + // Nearly JSON: a judge that quotes the agent inside its own reason writes those quotes + // unescaped, which is invalid and also the most ordinary thing for it to say. The + // number is what the column plots, so it is read out of the text rather than lost with + // the object around it. + let (score, reason) = salvage_verdict(text); + return Verdict { score, reason, checks: None, not_applicable: false }; + } + return Verdict::default(); + }; + let reason = || { + map.get("reason") + .or_else(|| map.get("comment")) + .and_then(|r| r.as_str()) + .map(|r| r.to_string()) + }; + if let Some(score) = map.get("score").and_then(as_number) { + return Verdict { + score: Some(score), + reason: reason(), + checks: map.get("checks").cloned(), + not_applicable: false, + }; + } + // Written out rather than merely absent, which is what separates it from a scorer that + // returned an object with no verdict in it at all. + if map.get("score").is_some_and(|s| s.is_null()) { + return Verdict { + score: None, + reason: reason(), + checks: map.get("checks").cloned(), + not_applicable: true, + }; + } + match map.get("output") { + Some(output) => match serde_json::value::to_raw_value(output) { + Ok(raw) => extract_verdict(&raw), + Err(_) => Verdict::default(), + }, + None => Verdict::default(), + } +} + +#[cfg(test)] +mod tests { + use super::*; + + fn raw(json: &str) -> Box { + serde_json::from_str(json).unwrap() + } + + /// A scorer's answer arrives in whatever shape its runnable returns: a script's bare value or + /// object, or a judge's answer wrapped in `output` and often stringified. A shape that goes + /// unrecognised is a silently empty cell rather than an error. + #[test] + fn extract_verdict_reads_every_documented_scorer_shape() { + let score = |json: &str| extract_verdict(&raw(json)).score; + assert_eq!(score("0.75"), Some(0.75)); + assert_eq!(score("true"), Some(1.0)); + assert_eq!(score(r#"{"score": 0.5}"#), Some(0.5)); + assert_eq!(score(r#"{"score": false}"#), Some(0.0)); + + // judges and agent scorers: the answer is under `output`, sometimes as a string + assert_eq!(score(r#"{"output": 0.25}"#), Some(0.25)); + assert_eq!(score(r#"{"output": "0.9"}"#), Some(0.9)); + assert_eq!(score(r#"{"output": {"score": 0.8}}"#), Some(0.8)); + assert_eq!(score(r#"{"output": "{\"score\": 0.4}"}"#), Some(0.4)); + + // a judge told to reply with JSON only, replying with JSON only, in a code fence + assert_eq!( + score("{\"output\": \"```json\\n{\\\"score\\\": 0.15}\\n```\"}"), + Some(0.15) + ); + assert_eq!(score("{\"output\": \"```\\n0.6\\n```\"}"), Some(0.6)); + + // A judge quoting the agent inside its own reason, which is invalid JSON. + let quoted = extract_verdict(&raw( + r#"{"output": "{\"score\": 0.8, \"reason\": \"invented context (\"stop asking me\", never said) here\"}"}"#, + )); + assert_eq!(quoted.score, Some(0.8)); + assert_eq!( + quoted.reason.as_deref(), + Some(r#"invented context ("stop asking me", never said) here"#) + ); + + // nothing numeric to plot: left empty rather than guessed at + assert_eq!(score(r#"{"output": "not a score"}"#), None); + assert_eq!(score(r#"{"verdict": "good"}"#), None); + + let full = extract_verdict(&raw( + r#"{"score": 0.5, "reason": "half", "checks": [{"name": "a"}]}"#, + )); + assert_eq!( + (full.score, full.reason), + (Some(0.5), Some("half".to_string())) + ); + assert!(full.checks.is_some()); + assert!(!full.not_applicable); + + // `comment` as the rationale, which is what a scorer written for LangSmith or Langfuse + // returns. Read rather than dropped, since the number arrives either way. + assert_eq!( + extract_verdict(&raw(r#"{"score": 1, "comment": "fine"}"#)) + .reason + .as_deref(), + Some("fine") + ); + } + + /// A score is a fraction: anything outside 0..=1 (a scorer that returned a count, say) is + /// recorded as an error naming the value rather than plotted as a bogus point. + #[test] + fn an_out_of_range_score_is_recorded_as_an_error_not_a_value() { + // In range: recorded as the score it is. + let (v, e) = settle_verdict(Some(&raw("0.5")), Some("success"), None).unwrap(); + assert_eq!(v.score, Some(0.5)); + assert!(e.is_none()); + // Out of range (a scorer returning a count, say): no score, an error naming the value. + let (v, e) = settle_verdict(Some(&raw("100")), Some("success"), None).unwrap(); + assert_eq!(v.score, None); + assert!(e.unwrap().contains("100")); + let (v, _) = settle_verdict(Some(&raw("-5")), Some("success"), None).unwrap(); + assert_eq!(v.score, None); + // No result at all once the iteration is over: an error, not a cell pending forever. + let (v, e) = settle_verdict(None, Some("success"), None).unwrap(); + assert_eq!(v.score, None); + assert!(e.is_some()); + // Still running: nothing to settle yet. + assert!(settle_verdict(None, None, None).is_none()); + } + + /// A scorer saying it has nothing to measure on a case is a verdict rather than a failure: the + /// cell is left out of the mean instead of counted as a zero. Spelled out, so a scorer that + /// returns nothing at all is still an error rather than silently excused. + #[test] + fn an_explicit_null_score_is_not_applicable_rather_than_missing() { + let na = extract_verdict(&raw(r#"{"score": null, "reason": "no sources to cite"}"#)); + assert!(na.not_applicable); + assert_eq!(na.score, None); + assert_eq!(na.reason.as_deref(), Some("no sources to cite")); + + // Through a judge's wrapper, as any other verdict is. + assert!(extract_verdict(&raw(r#"{"output": {"score": null}}"#)).not_applicable); + assert!(extract_verdict(&raw(r#"{"output": "{\"score\": null}"}"#)).not_applicable); + + // Not the same as a scorer that returned nothing, or an object with no verdict in it. + assert!(!extract_verdict(&raw("null")).not_applicable); + assert!(!extract_verdict(&raw(r#"{"verdict": "good"}"#)).not_applicable); + } +} diff --git a/backend/windmill-api/src/ai_evals/subject.rs b/backend/windmill-api/src/ai_evals/subject.rs new file mode 100644 index 0000000000..52b8ff0519 --- /dev/null +++ b/backend/windmill-api/src/ai_evals/subject.rs @@ -0,0 +1,135 @@ +use super::*; + +/// What a run is executed against. Kept as `(kind, path, version)` rather than a bare agent +/// path so flow-scoped evaluation is a later superset instead of a rewrite. +#[derive(Serialize, Deserialize, Debug, Clone)] +pub struct EvalSubject { + #[serde(default = "default_subject_kind")] + pub kind: EvalSubjectKind, + /// The agent resource under test. + pub path: String, + /// Which version of the agent, counted per path: how many times it had been saved. The + /// request's to choose for a pinned run, and otherwise the version the run was enqueued + /// against. + #[serde(default, skip_serializing_if = "Option::is_none")] + pub version: Option, + /// The agent's unsaved edits, as the editor holds them. Present exactly when `kind` is + /// `agent_draft`, since the edits exist nowhere else. + #[serde(default, skip_serializing_if = "Option::is_none")] + pub draft: Option, + /// Hash of that configuration. A draft moves without the version moving, so this is the only + /// thing that can say a run describes an agent that has since been edited. Stamped + /// server-side. + #[serde(default, skip_serializing_if = "Option::is_none")] + pub draft_hash: Option, +} + +/// Key order is not meaningful and `serde_json` preserves insertion order here, so it is sorted +/// away before hashing: the same configuration must hash the same however it was assembled. +fn canonical_json(value: &serde_json::Value) -> String { + match value { + serde_json::Value::Object(map) => { + let sorted = map + .iter() + .collect::>() + .into_iter() + .map(|(k, v)| { + format!( + "{}:{}", + serde_json::to_string(k).unwrap_or_default(), + canonical_json(v) + ) + }) + .collect::>() + .join(","); + format!("{{{}}}", sorted) + } + serde_json::Value::Array(items) => format!( + "[{}]", + items + .iter() + .map(canonical_json) + .collect::>() + .join(",") + ), + other => other.to_string(), + } +} + +pub(crate) fn draft_hash(draft: &AgentDraft) -> String { + use sha2::{Digest, Sha256}; + let mut hasher = Sha256::new(); + hasher.update(canonical_json(&draft.input_transforms).as_bytes()); + hasher.update(b"|"); + hasher.update(canonical_json(&serde_json::Value::Array(draft.tools.clone())).as_bytes()); + hex::encode(hasher.finalize())[..32].to_string() +} + +fn default_subject_kind() -> EvalSubjectKind { + EvalSubjectKind::Agent +} + +#[derive(Serialize, Deserialize, Debug, Clone, PartialEq)] +#[serde(rename_all = "snake_case")] +pub enum EvalSubjectKind { + Agent, + /// A saved agent's unsaved edits, carried by the request and inlined: a linked step resolves + /// the resource live and so would run what the edits replace. + AgentDraft, + /// One past version of a saved agent, inlined for the same reason. `version` says which, and + /// it is the request's to choose rather than the server's. + AgentVersion, +} + +/// The brain and tools of an agent, as the flow editor holds them. +#[derive(Serialize, Deserialize, Debug, Clone)] +pub struct AgentDraft { + /// The agent's input transforms: provider, system prompt, output type and the rest. The + /// message and attachments are supplied by the case and override anything named here. + #[serde(default)] + pub input_transforms: serde_json::Value, + #[serde(default)] + pub tools: Vec, +} + +impl EvalSubject { + /// What is recorded of a subject: enough to say what ran, without the configuration itself. + pub(crate) fn stamp(&self) -> EvalSubject { + EvalSubject { + kind: self.kind.clone(), + path: self.path.clone(), + version: self.version, + draft: None, + // Only ever derived from the draft this request carries: a hash the client supplies on + // its own could relabel a run as the deployed version. + draft_hash: self.draft.as_ref().map(draft_hash), + } + } +} + +#[derive(Deserialize)] +pub struct SubjectStateQuery { + pub path: String, +} + +#[derive(Serialize)] +pub struct SubjectState { + /// The version the agent is on now. + #[serde(skip_serializing_if = "Option::is_none")] + pub version: Option, +} + +/// The version the agent is deployed at. Small on purpose: the results endpoint reports the same +/// thing, but it harvests scores and reads every job to do it. +pub async fn subject_state( + authed: ApiAuthed, + Extension(user_db): Extension, + Path(w_id): Path, + Query(query): Query, +) -> JsonResult { + let Some((_, version)) = readable_agent_state(&authed, &user_db, &w_id, &query.path).await? + else { + return Err(Error::NotFound(format!("Agent {} not found", query.path))); + }; + Ok(Json(SubjectState { version: Some(version) })) +} diff --git a/backend/windmill-api/src/ai_evals/template.rs b/backend/windmill-api/src/ai_evals/template.rs new file mode 100644 index 0000000000..22d37d1e85 --- /dev/null +++ b/backend/windmill-api/src/ai_evals/template.rs @@ -0,0 +1,132 @@ +use super::*; + +/// What a script scorer starts from. +pub const SCORER_SCRIPT_TEMPLATE: &str = r#"// A scorer receives one run and returns a number between 0 and 1, a boolean, or +// { score, reason, checks } — checks show up in the case detail. +// Return { score: null } for a case this scorer has nothing to measure on: the cell +// is left out of the column's mean and pass rate rather than counted as a zero. +// +// The run is also handed to you spelled out, so a short scorer can skip the type below +// entirely: export async function main(output: unknown, expected: unknown) { ... } +type ToolCall = { + name: string + args?: Record + result?: unknown + error?: string + duration_ms?: number + truncated?: boolean +} + +type EvalRun = { + input: { user_message?: string; user_attachments?: unknown[] } + output?: unknown + expected?: unknown + tool_calls: ToolCall[] + tools: { name: string; schema?: Record }[] + metrics: { steps: number; duration_ms?: number; usage?: Record } + status: string + job_id: string +} + +export async function main(run: EvalRun) { + // How the agent got to its answer. Reported rather than scored: checks render in the case + // detail either way, so they explain the number without being averaged into it. + const checks = [ + check('arguments match the schema', args_schema_valid(run)), + check('no repeated calls', no_repeated_calls(run)), + check('no failed tool calls', no_step_errors(run)), + check('under 6 steps', run.metrics.steps <= 6, `${run.metrics.steps} steps`), + check('under 30 seconds', under_ms(run, 30_000), `${run.metrics.duration_ms ?? '?'} ms`) + ] + + // Nothing to compare the answer against, so this column has no verdict on this case rather + // than a failing one. The cell reads n/a and the column's mean is of the cases it measured. + if (run.expected == undefined) { + return { score: null, reason: 'this case has no expected answer', checks } + } + + // One question per column, and this column's question is whether the answer is right. + // Deliberately not the share of checks above that passed: a right answer that was slow and a + // wrong answer that was fast would score the same, and the column could not say which it was. + const correct = contains(run.output, text(run.expected)) + return { + score: correct ? 1 : 0, + reason: correct ? undefined : `expected ${text(run.expected)}`, + checks + } +} + +// Helpers. Edit or delete freely. + +function check(name: string, passed: boolean, detail?: string) { + return { name, passed, detail } +} + +function text(value: unknown): string { + return typeof value === 'string' ? value : JSON.stringify(value ?? '') +} + +function contains(output: unknown, needle: string): boolean { + return needle.trim().length > 0 && text(output).toLowerCase().includes(needle.trim().toLowerCase()) +} + +// Every call validated against the schema of the tool it called. A tool whose schema could not be +// resolved is not checked rather than failed. +function args_schema_valid(run: EvalRun): boolean { + return run.tool_calls.every((call) => { + const schema = run.tools.find((tool) => tool.name === call.name)?.schema as + | { properties?: Record; required?: string[] } + | undefined + if (!schema?.properties) return true + const args = call.args ?? {} + for (const key of schema.required ?? []) { + if (args[key] === undefined || args[key] === null) return false + } + for (const [key, value] of Object.entries(args)) { + const expected = schema.properties[key]?.type + if (!expected) continue + const actual = Array.isArray(value) ? 'array' : value === null ? 'null' : typeof value + if (expected === 'integer' ? !Number.isInteger(value) : expected !== actual) return false + } + return true + }) +} + +// The same tool called twice with the same arguments. +function no_repeated_calls(run: EvalRun): boolean { + const seen = new Set() + for (const call of run.tool_calls) { + const key = `${call.name}:${JSON.stringify(call.args ?? {})}` + if (seen.has(key)) return false + seen.add(key) + } + return true +} + +function no_step_errors(run: EvalRun): boolean { + return run.status === 'success' && run.tool_calls.every((call) => !call.error) +} + +// A run with no recorded duration is not under the limit: a check that could not be evaluated +// should not report as one that passed. +function under_ms(run: EvalRun, max: number): boolean { + const ms = run.metrics.duration_ms + return ms != undefined && ms <= max +} +"#; + +#[derive(Serialize)] +pub struct ScorerDefaults { + /// The system prompt a judge agent is created with. It lives on that agent afterwards. + pub judge_prompt: String, + /// The starting point for a script scorer, held here so the shape a scorer is handed and the + /// template that reads it cannot drift apart. + pub script_template: String, +} + +pub async fn scorer_defaults() -> JsonResult { + Ok(Json(ScorerDefaults { + judge_prompt: JUDGE_SYSTEM_PROMPT.to_string(), + script_template: SCORER_SCRIPT_TEMPLATE.to_string(), + })) +} diff --git a/backend/windmill-api/src/lib.rs b/backend/windmill-api/src/lib.rs index 9963219993..de49949043 100644 --- a/backend/windmill-api/src/lib.rs +++ b/backend/windmill-api/src/lib.rs @@ -113,6 +113,7 @@ pub mod storage_list_ee; mod storage_list_oss; mod workspace_dependencies; +mod ai_evals; mod approvals; #[cfg(all(feature = "enterprise", feature = "private"))] pub mod apps_ee; @@ -674,6 +675,7 @@ pub async fn run_server( .route("/labels/list", get(list_workspace_labels)) .nest("/job_metrics", job_metrics::workspaced_service()) .nest("/job_helpers", job_helpers_service) + .nest("/ai_evals", ai_evals::workspaced_service()) .nest("/jobs", jobs::workspaced_service()) .nest("/debug", windmill_api_debug::workspaced_service()) .nest("/native_triggers", { diff --git a/backend/windmill-api/src/offboarding.rs b/backend/windmill-api/src/offboarding.rs index 0d9529c0ce..a69441ea75 100644 --- a/backend/windmill-api/src/offboarding.rs +++ b/backend/windmill-api/src/offboarding.rs @@ -52,6 +52,8 @@ struct OffboardAffectedPaths { variables: Vec, #[serde(skip_serializing_if = "Vec::is_empty")] schedules: Vec, + #[serde(skip_serializing_if = "Vec::is_empty")] + eval_datasets: Vec, #[serde(skip_serializing_if = "HashMap::is_empty")] triggers: HashMap>, } @@ -91,6 +93,7 @@ struct OffboardSummary { flows_reassigned: i64, apps_reassigned: i64, resources_reassigned: i64, + eval_datasets_reassigned: i64, variables_reassigned: i64, schedules_reassigned: i64, triggers_reassigned: i64, @@ -168,6 +171,14 @@ async fn get_offboard_preview( .fetch_all(db) .await?; + let eval_datasets = sqlx::query_scalar!( + "SELECT path FROM eval_dataset WHERE path LIKE $1 AND workspace_id = $2", + &user_prefix, + w_id + ) + .fetch_all(db) + .await?; + let variables = sqlx::query_scalar!( "SELECT path FROM variable WHERE path LIKE $1 AND workspace_id = $2", &user_prefix, @@ -285,6 +296,16 @@ async fn get_offboard_preview( &ref_pattern, &user_prefix, w_id ).fetch_all(db).await?; + let ref_eval_datasets = sqlx::query_scalar!( + "SELECT path FROM eval_dataset + WHERE scorers::text LIKE $1 AND NOT path LIKE $2 AND workspace_id = $3", + &ref_pattern, + &user_prefix, + w_id + ) + .fetch_all(db) + .await?; + let ref_resources = sqlx::query_scalar!( "SELECT DISTINCT path FROM resource WHERE value::text LIKE $1 AND NOT path LIKE $2 AND workspace_id = $3", &ref_pattern, &user_prefix, w_id @@ -317,6 +338,7 @@ async fn get_offboard_preview( resources, variables, schedules, + eval_datasets, triggers, }, executing_on_behalf: OffboardAffectedPaths { @@ -332,6 +354,7 @@ async fn get_offboard_preview( flows: ref_flows, apps: ref_apps, resources: ref_resources, + eval_datasets: ref_eval_datasets, ..Default::default() }, tokens, @@ -548,6 +571,7 @@ pub(crate) async fn offboard_global_user( flows_reassigned: 0, apps_reassigned: 0, resources_reassigned: 0, + eval_datasets_reassigned: 0, variables_reassigned: 0, schedules_reassigned: 0, triggers_reassigned: 0, @@ -574,6 +598,7 @@ pub(crate) async fn offboard_global_user( total_summary.flows_reassigned += ws_summary.flows_reassigned; total_summary.apps_reassigned += ws_summary.apps_reassigned; total_summary.resources_reassigned += ws_summary.resources_reassigned; + total_summary.eval_datasets_reassigned += ws_summary.eval_datasets_reassigned; total_summary.variables_reassigned += ws_summary.variables_reassigned; total_summary.schedules_reassigned += ws_summary.schedules_reassigned; total_summary.triggers_reassigned += ws_summary.triggers_reassigned; @@ -746,6 +771,7 @@ async fn check_path_conflicts( "flow", "app", "resource", + "eval_dataset", "variable", "schedule", "http_trigger", @@ -959,6 +985,47 @@ async fn offboard_user_from_workspace<'c>( .await? .unwrap_or(0); + // ---- eval datasets ---- + // The foreign keys cascade the rename onto cases and experiments; the paths held inside JSONB + // (an experiment's subject, a dataset's scorers) are rewritten separately since the cascade + // cannot reach them and those runnables move with the user. + let eval_datasets_reassigned = sqlx::query_scalar!( + r#"WITH updated AS ( + UPDATE eval_dataset SET path = REGEXP_REPLACE(path, 'u/' || $2 || '/(.*)', $1 || '/\1') + WHERE path LIKE ('u/' || $2 || '/%') AND workspace_id = $3 + RETURNING 1 + ) SELECT COUNT(*) FROM updated"#, + &new_prefix, + username, + w_id + ) + .fetch_one(&mut **tx) + .await? + .unwrap_or(0); + sqlx::query!( + r#"UPDATE eval_experiment SET subject = jsonb_set(subject, '{path}', to_jsonb(REGEXP_REPLACE(subject->>'path', 'u/' || $2 || '/(.*)', $1 || '/\1'))) WHERE subject->>'path' LIKE ('u/' || $2 || '/%') AND workspace_id = $3"#, + &new_prefix, + username, + w_id + ) + .execute(&mut **tx) + .await?; + sqlx::query!( + r#"UPDATE eval_dataset SET scorers = COALESCE(( + SELECT jsonb_agg( + CASE WHEN elem->>'path' LIKE ('u/' || $2 || '/%') + THEN jsonb_set(elem, '{path}', to_jsonb(REGEXP_REPLACE(elem->>'path', 'u/' || $2 || '/(.*)', $1 || '/\1'))) + ELSE elem END) + FROM jsonb_array_elements(scorers) elem), '[]'::jsonb) + WHERE workspace_id = $3 + AND EXISTS (SELECT 1 FROM jsonb_array_elements(scorers) e WHERE e->>'path' LIKE ('u/' || $2 || '/%'))"#, + &new_prefix, + username, + w_id + ) + .execute(&mut **tx) + .await?; + // ---- variables (with Vault secret handling) ---- let old_var_prefix = format!("u/{}/", username); let new_var_prefix = format!("{}/", reassign_to); @@ -1152,6 +1219,7 @@ async fn offboard_user_from_workspace<'c>( flows_reassigned, apps_reassigned, resources_reassigned, + eval_datasets_reassigned, variables_reassigned, schedules_reassigned, triggers_reassigned, diff --git a/backend/windmill-api/src/token.rs b/backend/windmill-api/src/token.rs index fc808767ec..2268a63894 100644 --- a/backend/windmill-api/src/token.rs +++ b/backend/windmill-api/src/token.rs @@ -100,6 +100,12 @@ fn build_standard_scope_domains() -> Vec { ("oauth", "OAuth", "OAuth management", false), ("ai", "AI", "AI feature management", false), ("ai_skills", "AI Skills", "AI skill management", false), + ( + "ai_evals", + "AI Evals", + "AI agent eval datasets and standalone runs", + false, + ), ( "agent_workers", "Agent Workers", diff --git a/backend/windmill-api/src/users.rs b/backend/windmill-api/src/users.rs index 37946f68a4..8804004f2b 100644 --- a/backend/windmill-api/src/users.rs +++ b/backend/windmill-api/src/users.rs @@ -373,6 +373,87 @@ async fn update_username_in_workpsace<'c>( .execute(&mut **tx) .await?; + // Eval datasets are path-addressed like every other object, so a username change moves them + // too. The foreign keys cascade the rename onto their cases and experiments; the experiment + // subject (the agent a run was of) is a `u//` path of its own inside JSONB, so it is + // rewritten separately or a user's own runs would detach from their renamed agent. + sqlx::query!( + r#"UPDATE eval_dataset SET path = REGEXP_REPLACE(path,'u/' || $2 || '/(.*)','u/' || $1 || '/\1') WHERE path LIKE ('u/' || $2 || '/%') AND workspace_id = $3"#, + new_username, + old_username, + w_id + ).execute(&mut **tx) + .await?; + + sqlx::query!( + "UPDATE eval_dataset SET extra_perms = extra_perms - ('u/' || $2) || jsonb_build_object(('u/' || $1), extra_perms->('u/' || $2)) WHERE extra_perms ? ('u/' || $2) AND workspace_id = $3", + new_username, + old_username, + w_id + ) + .execute(&mut **tx) + .await?; + + sqlx::query!( + r#"UPDATE eval_experiment SET subject = jsonb_set(subject, '{path}', to_jsonb(REGEXP_REPLACE(subject->>'path','u/' || $2 || '/(.*)','u/' || $1 || '/\1'))) WHERE subject->>'path' LIKE ('u/' || $2 || '/%') AND workspace_id = $3"#, + new_username, + old_username, + w_id + ).execute(&mut **tx) + .await?; + + sqlx::query!( + "UPDATE eval_dataset SET created_by = $1 WHERE created_by = $2 AND workspace_id = $3", + new_username, + old_username, + w_id + ) + .execute(&mut **tx) + .await?; + sqlx::query!( + "UPDATE eval_dataset SET edited_by = $1 WHERE edited_by = $2 AND workspace_id = $3", + new_username, + old_username, + w_id + ) + .execute(&mut **tx) + .await?; + sqlx::query!( + "UPDATE eval_case SET created_by = $1 WHERE created_by = $2 AND workspace_id = $3", + new_username, + old_username, + w_id + ) + .execute(&mut **tx) + .await?; + sqlx::query!( + "UPDATE eval_experiment SET created_by = $1 WHERE created_by = $2 AND workspace_id = $3", + new_username, + old_username, + w_id + ) + .execute(&mut **tx) + .await?; + + // A dataset's scorers name scripts and agents by path in a JSONB array, which the rewrites + // above do not reach; those runnables are renamed elsewhere in this transaction, so each + // scorer path under the old username is rewritten too or the dataset points at a runnable that + // no longer exists. + sqlx::query!( + r#"UPDATE eval_dataset SET scorers = COALESCE(( + SELECT jsonb_agg( + CASE WHEN elem->>'path' LIKE ('u/' || $2 || '/%') + THEN jsonb_set(elem, '{path}', to_jsonb(REGEXP_REPLACE(elem->>'path','u/' || $2 || '/(.*)','u/' || $1 || '/\1'))) + ELSE elem END) + FROM jsonb_array_elements(scorers) elem), '[]'::jsonb) + WHERE workspace_id = $3 + AND EXISTS (SELECT 1 FROM jsonb_array_elements(scorers) e WHERE e->>'path' LIKE ('u/' || $2 || '/%'))"#, + new_username, + old_username, + w_id + ).execute(&mut **tx) + .await?; + // ---- variables ---- // Handle Vault secret renames before updating paths in DB diff --git a/backend/windmill-store/src/resources.rs b/backend/windmill-store/src/resources.rs index 1a051db8fe..b107d34be4 100644 --- a/backend/windmill-store/src/resources.rs +++ b/backend/windmill-store/src/resources.rs @@ -81,11 +81,8 @@ pub fn workspaced_service() -> Router { "/history/p/{*path}", get(get_resource_history).delete(clear_resource_history), ) - .route("/history/v/{version}", get(get_resource_version)) - .route( - "/history/restore/v/{version}", - post(restore_resource_version), - ) + .route("/history/v/{id}", get(get_resource_version)) + .route("/history/restore/v/{id}", post(restore_resource_version)) .route("/delete/{*path}", delete(delete_resource)) .route("/delete_bulk", delete(delete_resources_bulk)) .route("/create", post(create_resource)) @@ -2200,7 +2197,9 @@ async fn set_resource_value( #[derive(Serialize)] struct ResourceVersion { + /// Addresses a version; `version` is the per-resource number it is presented by. id: i64, + version: i64, created_at: chrono::DateTime, created_by: Option, } @@ -2208,6 +2207,7 @@ struct ResourceVersion { #[derive(Serialize)] struct ResourceVersionWithValue { id: i64, + version: i64, created_at: chrono::DateTime, created_by: Option, value: Option, @@ -2243,7 +2243,7 @@ async fn get_resource_history( let versions = sqlx::query_as!( ResourceVersion, - "SELECT id, created_at, created_by FROM resource_version + "SELECT id, version, created_at, created_by FROM resource_version WHERE workspace_id = $1 AND path = $2 ORDER BY id DESC LIMIT $3", w_id, path, @@ -2357,19 +2357,19 @@ async fn missing_references( async fn get_resource_version( authed: ApiAuthed, Extension(user_db): Extension, - Path((w_id, version)): Path<(String, i64)>, + Path((w_id, id)): Path<(String, i64)>, ) -> JsonResult { let mut tx = user_db.begin(&authed).await?; let row = sqlx::query!( - "SELECT id, path, created_at, created_by, value FROM resource_version + "SELECT id, version, path, created_at, created_by, value FROM resource_version WHERE workspace_id = $1 AND id = $2", w_id, - version + id ) .fetch_optional(&mut *tx) .await?; - let row = not_found_if_none(row, "ResourceVersion", version.to_string())?; + let row = not_found_if_none(row, "ResourceVersion", id.to_string())?; check_scopes(&authed, || format!("resources:read:{}", row.path))?; let missing = missing_references(&mut tx, &w_id, row.value.as_ref()).await?; @@ -2377,6 +2377,7 @@ async fn get_resource_version( Ok(Json(ResourceVersionWithValue { id: row.id, + version: row.version, created_at: row.created_at, created_by: row.created_by, value: row.value, @@ -2455,17 +2456,17 @@ async fn restore_resource_version( Extension(db): Extension, Extension(user_db): Extension, Extension(webhook): Extension, - Path((w_id, version)): Path<(String, i64)>, + Path((w_id, id)): Path<(String, i64)>, ) -> Result { let mut tx = user_db.clone().begin(&authed).await?; let row = sqlx::query!( - "SELECT path, value FROM resource_version WHERE workspace_id = $1 AND id = $2", + "SELECT path, value, version FROM resource_version WHERE workspace_id = $1 AND id = $2", w_id, - version + id ) .fetch_optional(&mut *tx) .await?; - let row = not_found_if_none(row, "ResourceVersion", version.to_string())?; + let row = not_found_if_none(row, "ResourceVersion", id.to_string())?; tx.commit().await?; check_scopes(&authed, || format!("resources:write:{}", row.path))?; @@ -2486,7 +2487,7 @@ async fn restore_resource_version( Ok(format!( "resource {} restored to version {}", - row.path, version + row.path, row.version )) } diff --git a/backend/windmill-worker/src/ai_executor.rs b/backend/windmill-worker/src/ai_executor.rs index b11ecebc6c..d948fee088 100644 --- a/backend/windmill-worker/src/ai_executor.rs +++ b/backend/windmill-worker/src/ai_executor.rs @@ -451,9 +451,9 @@ pub async fn handle_ai_agent_job( (args, tools) } else { let args = serde_json::from_str::(&serde_json::to_string(&local_args)?)?; - // "Edit" on a linked step clears `agent` but keeps the host's `tool_inputs` until Save or - // Cancel folds them back, so overlay them here too: a flow persisted mid-edit must still - // bind its tools to this flow's context rather than the agent author's. + // "Edit" on a linked step clears `agent` but keeps the host's `tool_inputs`, so overlay them + // here too: a flow persisted mid-edit must still bind its tools to this flow's context + // rather than the agent author's. let mut tools = module_tools; overlay_tool_inputs(&mut tools, &tool_inputs); (args, tools) diff --git a/docs/ai-agent-evals.md b/docs/ai-agent-evals.md new file mode 100644 index 0000000000..53d6ccf272 --- /dev/null +++ b/docs/ai-agent-evals.md @@ -0,0 +1,482 @@ +# AI agent evals + +A reusable AI agent (`docs/reusable-ai-agents.md`) can be run on its own, against a curated set +of **cases** — the inputs it is expected to keep handling. + +Three words, and no fourth: + +- a **case** is one input the agent should handle, held in a **dataset**; +- a **run** (stored as an **experiment**) is one execution of a whole dataset: a single flow job + that answers every case, which is what the UI labels "Run N"; +- each case is answered as one **iteration** of that run. + +The surface is a dialog, two screens deep: **this agent's runs** across every dataset it has been +measured on (one row per run, a badge per scorer), and, on opening a run, **one table** — a row per +case, a column per scorer, the cell being that scorer's verdict — with a case's detail beside it. +Editing a dataset is a drawer over both. It opens from where an agent already is: the agent card at +the top of an AI agent step's inputs in the flow editor, and the `ai_agent` row on `/resources`. + +**Evals belong to a saved agent.** A dataset and its runs hang off an `ai_agent` resource, so they +outlive the step being renamed, copied or deleted, and two runs are comparable because they name +the same thing. A step whose agent is written inline has nothing to hang them on: what stands in +its place is **Save as reusable agent**, which is the only setup evals ask for. + +## What runs + +**A run is one flow**: a loop over the dataset's cases, each iteration answering its case and then +scoring the answer. Pushed as a `RawFlow`, so the agent step is the same vehicle +`ModuleTest.svelte` uses to test an agent step and a case exercises the production branch of +`ai_executor.rs` rather than a parallel one. + +One flow rather than one job per case, because **a run outlives the tab that started it**: a +dataset of two hundred cases against a slow provider takes long enough that nobody watches it, and +only a worker can notice that the last case finished. Scoring is therefore a step, not something +the client does afterwards, and the run is one thing to watch, cancel, or point a schedule at. Each +iteration is read back by node id (`get_result_and_success_by_id_from_flow`), so an answer is +fetched without walking the loop's status. + +The loop is `parallel` with a bounded `parallelism` and `skip_failures`: a dataset is a burst of +calls to one provider, and one case failing is one cell of the run rather than the end of it. + +The cases are the loop's **static iterator**, so they live in the flow's value, which is stored +once. Passing them as an argument would put a copy of the whole dataset in every iteration's +arguments. + +Whichever state of the agent is chosen — what is deployed, the edits in progress, or a past version +— its configuration is fixed once, when the run is opened, and inlined into the step every case +runs. A linked step would resolve the resource when each case reaches it, so a deploy part-way +through a run would be executed by the cases after it while every row still named the version the +run started against. One run measures one configuration; the cost is that a run does not exercise +the linked branch the production step takes. The edits and a past version could not be run any +other way: a reference resolves to what is deployed, which is exactly what neither of them is. + +A saved agent's and a past version's configuration are never taken from the request: both are read +from the workspace by the path they name, and a subject carrying one is refused. The edits in +progress are the one kind the request has to carry — they exist only in the editor — and the run +records what it was handed, inlined into its flow and hashed, so it is reproducible and +attributable to "this version plus these edits"; what the server cannot assert about them is that +they derive from that version. + +Each iteration is an agent step, then a **payload step**, then one step per scorer. The payload +step exists because the flow cannot see what it needs to: the agent's own result carries the +answer and every message, but each tool call's arguments, result, status, duration and schema +belong to the job that ran it. The step reads them back through `GET /ai_evals/run_payload`, whose +one argument is the iteration's own job id, and hands the scorers exactly what a scorer receives +anywhere else. A scorer therefore measures the agent's latency and not its own: the payload +reports the *agent step's* duration, never the iteration's. + +The edits are the transforms as authored, expressions included. One that reads `results..x` +or a `flow_input` the case does not supply resolves to nothing here, the same way it would in any +run of that step outside its flow. + +A linked agent is not fully self-contained: a host flow can override its tools' inputs through +the step's `tool_inputs`. A run does not reproduce that wiring — the agent runs with its own +authored defaults — so an agent whose behaviour depends on one flow's overrides is measured here +without them. + +A case carries **no conversation**: one question and the answer it should produce, so a run starts +from the agent's own memory configuration and nothing is replayed into it. + +## Where results live + +Results are jobs. A run's logs, trajectory, tool-call child jobs, permissions and retention are +already `v2_job` / `v2_job_completed` and the flow status's `agent_actions`, and none of that is +stored a second time. + +What the table itself is made of is the exception: each cell's answer, its outcome and every +scorer's verdict are copied into the run's own rows the first time they can be read. Jobs have +their own retention, and a recorded run is meant to still read as the run it was long after the +jobs that produced it are gone. + +The pane shows the **answer** and nothing else of a job; the trajectory is the run page's, and +`job_id` is the way there. For a recorded row the answer is read off the row rather than out of +the job, because `job_id` is the whole iteration — the agent and then the scorers that measured it +— and its result is the last scorer's verdict, not the answer. + +What makes a job findable again is stamped on it at push: + +- `runnable_path` is the agent's own path, so the existing `script_path_start` job filter + answers "every run of this agent" with no new state. +- `_eval` in the flow's args records `{subject: {kind, path, version}, dataset, experiment_id}`, + and every iteration inherits it, so a job opened cold from the runs page explains itself. Which + case an iteration ran is in its own `iter.value`, which is also how a cell finds its job again. + Extra flow inputs are inert — the agent step reads only `user_message`/`user_attachments`. + +## Versioning + +`subject.version` is the agent's version number: how many times it has been saved. It is counted +per resource rather than read off `resource_version.id`, which is one identity sequence for the +whole table — an agent saved nine times reads v4 … v24 under it, and the gaps count writes in +workspaces the reader cannot see. The id stays how a version is addressed, by the history routes +and by restore; the number is what a version is called, and what runs are named and compared by. + +The number is stored on the row rather than counted when read, because both ways of deleting +versions take the oldest: the monitor's trim past `MAX_RESOURCE_VERSIONS`, and clearing a history +down to its current value. Counting the survivors would renumber under either, so a run recorded +against v3 would later name a different version. + +For an `agent` run the version names the configuration the run read when it opened, which is the +one every case executes. Pinning an *older* version is a subject kind of its own — an +`agent_version` run says which version to read, where an `agent` run reads whatever is deployed at +the moment it starts. + +A version captures the resource, not its transitive closure. Two byte-identical versions can +behave differently because a `$var:`/`$res:` they reference changed underneath them, so a +recorded version is necessary for attribution but not sufficient. + +## Experiments + +Running a dataset produces an experiment: every case executed against one subject, with a row per +case. The experiment records the **exact case set it ran**, by value — a dataset keeps changing, +and a result set that cannot say which inputs produced it is not reproducible. + +### What a run is called + +An experiment is `Run N`: `run_number` is allocated per `(dataset, agent path)` when the run is +opened, once, and never reused. Stored rather than counted at read time, so a run keeps the name it +was given as history is pruned around it. There is no user-given label. + +Numbering is per agent and not per subject kind, so runs of what is deployed and runs of the edits +on top of it share one sequence: "Run 7" means one thing, and which of the two ran it is what the +run says beside its number. + +### A run is permanent + +Every run is written once and then only ever read: there is no writable experiment, no partial +rerun, and no cell that can be edited after the fact. A run in which some cells came from one +version and some from another would not be worth comparing, and running the dataset is the only +way a run appears. + +- **One experiment holds one subject, and one agent keeps one history.** The experiment list is + filtered to the agent the pane was opened on, across both kinds, so a dataset shared by two + agents never shows one agent's runs when the other is opened. +- **A version is per cell** (`eval_experiment_case.subject_version`), not per experiment. The + subject is resolved once when the run is opened and every cell is stamped from it, so the column + is uniform today; it is per cell so that a run which one day executes cell by cell can say so + rather than averaging two versions silently. +- **Edits are dated by their hash**, not by a version, because editing moves nothing a version + could record. Each cell carries `subject_draft_hash`, the hash of the configuration it ran + (canonicalised: key order is not meaningful and `serde_json` preserves insertion order). +- **An agent's unsaved edits are their own subject.** Their runs are keyed under `agent_draft`, so + a number produced by edits is never quietly read as the deployed agent's. The run dialog offers + them only when it was opened from the editing card, preselected there; from anywhere else the + agent is what is deployed. + +The table asks what version the agent is on when it opens and whenever the tab regains focus — a +small `subject_state` read — rather than polling for it; the results endpoint reports the same +version but collects the run as it goes, so it is polled only while a run is in flight, one pass +at a time. An agent saved in another tab while this one stays focused is noticed on the next focus +or the next run. + +A run of an older version is history and says so (`Run 14 · v23` beside an agent on v24); nothing +flags it, since that would flag every past run the moment anything is deployed. A run whose edits +were later deployed is a run of that version, and the results endpoint recognises and restamps it +(see "What a run says it ran"). The hash each run carries is recorded, not shown. + +In the flow editor, editing a linked agent forks the configuration into the step and clears the +link; the step is the only copy of the edits until Save changes, Cancel or Discard +(`docs/reusable-ai-agents.md`). Evals open from the agent card in both of its states: from the +editing card they run the edits as the step holds them when Run is pressed; from the linked card +they run the deployed agent, the same reading a linked step makes at run time. The agent's own +resource draft — the one the resource editor writes — is never read by evals. The card also names +the version a run is recorded against (`v24`, and `v24` beside an unsaved-changes badge for edits +on top of it), read from the resource's newest history entry since the resource itself does not +carry its version. + +## Scoring + +Scoring is not a second act with a button of its own: **Run** produces an answer and then scores +it. Each iteration of the run's flow scores its own answer as a step, and the numbers are harvested +into rows when results are read. + +A run's cells are therefore measured by the scorers as they stood when it ran, and never again. +A scorer edited or added afterwards has no cell in the runs that predate it: rescoring a run in +place would make a permanent run editable. The one thing read through the present is the pass line +— `pass_if` is applied when a score is read, so moving it re-reads every run with no model call. + +The columns themselves are the dataset's current scorers, so the table stays comparable across the +runs it lists rather than growing a column per run. Removing a scorer therefore takes its column +off the runs already recorded as well: the rows it produced are not deleted, but nothing renders +them, and adding the scorer back mints a new column that fills from the next run on. The removal +asks first, and says that. + +Two things are deliberately absent. Rescoring stored answers under edited scorers would need a run +of its own that reuses a parent run's answers and is attributed to the version that produced them, +so it never reads as the agent having answered again. A result cache keyed on (agent configuration, +case, scorer definition) would assert the agent is deterministic, which it is not, so it has to be +an explicit choice with its own answer to what a run means when half of it was computed last week. + +### A score is a number, and optionally a line through it + +Every scorer returns a number **between 0 and 1** — both templates say so, and the mean and the +pass rate read it as a fraction; a scorer returning anything outside that range has its result +recorded as an error rather than counted, and a `pass_if` threshold is held to the same range. Pass +or fail is not a second kind of score: a column carries an optional `pass_if`, and a case scoring at +or above it counts as a pass. A boolean scorer is one that returns 0 or 1 with the line at 0.5. + +A column with a threshold reports a **pass rate** beside its mean and marks each cell; a column +without one is a plain number and is not dressed up as a verdict. + +The line is deliberately outside the score's **definition** hash: where it sits is an +interpretation of a score rather than part of producing it, so moving it re-reads every run already +recorded with nothing re-run. It is set when the column is added and changed later under **Scorer +settings** in the dataset drawer, which is also where the column is named. + +The name is this dataset's own name for the scorer, seeded from the summary given when it was +added. It is a copy, not a link: the script or judge agent keeps whatever it is called, so renaming +a column here does not rename anything a second dataset shows. Reading it live from the runnable +would cost a fetch per column and leave a column blank for anyone who cannot read what it points +at. + +### What a scorer receives + +An agent is judged on its behaviour, so the final answer is the smaller half of the evidence. Every +scorer — a judge prompt or a script — is handed the same `EvalRun`, built from the job the run +already stored: + +| field | from | +|---|---| +| `input`, `expected` | the case as the experiment recorded it | +| `output` | the agent step's own result | +| `tool_calls` | every message carrying an `agent_action`, in order, with the arguments, result, error and duration of the job that call ran | +| `tools` | the tools that were called, with the schema of the script version that ran | +| `metrics` | `steps`, `duration_ms`, and the provider's `usage` when it reported any | + +Tool results are truncated at 4 KiB with `truncated: true`, so a large one cannot swamp a judge's +context, and a check that reads a truncated result can say so rather than failing on the missing +tail. A tool whose schema could not be resolved carries `null`, and a scorer validating arguments +must treat that as unchecked rather than as a failure. There is no cost field: Windmill keeps no +provider price table — the script template takes a rate as an argument instead. + +### The kinds + +Two, and both are runnables: + +| kind | is | receives | +|---|---|---| +| `agent` | an `ai_agent` resource used as a judge | the run, rendered as a message | +| `script` | a workspace script | `run`, with `input`, `output` and `expected` also spelled out | + +Keeping every scorer a runnable is what makes columns comparable: each has a path, a version, and +code you can open. There is no third kind stored as configuration on the dataset — a judge's model +and grading prompt live on the agent resource, so editing a judge is editing that agent, and the +column is not something you edit at all. Editing a column is editing the runnable it points at, so +the dataset drawer opens it in place: a script in the script editor, a judge in the resource +editor. + +Adding a scorer chooses the kind before the form opens: a judge is created next to the dataset +from the model you pick and a grading prompt that starts at the default; a script is created from +the template and opened in the editor. Both are named by a summary of what they score, which +becomes the column header and, prefixed with the dataset, the path. + +A `reason` is worth returning: it is what the cell shows on hover, together with the per-assertion +`checks`, so a number that looks wrong can be read rather than re-derived from the trajectory. + +A scorer may return a bare number, a boolean, or `{score, reason, checks}`; a judge's answer arrives +under `output`, sometimes as a string holding one of those, and often as a markdown code fence +around it, which is still read. `comment` is read as `reason`, so a scorer written for another +platform keeps its rationale. Anything with no number in it is left empty rather than guessed at, +and means skip the empty ones — a missing score counted as zero would read as a regression. + +`{score: null}` is the one exception, and it means the scorer read the case and had nothing to +measure on it: a column asking whether sources were cited has no verdict on a case with nothing to +cite. The cell shows `n/a` and is left out of the column's mean and pass rate, which is not the +same as the scorer failing — that is an error, and the column reports it as one. Written out +rather than merely absent, since a scorer that returns nothing at all is a scorer that is broken. + +### What a run says it ran + +A run is `v15` when it ran the deployed agent and `v15 + edits` when it ran that version with +undeployed changes on top. + +An `agent_draft` run records the version it is an edit of, because "the draft" is not attributable +without saying which deployed state it is a draft of. It also stops being a draft by itself: the +agent is hashed as deployed, in the same shape a draft is hashed in, so a run whose configuration +was later saved is recognised as the version it became. Edit, run, deploy, and the run you made +reads as `v16` rather than staying an edit of `v15` forever. + +That recognition is **written, not derived**. When the hashes match, the run's subject is rewritten +to `agent` at that version, once, keeping the hash it is founded on. Deriving it on every read +would make the answer expire: it would only ever mean "this ran what is deployed right now", so the +next deployment would send a run that already read `v16` back to `v15 + edits`. The write goes to +the unrestricted pool alongside the scores harvested in the same read, and nothing in it comes from +the caller — the hash is the proof, and a run of a configuration that was never deployed simply +stays an edit. + +It follows that the resolution needs someone to look: a run is stamped by the first results read +after its configuration is deployed. A run whose configuration was deployed and then replaced +without anyone opening the table keeps saying `+ edits`, which is the honest answer when the only +evidence is a hash that matches nothing deployed. + +### Reusing a scorer + +The add form lists the scorers this workspace already uses, most recently edited dataset first, +read out of the datasets' own `scorers` rather than stored anywhere new. It is filtered twice, +both times by what the caller can read: the datasets are read through `user_db`, so a scorer only +appears if the dataset carrying it does; then the runnables themselves are checked the same way, so +a script or agent the caller cannot open is never suggested. + +### A scorer is a column + +A scorer is stored on the dataset as `{id, name?, pass_if?, kind, path}`, with the `id` assigned +once and never reused: on a write, an incoming id is kept only when it names a column the dataset +already holds, and anything else is minted, so a column that was removed cannot come back under +its old id and inherit the scores recorded against it. That id is what makes a column the same +column across experiments when the scorer is renamed or its definition edited, and a delta is only +ever computed between two scores carrying the same id. Two scorers pointing at the same script are +two columns. + +A score is keyed `(experiment_id, ordinal, scorer_id)`, not baked into the experiment, so a frozen +experiment can gain a score without becoming mutable in any way that matters: what is frozen is +which runs are in it. + +Each score also records the **definition** that produced it — the kind, the path, and the script +hash or resource version that actually ran, so a path alone cannot hide an edit. When two scores +of one column carry different definitions the delta is still shown, marked: hiding the number +would force model calls just to see anything, and showing it unmarked would let a change of judge +read as a change of agent. + +### The surface + +Opening evals selects the dataset this agent was last worked in, remembered per agent in +`localStorage` and only restored while it still exists and is still readable; no run is opened for +you. The picker lists this agent's own datasets first and everyone else's below, sorted rather +than filtered, since running one dataset against a second agent is a comparison the picker exists +for. + +A dataset is named the way a script is: a **summary** of what the cases are for, from which the +path follows, prefixed with the agent so it sorts with the agent's own. With no summary the +fallback is `_dataset1`, taking the next free number. One path segment rather than a folder +under the agent, because a Windmill path is `//` and the picker that edits it +cannot express a deeper one. + +The dataset is edited in a drawer over the table: the summary and the path, the **scorers**, then +the cases in a grid. **Every way of managing a scorer is in that drawer** — adding, renaming, +moving its pass line, opening the runnable behind it, removing it; the column header over a run's +table reports and does not edit, since a run is permanent. Creating a dataset is the same drawer +with no cases yet, reached from the dataset named on a row of the runs list and from the run +dialog. Renaming moves the dataset, and its cases and its runs follow through the foreign keys. +The drawer edits a working copy and writes it in one request when **Save** is pressed — the +rename, the summary and the cases together — so a rename the server refuses leaves the cases as +they were, and a half-finished edit is never what the next run executes. A row's panel in the +results table is read-only and shows the case *as the run executed it*, not as the dataset holds +it now; deleting a case is in the drawer, and asks first. + +A case is its message, what it expects, and nothing else; the message is what identifies it. +`expected` is what a scorer compares an answer against: plain text, or JSON when the answer has +structure. + +The runs list is one row per run of this agent, newest first, whichever dataset it was of: the +run's number and what executed it (`v24`, `v24 + edits`, or a pinned `v18`), how many cases, one +badge per scorer, the dataset, and when. Each badge is the headline that column reports — a pass +rate where the column has a line, the mean where it does not — read through the thresholds as they +are **now**. A column that never scored a run reads `—`; a run still going spins. The badges are +named and resolved server-side: a list spanning datasets cannot hold every dataset's scorers to +look a column's name up, so the name and the kind ride along with the number, and the thresholds +are joined in per (run, column) — one grouped query over `eval_score` rather than a read of each +run's cells. A run whose scores are still in its flow is read out of it by the list itself, capped +per call and skipped for runs already collected, so the steady state is one query. + +**Run** asks two questions: which state of the agent (`v24 (latest deployed)` as it is saved when +you press Run; a past version as it was then; `v24 + edits (current)` running the step's edits as +they are when you press Run, offered and preselected only from the editing card), and which +dataset, with an edit button on the row and a way to start a new one without leaving. A pinned +version reproduces the configuration, not the world around it: `$var:` and `$res:` references +inside it still resolve at run time. The run that was just started opens straight away. + +The results table's rows are the dataset's cases, in dataset order, each carrying its result in +the selected experiment when it has one, so a dataset that has never been run is not an empty +table. A case the experiment ran but the dataset no longer holds keeps its row at the end: the run +happened, and deleting the case does not unmake it. Each column's mean sits under its header, with +its delta beside it when a baseline is selected. + +Picking a baseline adds a per-scorer delta to every cell and to each column's mean, and counts the +cells that regressed. Every delta names its scorer; there is no single number for a dataset, since +averaging a judge with an exact match would invent one. Rows are joined by case id, so a case added +after the baseline ran has no delta rather than counting as a change, and a column the baseline +was never scored with reports that rather than a difference that does not exist. + +## Storage + +Datasets, cases and experiments are rows: + +| table | holds | +|---|---| +| `eval_dataset` | one dataset, addressed by a workspace path, and the scorers that are its columns | +| `eval_case` | one case: its inputs and the answer it was expected to produce | +| `eval_experiment` | one run over a dataset, against one subject; written once, then only read | +| `eval_experiment_case` | the case set it executed, the job each case became, and the version or draft hash each ran against | +| `eval_score` | one scorer's verdict on one run, with the definition that produced it | + +An experiment records its cases by value instead of pointing at `eval_case`, because a dataset +keeps changing and a result set that cannot say which inputs produced it is not reproducible. For +the same reason `case_id` is a plain column rather than a foreign key: deleting a case must not +rewrite the history of the runs that used it. + +Deleting a dataset takes its cases, its experiments, their recorded case sets and every score with +it through the foreign keys. The jobs those experiments produced are left alone — they are jobs, +with their own retention. + +A case is text: a message and an expected answer. Attachments are S3 references rather than inline +bytes, so nothing in a case is meant to be large, and three caps keep it that way — 256 KiB per +case, 16 MiB and 1 000 cases per dataset — all refused at the API rather than truncated. A run +scores every case by every scorer, so a dataset also holds at most 20 scorers, refused the same +way. + +### Permissions + +A dataset is permissioned like any other path-addressed object: row-level security on +`eval_dataset` decides who may see it (readers of its folder, `u/`, a group, or an +`extra_perms` grant) and who may change it. Operators cannot write at all. Recording an experiment +counts as a write, since it persists into the dataset. + +Cases are the contents of a dataset rather than objects in their own right. `eval_case` carries a +read policy derived from its dataset (`see_parent_dataset`) and write policies that check the +dataset is *writable* — `eval_dataset_writable`, one function holding the same disjunction the +dataset's own write policies use, so a read-only grant can list a dataset's cases but not edit +them. A dataset and its cases therefore move in one `user_db` transaction, governed by the same +policies, and a rename is checked against the destination path the same way. The experiment tables +are the exception: their rows are written both by a launch (which holds dataset write) and by the +harvest (which holds only *read* of the run it copies onto its rows), so they carry read policies +only and are written on the unrestricted pool after the API has checked the right access. + +### Why an experiment is recorded before it is launched + +Launching picks the run job's id up front, writes the experiment, its case set and a pending score +per cell in one transaction, and only then queues the flow. Queueing first and recording afterwards +leaves a window in which a flow is running that no experiment accounts for, that nothing will +collect and that a retry would silently duplicate. In this order, a launch that dies before the +push leaves an experiment naming a job that never started — a run that did not run — and a push +that fails deletes it, because one failed push is the whole run. + +The dataset's foreign key guards a delete that races the assembly: the transaction fails, and at +that point nothing has been queued. It does not cover a delete that lands after this transaction +commits and before the flow is queued, which cascades the experiment away while the run still +starts. + +### How a cell finds its job, and its score + +The flow engine mints the iteration job ids, so a case is recorded before it has one. Three things +fill the gap, each copied out of the flow the first time it can be read: + +- **Which iteration ran which case.** The case is what the loop iterates over, so it is in the + iteration's own arguments by construction: `args -> 'iter' -> 'value' ->> 'case_id'` matches the + cell, whatever order the iterations finish in. +- **What the agent answered.** The agent step's result and outcome, copied onto the cell as soon + as that step is done — which is well before the iteration around it, since the scorers are still + reading it. +- **What the scorers returned.** Each scorer step's result is read out of the iteration's flow + status into the pending row that was written for it at launch. + +All three are written once, when they first become readable, and every later read is of the rows. +A job that was retained away before anything read it leaves the cell saying so, rather than +looking like a case still being answered. + +The flow itself cannot write them: it runs on workers that know nothing about these tables. So two +things call the collector. A run's flow ends with a step that calls `POST +/ai_evals/experiments/collect` on itself, which is what records a run nobody watched finish. +Reading a run collects it too, which covers the run whose flow never reached that step: one +cancelled part-way, or started while nothing served the `nativets` tag. + +That step is bookkeeping, so it is `continue_on_error`: a run whose every case answered and scored +does not become a failed job because the call did not land. diff --git a/docs/reusable-ai-agents.md b/docs/reusable-ai-agents.md index 8fa9facd74..8383bc2836 100644 --- a/docs/reusable-ai-agents.md +++ b/docs/reusable-ai-agents.md @@ -28,6 +28,11 @@ In the flow editor, the AI agent step's **Step Input** tab shows a single read-o (*linked to *, with the inherited brain + tools and an explanatory tooltip) plus *Edit* (fork into the editable step, Save changes upserts back and re-links) and *Unlink* (fork the resolved config — including any `tool_inputs` — back into the step as a one-off). +While editing, the step is the only copy of the edits: Cancel drops them and re-links (asking +first when there is something to drop), and the unsaved-changes badge opens a diff against the +deployed agent whose Discard changes is Cancel without the question. What a fork is an edit of, +and the deployed baseline the edits are judged against, live in `agentEditStore` (in memory), so +a reload brings the step back as a standalone agent with no path to save back to. A linked agent's tools appear as display-only graph tool nodes (clicking one selects the agent step); below the step's inputs, each tool gets a section with the standard schema-aware input editors (prop picker included) and a read-only view of its code — edits persist into @@ -62,6 +67,9 @@ as the reference, so two versions can be byte-identical while the agent behaves because the referenced variable changed underneath them. Anything comparing agent runs across versions has to account for that. +An eval run records the version its agent was at when the run was enqueued, which is what makes a +result attributable to a prompt state — see `docs/ai-agent-evals.md`. + A superseded value is retained for up to 100 versions. Values written through the UI keep their secrets in linked variables, but one pushed by `wmill` or written by `setResource` can hold an inline credential, and overwriting it no longer removes it from the database — anyone who can diff --git a/frontend/src/lib/components/Path.svelte b/frontend/src/lib/components/Path.svelte index 639081cc3d..37eef75bf5 100644 --- a/frontend/src/lib/components/Path.svelte +++ b/frontend/src/lib/components/Path.svelte @@ -89,6 +89,9 @@ * already have written there itself — a setup flow correcting its own failed attempt. * Every other existing path is still refused. */ allowedExistingPath?: string + /** Show the "moving may break other items" warning on a rename. Off for items nothing + * can reference by path and whose dependents move with them (eval datasets). */ + warnOnRename?: boolean } let { @@ -107,7 +110,8 @@ size = 'md', drawerOffset = 0, workspaceOverride = undefined, - allowedExistingPath = undefined + allowedExistingPath = undefined, + warnOnRename = true }: Props = $props() let ws = $derived(workspaceOverride ?? $workspaceStore) @@ -430,7 +434,8 @@ // rename. `checkInitialPathExistence` is what callers set when they are creating something, // which is the same question asked the other way round. let displayPathChangedWarning = $derived( - (['flow', 'script', 'resource', 'variable'] as PathKind[]).includes(kind) && + warnOnRename && + (['flow', 'script', 'resource', 'variable'] as PathKind[]).includes(kind) && !checkInitialPathExistence && initialPath && initialPath !== path diff --git a/frontend/src/lib/components/ResourceVersionHistory.svelte b/frontend/src/lib/components/ResourceVersionHistory.svelte index 90e41b74f0..9f73e7dda1 100644 --- a/frontend/src/lib/components/ResourceVersionHistory.svelte +++ b/frontend/src/lib/components/ResourceVersionHistory.svelte @@ -34,7 +34,11 @@ // moves and only reinstated once its own fetch lands, so the pane can never show one version's // JSON under another version's highlight. let selectedId = $state(undefined) - let loaded = $state<{ id: number; value: string; missing: string[] } | undefined>(undefined) + // `id` addresses the version, `version` is what it is called: the id is unique across every + // resource, so it is no indication of how many times this one has been saved. + let loaded = $state< + { id: number; version: number; value: string; missing: string[] } | undefined + >(undefined) // Undefined until the newest version's value arrives, which is what "Diff with current" needs. // Fetched without blocking the list, so an absent baseline disables the diff rather than // holding up the drawer everyone else opened to read. @@ -105,12 +109,21 @@ async function fetchVersion(id: number) { const version = await ResourceService.getResourceVersion({ workspace: effectiveWorkspace, - version: id + id }) - return { id, value: pretty(version.value), missing: version.missing_references ?? [] } + return { + id, + version: version.version, + value: pretty(version.value), + missing: version.missing_references ?? [] + } } - async function selectVersion(id: number | undefined, generation = loadGeneration) { + async function selectVersion( + id: number | undefined, + number: number | undefined, + generation = loadGeneration + ) { selectedId = id // Dropped up front rather than left in place while the new value is in flight: keeping it // would highlight the clicked row while the pane still rendered the previous version, and @@ -129,7 +142,7 @@ } catch (err) { if (selectedId === id && generation === loadGeneration) { selectedId = undefined - sendUserToast(`Could not load version ${id}`, true) + sendUserToast(`Could not load version ${number}`, true) } } } @@ -138,15 +151,15 @@ // loaded.id, never selectedId: restoring what the pane is showing. A selection whose value // has not arrived leaves `loaded` undefined, so this writes nothing rather than restoring a // version the user has not seen. - const id = loaded?.id - if (id === undefined) return + const target = loaded + if (target === undefined) return restoring = true try { await ResourceService.restoreResourceVersion({ workspace: effectiveWorkspace, - version: id + id: target.id }) - sendUserToast(`Restored ${path} to version ${id}`) + sendUserToast(`Restored ${path} to version ${target.version}`) onRestore?.() await loadVersions() } finally { @@ -222,14 +235,14 @@ {#each versions as version, index (version.id)} selectVersion(version.id)} + onclick={() => selectVersion(version.id, version.version)} >
{#if index === 0} {/if} - {index === 0 ? 'Current' : `Version ${version.id}`} + {index === 0 ? 'Current' : `Version ${version.version}`}
{displayDate(version.created_at)}{version.created_by diff --git a/frontend/src/lib/components/aiEvals/AddScorer.svelte b/frontend/src/lib/components/aiEvals/AddScorer.svelte new file mode 100644 index 0000000000..590a5ed573 --- /dev/null +++ b/frontend/src/lib/components/aiEvals/AddScorer.svelte @@ -0,0 +1,410 @@ + + +
+ {#if mode === 'new'} + {#if kind === 'agent'} + + An agent handed one whole run to grade. It is an ordinary AI agent resource: this creates it + with the prompt below, and editing the column later means editing that agent. + + {:else} + + A script handed the same run, returning a number, a boolean or {'{ score, reason, checks }'}. + The template scores the answer against the case's expected one, reports how the agent got + there as checks beside it, and leaves a case with no expected answer unmeasured. Helpers + below it cover exact and structural matches, which tools were called, arguments against each + tool's schema, repeated calls, step errors, latency and cost. + + {/if} + + + + + + + + {#if kind === 'agent'} + + + + + + {/if} + {:else} + {#if recent.length > 0} + + {#snippet children({ item })} + + + {/snippet} + + {/if} + + {#if usingRecent} +
+ {#each recent as scorer (scorer.path)} + {@const measures = datasetSummary(datasets, scorer.dataset)} + + {/each} +
+ {:else if kind === 'agent'} + + {:else} + + {/if} + {/if} +
diff --git a/frontend/src/lib/components/aiEvals/AgentEvalModal.svelte b/frontend/src/lib/components/aiEvals/AgentEvalModal.svelte new file mode 100644 index 0000000000..ff014f403a --- /dev/null +++ b/frontend/src/lib/components/aiEvals/AgentEvalModal.svelte @@ -0,0 +1,68 @@ + + + + + {#snippet titleBadge()} + Beta + {/snippet} +
+ {#if agentPath} + + {#key `${opWorkspace ?? ''}:${agentPath}`} + + {/key} + {:else} +
+ Evals run against a saved agent + + This agent is written into the flow step rather than saved as its own agent, so there is + nothing for a dataset and its runs to belong to. Save it as a reusable agent from the + step, and its evals start there. + +
+ {/if} +
+ diff --git a/frontend/src/lib/components/aiEvals/EvalCasesGrid.svelte b/frontend/src/lib/components/aiEvals/EvalCasesGrid.svelte new file mode 100644 index 0000000000..f7fec7a89f --- /dev/null +++ b/frontend/src/lib/components/aiEvals/EvalCasesGrid.svelte @@ -0,0 +1,132 @@ + + + + +
diff --git a/frontend/src/lib/components/aiEvals/EvalDatasetDrawer.svelte b/frontend/src/lib/components/aiEvals/EvalDatasetDrawer.svelte new file mode 100644 index 0000000000..3da84cf1ad --- /dev/null +++ b/frontend/src/lib/components/aiEvals/EvalDatasetDrawer.svelte @@ -0,0 +1,419 @@ + + + onClosed?.()}> + + (removingCase = undefined)} + on:confirmed={() => { + const target = removingCase + removingCase = undefined + if (target?.id) deleteCase(target.id) + }} + > + + {caseLabel(removingCase ?? { input: {} })} goes from the dataset. The runs that executed it keep + their results: a run that happened is not undone by curating the case away. + + + (removingDataset = false)} + on:confirmed={() => { + removingDataset = false + deleteDataset() + }} + > + + {datasetPath} goes with its cases and every run recorded against it. The jobs those runs produced + are kept. + + + drawer?.closeDrawer()} + > +
+ + {mode === 'edit' + ? 'The cases this agent is measured on. Editing them leaves the runs that already executed them as they were.' + : 'A set of cases to measure this agent on, and the scorers that read them.'} + + {#key formGeneration} + +
+ + +
+ {/key} + + +
+ (scorersWriting = w)} + /> +
+
+
+ Cases + {workingCases.length} +
+ +
+
+ (casesEditing = v)} + /> +
+
+
+ {#snippet actions()} + {#if mode === 'edit'} + + + {:else} + + {/if} + {/snippet} + + diff --git a/frontend/src/lib/components/aiEvals/EvalRunDialog.svelte b/frontend/src/lib/components/aiEvals/EvalRunDialog.svelte new file mode 100644 index 0000000000..d4d1714fd0 --- /dev/null +++ b/frontend/src/lib/components/aiEvals/EvalRunDialog.svelte @@ -0,0 +1,279 @@ + + + +
+ + + + + + +
+ {#snippet actions()} + + {/snippet} + diff --git a/frontend/src/lib/components/aiEvals/EvalRunsList.svelte b/frontend/src/lib/components/aiEvals/EvalRunsList.svelte new file mode 100644 index 0000000000..5d84ef88ce --- /dev/null +++ b/frontend/src/lib/components/aiEvals/EvalRunsList.svelte @@ -0,0 +1,170 @@ + + + + + + + + + + + + + Run + Dataset + Cases + Scores + When + + + + {#each experiments as experiment (experiment.id)} + onOpen(experiment)}> + +
+
+ {experimentName(experiment)} + + {subjectLabel(experiment, deployedHash, currentVersion)} + +
+ {experiment.created_by} +
+ + + {@const summary = datasetSummary(datasets, experiment.dataset)} + + + + {experiment.case_count} + + +
+ {#each experiment.scores ?? [] as score (score.scorer_id)} + {@const value = headline(score)} + + + {#if score.kind === 'agent'} + + {:else} + + {/if} + {score.name} + {#if value != undefined} + {value} + {:else if score.failed > 0} + failed + {:else if experiment.running} + + {:else} + — + {/if} + + + {/each} + {#if (experiment.scores ?? []).length === 0} + {#if experiment.running} + + + scoring + + {:else} + not scored + {/if} + {/if} +
+ + + + + + + + {/each} + {#if experiments.length === 0 && !loaded} + + + + + + {:else if experiments.length === 0} + + +
+ No runs yet + + A run answers every case of a dataset and scores the answers. Each one is kept, so the + next has something to be compared against. + + +
+ + + {/if} + + diff --git a/frontend/src/lib/components/aiEvals/EvalScorers.svelte b/frontend/src/lib/components/aiEvals/EvalScorers.svelte new file mode 100644 index 0000000000..68287b0fbc --- /dev/null +++ b/frontend/src/lib/components/aiEvals/EvalScorers.svelte @@ -0,0 +1,383 @@ + + +
+
+ Scorers + {scorers.length} +
+ openAdd('agent', 'new') }, + { + displayName: 'Existing AI judge', + icon: Bot, + action: () => openAdd('agent', 'existing') + }, + { displayName: 'New code scorer', icon: Code2, action: () => openAdd('script', 'new') }, + { + displayName: 'Existing code scorer', + icon: Code2, + action: () => openAdd('script', 'existing') + } + ]} + placement="bottom-end" + > + {#snippet buttonReplacement()} + + {/snippet} + +
+ +
+ {#if scorers.length === 0} +
+ A scorer reads one run and returns a number. Every run of this dataset is measured by all of + them, which is what makes two runs comparable. +
+ {:else} +
+ {#each scorers as scorer (scorer.id)} +
+ {#if scorer.kind === 'agent'} + + {:else} + + {/if} +
+ + {scorerLabel(scorer)} + + {scorer.path} +
+ {#if scorer.pass_if != undefined} + + ≥ {scorer.pass_if} + + {/if} +
+ {/each} +
+ {/if} +
+
+ + + scorerDrawer?.closeDrawer()} + > + {#if workspace && datasetPath} + {#key scorerFormGeneration} + + scriptEditorDrawer + ?.openDrawer(hash, onChanged) + .catch((e) => sendUserToast(`Failed to open the scorer: ${e}`, true))} + /> + {/key} + {/if} + {#snippet actions()} + {@const state = addScorerForm?.submitState()} + + {/snippet} + + + + + settingsDrawer?.closeDrawer()}> + {#if settingsScorer} +
+ + + +
+ {/if} + {#snippet actions()} + + {/snippet} + + + + + + + + (removingScorer = undefined)} + on:confirmed={async () => { + const target = removingScorer + removingScorer = undefined + if (!target) return + try { + await saveScorers(scorers.filter((s) => s.id !== target.id)) + } catch (e) { + sendUserToast(`Failed to remove the scorer: ${e}`, true) + } + }} +> + + The column goes from every run of this dataset, the ones already recorded included. Adding it + again starts a new column, which fills from the next run on. + + diff --git a/frontend/src/lib/components/aiEvals/EvalsPane.svelte b/frontend/src/lib/components/aiEvals/EvalsPane.svelte new file mode 100644 index 0000000000..d9e5b16fe4 --- /dev/null +++ b/frontend/src/lib/components/aiEvals/EvalsPane.svelte @@ -0,0 +1,978 @@ + + +
+
+ {#if viewingRun} + + {/if} +
+ {#if viewingRun && experiment?.run_job_id} + + Open the job + + + {/if} + {#if !viewingRun && loaded && datasets.length > 0} + + {#if experiments.length > 0} + + + {/if} + {/if} +
+ +
+ + +
+ {#if loaded && loadError} +
+ Could not load evals + + The datasets or runs could not be read. Check your access to this agent and reload. + +
+ {:else if loaded && datasets.length === 0} +
+ No dataset yet + + A dataset is the set of cases this agent is measured on. Runs are of a dataset, so + it is the first thing to make. + + +
+ {:else if !viewingRun || !loaded} + openRun(e.id)} + onEditDataset={async (path) => { + if (await useDataset(path)) datasetDrawer?.openDrawer('edit') + }} + onNew={() => (runDialogOpen = true)} + /> + {:else} + + + + + {#each scorers as scorer (scorer.id)} + + {/each} + + + + Case + Answer + {#each scorers as scorer, index (scorer.id)} + {@const mean = means.find((m) => m.scorer_id === scorer.id)} + {@const headline = columnHeadline(scorer, mean)} + + +
+ + {#if scorer.kind === 'agent'} + + {:else} + + {/if} + {scorerLabel(scorer)} + + + {#if headline} + + {headline.value} + + {#if headline.delta && headline.direction !== 0} + 0 ? 'text-green-500' : headline.direction < 0 ? 'text-red-500' : 'text-tertiary'}`} + > + {headline.delta} + + {/if} + {/if} + +
+ + {/each} + + + + {#each displayRows as row (row.case_id)} + {@const status = statusOf(row.status)} + openCase(row)} + > + + {caseLabel(row)} + + + + + {#if row.output != undefined} + {row.output} + {:else if status === STATUS.not_run} + not run + {:else} + {status.label.toLowerCase()} + {/if} + + + {#each scorers as scorer, index (scorer.id)} + {@const cell = row.scores.find((s) => s.scorer_id === scorer.id)} + + {#if cell?.pending} + + + + {:else if cell?.score != undefined} + + {#snippet text()} +
+ {#if cell.reason} + {cell.reason} + {/if} + {#each checksOf(cell) as check (check.name)} + + + {check.passed ? '✓' : '✗'} + + {check.name} + {#if check.detail} + {check.detail} + {/if} + + {/each} +
+ {/snippet} + + {#if cell.passed != undefined} + + {cell.passed ? '✓' : '✗'} + + {/if} + + {formatScore(cell.score)} + + {#if cell.baseline != undefined && cell.score !== cell.baseline} + {@const delta = cell.score - cell.baseline} + 0 ? 'text-green-500' : 'text-red-500'}`} + > + {formatDelta(delta)} + + {/if} + + + {:else if cell?.not_applicable} + + {#snippet text()} + {cell.reason} + {/snippet} + + n/a + + + {:else if cell?.error} + + {#snippet text()} + {cell.error} + {/snippet} + failed + + {:else} + — + {/if} + + {/each} + + {/each} + + + {/if} +
+ + {#if selectedRow} + {@const openRow = selectedRow} + +
+
+ + {openRow.input?.user_message ?? caseLabel(openRow)} + +
+ {#if openRow.job_id} + + Open the case job + + + {/if} +
+
+ {#if openRow.expected != undefined && openRow.expected !== ''} + + {/if} + {#if scorers.length > 0 && openRow.scores.length > 0} + + {/if} + {#if experiment && (openRow.job_id || openRow.output != undefined)} +
+
+ + Case result + +
+
+ {#if openRow.output != undefined} +
+ +
+ {:else if openRow.status === 'running'} + + + Running + + {:else} + {statusOf(openRow.status).label} + {/if} +
+
+ {/if} +
+
+ + {/if} + +
+
+ + { + if (await useDataset(path)) { + resumeRunDialog = true + datasetDrawer?.openDrawer('edit') + } + }} + onNewDataset={() => { + resumeRunDialog = true + datasetDrawer?.openDrawer('new') + }} +/> + + { + if (!resumeRunDialog) return + resumeRunDialog = false + // On the dataset the drawer was just in: the dialog opens on the pane's own, which + // creating or editing one has already moved to it. + runDialogOpen = true + }} +/> diff --git a/frontend/src/lib/components/aiEvals/evalUtils.test.ts b/frontend/src/lib/components/aiEvals/evalUtils.test.ts new file mode 100644 index 0000000000..d1f65e0d96 --- /dev/null +++ b/frontend/src/lib/components/aiEvals/evalUtils.test.ts @@ -0,0 +1,46 @@ +import { describe, expect, it } from 'vitest' +import type { EvalExperiment } from '$lib/gen' +import { parseThreshold, subjectLabel } from './evalUtils' + +describe('parseThreshold', () => { + it('keeps 0 as a threshold and reads only empty text as no threshold', () => { + expect(parseThreshold(0)).toEqual({ value: 0, error: false }) + expect(parseThreshold('0')).toEqual({ value: 0, error: false }) + expect(parseThreshold('')).toEqual({ error: false }) + expect(parseThreshold(' ')).toEqual({ error: false }) + expect(parseThreshold(null)).toEqual({ error: false }) + expect(parseThreshold(undefined)).toEqual({ error: false }) + }) + + it('refuses anything outside 0 to 1 or not a number', () => { + expect(parseThreshold('0.5')).toEqual({ value: 0.5, error: false }) + expect(parseThreshold('1')).toEqual({ value: 1, error: false }) + expect(parseThreshold('1.5')).toEqual({ error: true }) + expect(parseThreshold('-0.1')).toEqual({ error: true }) + expect(parseThreshold('abc')).toEqual({ error: true }) + }) +}) + +describe('subjectLabel', () => { + function run(subject: Record): EvalExperiment { + return { subject: { path: 'u/me/agent', ...subject } } as unknown as EvalExperiment + } + + it('names a deployed run and a pinned version by their number', () => { + expect(subjectLabel(run({ kind: 'agent', version: 4 }))).toBe('v4') + expect(subjectLabel(run({ kind: 'agent_version', version: 2 }))).toBe('v2') + }) + + it('says a draft run is edits on top of the version it was an edit of', () => { + expect(subjectLabel(run({ kind: 'agent_draft', version: 4, draft_hash: 'h1' }))).toBe( + 'v4 + edits' + ) + expect(subjectLabel(run({ kind: 'agent_draft', draft_hash: 'h1' }))).toBe('edits') + }) + + it('reads a draft whose configuration is now deployed as the current version', () => { + const draft = run({ kind: 'agent_draft', version: 4, draft_hash: 'h1' }) + expect(subjectLabel(draft, 'h1', 5)).toBe('v5') + expect(subjectLabel(draft, 'other', 5)).toBe('v4 + edits') + }) +}) diff --git a/frontend/src/lib/components/aiEvals/evalUtils.ts b/frontend/src/lib/components/aiEvals/evalUtils.ts new file mode 100644 index 0000000000..fcc5419f48 --- /dev/null +++ b/frontend/src/lib/components/aiEvals/evalUtils.ts @@ -0,0 +1,107 @@ +import type { + EvalCase, + EvalCaseInput, + EvalDataset, + EvalExperiment, + NewEvalCase, + Scorer +} from '$lib/gen' + +/** The case being edited in the drawer, before it is either run or saved to a dataset. */ +export type CaseDraft = NewEvalCase & { id?: string } + +/** A level the evals pane is on, and the way out of it. */ +export type EvalsLocation = { label: string; back: () => void } + +export type ScorerKind = Scorer['kind'] + +export function emptyCase(): CaseDraft { + return { input: { user_message: '' } } +} + +export function fromStoredCase(c: EvalCase): CaseDraft { + const { created_at: _created_at, created_by: _created_by, ...rest } = c + return rest +} + +export function caseLabel(c: { input?: EvalCaseInput }): string { + const message = c.input?.user_message?.trim() + if (message) return message.length > 60 ? message.slice(0, 60) + '…' : message + return 'Untitled case' +} + +export function experimentName(experiment: EvalExperiment): string { + return `Run ${experiment.run_number}` +} + +/** + * What ran: a deployed version, or a version with edits sitting on top of it. + * + * The list and the results endpoint restamp a draft run whose configuration was later deployed, so + * the kind is usually enough; `deployedHash` and `currentVersion` resolve the one still unstamped. + */ +export function subjectLabel( + experiment: EvalExperiment, + deployedHash?: string, + currentVersion?: number +): string { + if (experiment.subject.kind === 'agent_version') { + return experiment.subject.version ? `v${experiment.subject.version}` : 'a past version' + } + const deployed = + experiment.subject.kind === 'agent' || + (experiment.subject.draft_hash != undefined && experiment.subject.draft_hash === deployedHash) + if (deployed) { + const version = + experiment.subject.kind === 'agent' ? experiment.subject.version : currentVersion + return version ? `v${version}` : 'deployed' + } + return experiment.subject.version ? `v${experiment.subject.version} + edits` : 'edits' +} + +/** A scorer keeps its id when renamed, so its name is the column header and nothing else. */ +export function scorerLabel(scorer: Scorer): string { + return scorer.name || scorer.path.split('/').pop() || scorer.path +} + +export function kindLabel(kind: ScorerKind): string { + return kind === 'agent' ? 'Judge agent' : 'Script' +} + +export function formatScore(score: number | undefined): string { + return score == undefined ? '—' : score.toFixed(2) +} + +export function formatDelta(delta: number): string { + if (delta === 0) return '0.00' + return `${delta > 0 ? '+' : '−'}${Math.abs(delta).toFixed(2)}` +} + +/** What a dataset is for, where it says so: the path names it either way. */ +export function datasetSummary(datasets: EvalDataset[], path: unknown): string | undefined { + return datasets.find((d) => d.path === path)?.summary || undefined +} + +/** + * A pass threshold, as a field holds it. Empty is `''` or null, never a number: a number input + * coerces the text, so a valid threshold of 0 would otherwise read as empty and be dropped. The + * server refuses anything outside 0 to 1, caught here so the form blocks instead of the save. + */ +export function parseThreshold(text: string | number | null | undefined): { + value?: number + error: boolean +} { + const trimmed = typeof text === 'string' ? text.trim() : text + if (trimmed === '' || trimmed == undefined) return { error: false } + const value = Number(trimmed) + if (Number.isNaN(value) || value < 0 || value > 1) return { error: true } + return { value, error: false } +} + +export function summaryToName(summary: string): string { + return summary + .toLowerCase() + .replace(/[^a-z0-9_]/g, '_') + .replace(/_+/g, '_') + .replace(/^_|_$/g, '') +} diff --git a/frontend/src/lib/components/apps/components/display/table/multilineCellEditor.css b/frontend/src/lib/components/apps/components/display/table/multilineCellEditor.css new file mode 100644 index 0000000000..1ac9f8b31c --- /dev/null +++ b/frontend/src/lib/components/apps/components/display/table/multilineCellEditor.css @@ -0,0 +1,26 @@ +/* MultilineCellEditor: a popup positioned over the cell, so it has to paint the cell's own frame + rather than inherit it. */ +.ag-theme-alpine .wm-multiline-cell-editor, +.ag-theme-alpine-dark .wm-multiline-cell-editor { + background-color: var(--ag-background-color); +} +.ag-theme-alpine .wm-multiline-cell-editor textarea, +.ag-theme-alpine-dark .wm-multiline-cell-editor textarea { + display: block; + box-sizing: border-box; + /* Horizontal only: the vertical padding is set by the editor, which knows the height of the row + it is replacing. `line-height` here is what it computes against. */ + padding: 0 calc(var(--ag-cell-horizontal-padding) - 1px); + border: 1px solid var(--ag-input-focus-border-color); + border-radius: 3px; + outline: none; + resize: none; + /* Past this it scrolls rather than growing. */ + max-height: 40vh; + overflow-y: auto; + background-color: var(--ag-background-color); + color: var(--ag-foreground-color); + font: inherit; + line-height: 20px; + white-space: pre-wrap; +} diff --git a/frontend/src/lib/components/apps/components/display/table/multilineCellEditor.ts b/frontend/src/lib/components/apps/components/display/table/multilineCellEditor.ts new file mode 100644 index 0000000000..00975955c9 --- /dev/null +++ b/frontend/src/lib/components/apps/components/display/table/multilineCellEditor.ts @@ -0,0 +1,108 @@ +import type { ColDef, ICellEditorComp, ICellEditorParams } from 'ag-grid-community' +// Beside the editor rather than in the AgGrid theme: that file is the vendored theme, and a rule +// added to it is one the next copy of it drops. +import './multilineCellEditor.css' + +/** Kept in step with the `line-height` the stylesheet gives the textarea. */ +const LINE_HEIGHT = 20 + +/** + * A text cell editor that starts the height of the cell and grows as lines are added, for columns + * holding prose rather than a value. Enter commits, Shift+Enter adds a line, Escape cancels. + * + * Rendered as a popup positioned over the cell: an in-cell editor is clipped to the row height, so + * growing is only visible if the editor is allowed to paint outside it. + */ +export class MultilineCellEditor implements ICellEditorComp { + private eGui!: HTMLDivElement + private textarea!: HTMLTextAreaElement + private params!: ICellEditorParams + private wasEmpty = false + + init(params: ICellEditorParams) { + this.params = params + this.eGui = document.createElement('div') + this.eGui.className = 'wm-multiline-cell-editor' + + this.wasEmpty = params.value == undefined + + this.textarea = document.createElement('textarea') + this.textarea.rows = 1 + // A keystroke that opened the edit replaces the value, as it does in every other cell; F2 + // and double-click keep it to be edited. + this.textarea.value = params.eventKey?.length === 1 ? params.eventKey : (params.value ?? '') + this.textarea.style.width = `${params.column.getActualWidth() - 2}px` + // Padded so one line fills the cell it replaces and a second costs a line rather than a row. + // From the row rather than from `--ag-row-height`, which is the theme's figure and not + // necessarily this grid's. + const rowHeight = params.node.rowHeight ?? 28 + const padding = Math.max(0, (rowHeight - LINE_HEIGHT - 2) / 2) + this.textarea.style.paddingTop = `${padding}px` + this.textarea.style.paddingBottom = `${padding}px` + + this.textarea.addEventListener('input', () => this.resize()) + this.textarea.addEventListener('keydown', (e) => { + if (e.key === 'Escape') { + // Kept from whatever is around the grid: a grid in a drawer or a dialog is under a + // surface that closes on Escape, and leaving an edit is not asking to leave that. + e.preventDefault() + e.stopPropagation() + this.params.api.stopEditing(true) + return + } + if (e.key !== 'Enter' || e.isComposing) return + // Both branches keep the key from the grid, which ends the edit on Enter whether or not + // Shift is held: Shift+Enter falls through to the textarea's own newline, and plain Enter + // ends the edit here instead. + e.stopPropagation() + if (!e.shiftKey) { + e.preventDefault() + this.params.stopEditing() + } + }) + this.eGui.appendChild(this.textarea) + } + + private resize() { + this.textarea.style.height = 'auto' + this.textarea.style.height = `${this.textarea.scrollHeight}px` + } + + getGui() { + return this.eGui + } + + afterGuiAttached() { + this.resize() + this.textarea.focus() + // At the end rather than selected: a selection is a keystroke away from erasing the cell. + const end = this.textarea.value.length + this.textarea.setSelectionRange(end, end) + } + + getValue() { + // Nothing typed into a cell that held nothing is not an edit: returning '' here would write + // an empty string over a null, which the grid would see as a change and commit. + if (this.wasEmpty && this.textarea.value === '') return this.params.value + return this.textarea.value + } + + isPopup() { + return true + } + + getPopupPosition(): 'over' | 'under' { + return 'over' + } +} + +/** + * What a column of prose needs, ready to spread into a colDef. `suppressKeyboardEvent` as well as + * the editor: the grid ends an edit on Enter from a handler a popup editor's DOM does not sit + * under, so the editor cannot keep Shift+Enter for itself on its own. + */ +export const multilineCellColDef: Pick = { + cellEditor: MultilineCellEditor, + suppressKeyboardEvent: (p) => + p.editing && (p.event as KeyboardEvent).key === 'Enter' && (p.event as KeyboardEvent).shiftKey +} diff --git a/frontend/src/lib/components/common/drawer/Disposable.svelte b/frontend/src/lib/components/common/drawer/Disposable.svelte index 6a27357cf2..642d1366f6 100644 --- a/frontend/src/lib/components/common/drawer/Disposable.svelte +++ b/frontend/src/lib/components/common/drawer/Disposable.svelte @@ -94,6 +94,13 @@ return open } + /** Whether this is the overlay on top, i.e. the one a key press is for. Overlays that keep + * Escape for themselves (`preventEscape`) have to ask, or they answer keys aimed at whatever + * is stacked above them. Same condition the handler below arbitrates on. */ + export function isTopmost() { + return stack.val.length === 0 || stack.val[stack.val.length - 1] === id + } + function handleClickAway(e) { const last = stack.val[stack.val.length - 1] if (last === id) { diff --git a/frontend/src/lib/components/common/modal/Modal.svelte b/frontend/src/lib/components/common/modal/Modal.svelte index 20821a33f9..9cfa11b84e 100644 --- a/frontend/src/lib/components/common/modal/Modal.svelte +++ b/frontend/src/lib/components/common/modal/Modal.svelte @@ -1,5 +1,16 @@ + + -
+
{#if agent} -
-
+
+ +
(showDetail = !showDetail)} + onkeydown={(e) => { + // Keys aimed at the buttons inside the row bubble through here; leave them theirs. + if (e.target !== e.currentTarget) return + if (e.key === 'Enter' || e.key === ' ') { + e.preventDefault() + showDetail = !showDetail + } + }} + > - Linked to - +
{agent} e.stopPropagation()}>{agent} - - {#snippet text()} - Read-only: the configuration comes from this saved agent, and only the message and - inputs are set in this flow. Edit changes the agent everywhere it's used. Unlink forks - an editable copy into just this step. - {/snippet} - - -
+ {#if version != undefined} + + v{version} + + {/if} +
+
+ {#if brainParams.length > 0 || inheritedTools.length > 0} + + {#if showDetail} + + {:else} + + {/if} + + {/if} +
- {#if brainParams.length > 0 || inheritedTools.length > 0} -
+ {#if showDetail && (brainParams.length > 0 || inheritedTools.length > 0)} +
{#each brainParams as param (param.label)}
{param.label}
@@ -478,12 +609,7 @@
Tools
{#each inheritedTools as tool (tool.id)} - - {toolLabel(tool)} - + {toolLabel(tool)} {/each}
@@ -502,14 +628,56 @@ {/if} {:else if editingPath}
- - Editing - {editingPath} -
+
+ +
+
+ {editingPath} + {#if version != undefined} + + v{version} + + {/if} + {#if edited} + + unsaved changes + + {/if} +
+
+ saving updates every flow using it + {#snippet text()} + The edits live in this step until you decide: Evals runs them as they are here, Save + changes writes them to the agent, Cancel drops them and re-links the step. + {/snippet} + +
+
+
+
+ + -
-

- Editing the saved agent. Save changes updates it and re-links this step — the update - propagates to every flow that links to it. Cancel keeps your edits here as a standalone step - instead. -

{#if providerSaveError} -

+

{providerSaveError}

{/if} {:else} -
-
- -
- or - -
+ {/if}
@@ -552,7 +711,8 @@

Save this AI agent's configuration and tools as a reusable resource. Other flows can then - link to it, and updates propagate automatically. + link to it, updates propagate automatically, and it gains a dataset of eval cases of its + own.

+ + + + + + + { + confirmCancel = false + const path = editingPath + if (path) relink(path) + }} + onCanceled={() => (confirmCancel = false)} +> + + The step goes back to {editingPath} as it is deployed, and the edits are not kept anywhere. Save + changes writes them to the agent instead. + + diff --git a/frontend/src/lib/components/flows/content/ScriptEditorDrawer.svelte b/frontend/src/lib/components/flows/content/ScriptEditorDrawer.svelte index 538bcf26c7..362b7bca37 100644 --- a/frontend/src/lib/components/flows/content/ScriptEditorDrawer.svelte +++ b/frontend/src/lib/components/flows/content/ScriptEditorDrawer.svelte @@ -175,52 +175,6 @@ let settingsDrawer: Drawer | undefined = $state() - { - unsavedModalOpen = false - }} - on:confirmed={() => { - console.log('confirmed') - closeAnyway = true - unsavedModalOpen = false - scriptEditorDrawer?.closeDrawer() - }} -> -
- Are you sure you want to discard the changes you have made? - -
- + + { + unsavedModalOpen = false + }} + on:confirmed={() => { + closeAnyway = true + unsavedModalOpen = false + scriptEditorDrawer?.closeDrawer() + }} + > +
+ Are you sure you want to discard the changes you have made? + +
+ { +export async function createAiAgent( + id: string, + agentPath?: string +): Promise<[FlowModule, FlowModuleState]> { const storedConfig = loadStoredConfig() const providerValue = storedConfig ?? { kind: 'openai', resource: '', model: '' } + // A step linked to a saved agent reads its brain and tools from the resource, so it carries only + // the flow-local inputs: seeding `provider`/`output_type` would leave transforms it never reads. const aiAgentFlowModules: FlowModule = { id, value: { type: 'aiagent', + ...(agentPath ? { agent: agentPath } : {}), tools: [], input_transforms: { - provider: { type: 'static', value: providerValue }, - output_type: { type: 'static', value: 'text' }, + ...(agentPath + ? {} + : { + provider: { type: 'static', value: providerValue }, + output_type: { type: 'static', value: 'text' } + }), user_message: { type: 'static', value: undefined } } } diff --git a/frontend/src/lib/components/flows/map/FlowModuleSchemaMap.svelte b/frontend/src/lib/components/flows/map/FlowModuleSchemaMap.svelte index c98c10b1a2..ef2dd38c39 100644 --- a/frontend/src/lib/components/flows/map/FlowModuleSchemaMap.svelte +++ b/frontend/src/lib/components/flows/map/FlowModuleSchemaMap.svelte @@ -165,7 +165,8 @@ kind: InsertKind, wsScript?: { path: string; summary: string; hash: string | undefined }, wsFlow?: { path: string; summary: string }, - inlineScript?: InlineScript + inlineScript?: InlineScript, + agentPath?: string ): Promise { let module = emptyModule(flowStateStore.val, flowStore.val, kind == 'flow') let state = emptyFlowModuleState() @@ -190,7 +191,7 @@ } else if (kind == 'branchall') { ;[module, state] = await createBranchAll(module.id) } else if (kind == 'aiagent') { - ;[module, state] = await createAiAgent(module.id) + ;[module, state] = await createAiAgent(module.id, agentPath) } else if (inlineScript) { const { language, kind, subkind, summary } = inlineScript ;[module, state] = await createInlineScriptModule(language, kind, subkind, module.id, summary) @@ -751,7 +752,8 @@ detail.kind as InsertKind, detail.script, detail.flow ? { path: detail.flow.path, summary: detail.flow.summary } : undefined, - detail.inlineScript + detail.inlineScript, + detail.agentPath ) const index = detail.index ?? 0 const extraModules: FlowModule[] = [module] diff --git a/frontend/src/lib/components/flows/map/InsertModuleInner.svelte b/frontend/src/lib/components/flows/map/InsertModuleInner.svelte index 0b3e77a848..3316463bbb 100644 --- a/frontend/src/lib/components/flows/map/InsertModuleInner.svelte +++ b/frontend/src/lib/components/flows/map/InsertModuleInner.svelte @@ -10,6 +10,11 @@ import ToggleHubWorkspaceQuick from '$lib/components/ToggleHubWorkspaceQuick.svelte' import TopLevelNode from '../pickers/TopLevelNode.svelte' import RefreshButton from '$lib/components/common/button/RefreshButton.svelte' + import Button from '$lib/components/common/button/Button.svelte' + import { ResourceService } from '$lib/gen' + import { workspaceStore } from '$lib/stores' + import type { FlowEditorContext } from '../types' + import { BotIcon, Loader2, Plus } from 'lucide-svelte' const dispatch = createEventDispatcher() interface Props { @@ -42,11 +47,44 @@ | 'approval' | 'flow' | 'failure' - | 'aisandbox' = $state(untrack(() => kind)) + | 'aisandbox' + | 'aiagent' = $state(untrack(() => kind)) let preFilter: 'all' | 'workspace' | 'hub' = $state('all') let loading = $state(false) let small = $derived(smallProp ?? (kind === 'preprocessor' || kind === 'failure')) + // Optional: this picker also renders outside the flow editor's context (the triggers wrapper). + const flowEditorContext = getContext('FlowEditorContext') + let ws = $derived(flowEditorContext?.opWorkspace?.() ?? $workspaceStore) + + let savedAgents = $state<{ path: string; description?: string }[]>([]) + let savedAgentsLoading = $state(false) + let savedAgentsWs: string | undefined = undefined + async function loadSavedAgents() { + if (!ws || savedAgentsWs === ws) { + return + } + savedAgentsLoading = true + try { + const rs = await ResourceService.listResource({ + workspace: ws, + resourceType: 'ai_agent', + perPage: 1000 + }) + savedAgents = rs.map((r) => ({ path: r.path, description: r.description })) + savedAgentsWs = ws + } catch { + savedAgents = [] + } finally { + savedAgentsLoading = false + } + } + let filteredAgents = $derived( + funcDesc + ? savedAgents.filter((a) => a.path.toLowerCase().includes(funcDesc.toLowerCase())) + : savedAgents + ) + let height = $state(0) let owners = $state([]) // Only the content-sized host (TriggersWrapper) grows past this. The fixed-height hosts top out @@ -81,6 +119,10 @@ {loading} onClick={() => { refreshCount.val += 1 + if (selectedKind === 'aiagent') { + savedAgentsWs = undefined + loadSavedAgents() + } }} />
@@ -184,9 +226,10 @@ {#if customUi?.aiAgent != false} { - dispatch('close') - dispatch('new', { kind: 'aiagent' }) + selectedKind = 'aiagent' + loadSavedAgents() }} /> {/if} @@ -203,7 +246,52 @@
{/if} - {#if selectedKind === 'aisandbox'} + {#if selectedKind === 'aiagent'} +
+ + {#if savedAgentsLoading} +
+ Loading saved agents +
+ {:else if filteredAgents.length > 0} +
Saved agents
+ {#each filteredAgents as agent (agent.path)} + + {/each} + {:else} +
+ {savedAgents.length > 0 + ? 'No saved agent matches this search' + : 'No saved agent in this workspace yet. Configure a blank one, then Save as reusable agent to reuse it.'} +
+ {/if} +
+ {:else if selectedKind === 'aisandbox'}
Promise diff --git a/frontend/src/lib/components/graph/graphBuilder.svelte.ts b/frontend/src/lib/components/graph/graphBuilder.svelte.ts index 450debbef4..3491cd3126 100644 --- a/frontend/src/lib/components/graph/graphBuilder.svelte.ts +++ b/frontend/src/lib/components/graph/graphBuilder.svelte.ts @@ -58,6 +58,8 @@ export type GraphEventHandlers = { inlineScript?: InlineScript script?: PathScript flow?: { path: string; summary: string } + /** Saved `ai_agent` resource the inserted agent step links to, for `kind: 'aiagent'`. */ + agentPath?: string isPreprocessor?: boolean }) => void deleteBranch: (detail: { id: string; index: number }, label: string) => void diff --git a/frontend/src/lib/components/graph/renderers/edges/BaseEdge.svelte b/frontend/src/lib/components/graph/renderers/edges/BaseEdge.svelte index 89e7c88c94..5678621062 100644 --- a/frontend/src/lib/components/graph/renderers/edges/BaseEdge.svelte +++ b/frontend/src/lib/components/graph/renderers/edges/BaseEdge.svelte @@ -192,7 +192,8 @@ branch: data.branch, index: data.index, kind: e.detail.kind, - inlineScript: e.detail.inlineScript + inlineScript: e.detail.inlineScript, + agentPath: e.detail.agentPath }) }} on:pickScript={(e) => { diff --git a/frontend/src/lib/components/select/SelectDropdown.svelte b/frontend/src/lib/components/select/SelectDropdown.svelte index d9fc2dcf70..8dd952e5bf 100644 --- a/frontend/src/lib/components/select/SelectDropdown.svelte +++ b/frontend/src/lib/components/select/SelectDropdown.svelte @@ -145,17 +145,21 @@ }} > {@render startSnippet?.({ item, close: () => (open = false) })} - - {item.label || '\xa0'} - + +
+ + {item.label || '\xa0'} + + {#if item.subtitle} +
{item.subtitle}
+ {/if} +
{#if item.__is_create} {:else} {@render endSnippet?.({ item, close: () => (open = false) })} {/if} - {#if item.subtitle} -
{item.subtitle}
- {/if} {/each} diff --git a/frontend/src/routes/(root)/(logged)/resources/+page.svelte b/frontend/src/routes/(root)/(logged)/resources/+page.svelte index 5dbe7378fe..6f3fd1a6dc 100644 --- a/frontend/src/routes/(root)/(logged)/resources/+page.svelte +++ b/frontend/src/routes/(root)/(logged)/resources/+page.svelte @@ -68,6 +68,7 @@ Plus, RotateCw, Save, + FlaskConical, SearchX, Shield, Trash, @@ -82,6 +83,7 @@ assetCanBeExplored } from '../../../../lib/components/ExploreAssetButton.svelte' import NoDirectDeployAlert from '$lib/components/NoDirectDeployAlert.svelte' + import AgentEvalModal from '$lib/components/aiEvals/AgentEvalModal.svelte' type ResourceW = ListableResource & { canWrite: boolean; marked?: string } type ResourceTypeW = ResourceType & { canWrite: boolean } @@ -133,6 +135,8 @@ let deleteConfirmedCallback: (() => void) | undefined = $state(undefined) let deleteIsLinked = $state(false) let deletePath = $state('') + let evalsOpen = $state(false) + let evalsAgentPath = $state(undefined) let loading = $state({ resources: true, types: true @@ -1262,6 +1266,18 @@ { + evalsAgentPath = path + evalsOpen = true + } + } + ] + : []), { displayName: 'Permissions', icon: Shield, @@ -1462,6 +1478,8 @@ + + Date: Mon, 24 Aug 2026 22:29:44 +0200 Subject: [PATCH 41/48] fix: patch sqlx so a cancelled BEGIN cannot poison a pooled connection (#10823) * fix: patch sqlx so a cancelled BEGIN cannot poison a pooled connection Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_01Qzqmh52NU8fB9RBQNNkJGt * test: drop the migration run and fixed sleep from the sqlx patch guard Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_01Qzqmh52NU8fB9RBQNNkJGt * test: ignore the sqlx patch guard by default and point at it from where sqlx is changed Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_01Qzqmh52NU8fB9RBQNNkJGt --------- Co-authored-by: Claude Opus 5 (1M context) --- backend/Cargo.lock | 21 ++---- backend/Cargo.toml | 20 ++++++ .../tests/sqlx_begin_cancel_safe.rs | 72 +++++++++++++++++++ docs/validation.md | 1 + 4 files changed, 100 insertions(+), 14 deletions(-) create mode 100644 backend/windmill-common/tests/sqlx_begin_cancel_safe.rs diff --git a/backend/Cargo.lock b/backend/Cargo.lock index 43ff602ec1..6b76f995f9 100644 --- a/backend/Cargo.lock +++ b/backend/Cargo.lock @@ -11849,8 +11849,7 @@ dependencies = [ [[package]] name = "sqlx" version = "0.8.6" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "1fefb893899429669dcdd979aff487bd78f4064e5e7907e4269081e0ef7d97dc" +source = "git+https://github.com/windmill-labs/sqlx?rev=6bdaee94fa62a01561125646da3f99eb341f2457#6bdaee94fa62a01561125646da3f99eb341f2457" dependencies = [ "sqlx-core", "sqlx-macros", @@ -11862,8 +11861,7 @@ dependencies = [ [[package]] name = "sqlx-core" version = "0.8.6" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "ee6798b1838b6a0f69c007c133b8df5866302197e404e8b6ee8ed3e3a5e68dc6" +source = "git+https://github.com/windmill-labs/sqlx?rev=6bdaee94fa62a01561125646da3f99eb341f2457#6bdaee94fa62a01561125646da3f99eb341f2457" dependencies = [ "base64 0.22.1", "bigdecimal", @@ -11901,8 +11899,7 @@ dependencies = [ [[package]] name = "sqlx-macros" version = "0.8.6" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "a2d452988ccaacfbf5e0bdbc348fb91d7c8af5bee192173ac3636b5fb6e6715d" +source = "git+https://github.com/windmill-labs/sqlx?rev=6bdaee94fa62a01561125646da3f99eb341f2457#6bdaee94fa62a01561125646da3f99eb341f2457" dependencies = [ "proc-macro2", "quote", @@ -11914,8 +11911,7 @@ dependencies = [ [[package]] name = "sqlx-macros-core" version = "0.8.6" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "19a9c1841124ac5a61741f96e1d9e2ec77424bf323962dd894bdb93f37d5219b" +source = "git+https://github.com/windmill-labs/sqlx?rev=6bdaee94fa62a01561125646da3f99eb341f2457#6bdaee94fa62a01561125646da3f99eb341f2457" dependencies = [ "dotenvy", "either", @@ -11939,8 +11935,7 @@ dependencies = [ [[package]] name = "sqlx-mysql" version = "0.8.6" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "aa003f0038df784eb8fecbbac13affe3da23b45194bd57dba231c8f48199c526" +source = "git+https://github.com/windmill-labs/sqlx?rev=6bdaee94fa62a01561125646da3f99eb341f2457#6bdaee94fa62a01561125646da3f99eb341f2457" dependencies = [ "atoi", "base64 0.22.1", @@ -11984,8 +11979,7 @@ dependencies = [ [[package]] name = "sqlx-postgres" version = "0.8.6" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "db58fcd5a53cf07c184b154801ff91347e4c30d17a3562a635ff028ad5deda46" +source = "git+https://github.com/windmill-labs/sqlx?rev=6bdaee94fa62a01561125646da3f99eb341f2457#6bdaee94fa62a01561125646da3f99eb341f2457" dependencies = [ "atoi", "base64 0.22.1", @@ -12025,8 +12019,7 @@ dependencies = [ [[package]] name = "sqlx-sqlite" version = "0.8.6" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "c2d12fe70b2c1b4401038055f90f151b78208de1f9f89a7dbfd41587a10c3eea" +source = "git+https://github.com/windmill-labs/sqlx?rev=6bdaee94fa62a01561125646da3f99eb341f2457#6bdaee94fa62a01561125646da3f99eb341f2457" dependencies = [ "atoi", "chrono", diff --git a/backend/Cargo.toml b/backend/Cargo.toml index 0bfa52d3e6..0e08b0f060 100644 --- a/backend/Cargo.toml +++ b/backend/Cargo.toml @@ -214,6 +214,26 @@ all_sqlx_features = ["all_languages", "enterprise", "enterprise_saml", "embeddin "windmill-git-sync/all_sqlx_features"] [patch.crates-io] +# v0.8.6 plus one commit: `Pool::begin` is not cancel-safe on Postgres. sqlx raises the +# transaction depth its rollback-on-drop guard keys on only *after* the BEGIN round trip, so +# a cancelled caller (a disconnecting API client, a `timeout`, an aborted task) leaves the +# session in a transaction nothing will end, and the pool hands that connection out again — +# every later query on it fails with 25P02 until max_lifetime recycles it 30 minutes on. +# Reported upstream in 2022 (launchbadge/sqlx#2054), fixed for SQLite only, and still present +# in 0.9.0. Drop this the moment upstream carries the fix. +# The whole family has to move together: `sqlx-postgres` depends on `sqlx-core` by path +# inside the sqlx workspace, so patching it alone leaves two incompatible `sqlx-core`s and +# `Postgres` stops implementing the `Database` the macros expect. +# Changing any of this — a bump, a rebase of the fork, dropping these lines — still compiles +# clean, so run the guard that actually checks the behaviour is still there: +# cargo test -p windmill-common --test sqlx_begin_cancel_safe -- --ignored +sqlx = { git = "https://github.com/windmill-labs/sqlx", rev = "6bdaee94fa62a01561125646da3f99eb341f2457" } +sqlx-core = { git = "https://github.com/windmill-labs/sqlx", rev = "6bdaee94fa62a01561125646da3f99eb341f2457" } +sqlx-macros = { git = "https://github.com/windmill-labs/sqlx", rev = "6bdaee94fa62a01561125646da3f99eb341f2457" } +sqlx-macros-core = { git = "https://github.com/windmill-labs/sqlx", rev = "6bdaee94fa62a01561125646da3f99eb341f2457" } +sqlx-postgres = { git = "https://github.com/windmill-labs/sqlx", rev = "6bdaee94fa62a01561125646da3f99eb341f2457" } +sqlx-mysql = { git = "https://github.com/windmill-labs/sqlx", rev = "6bdaee94fa62a01561125646da3f99eb341f2457" } +sqlx-sqlite = { git = "https://github.com/windmill-labs/sqlx", rev = "6bdaee94fa62a01561125646da3f99eb341f2457" } object_store = { git = "https://github.com/apache/arrow-rs-object-store", rev = "36752c975d4f29e20b57c91f81a10872dcd48ae7" } # Use tiberius main branch for libgssapi 0.8.1 fix (https://github.com/prisma/tiberius/issues/343) tiberius = { git = "https://github.com/prisma/tiberius", rev = "59db57960a14b422fb3a1309aa4aa47880896ff8" } diff --git a/backend/windmill-common/tests/sqlx_begin_cancel_safe.rs b/backend/windmill-common/tests/sqlx_begin_cancel_safe.rs new file mode 100644 index 0000000000..1f387984b0 --- /dev/null +++ b/backend/windmill-common/tests/sqlx_begin_cancel_safe.rs @@ -0,0 +1,72 @@ +//! Guards the `sqlx` entries in `[patch.crates-io]` — `backend/Cargo.toml` carries the why. +//! Dropping the patch still compiles, so a test is what notices. +//! +//! Ignored by default: it only has something to say when the sqlx dependency moves, and it +//! spends a couple of seconds waiting on a deliberately slow round trip. Run it whenever you +//! touch sqlx — a version bump, a change to the patch entries, a fork rebase: +//! +//! ```text +//! cargo test -p windmill-common --test sqlx_begin_cancel_safe -- --ignored +//! ``` + +use sqlx::{Connection, PgConnection, Pool, Postgres}; +use std::time::{Duration, Instant}; + +#[sqlx::test] +#[ignore = "run with --ignored after any sqlx bump or change to [patch.crates-io]"] +async fn begin_cancelled_mid_round_trip_leaves_no_open_transaction(db: Pool) { + // One connection, so the session inspected below is the one the cancelled begin used. + let pool = sqlx::postgres::PgPoolOptions::new() + .max_connections(1) + .min_connections(0) + .connect_with((*db.connect_options()).clone()) + .await + .expect("failed to build pool"); + let pid: i32 = sqlx::query_scalar("SELECT pg_backend_pid()") + .fetch_one(&pool) + .await + .unwrap(); + + // A plain `BEGIN` answers in well under a millisecond, which is too narrow to cancel + // reliably; appending a sleep widens the round trip and runs through the same + // `PgTransactionManager::begin` the patch fixes. + let cancelled = tokio::time::timeout( + Duration::from_millis(300), + pool.begin_with("BEGIN; SELECT pg_sleep(2);"), + ) + .await; + assert!(cancelled.is_err(), "the begin must not have completed"); + + let mut admin = PgConnection::connect_with(&(*db.connect_options()).clone()) + .await + .expect("failed to open an observing connection"); + + // sqlx only flushes the queued ROLLBACK once the abandoned statement has answered, so + // wait for the session to stop running rather than sleeping a fixed time a loaded runner + // could overshoot. + let deadline = Instant::now() + Duration::from_secs(30); + let state = loop { + let state: String = sqlx::query_scalar("SELECT state FROM pg_stat_activity WHERE pid = $1") + .bind(pid) + .fetch_optional(&mut admin) + .await + .unwrap() + .flatten() + .unwrap_or_default(); + if state != "active" || Instant::now() >= deadline { + break state; + } + tokio::time::sleep(Duration::from_millis(100)).await; + }; + + assert!( + !state.starts_with("idle in transaction"), + "connection returned to the pool still inside a transaction (state {state:?}) — is \ + the sqlx patch in backend/Cargo.toml still applied?" + ); + + sqlx::query_scalar::<_, i32>("SELECT 1") + .fetch_one(&pool) + .await + .expect("pool must still serve queries"); +} diff --git a/docs/validation.md b/docs/validation.md index 050eacd75b..74e9cf5677 100644 --- a/docs/validation.md +++ b/docs/validation.md @@ -16,6 +16,7 @@ After making changes, run the appropriate checks and fix all errors before consi | Multiple gated modules | `cargo check --features enterprise,parquet` | Combine only the flags you need | | API route changes | `cargo check` | Then update `openapi.yaml` and run `npm run generate-backend-client` | | Database migrations | `cargo check` | Test migration applies cleanly with `sqlx migrate run` | +| The `sqlx` dependency (version bump, `[patch.crates-io]` entries, fork rebase) | `cargo test -p windmill-common --test sqlx_begin_cancel_safe -- --ignored` | Windmill runs a patched `sqlx`: upstream's `Pool::begin` is not cancel-safe on Postgres, and a cancelled one poisons the pooled connection for 30 minutes. Losing the patch still compiles, so this ignored test is the only thing that notices. `backend/Cargo.toml` has the detail | **Never** use `--features all_sqlx_features` — it compiles everything and is very slow. Check `backend/Cargo.toml` `[features]` to find the right flags. From 541b6c849657d13fed3580407a00a996e891ad9e Mon Sep 17 00:00:00 2001 From: hugocasa Date: Mon, 24 Aug 2026 22:30:45 +0200 Subject: [PATCH 42/48] fix: keep ai chat messages when leaving the page mid-generation (#10809) * fix: persist ai chat turns mid-generation so leaving the page keeps them * fix: stop chat checkpoints once the turn commits, keep streamed text visible * fix: checkpoint streamed answers as they grow and keep half-run tool batches * fix: checkpoint text as received so a backgrounded tab keeps capturing * fix: keep buffered tool screenshots in mid-batch chat checkpoints * fix: decide committed-text at the flush site, condense checkpoint comments * fix: checkpoint only live streamed text, never text the parser owns * fix: don't swap the chat transcript out from under a running turn * fix: close the pre-loading window in the conversation-switch guard --- .../copilot/chat/AIChatDisplay.svelte | 6 +- .../copilot/chat/AIChatManager.svelte.ts | 181 ++++++++- .../copilot/chat/AIChatManager.test.ts | 362 ++++++++++++++++++ .../components/copilot/chat/chatLoop.test.ts | 26 +- .../lib/components/copilot/chat/chatLoop.ts | 32 ++ .../src/lib/components/copilot/chat/shared.ts | 25 +- .../copilot/chat/typewriterReveal.test.ts | 20 + .../copilot/chat/typewriterReveal.ts | 9 + 8 files changed, 635 insertions(+), 26 deletions(-) diff --git a/frontend/src/lib/components/copilot/chat/AIChatDisplay.svelte b/frontend/src/lib/components/copilot/chat/AIChatDisplay.svelte index 2dd8f7bef7..9b55bd9217 100644 --- a/frontend/src/lib/components/copilot/chat/AIChatDisplay.svelte +++ b/frontend/src/lib/components/copilot/chat/AIChatDisplay.svelte @@ -611,7 +611,11 @@ the panel, or the Escape-to-stop focus check would wrongly reject them. -->
{#each pastChats as chat (chat.id)}