Files
windmill/ai_evals
2044656d74 refactor: run the ai session's tools on the backend (#11899)
* feat: add a workspace mcp server with the ai session toolset

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix: keep workspace mcp tool calls on their own routes

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix: keep a token's job tag fence on workspace mcp tool calls

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix: state the workspace mcp token rule in its refusal message

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* feat: let the workspace mcp server span workspaces chosen per token

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* feat: let the mcp url form choose between the tools and workspace servers

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* feat: serve the workspace mcp server as windmill mcp v2 at /api/mcp/v2

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* feat: add read-and-run access to windmill mcp v2

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* feat: serve the full windmill mcp experience at /api/mcp, sign-in first

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix: align the mcp url form with the brand guidelines

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* feat: serve every mcp token from /api/mcp, with one access picker

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* feat: rename mcp access levels and keep selections to scripts and flows on /api/mcp

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix: name the mcp access levels in the url caption

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* chore: name the mcp server windmill

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix: lead the mcp access picker with build and run, and spell out what a selection excludes

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* feat: show the tools each mcp access level gives

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix: list the mcp tools in a collapsible section under the access row

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix: offer all scripts and flows as typed mcp tools again, as wildcard scopes

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix: label the mcp selection access as scripts and flows as tools

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* feat: link mcp run, app draft and deploy results to their windmill pages

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix: mirror the session tool changes from main in the mcp server

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix: keep an ai agent's identity when the ai chat deploys its resource

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* refactor: call the api in-process from the mcp session tools

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix: keep a draft that rebase_draft could not return whole

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix: let read-and-run mcp tokens run scripts and flows that take secrets

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* feat: ask mcp clients to open windmill links in a browser the user sees

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* feat: send mcp clients condensed instructions and serve the full rules on demand

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* refactor: put the core rules first in the mcp instructions instead of condensing them

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix: go straight to the mcp sign-in page after login, without the workspace picker

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix: no workspace error toast on the mcp sign-in page while logged out

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* refactor: run the ai session's reads and draft tools on the backend

* feat: open raw app previews on the preview alone, from mcp links and in sessions

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* feat: link mcp script, flow and agent drafts to their editors, and agents to their page

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix: keep the session preview's saved sidebar choice, fold the build log in preview-only views

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* refactor: resolve and start the ai session's runs on the backend

* fix: hold mcp tokens below full access to their level on the per-workspace urls

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix: revoke tokens with the new mcp scopes when the migration is reverted

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix: keep an mcp token's scopes when its editor is saved without changing access

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix: mint one jwt per mcp tool call, accept the old approve body, list the new scopes

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* refactor: run delete on the backend and drop the chat's moved tool logic

* fix: reset the mcp token editor's picker state for each token

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix: close the session tools' parity gaps and drop the tests of moved logic

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* refactor: mark only mcp selections, read levels imply no endpoint tools

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* feat: run the global ai evals against a real workspace per attempt

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix: allow the skill description that only the mcp instructions read

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* ci: build the scheduled ai evals' backend with the hub embeddings index

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix: give migrations their own wait, bound the npm types fetch, keep draft links mcp-only

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix: seed hub-synced resource types for global eval fixtures that set none

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix: make the global eval workspaces match a production instance

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix: settle a linked agent's unsaved draft before testing its step

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix: seed non-admin eval users in the database, give standalone eval chats no session

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix: let a step test run past a deleted or unreadable linked agent, as the server does

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix: judge npm search matches against the best one, not a fixed score

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* chore: update ee-repo-ref to b3fd7d49f2ba58f5080fabc3a5686c34a0b2684c

This commit updates the EE repository reference after PR #877 was merged in windmill-ee-private.

Previous ee-repo-ref: 9e960246d51401a2b0487dcffcadd0b43cd67d39

New ee-repo-ref: b3fd7d49f2ba58f5080fabc3a5686c34a0b2684c

Automated by sync-ee-ref workflow.

* fix: show the draft save label when the server builds the draft, as before

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix: keep a browser caller's session on the session tools' internal calls, so its runs and trigger edits are attributed to the UI as before

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* refactor: keep the chat's npm search in the browser, so its registry requests leave from the user's browser as before

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test: share the run tools' argument cases between the chat's and the server's handling, so a change on one side only fails

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: windmill-internal-app[bot] <windmill-internal-app[bot]@users.noreply.github.com>
2026-11-02 23:57:00 +01:00
..

AI Evals

Small benchmark runner for the Windmill AI generation modes:

  • cli
  • flow
  • script
  • app
  • global

The benchmark always tests the current production prompts, tools, and guidance in this checkout.

Each attempt runs:

  1. the real production path
  2. deterministic validation
  3. LLM judging

Install

cd ai_evals
bun install

Frontend modes also require frontend dependencies:

cd frontend
bun install

Commands

List model aliases:

cd ai_evals
bun run cli -- models

List cases:

cd ai_evals
bun run cli -- cases
bun run cli -- cases flow

Run benchmarks:

cd ai_evals
bun run cli -- run flow
bun run cli -- run flow flow-test4-order-processing-loop --model opus
bun run cli -- run flow flow-test0-sum-two-numbers --models haiku,opus,4o
bun run cli -- run flow flow-test0-sum-two-numbers --runs 3 --verbose
bun run cli -- run flow --record
GEMINI_API_KEY=... bun run cli -- run app app-test1-counter-create --model gemini-3-flash-preview
WMILL_AI_EVAL_BACKEND_URL=http://127.0.0.1:8000 bun run cli -- run flow --backend-validation preview
bun run cli -- run global global-test1-script-create
bun run cli -- run cli bun-hello-script

Public CLI surface:

  • models
  • cases [mode]
  • run <mode> [caseIds...]

run options:

  • --runs <n>: repeat each case n times
  • --output <path>: custom result JSON path
  • --model <alias>: choose the model under test
  • --models <a,b,c>: run the same cases sequentially against several model aliases
  • --verbose: stream assistant output for frontend runs
  • --skip-judge: skip LLM judge scoring for the run
  • --execution-only: only require the model/proxy/frontend loop to complete; skip validators, tool expectations, backend artifact validation, and judge scoring
  • --reasoning <effort>: reasoning effort for frontend modes (off, low, medium, high, max, …); without it the product's default applies (high on models that can reason). The effort is appended to the recorded model label (anthropic:claude-sonnet-5-5@max)
  • --record: append a compact tracked summary line to ai_evals/history/<mode>.jsonl for full-suite runs only
  • --backend-validation <mode>: optional backend smoke validation (off or preview) for script and flow evals

Models

Use bun run cli -- models to see the current aliases; core/models.ts is the list.

Notes:

  • the command also prints accepted alias spellings such as gpt-4o, gpt-55, claude-opus-4.6, and claude-haiku-4.5

  • frontend modes (flow, script, app, global) can use Anthropic, OpenAI, Gemini, and DeepSeek-backed aliases

  • cli mode runs a coding agent over the workspace, chosen by the alias's cli entry; aliases without one are rejected there:

    Agent Models Label Needs
    Claude Code (@anthropic-ai/claude-agent-sdk) Claude aliases anthropic:<model> ANTHROPIC_API_KEY
    Claude Code on DeepSeek's Anthropic-compatible endpoint deepseek-v4-pro claude-code:<model> DEEPSEEK_API_KEY
    Codex (codex exec --json) gpt-6.1-sol, gpt-6-astra codex:<model> codex on PATH (CI pins 0.159.3), OPENAI_API_KEY

    Codex runs with a throwaway HOME, so your own ~/.codex config and skills stay out of the run. It has no Skill tool: it finds .agents/skills on its own and loads a skill by reading its SKILL.md, so the trace records any command that touches .agents/skills/<name>/SKILL.md as a Skill call to <name>, just before that command, or just after it when the command mutates something before reading the skill. Its shell commands are recorded as Bash and its patches as Edit, so the invokes skill … before first mutation checks compare the same things for both agents. Codex has no turn cap; a run stops at maxTurns rounds of tool calls (parallel calls count once) and fails, as a Claude Code run does at its cap.

  • the judge model is separate and currently defaults to claude-sonnet-5-5; use --skip-judge for deterministic-only runs

Case Format

Cases live in one YAML file per mode under ai_evals/cases/.

Minimal shape:

- id: flow-test0-sum-two-numbers
  prompt: |-
    Create a flow that takes two numbers, `a` and `b`, and returns their sum.
  initial: ai_evals/fixtures/...
  expected: ai_evals/fixtures/...

Optional fields:

  • initial: starting state fixture
  • expected: expected artifact fixture
  • validate: extra deterministic validation rules
  • runtime.backendPreview: optional real backend preview config for smoke validation

For flow mode, validate can express requirements such as:

  • accepted input schema shapes
  • required results.* reference validity
  • required module/code/input characteristics

For app mode, validate can express narrow hard requirements such as:

  • required frontend file paths or backend runnable keys
  • minimum backend runnable counts
  • required backend runnable types
  • minimum datatable / datatable-table counts
  • specific required datatable tables

For global mode, validate can express draft-level requirements such as:

  • required draft type/path/language
  • required or forbidden snippets in draft values
  • required or forbidden draft counts
  • forbidden draft paths

Global initial fixtures can also seed liveEditorDrafts with type, storagePath, effectivePath, and value fields. These drafts emulate the currently open script, flow, or raw app editor so cases can test prompts that refer to "this" or the "current" item.

Global initial fixtures can seed the session's artifacts — { name, versions: [{ content, note? }], role?, approvedVersion? }, oldest version first, so the artifact starts with the history list_artifact_versions reports — and the previewTabs open in its side panel, for cases that run with runtime.sessionChat: true. A tab entry names one destination and may be the active one:

"previewTabs": [{ "artifact": { "name": "Onboarding plan", "version": 2 }, "active": true }]

page ({ href, label }) and item ({ kind, path }) tabs work the same way. Tabs are driven by the production tab model, so open_preview, get_preview_status and close_page really open, report and close them, and a version is the pin a reader chose in the artifact's version picker — which only get_preview_status reports.

Global initial fixtures can seed workspace.variables with { path, value, is_secret, description?, labels?, ws_specific? } entries so cases can read and edit variables that already exist in the workspace. Seed a recognizable secret (the existing fixture uses sk_live_do_not_leak_me) and assert it via valueExcludes to catch a leak.

Global mode runs against a real workspace

The session's tools run on the backend, so each global attempt creates its own workspace on the eval backend, seeds the fixture there through the API, and deletes it afterwards. What that backend needs:

  • a worker serving the postgresql tag (the native group) for datatable fixtures and the SQL tools, and WMILL_AI_EVAL_DATATABLE_PG_URL (default postgres://postgres:changeme@localhost:5432/postgres) naming a Postgres the worker can reach, where each datatable fixture gets a scratch database;
  • a superadmin login (the default admin@windmill.dev), since it creates and deletes workspaces. WMILL_AI_EVAL_BACKEND_WORKSPACE does not apply to global mode. A CE backend allows two workspaces besides admins, so attempts past that wait for a free slot;
  • WMILL_AI_EVAL_DATABASE_URL, the eval backend's own database, for fixtures whose user is not the admin: the harness writes that user and a token of theirs there with psql (CE cannot create users through the API), adds them to the workspace, and revokes the token afterwards;
  • network access to the hub, which the hub tools reach through the backend, and the embedding cargo feature, without which search_hub_scripts has no index to search;
  • the http_trigger cargo feature, which mounts the routes HTTP trigger drafts are checked against;
  • RESOURCE_TYPE_EMBEDDINGS_CHECK_INTERVAL_SECS set low (CI uses 2): each attempt seeds the hub's resource types into its workspace and waits until the resource type search sees them, which takes up to that interval (60 s by default).

A fixture's jobs become completed preview runs with the given id, path, args, logs and outcome; createdBy is not reproduced. Its resourceTypes are created as workspace types, alongside whatever types the instance already holds; a fixture that sets none gets the types of an instance synced with the hub.

runtime.userAnswers scripts the user's side of askUserQuestion in a global case: one answer per question, in the order they are asked, and a question past the last answer is dismissed, so userAnswers: [] dismisses every question. Without it the tool reports that it cannot ask, and the model usually carries on with its own guess, which makes "asks before creating anything" impossible to assert.

runtime.profile runs a session-chat case (sessionChat: true) as a non-admin role instead of full access: the prompt, toolset and tool schemas are cut as production's capabilitiesForRole cuts them, and the workspace's operator_settings are seeded for the tools that read them directly:

profile:
  operator: true
  operatorSettings: { builder_apps: true }   # also builder_flows, manage_schedules, manage_triggers

A role that builds nothing (no builder rights, manage_schedules and manage_triggers both false) holds the session in Use, as the chat does in production: the toolset is the read-only one and the prompt carries the session section that names the role as the reason.

judgeAssistantText: true hands the judge what the assistant said alongside the mode's output. Use it when the deliverable is the answer itself, such as a refusal and where to go instead: the global judge otherwise reads only the drafts, and a substring check cannot tell "you can't create it yourself" from "create it yourself".

toolExpect.toolCallArgs entries support sharedByAtLeast: <n>: at least n recorded calls to that tool must carry the same non-blank string in the field. Use it for calls that have to share an identifier, like two test runs of one chat conversation.

toolExpect.toolCallArgs entries additionally support fieldMustBeAbsent: true: no recorded call to that tool may pass the field at all (an explicit null counts as passing it). Use it for partial-update tools, where supplying a field the model could not have read is itself the failure — e.g. write_variable.value on a secret variable.

Global initial fixtures can seed workspace.datatables. Each entry is { datatable_name, schemas: { <schema>: { <table>: { columns, rows? } } } }, plus migrations_enabled: true and migrations: [{ timestamp, name, code_up, code_down?, applied? }] for a datatable that keeps its schema as migrations. Each one becomes a real datatable of the case's workspace, backed by its scratch database, so the datatable and SQL tools see exactly what a query would. An absent datatables seed leaves the workspace with none, which is what the "no datatable configured" blocking cases rely on.

Set WMILL_AI_EVAL_DISABLE_ACTIVE_EDITOR_CONTEXT=1 to run those cases with the old behavior where the live editor is only discoverable through list_workspace_items.

App fixtures can also include an optional datatables.json file at the fixture root.

For flow mode, an initial fixture can also include a benchmark workspace catalog of existing scripts and flows. That lets the real search_workspace and get_runnable_details tools discover reusable workspace runnables during evals.

If --backend-validation preview is enabled:

  • script evals run a real backend script preview in an isolated temp workspace
  • flow evals run a real backend flow preview only for cases that define runtime.backendPreview
  • flow cases with initial.workspace fixtures seed those scripts and flows into the preview workspace before preview
  • when WMILL_AI_EVAL_BACKEND_WORKSPACE is set, ai_evals creates or reuses that workspace as a dedicated test workspace, clears managed eval assets under f/evals/* before each preview run, and then reseeds the current case fixtures

Supported backend env vars:

  • WMILL_AI_EVAL_BACKEND_VALIDATION=preview
  • WMILL_AI_EVAL_BACKEND_URL=http://127.0.0.1:8000
  • WMILL_AI_EVAL_BACKEND_EMAIL=admin@windmill.dev
  • WMILL_AI_EVAL_BACKEND_PASSWORD=changeme
  • WMILL_AI_EVAL_BACKEND_WORKSPACE=integration-tests to reuse an existing workspace on CE installs with low workspace limits

Frontend modes require a reachable Windmill backend and send model requests through the workspace AI proxy at /api/w/{workspace}/ai/proxy. At startup, ai_evals checks the resolved backend URL and fails early with setup guidance if the backend cannot be reached or login fails.

For frontend modes:

  • ai_evals creates a temporary backend workspace, or creates/reuses WMILL_AI_EVAL_BACKEND_WORKSPACE when it is set
  • it upserts a provider resource under f/evals/ai/<provider>
  • frontend requests go through /api/w/{workspace}/ai/proxy

Results And Artifacts

Every run writes:

  • a summary JSON under ai_evals/results/
  • generated artifacts in a sibling directory

If --record is used, the CLI also appends one compact JSON line to:

  • ai_evals/history/flow.jsonl
  • ai_evals/history/script.jsonl
  • ai_evals/history/app.jsonl
  • ai_evals/history/global.jsonl
  • ai_evals/history/cli.jsonl

Each recorded line contains:

  • run metadata (createdAt, gitSha, mode, runModel, judgeModel)
  • suite totals (caseCount, attemptCount, passedAttempts, passRate, averageDurationMs, averagePassedDurationMs, averageJudgeScore)
  • average token usage (averageTokenUsagePerAttempt, averageTokenUsagePerPassedAttempt)
  • per-case metrics under cases[] (averageDurationMs, averagePassedDurationMs, averageJudgeScore, averageTokenUsagePerAttempt, averageTokenUsagePerPassedAttempt, pass rate)
  • failedCaseIds

The CLI headline duration and token averages use passed attempts only. All-attempt averages are still recorded to make failures auditable without letting failed attempts skew success cost comparisons.

Example:

  • summary: ai_evals/results/2026-04-09T09-40-33.051Z__flow.json
  • artifacts: ai_evals/results/2026-04-09T09-40-33.051Z__flow/

Typical artifacts by mode:

  • flow: flow.json
  • script: script.json plus the generated script file
  • app: app.json plus frontend/backend files
  • global: global-drafts.json
  • cli: assistant-output.txt, trace.json, wmill-invocations.jsonl, plus generated workspace files
  • backend-validated attempts also include backend-preview.json

Layout

  • cases/: one YAML file per mode
  • fixtures/: initial and expected fixtures
  • core/: shared loading, model resolution, validation, judging, and result writing
  • modes/: one runner per mode
  • history/: optional tracked pass-rate history written by run --record, one JSONL file per mode
  • results/: local benchmark output and artifacts

Harness unit tests run in two lanes: bun test adapters/ for plain TypeScript, and bun run test:frontend-graph for *.vitest.ts files, which exercise adapters built on frontend code (Svelte runes, SvelteKit aliases) that bun cannot load.

Notes

  • Frontend modes reuse the production frontend chat code through the Vitest bridge.
  • Global mode evaluates the production global AI tools and validates the resulting AI draft store.
  • CLI mode creates an isolated workspace, writes the current checkout guidance into it, and benchmarks the real skills / AGENTS.md flow.
  • CLI mode now also records a structured trace of invoked skills, tool calls, proposed wmill commands, and any attempted wmill executions.
  • Frontend progress streams live while the benchmark is running.
  • Deterministic validators should stay focused on real correctness constraints, not one exact implementation shape.