* feat: add a workspace mcp server with the ai session toolset Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix: keep workspace mcp tool calls on their own routes Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix: keep a token's job tag fence on workspace mcp tool calls Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix: state the workspace mcp token rule in its refusal message Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * feat: let the workspace mcp server span workspaces chosen per token Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * feat: let the mcp url form choose between the tools and workspace servers Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * feat: serve the workspace mcp server as windmill mcp v2 at /api/mcp/v2 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * feat: add read-and-run access to windmill mcp v2 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * feat: serve the full windmill mcp experience at /api/mcp, sign-in first Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix: align the mcp url form with the brand guidelines Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * feat: serve every mcp token from /api/mcp, with one access picker Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * feat: rename mcp access levels and keep selections to scripts and flows on /api/mcp Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix: name the mcp access levels in the url caption Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * chore: name the mcp server windmill Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix: lead the mcp access picker with build and run, and spell out what a selection excludes Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * feat: show the tools each mcp access level gives Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix: list the mcp tools in a collapsible section under the access row Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix: offer all scripts and flows as typed mcp tools again, as wildcard scopes Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix: label the mcp selection access as scripts and flows as tools Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * feat: link mcp run, app draft and deploy results to their windmill pages Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix: mirror the session tool changes from main in the mcp server Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix: keep an ai agent's identity when the ai chat deploys its resource Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * refactor: call the api in-process from the mcp session tools Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix: keep a draft that rebase_draft could not return whole Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix: let read-and-run mcp tokens run scripts and flows that take secrets Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * feat: ask mcp clients to open windmill links in a browser the user sees Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * feat: send mcp clients condensed instructions and serve the full rules on demand Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * refactor: put the core rules first in the mcp instructions instead of condensing them Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix: go straight to the mcp sign-in page after login, without the workspace picker Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix: no workspace error toast on the mcp sign-in page while logged out Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * refactor: run the ai session's reads and draft tools on the backend * feat: open raw app previews on the preview alone, from mcp links and in sessions Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * feat: link mcp script, flow and agent drafts to their editors, and agents to their page Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix: keep the session preview's saved sidebar choice, fold the build log in preview-only views Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * refactor: resolve and start the ai session's runs on the backend * fix: hold mcp tokens below full access to their level on the per-workspace urls Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix: revoke tokens with the new mcp scopes when the migration is reverted Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix: keep an mcp token's scopes when its editor is saved without changing access Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix: mint one jwt per mcp tool call, accept the old approve body, list the new scopes Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * refactor: run delete on the backend and drop the chat's moved tool logic * fix: reset the mcp token editor's picker state for each token Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix: close the session tools' parity gaps and drop the tests of moved logic Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * refactor: mark only mcp selections, read levels imply no endpoint tools Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * feat: run the global ai evals against a real workspace per attempt Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix: allow the skill description that only the mcp instructions read Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * ci: build the scheduled ai evals' backend with the hub embeddings index Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix: give migrations their own wait, bound the npm types fetch, keep draft links mcp-only Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix: seed hub-synced resource types for global eval fixtures that set none Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix: make the global eval workspaces match a production instance Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix: settle a linked agent's unsaved draft before testing its step Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix: seed non-admin eval users in the database, give standalone eval chats no session Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix: let a step test run past a deleted or unreadable linked agent, as the server does Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix: judge npm search matches against the best one, not a fixed score Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * chore: update ee-repo-ref to b3fd7d49f2ba58f5080fabc3a5686c34a0b2684c This commit updates the EE repository reference after PR #877 was merged in windmill-ee-private. Previous ee-repo-ref: 9e960246d51401a2b0487dcffcadd0b43cd67d39 New ee-repo-ref: b3fd7d49f2ba58f5080fabc3a5686c34a0b2684c Automated by sync-ee-ref workflow. * fix: show the draft save label when the server builds the draft, as before Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix: keep a browser caller's session on the session tools' internal calls, so its runs and trigger edits are attributed to the UI as before Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * refactor: keep the chat's npm search in the browser, so its registry requests leave from the user's browser as before Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * test: share the run tools' argument cases between the chat's and the server's handling, so a change on one side only fails Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> Co-authored-by: windmill-internal-app[bot] <windmill-internal-app[bot]@users.noreply.github.com>
AI Evals
Small benchmark runner for the Windmill AI generation modes:
cliflowscriptappglobal
The benchmark always tests the current production prompts, tools, and guidance in this checkout.
Each attempt runs:
- the real production path
- deterministic validation
- LLM judging
Install
cd ai_evals
bun install
Frontend modes also require frontend dependencies:
cd frontend
bun install
Commands
List model aliases:
cd ai_evals
bun run cli -- models
List cases:
cd ai_evals
bun run cli -- cases
bun run cli -- cases flow
Run benchmarks:
cd ai_evals
bun run cli -- run flow
bun run cli -- run flow flow-test4-order-processing-loop --model opus
bun run cli -- run flow flow-test0-sum-two-numbers --models haiku,opus,4o
bun run cli -- run flow flow-test0-sum-two-numbers --runs 3 --verbose
bun run cli -- run flow --record
GEMINI_API_KEY=... bun run cli -- run app app-test1-counter-create --model gemini-3-flash-preview
WMILL_AI_EVAL_BACKEND_URL=http://127.0.0.1:8000 bun run cli -- run flow --backend-validation preview
bun run cli -- run global global-test1-script-create
bun run cli -- run cli bun-hello-script
Public CLI surface:
modelscases [mode]run <mode> [caseIds...]
run options:
--runs <n>: repeat each casentimes--output <path>: custom result JSON path--model <alias>: choose the model under test--models <a,b,c>: run the same cases sequentially against several model aliases--verbose: stream assistant output for frontend runs--skip-judge: skip LLM judge scoring for the run--execution-only: only require the model/proxy/frontend loop to complete; skip validators, tool expectations, backend artifact validation, and judge scoring--reasoning <effort>: reasoning effort for frontend modes (off,low,medium,high,max, …); without it the product's default applies (highon models that can reason). The effort is appended to the recorded model label (anthropic:claude-sonnet-5-5@max)--record: append a compact tracked summary line toai_evals/history/<mode>.jsonlfor full-suite runs only--backend-validation <mode>: optional backend smoke validation (offorpreview) forscriptandflowevals
Models
Use bun run cli -- models to see the current aliases; core/models.ts is the list.
Notes:
-
the command also prints accepted alias spellings such as
gpt-4o,gpt-55,claude-opus-4.6, andclaude-haiku-4.5 -
frontend modes (
flow,script,app,global) can use Anthropic, OpenAI, Gemini, and DeepSeek-backed aliases -
climode runs a coding agent over the workspace, chosen by the alias'sclientry; aliases without one are rejected there:Agent Models Label Needs Claude Code ( @anthropic-ai/claude-agent-sdk)Claude aliases anthropic:<model>ANTHROPIC_API_KEYClaude Code on DeepSeek's Anthropic-compatible endpoint deepseek-v4-proclaude-code:<model>DEEPSEEK_API_KEYCodex ( codex exec --json)gpt-6.1-sol,gpt-6-astracodex:<model>codexonPATH(CI pins 0.159.3),OPENAI_API_KEYCodex runs with a throwaway
HOME, so your own~/.codexconfig and skills stay out of the run. It has no Skill tool: it finds.agents/skillson its own and loads a skill by reading itsSKILL.md, so the trace records any command that touches.agents/skills/<name>/SKILL.mdas aSkillcall to<name>, just before that command, or just after it when the command mutates something before reading the skill. Its shell commands are recorded asBashand its patches asEdit, so theinvokes skill … before first mutationchecks compare the same things for both agents. Codex has no turn cap; a run stops atmaxTurnsrounds of tool calls (parallel calls count once) and fails, as a Claude Code run does at its cap. -
the judge model is separate and currently defaults to
claude-sonnet-5-5; use--skip-judgefor deterministic-only runs
Case Format
Cases live in one YAML file per mode under ai_evals/cases/.
Minimal shape:
- id: flow-test0-sum-two-numbers
prompt: |-
Create a flow that takes two numbers, `a` and `b`, and returns their sum.
initial: ai_evals/fixtures/...
expected: ai_evals/fixtures/...
Optional fields:
initial: starting state fixtureexpected: expected artifact fixturevalidate: extra deterministic validation rulesruntime.backendPreview: optional real backend preview config for smoke validation
For flow mode, validate can express requirements such as:
- accepted input schema shapes
- required
results.*reference validity - required module/code/input characteristics
For app mode, validate can express narrow hard requirements such as:
- required frontend file paths or backend runnable keys
- minimum backend runnable counts
- required backend runnable types
- minimum datatable / datatable-table counts
- specific required datatable tables
For global mode, validate can express draft-level requirements such as:
- required draft type/path/language
- required or forbidden snippets in draft values
- required or forbidden draft counts
- forbidden draft paths
Global initial fixtures can also seed liveEditorDrafts with type,
storagePath, effectivePath, and value fields. These drafts emulate the
currently open script, flow, or raw app editor so cases can test prompts that
refer to "this" or the "current" item.
Global initial fixtures can seed the session's artifacts — { name, versions: [{ content, note? }], role?, approvedVersion? }, oldest version first, so the artifact starts with the
history list_artifact_versions reports — and the previewTabs open in its side panel, for
cases that run with runtime.sessionChat: true. A tab entry names one destination and may
be the active one:
"previewTabs": [{ "artifact": { "name": "Onboarding plan", "version": 2 }, "active": true }]
page ({ href, label }) and item ({ kind, path }) tabs work the same way. Tabs are
driven by the production tab model, so open_preview, get_preview_status and
close_page really open, report and close them, and a version is the pin a reader
chose in the artifact's version picker — which only get_preview_status reports.
Global initial fixtures can seed workspace.variables with
{ path, value, is_secret, description?, labels?, ws_specific? } entries so cases can
read and edit variables that already exist in the workspace. Seed a recognizable secret
(the existing fixture uses sk_live_do_not_leak_me) and assert it via valueExcludes to
catch a leak.
Global mode runs against a real workspace
The session's tools run on the backend, so each global attempt creates its own workspace on the eval backend, seeds the fixture there through the API, and deletes it afterwards. What that backend needs:
- a worker serving the
postgresqltag (thenativegroup) for datatable fixtures and the SQL tools, andWMILL_AI_EVAL_DATATABLE_PG_URL(defaultpostgres://postgres:changeme@localhost:5432/postgres) naming a Postgres the worker can reach, where each datatable fixture gets a scratch database; - a superadmin login (the default
admin@windmill.dev), since it creates and deletes workspaces.WMILL_AI_EVAL_BACKEND_WORKSPACEdoes not apply to global mode. A CE backend allows two workspaces besidesadmins, so attempts past that wait for a free slot; WMILL_AI_EVAL_DATABASE_URL, the eval backend's own database, for fixtures whoseuseris not the admin: the harness writes that user and a token of theirs there withpsql(CE cannot create users through the API), adds them to the workspace, and revokes the token afterwards;- network access to the hub, which the hub tools reach through the backend, and the
embeddingcargo feature, without whichsearch_hub_scriptshas no index to search; - the
http_triggercargo feature, which mounts the routes HTTP trigger drafts are checked against; RESOURCE_TYPE_EMBEDDINGS_CHECK_INTERVAL_SECSset low (CI uses 2): each attempt seeds the hub's resource types into its workspace and waits until the resource type search sees them, which takes up to that interval (60 s by default).
A fixture's jobs become completed preview runs with the given id, path, args, logs and
outcome; createdBy is not reproduced. Its resourceTypes are created as workspace types,
alongside whatever types the instance already holds; a fixture that sets none gets the types
of an instance synced with the hub.
runtime.userAnswers scripts the user's side of askUserQuestion in a global case: one
answer per question, in the order they are asked, and a question past the last answer is
dismissed, so userAnswers: [] dismisses every question. Without it the tool reports that
it cannot ask, and the model usually carries on with its own guess, which makes "asks
before creating anything" impossible to assert.
runtime.profile runs a session-chat case (sessionChat: true) as a non-admin role
instead of full access: the prompt, toolset and tool schemas are cut as production's
capabilitiesForRole cuts them, and the workspace's operator_settings are seeded for the
tools that read them directly:
profile:
operator: true
operatorSettings: { builder_apps: true } # also builder_flows, manage_schedules, manage_triggers
A role that builds nothing (no builder rights, manage_schedules and manage_triggers both
false) holds the session in Use, as the chat does in production: the toolset is the
read-only one and the prompt carries the session section that names the role as the reason.
judgeAssistantText: true hands the judge what the assistant said alongside the mode's
output. Use it when the deliverable is the answer itself, such as a refusal and where to go
instead: the global judge otherwise reads only the drafts, and a substring check cannot tell
"you can't create it yourself" from "create it yourself".
toolExpect.toolCallArgs entries support sharedByAtLeast: <n>: at least n recorded
calls to that tool must carry the same non-blank string in the field. Use it for calls that
have to share an identifier, like two test runs of one chat conversation.
toolExpect.toolCallArgs entries additionally support fieldMustBeAbsent: true: no
recorded call to that tool may pass the field at all (an explicit null counts as
passing it). Use it for partial-update tools, where supplying a field the model could
not have read is itself the failure — e.g. write_variable.value on a secret variable.
Global initial fixtures can seed workspace.datatables. Each entry is
{ datatable_name, schemas: { <schema>: { <table>: { columns, rows? } } } },
plus migrations_enabled: true and migrations: [{ timestamp, name, code_up, code_down?, applied? }]
for a datatable that keeps its schema as migrations. Each one becomes a real datatable of
the case's workspace, backed by its scratch database, so the datatable and SQL tools see
exactly what a query would. An absent datatables seed leaves the workspace with none,
which is what the "no datatable configured" blocking cases rely on.
Set WMILL_AI_EVAL_DISABLE_ACTIVE_EDITOR_CONTEXT=1 to run those cases with
the old behavior where the live editor is only discoverable through
list_workspace_items.
App fixtures can also include an optional datatables.json file at the fixture root.
For flow mode, an initial fixture can also include a benchmark workspace catalog of
existing scripts and flows. That lets the real search_workspace and
get_runnable_details tools discover reusable workspace runnables during evals.
If --backend-validation preview is enabled:
scriptevals run a real backend script preview in an isolated temp workspaceflowevals run a real backend flow preview only for cases that defineruntime.backendPreviewflowcases withinitial.workspacefixtures seed those scripts and flows into the preview workspace before preview- when
WMILL_AI_EVAL_BACKEND_WORKSPACEis set,ai_evalscreates or reuses that workspace as a dedicated test workspace, clears managed eval assets underf/evals/*before each preview run, and then reseeds the current case fixtures
Supported backend env vars:
WMILL_AI_EVAL_BACKEND_VALIDATION=previewWMILL_AI_EVAL_BACKEND_URL=http://127.0.0.1:8000WMILL_AI_EVAL_BACKEND_EMAIL=admin@windmill.devWMILL_AI_EVAL_BACKEND_PASSWORD=changemeWMILL_AI_EVAL_BACKEND_WORKSPACE=integration-teststo reuse an existing workspace on CE installs with low workspace limits
Frontend modes require a reachable Windmill backend and send model requests through the workspace AI proxy at /api/w/{workspace}/ai/proxy. At startup, ai_evals checks the resolved backend URL and fails early with setup guidance if the backend cannot be reached or login fails.
For frontend modes:
ai_evalscreates a temporary backend workspace, or creates/reusesWMILL_AI_EVAL_BACKEND_WORKSPACEwhen it is set- it upserts a provider resource under
f/evals/ai/<provider> - frontend requests go through
/api/w/{workspace}/ai/proxy
Results And Artifacts
Every run writes:
- a summary JSON under
ai_evals/results/ - generated artifacts in a sibling directory
If --record is used, the CLI also appends one compact JSON line to:
ai_evals/history/flow.jsonlai_evals/history/script.jsonlai_evals/history/app.jsonlai_evals/history/global.jsonlai_evals/history/cli.jsonl
Each recorded line contains:
- run metadata (
createdAt,gitSha,mode,runModel,judgeModel) - suite totals (
caseCount,attemptCount,passedAttempts,passRate,averageDurationMs,averagePassedDurationMs,averageJudgeScore) - average token usage (
averageTokenUsagePerAttempt,averageTokenUsagePerPassedAttempt) - per-case metrics under
cases[](averageDurationMs,averagePassedDurationMs,averageJudgeScore,averageTokenUsagePerAttempt,averageTokenUsagePerPassedAttempt, pass rate) failedCaseIds
The CLI headline duration and token averages use passed attempts only. All-attempt averages are still recorded to make failures auditable without letting failed attempts skew success cost comparisons.
Example:
- summary:
ai_evals/results/2026-04-09T09-40-33.051Z__flow.json - artifacts:
ai_evals/results/2026-04-09T09-40-33.051Z__flow/
Typical artifacts by mode:
flow:flow.jsonscript:script.jsonplus the generated script fileapp:app.jsonplus frontend/backend filesglobal:global-drafts.jsoncli:assistant-output.txt,trace.json,wmill-invocations.jsonl, plus generated workspace files- backend-validated attempts also include
backend-preview.json
Layout
cases/: one YAML file per modefixtures/: initial and expected fixturescore/: shared loading, model resolution, validation, judging, and result writingmodes/: one runner per modehistory/: optional tracked pass-rate history written byrun --record, one JSONL file per moderesults/: local benchmark output and artifacts
Harness unit tests run in two lanes: bun test adapters/ for plain TypeScript, and
bun run test:frontend-graph for *.vitest.ts files, which exercise adapters built on
frontend code (Svelte runes, SvelteKit aliases) that bun cannot load.
Notes
- Frontend modes reuse the production frontend chat code through the Vitest bridge.
- Global mode evaluates the production global AI tools and validates the resulting AI draft store.
- CLI mode creates an isolated workspace, writes the current checkout guidance into it, and benchmarks the real skills /
AGENTS.mdflow. - CLI mode now also records a structured trace of invoked skills, tool calls, proposed
wmillcommands, and any attemptedwmillexecutions. - Frontend progress streams live while the benchmark is running.
- Deterministic validators should stay focused on real correctness constraints, not one exact implementation shape.