refactor: run the ai session's tools on the backend (#11899)

* feat: add a workspace mcp server with the ai session toolset

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix: keep workspace mcp tool calls on their own routes

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix: keep a token's job tag fence on workspace mcp tool calls

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix: state the workspace mcp token rule in its refusal message

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* feat: let the workspace mcp server span workspaces chosen per token

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* feat: let the mcp url form choose between the tools and workspace servers

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* feat: serve the workspace mcp server as windmill mcp v2 at /api/mcp/v2

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* feat: add read-and-run access to windmill mcp v2

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* feat: serve the full windmill mcp experience at /api/mcp, sign-in first

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix: align the mcp url form with the brand guidelines

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* feat: serve every mcp token from /api/mcp, with one access picker

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* feat: rename mcp access levels and keep selections to scripts and flows on /api/mcp

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix: name the mcp access levels in the url caption

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* chore: name the mcp server windmill

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix: lead the mcp access picker with build and run, and spell out what a selection excludes

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* feat: show the tools each mcp access level gives

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix: list the mcp tools in a collapsible section under the access row

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix: offer all scripts and flows as typed mcp tools again, as wildcard scopes

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix: label the mcp selection access as scripts and flows as tools

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* feat: link mcp run, app draft and deploy results to their windmill pages

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix: mirror the session tool changes from main in the mcp server

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix: keep an ai agent's identity when the ai chat deploys its resource

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* refactor: call the api in-process from the mcp session tools

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix: keep a draft that rebase_draft could not return whole

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix: let read-and-run mcp tokens run scripts and flows that take secrets

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* feat: ask mcp clients to open windmill links in a browser the user sees

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* feat: send mcp clients condensed instructions and serve the full rules on demand

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* refactor: put the core rules first in the mcp instructions instead of condensing them

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix: go straight to the mcp sign-in page after login, without the workspace picker

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix: no workspace error toast on the mcp sign-in page while logged out

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* refactor: run the ai session's reads and draft tools on the backend

* feat: open raw app previews on the preview alone, from mcp links and in sessions

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* feat: link mcp script, flow and agent drafts to their editors, and agents to their page

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix: keep the session preview's saved sidebar choice, fold the build log in preview-only views

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* refactor: resolve and start the ai session's runs on the backend

* fix: hold mcp tokens below full access to their level on the per-workspace urls

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix: revoke tokens with the new mcp scopes when the migration is reverted

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix: keep an mcp token's scopes when its editor is saved without changing access

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix: mint one jwt per mcp tool call, accept the old approve body, list the new scopes

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* refactor: run delete on the backend and drop the chat's moved tool logic

* fix: reset the mcp token editor's picker state for each token

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix: close the session tools' parity gaps and drop the tests of moved logic

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* refactor: mark only mcp selections, read levels imply no endpoint tools

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* feat: run the global ai evals against a real workspace per attempt

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix: allow the skill description that only the mcp instructions read

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* ci: build the scheduled ai evals' backend with the hub embeddings index

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix: give migrations their own wait, bound the npm types fetch, keep draft links mcp-only

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix: seed hub-synced resource types for global eval fixtures that set none

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix: make the global eval workspaces match a production instance

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix: settle a linked agent's unsaved draft before testing its step

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix: seed non-admin eval users in the database, give standalone eval chats no session

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix: let a step test run past a deleted or unreadable linked agent, as the server does

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix: judge npm search matches against the best one, not a fixed score

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* chore: update ee-repo-ref to b3fd7d49f2ba58f5080fabc3a5686c34a0b2684c

This commit updates the EE repository reference after PR #877 was merged in windmill-ee-private.

Previous ee-repo-ref: 9e960246d51401a2b0487dcffcadd0b43cd67d39

New ee-repo-ref: b3fd7d49f2ba58f5080fabc3a5686c34a0b2684c

Automated by sync-ee-ref workflow.

* fix: show the draft save label when the server builds the draft, as before

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix: keep a browser caller's session on the session tools' internal calls, so its runs and trigger edits are attributed to the UI as before

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* refactor: keep the chat's npm search in the browser, so its registry requests leave from the user's browser as before

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test: share the run tools' argument cases between the chat's and the server's handling, so a change on one side only fails

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: windmill-internal-app[bot] <windmill-internal-app[bot]@users.noreply.github.com>
This commit is contained in:
hugocasa
2026-11-02 23:57:00 +01:00
committed by GitHub
co-authored by Claude Opus 5.5 windmill-internal-app[bot]
parent dd6ae44136
commit 2044656d74
70 changed files with 5635 additions and 13701 deletions
+9 -1
View File
@@ -105,16 +105,22 @@ jobs:
SQLX_OFFLINE: true
CARGO_BUILD_JOBS: 12
RUSTFLAGS: ""
run: cargo build --features quickjs
# The global suite's hub cases search the hub embeddings index; its trigger cases
# write HTTP triggers, whose routes the feature mounts.
run: cargo build --features quickjs,embedding,http_trigger
- name: Start Windmill
working-directory: ./backend
env:
DATABASE_URL: postgres://postgres:changeme@localhost:5432/windmill
RUST_LOG: info
# Each global case seeds resource types and waits until the search indexes them.
RESOURCE_TYPE_EMBEDDINGS_CHECK_INTERVAL_SECS: "2"
run: |
mkdir -p ../ai_evals/logs
./target/debug/windmill > ../ai_evals/logs/windmill.log 2>&1 &
# The global suite's datatable cases run SQL on the `postgresql` tag, a native worker's.
MODE=worker WORKER_GROUP=native PORT=8001 ./target/debug/windmill > ../ai_evals/logs/windmill-native.log 2>&1 &
for i in $(seq 1 60); do
curl -sf http://localhost:8000/api/version > /dev/null 2>&1 && break
sleep 2
@@ -142,6 +148,8 @@ jobs:
MODEL: ${{ matrix.model }}
WMILL_AI_EVAL_BACKEND_URL: http://localhost:8000
WMILL_AI_EVAL_BACKEND_WORKSPACE: integration-tests
# Global fixtures whose user is not the admin are created in the backend's database.
WMILL_AI_EVAL_DATABASE_URL: postgres://postgres:changeme@localhost:5432/windmill
ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }}
OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
GEMINI_API_KEY: ${{ secrets.GOOGLE_API_KEY }}
+7 -2
View File
@@ -3,8 +3,9 @@ name: AI Evals (global mode)
# Smoke-tests the production global AI chat proxy/frontend execution path via
# the ai_evals harness, one case across one cheap model per provider. Runs only
# when the eval harness or the global chat code change, since each run makes real
# (paid) LLM calls. The backend is built from source purely as the AI proxy the
# harness routes model calls through; the global tools/drafts run in-process in
# (paid) LLM calls. The backend is built from source as the AI proxy the harness
# routes model calls through and as the server the global tools run on, in a
# workspace the harness creates per attempt; the chat itself runs in-process in
# the Vitest bridge against production frontend code. To avoid spending on every
# commit, the PR side triggers only when a PR is marked ready for review (out of
# draft) — not on `synchronize` — plus push to main and manual dispatch.
@@ -15,6 +16,7 @@ on:
paths:
- "ai_evals/**"
- "backend/windmill-api/src/ai.rs"
- "backend/windmill-api/src/ai_session_tools/**"
- "backend/windmill-ai/**"
- "frontend/src/lib/components/copilot/**"
# The eval harness runs production frontend code in-process; these are the
@@ -29,6 +31,7 @@ on:
paths:
- "ai_evals/**"
- "backend/windmill-api/src/ai.rs"
- "backend/windmill-api/src/ai_session_tools/**"
- "backend/windmill-ai/**"
- "frontend/src/lib/components/copilot/**"
# The eval harness runs production frontend code in-process; these are the
@@ -137,6 +140,8 @@ jobs:
env:
WMILL_AI_EVAL_BACKEND_URL: http://localhost:8000
WMILL_AI_EVAL_BACKEND_WORKSPACE: integration-tests
# Global fixtures whose user is not the admin are created in the backend's database.
WMILL_AI_EVAL_DATABASE_URL: postgres://postgres:changeme@localhost:5432/windmill
# Anthropic backs the haiku model. Google AI uses GEMINI_API_KEY.
ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }}
OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}