mirror of
https://github.com/windmill-labs/windmill.git
synced 2026-11-17 00:20:51 +00:00
refactor: run the ai session's tools on the backend (#11899)
* feat: add a workspace mcp server with the ai session toolset Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix: keep workspace mcp tool calls on their own routes Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix: keep a token's job tag fence on workspace mcp tool calls Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix: state the workspace mcp token rule in its refusal message Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * feat: let the workspace mcp server span workspaces chosen per token Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * feat: let the mcp url form choose between the tools and workspace servers Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * feat: serve the workspace mcp server as windmill mcp v2 at /api/mcp/v2 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * feat: add read-and-run access to windmill mcp v2 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * feat: serve the full windmill mcp experience at /api/mcp, sign-in first Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix: align the mcp url form with the brand guidelines Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * feat: serve every mcp token from /api/mcp, with one access picker Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * feat: rename mcp access levels and keep selections to scripts and flows on /api/mcp Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix: name the mcp access levels in the url caption Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * chore: name the mcp server windmill Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix: lead the mcp access picker with build and run, and spell out what a selection excludes Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * feat: show the tools each mcp access level gives Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix: list the mcp tools in a collapsible section under the access row Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix: offer all scripts and flows as typed mcp tools again, as wildcard scopes Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix: label the mcp selection access as scripts and flows as tools Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * feat: link mcp run, app draft and deploy results to their windmill pages Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix: mirror the session tool changes from main in the mcp server Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix: keep an ai agent's identity when the ai chat deploys its resource Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * refactor: call the api in-process from the mcp session tools Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix: keep a draft that rebase_draft could not return whole Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix: let read-and-run mcp tokens run scripts and flows that take secrets Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * feat: ask mcp clients to open windmill links in a browser the user sees Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * feat: send mcp clients condensed instructions and serve the full rules on demand Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * refactor: put the core rules first in the mcp instructions instead of condensing them Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix: go straight to the mcp sign-in page after login, without the workspace picker Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix: no workspace error toast on the mcp sign-in page while logged out Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * refactor: run the ai session's reads and draft tools on the backend * feat: open raw app previews on the preview alone, from mcp links and in sessions Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * feat: link mcp script, flow and agent drafts to their editors, and agents to their page Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix: keep the session preview's saved sidebar choice, fold the build log in preview-only views Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * refactor: resolve and start the ai session's runs on the backend * fix: hold mcp tokens below full access to their level on the per-workspace urls Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix: revoke tokens with the new mcp scopes when the migration is reverted Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix: keep an mcp token's scopes when its editor is saved without changing access Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix: mint one jwt per mcp tool call, accept the old approve body, list the new scopes Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * refactor: run delete on the backend and drop the chat's moved tool logic * fix: reset the mcp token editor's picker state for each token Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix: close the session tools' parity gaps and drop the tests of moved logic Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * refactor: mark only mcp selections, read levels imply no endpoint tools Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * feat: run the global ai evals against a real workspace per attempt Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix: allow the skill description that only the mcp instructions read Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * ci: build the scheduled ai evals' backend with the hub embeddings index Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix: give migrations their own wait, bound the npm types fetch, keep draft links mcp-only Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix: seed hub-synced resource types for global eval fixtures that set none Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix: make the global eval workspaces match a production instance Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix: settle a linked agent's unsaved draft before testing its step Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix: seed non-admin eval users in the database, give standalone eval chats no session Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix: let a step test run past a deleted or unreadable linked agent, as the server does Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix: judge npm search matches against the best one, not a fixed score Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * chore: update ee-repo-ref to b3fd7d49f2ba58f5080fabc3a5686c34a0b2684c This commit updates the EE repository reference after PR #877 was merged in windmill-ee-private. Previous ee-repo-ref: 9e960246d51401a2b0487dcffcadd0b43cd67d39 New ee-repo-ref: b3fd7d49f2ba58f5080fabc3a5686c34a0b2684c Automated by sync-ee-ref workflow. * fix: show the draft save label when the server builds the draft, as before Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix: keep a browser caller's session on the session tools' internal calls, so its runs and trigger edits are attributed to the UI as before Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * refactor: keep the chat's npm search in the browser, so its registry requests leave from the user's browser as before Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * test: share the run tools' argument cases between the chat's and the server's handling, so a change on one side only fails Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> Co-authored-by: windmill-internal-app[bot] <windmill-internal-app[bot]@users.noreply.github.com>
This commit is contained in:
co-authored by
Claude Opus 5.5
windmill-internal-app[bot]
parent
dd6ae44136
commit
2044656d74
@@ -105,16 +105,22 @@ jobs:
|
||||
SQLX_OFFLINE: true
|
||||
CARGO_BUILD_JOBS: 12
|
||||
RUSTFLAGS: ""
|
||||
run: cargo build --features quickjs
|
||||
# The global suite's hub cases search the hub embeddings index; its trigger cases
|
||||
# write HTTP triggers, whose routes the feature mounts.
|
||||
run: cargo build --features quickjs,embedding,http_trigger
|
||||
|
||||
- name: Start Windmill
|
||||
working-directory: ./backend
|
||||
env:
|
||||
DATABASE_URL: postgres://postgres:changeme@localhost:5432/windmill
|
||||
RUST_LOG: info
|
||||
# Each global case seeds resource types and waits until the search indexes them.
|
||||
RESOURCE_TYPE_EMBEDDINGS_CHECK_INTERVAL_SECS: "2"
|
||||
run: |
|
||||
mkdir -p ../ai_evals/logs
|
||||
./target/debug/windmill > ../ai_evals/logs/windmill.log 2>&1 &
|
||||
# The global suite's datatable cases run SQL on the `postgresql` tag, a native worker's.
|
||||
MODE=worker WORKER_GROUP=native PORT=8001 ./target/debug/windmill > ../ai_evals/logs/windmill-native.log 2>&1 &
|
||||
for i in $(seq 1 60); do
|
||||
curl -sf http://localhost:8000/api/version > /dev/null 2>&1 && break
|
||||
sleep 2
|
||||
@@ -142,6 +148,8 @@ jobs:
|
||||
MODEL: ${{ matrix.model }}
|
||||
WMILL_AI_EVAL_BACKEND_URL: http://localhost:8000
|
||||
WMILL_AI_EVAL_BACKEND_WORKSPACE: integration-tests
|
||||
# Global fixtures whose user is not the admin are created in the backend's database.
|
||||
WMILL_AI_EVAL_DATABASE_URL: postgres://postgres:changeme@localhost:5432/windmill
|
||||
ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }}
|
||||
OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
|
||||
GEMINI_API_KEY: ${{ secrets.GOOGLE_API_KEY }}
|
||||
|
||||
@@ -3,8 +3,9 @@ name: AI Evals (global mode)
|
||||
# Smoke-tests the production global AI chat proxy/frontend execution path via
|
||||
# the ai_evals harness, one case across one cheap model per provider. Runs only
|
||||
# when the eval harness or the global chat code change, since each run makes real
|
||||
# (paid) LLM calls. The backend is built from source purely as the AI proxy the
|
||||
# harness routes model calls through; the global tools/drafts run in-process in
|
||||
# (paid) LLM calls. The backend is built from source as the AI proxy the harness
|
||||
# routes model calls through and as the server the global tools run on, in a
|
||||
# workspace the harness creates per attempt; the chat itself runs in-process in
|
||||
# the Vitest bridge against production frontend code. To avoid spending on every
|
||||
# commit, the PR side triggers only when a PR is marked ready for review (out of
|
||||
# draft) — not on `synchronize` — plus push to main and manual dispatch.
|
||||
@@ -15,6 +16,7 @@ on:
|
||||
paths:
|
||||
- "ai_evals/**"
|
||||
- "backend/windmill-api/src/ai.rs"
|
||||
- "backend/windmill-api/src/ai_session_tools/**"
|
||||
- "backend/windmill-ai/**"
|
||||
- "frontend/src/lib/components/copilot/**"
|
||||
# The eval harness runs production frontend code in-process; these are the
|
||||
@@ -29,6 +31,7 @@ on:
|
||||
paths:
|
||||
- "ai_evals/**"
|
||||
- "backend/windmill-api/src/ai.rs"
|
||||
- "backend/windmill-api/src/ai_session_tools/**"
|
||||
- "backend/windmill-ai/**"
|
||||
- "frontend/src/lib/components/copilot/**"
|
||||
# The eval harness runs production frontend code in-process; these are the
|
||||
@@ -137,6 +140,8 @@ jobs:
|
||||
env:
|
||||
WMILL_AI_EVAL_BACKEND_URL: http://localhost:8000
|
||||
WMILL_AI_EVAL_BACKEND_WORKSPACE: integration-tests
|
||||
# Global fixtures whose user is not the admin are created in the backend's database.
|
||||
WMILL_AI_EVAL_DATABASE_URL: postgres://postgres:changeme@localhost:5432/windmill
|
||||
# Anthropic backs the haiku model. Google AI uses GEMINI_API_KEY.
|
||||
ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }}
|
||||
OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
|
||||
|
||||
Reference in New Issue
Block a user