refactor: run the ai session's tools on the backend (#11899)

* feat: add a workspace mcp server with the ai session toolset

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix: keep workspace mcp tool calls on their own routes

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix: keep a token's job tag fence on workspace mcp tool calls

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix: state the workspace mcp token rule in its refusal message

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* feat: let the workspace mcp server span workspaces chosen per token

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* feat: let the mcp url form choose between the tools and workspace servers

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* feat: serve the workspace mcp server as windmill mcp v2 at /api/mcp/v2

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* feat: add read-and-run access to windmill mcp v2

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* feat: serve the full windmill mcp experience at /api/mcp, sign-in first

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix: align the mcp url form with the brand guidelines

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* feat: serve every mcp token from /api/mcp, with one access picker

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* feat: rename mcp access levels and keep selections to scripts and flows on /api/mcp

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix: name the mcp access levels in the url caption

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* chore: name the mcp server windmill

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix: lead the mcp access picker with build and run, and spell out what a selection excludes

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* feat: show the tools each mcp access level gives

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix: list the mcp tools in a collapsible section under the access row

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix: offer all scripts and flows as typed mcp tools again, as wildcard scopes

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix: label the mcp selection access as scripts and flows as tools

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* feat: link mcp run, app draft and deploy results to their windmill pages

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix: mirror the session tool changes from main in the mcp server

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix: keep an ai agent's identity when the ai chat deploys its resource

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* refactor: call the api in-process from the mcp session tools

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix: keep a draft that rebase_draft could not return whole

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix: let read-and-run mcp tokens run scripts and flows that take secrets

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* feat: ask mcp clients to open windmill links in a browser the user sees

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* feat: send mcp clients condensed instructions and serve the full rules on demand

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* refactor: put the core rules first in the mcp instructions instead of condensing them

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix: go straight to the mcp sign-in page after login, without the workspace picker

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix: no workspace error toast on the mcp sign-in page while logged out

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* refactor: run the ai session's reads and draft tools on the backend

* feat: open raw app previews on the preview alone, from mcp links and in sessions

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* feat: link mcp script, flow and agent drafts to their editors, and agents to their page

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix: keep the session preview's saved sidebar choice, fold the build log in preview-only views

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* refactor: resolve and start the ai session's runs on the backend

* fix: hold mcp tokens below full access to their level on the per-workspace urls

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix: revoke tokens with the new mcp scopes when the migration is reverted

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix: keep an mcp token's scopes when its editor is saved without changing access

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix: mint one jwt per mcp tool call, accept the old approve body, list the new scopes

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* refactor: run delete on the backend and drop the chat's moved tool logic

* fix: reset the mcp token editor's picker state for each token

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix: close the session tools' parity gaps and drop the tests of moved logic

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* refactor: mark only mcp selections, read levels imply no endpoint tools

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* feat: run the global ai evals against a real workspace per attempt

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix: allow the skill description that only the mcp instructions read

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* ci: build the scheduled ai evals' backend with the hub embeddings index

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix: give migrations their own wait, bound the npm types fetch, keep draft links mcp-only

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix: seed hub-synced resource types for global eval fixtures that set none

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix: make the global eval workspaces match a production instance

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix: settle a linked agent's unsaved draft before testing its step

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix: seed non-admin eval users in the database, give standalone eval chats no session

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix: let a step test run past a deleted or unreadable linked agent, as the server does

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix: judge npm search matches against the best one, not a fixed score

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* chore: update ee-repo-ref to b3fd7d49f2ba58f5080fabc3a5686c34a0b2684c

This commit updates the EE repository reference after PR #877 was merged in windmill-ee-private.

Previous ee-repo-ref: 9e960246d51401a2b0487dcffcadd0b43cd67d39

New ee-repo-ref: b3fd7d49f2ba58f5080fabc3a5686c34a0b2684c

Automated by sync-ee-ref workflow.

* fix: show the draft save label when the server builds the draft, as before

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix: keep a browser caller's session on the session tools' internal calls, so its runs and trigger edits are attributed to the UI as before

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* refactor: keep the chat's npm search in the browser, so its registry requests leave from the user's browser as before

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test: share the run tools' argument cases between the chat's and the server's handling, so a change on one side only fails

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: windmill-internal-app[bot] <windmill-internal-app[bot]@users.noreply.github.com>
This commit is contained in:
hugocasa
2026-11-02 23:57:00 +01:00
committed by GitHub
co-authored by Claude Opus 5.5 windmill-internal-app[bot]
parent dd6ae44136
commit 2044656d74
70 changed files with 5635 additions and 13701 deletions
+4
View File
@@ -36,6 +36,10 @@ WMILL_AI_EVAL_BACKEND_URL=http://127.0.0.1:<port> WMILL_AI_EVAL_BACKEND_WORKSPAC
`WMILL_AI_EVAL_BACKEND_WORKSPACE=integration-tests` (or any existing workspace) to
reuse one. The only side effect of a run is upserting an `f/evals/ai/<provider>`
resource there.
- **Global mode creates its own workspace per attempt** and seeds the case's fixture in it,
because the session's tools run on the backend; it ignores
`WMILL_AI_EVAL_BACKEND_WORKSPACE` and needs a `postgresql` (native) worker for datatable
cases. `ai_evals/README.md` ("Global mode runs against a real workspace") lists the rest.
- Provider keys live in `ai_evals/.env` and are auto-loaded by bun. The judge is a
separate Anthropic call (default `claude-sonnet-4-6`) regardless of the model under
test.
+9 -1
View File
@@ -105,16 +105,22 @@ jobs:
SQLX_OFFLINE: true
CARGO_BUILD_JOBS: 12
RUSTFLAGS: ""
run: cargo build --features quickjs
# The global suite's hub cases search the hub embeddings index; its trigger cases
# write HTTP triggers, whose routes the feature mounts.
run: cargo build --features quickjs,embedding,http_trigger
- name: Start Windmill
working-directory: ./backend
env:
DATABASE_URL: postgres://postgres:changeme@localhost:5432/windmill
RUST_LOG: info
# Each global case seeds resource types and waits until the search indexes them.
RESOURCE_TYPE_EMBEDDINGS_CHECK_INTERVAL_SECS: "2"
run: |
mkdir -p ../ai_evals/logs
./target/debug/windmill > ../ai_evals/logs/windmill.log 2>&1 &
# The global suite's datatable cases run SQL on the `postgresql` tag, a native worker's.
MODE=worker WORKER_GROUP=native PORT=8001 ./target/debug/windmill > ../ai_evals/logs/windmill-native.log 2>&1 &
for i in $(seq 1 60); do
curl -sf http://localhost:8000/api/version > /dev/null 2>&1 && break
sleep 2
@@ -142,6 +148,8 @@ jobs:
MODEL: ${{ matrix.model }}
WMILL_AI_EVAL_BACKEND_URL: http://localhost:8000
WMILL_AI_EVAL_BACKEND_WORKSPACE: integration-tests
# Global fixtures whose user is not the admin are created in the backend's database.
WMILL_AI_EVAL_DATABASE_URL: postgres://postgres:changeme@localhost:5432/windmill
ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }}
OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
GEMINI_API_KEY: ${{ secrets.GOOGLE_API_KEY }}
+7 -2
View File
@@ -3,8 +3,9 @@ name: AI Evals (global mode)
# Smoke-tests the production global AI chat proxy/frontend execution path via
# the ai_evals harness, one case across one cheap model per provider. Runs only
# when the eval harness or the global chat code change, since each run makes real
# (paid) LLM calls. The backend is built from source purely as the AI proxy the
# harness routes model calls through; the global tools/drafts run in-process in
# (paid) LLM calls. The backend is built from source as the AI proxy the harness
# routes model calls through and as the server the global tools run on, in a
# workspace the harness creates per attempt; the chat itself runs in-process in
# the Vitest bridge against production frontend code. To avoid spending on every
# commit, the PR side triggers only when a PR is marked ready for review (out of
# draft) — not on `synchronize` — plus push to main and manual dispatch.
@@ -15,6 +16,7 @@ on:
paths:
- "ai_evals/**"
- "backend/windmill-api/src/ai.rs"
- "backend/windmill-api/src/ai_session_tools/**"
- "backend/windmill-ai/**"
- "frontend/src/lib/components/copilot/**"
# The eval harness runs production frontend code in-process; these are the
@@ -29,6 +31,7 @@ on:
paths:
- "ai_evals/**"
- "backend/windmill-api/src/ai.rs"
- "backend/windmill-api/src/ai_session_tools/**"
- "backend/windmill-ai/**"
- "frontend/src/lib/components/copilot/**"
# The eval harness runs production frontend code in-process; these are the
@@ -137,6 +140,8 @@ jobs:
env:
WMILL_AI_EVAL_BACKEND_URL: http://localhost:8000
WMILL_AI_EVAL_BACKEND_WORKSPACE: integration-tests
# Global fixtures whose user is not the admin are created in the backend's database.
WMILL_AI_EVAL_DATABASE_URL: postgres://postgres:changeme@localhost:5432/windmill
# Anthropic backs the haiku model. Google AI uses GEMINI_API_KEY.
ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }}
OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
+1 -1
View File
@@ -69,7 +69,7 @@ Open-source platform for internal tools, workflows, API integrations, background
- **Domain vocabulary**: `CONTEXT.md` — the words this codebase uses for its own concepts (step, step setting, trigger step, …). Name things the way it does.
- **CLI commands**: when adding/modifying/removing a command, subcommand, option, or description in `cli/src/commands/`, run `python system_prompts/generate.py` to refresh `system_prompts/auto-generated/` and `cli/src/guidance/skills.gen.ts`. The CLI docs the agents use to operate `wmill` are derived from the source — stale generated files give agents the wrong flags.
- **AI guidance** (chat prompts and CLI skills): what the AI knows about Windmill itself — how flows, scripts, apps, pipelines and resources work — is written once, in `system_prompts/base/` and `system_prompts/languages/`, and `system_prompts/generate.py` builds both the chat's `$system_prompts` exports and the CLI's skills from it. Add or change such guidance there, never inline in one consumer: a sentence only one side should see goes in a `<!-- cli-only -->` / `<!-- chat-only -->` block, and a new file reaches both through the `TOPICS` table. The chat prompt builders under `frontend/src/lib/components/copilot/chat/` add only tool plumbing and runtime values, and shared markdown names no chat tool, since it reaches every chat mode. Rerun `generate.py` after editing; `system_prompts/README.md` has the details.
- **Windmill MCP server**: `/api/mcp` gives a token with `mcp:all` the AI session's GLOBAL toolset (`backend/windmill-api/src/mcp/session/`), and a token scoped to a selection of scripts and flows the tools server's per-runnable tools, without endpoint tools. The tools server's own URLs, `/api/mcp/w/{workspace}/mcp` and `/api/mcp/gateway`, keep serving existing tokens, endpoint scopes included, except that they give the session toolset to an `mcp:all` token carrying `mcp:session` (added by the OAuth approve, kept across scope edits) or `mcp:read_only`/`mcp:read_run`. Every tool takes a `workspace_id`; the token bounds which (its bound workspace, the `mcp:workspaces:` list, or all of its owner's). With the `mcp_external_idp` instance setting, every MCP URL also accepts an access token of an external OIDC issuer (`mcp/external_idp.rs`), told apart from Windmill's own by its `iss`: the token's client is mapped to a workspace and to the same access a Windmill token takes (an access level, served the session toolset as `mcp:session`, or a selection of scripts and flows), and the request acts as the existing Windmill user with the token's email, under that access, as a workspace-bound token does. The protected-resource metadata then names that issuer. What a model reads there (tool names, descriptions, input schemas, instructions, `get_instructions` texts) is generated from the session's TypeScript into `session/gen/`: after changing a GLOBAL tool definition, the global system prompt or `system_prompts/`, run `npm run gen:mcp-session` from `frontend/` (`mcpSessionGen.test.ts` fails otherwise). What a tool *does* is written twice, in `chat/global/core.ts` and in a Rust handler: a behaviour change on one side must be mirrored on the other, and a new session tool is either implemented in Rust or added to `EDITOR_ONLY` (a `cargo test` in `session/mod.rs` fails otherwise). For MCP Apps clients the server adds tools of its own, defined in Rust (`session/views.rs`) rather than generated: `show_run`, `show_runs`, `show_deploy_review` and `show_item` draw a card in the `ui://windmill/card-v<N>` view (`session/card.html`; bump `CARD_URI` when the `_meta` it reads changes shape, since hosts cache a view by URI) from their result's `_meta` (`windmill/run`, `windmill/runs`, `windmill/review`, `windmill/item`), which the model never reads. A host draws a card for every call of a tool that declares one, so only these show tools do: the tools the model reads and runs with draw nothing, and `get_run` builds no card data. The card acts only by calling the session's tools (`show_run` to refresh, `cancel_job`, `show_deploy_review` to re-read a draft, `deploy_workspace_item` from the review card's Deploy button, which the user presses), so it sees nothing the model could not. Every `DESTRUCTIVE` tool but those in `CARD_ACTIONS` is hidden from it. Every session tool carries a `title` and explicit read-only, destructive and open-world hints, which the Claude and ChatGPT directories require (`DESTRUCTIVE`, `OPEN_WORLD`). A client that declares capabilities without the MCP Apps extension is not offered the show tools. It exists only on the MCP side, so `core.ts` has nothing to mirror.
- **Windmill MCP server**: `/api/mcp` gives a token with `mcp:all` the AI session's GLOBAL toolset (`backend/windmill-api/src/ai_session_tools/`), and a token scoped to a selection of scripts and flows the tools server's per-runnable tools, without endpoint tools. The tools server's own URLs, `/api/mcp/w/{workspace}/mcp` and `/api/mcp/gateway`, keep serving existing tokens, endpoint scopes included, except that they give the session toolset to an `mcp:all` token carrying `mcp:session` (added by the OAuth approve, kept across scope edits) or `mcp:read_only`/`mcp:read_run`. Every tool takes a `workspace_id`; the token bounds which (its bound workspace, the `mcp:workspaces:` list, or all of its owner's). With the `mcp_external_idp` instance setting, every MCP URL also accepts an access token of an external OIDC issuer (`mcp/external_idp.rs`), told apart from Windmill's own by its `iss`: the token's client is mapped to a workspace and to the same access a Windmill token takes (an access level, served the session toolset as `mcp:session`, or a selection of scripts and flows), and the request acts as the existing Windmill user with the token's email, under that access, as a workspace-bound token does. The protected-resource metadata then names that issuer. What a model reads there (tool names, descriptions, input schemas, instructions, `get_instructions` texts) is generated from the session's TypeScript into `ai_session_tools/gen/`: after changing a GLOBAL tool definition, the global system prompt or `system_prompts/`, run `npm run gen:mcp-session` from `frontend/` (`mcpSessionGen.test.ts` fails otherwise). What a tool *does* is written once, in a Rust handler the chat also calls (`POST /w/{workspace}/ai_session_tools/{name}`, through `chat/global/backendSessionTools.ts`); the browser applies the effects it returns, saving drafts through its own draft store. A new session tool is either implemented in Rust or added to `EDITOR_ONLY` (a `cargo test` in `ai_session_tools/mod.rs` fails otherwise). Still written twice, so a behaviour change must be mirrored: `deploy_workspace_item` and `diff` (the chat uses the UI's shared deploy and diff code), and the run tools' form side (argument coercion, secrets, plan mode, the outcome text), where the chat resolves and starts the run on the server but owns the form in between; both sides test their argument handling against `ai_session_tools/run_args_cases.json`, so a change there comes with a case. For MCP Apps clients the server adds tools of its own, defined in Rust (`ai_session_tools/views.rs`) rather than generated, and not offered to the chat, which draws its own cards: `show_run`, `show_runs`, `show_deploy_review` and `show_item` draw a card in the `ui://windmill/card-v<N>` view (`mcp/card.html`; bump `CARD_URI` in `mcp/session.rs` when the `_meta` it reads changes shape, since hosts cache a view by URI) from their result's `_meta` (`windmill/run`, `windmill/runs`, `windmill/review`, `windmill/item`), which the model never reads. A host draws a card for every call of a tool that declares one, so only these show tools do: the tools the model reads and runs with draw nothing, and `get_run` builds no card data. The card acts only by calling the session's tools (`show_run` to refresh, `cancel_job`, `show_deploy_review` to re-read a draft, `deploy_workspace_item` from the review card's Deploy button, which the user presses), so it sees nothing the model could not. Every `DESTRUCTIVE` tool but those in `CARD_ACTIONS` is hidden from it. Every session tool carries a `title` and explicit read-only, destructive and open-world hints, which the Claude and ChatGPT directories require (`DESTRUCTIVE`, `OPEN_WORLD`). A client that declares capabilities without the MCP Apps extension is not offered the show tools.
- **Session recorder**: `frontend/src/lib/components/recording/` is also the recorder `wmill app dev --recording` serves, vendored into the CLI as `cli/src/commands/app/devRecorderBundle.gen.ts`. After changing `rawAppSnapshot.ts` or `rawAppRecording.svelte.ts`, run `bun run gen:dev-recorder` from `cli/` (`cli/test/dev_recorder_bundle_unit.test.ts` fails otherwise).
- **Raw-app policy**: `frontend/src/lib/components/raw_apps/rawAppPolicy.ts` also derives the policy the server's raw-app deploy stores, vendored into the bundle job as `backend/windmill-api/src/apps_raw_policy.gen.js`. After changing it or anything it imports, run `bun run gen:app-policy` from `cli/` (`cli/test/app_policy_bundle_unit.test.ts` fails otherwise). It rides in the job rather than being read from the CLI the job runs because the images install `windmill-cli` unpinned, so an image can carry one older than its server.
+36 -23
View File
@@ -170,13 +170,38 @@ chose in the artifact's version picker — which only `get_preview_status` repor
Global initial fixtures can seed `workspace.variables` with
`{ path, value, is_secret, description?, labels?, ws_specific? }` entries so cases can
read and edit variables that already exist in the workspace. The mock mirrors the real
`get_variable`, **decrypt-by-default included**: a secret's `value` is withheld only
when the caller explicitly passes `decryptSecret: false`, and omitting the flag returns
the decrypted value, exactly as against a real backend. The chat's read path passes
`decryptSecret: false`, so a case can verify it never invents a value it was not shown.
Seed a recognizable secret (the existing fixture uses `sk_live_do_not_leak_me`) and
assert it via `valueExcludes` to catch a leak.
read and edit variables that already exist in the workspace. Seed a recognizable secret
(the existing fixture uses `sk_live_do_not_leak_me`) and assert it via `valueExcludes` to
catch a leak.
### Global mode runs against a real workspace
The session's tools run on the backend, so each global attempt creates its own workspace on
the eval backend, seeds the fixture there through the API, and deletes it afterwards. What
that backend needs:
- a worker serving the `postgresql` tag (the `native` group) for datatable fixtures and the
SQL tools, and `WMILL_AI_EVAL_DATATABLE_PG_URL` (default
`postgres://postgres:changeme@localhost:5432/postgres`) naming a Postgres the worker can
reach, where each datatable fixture gets a scratch database;
- a superadmin login (the default `admin@windmill.dev`), since it creates and deletes
workspaces. `WMILL_AI_EVAL_BACKEND_WORKSPACE` does not apply to global mode. A CE backend
allows two workspaces besides `admins`, so attempts past that wait for a free slot;
- `WMILL_AI_EVAL_DATABASE_URL`, the eval backend's own database, for fixtures whose `user` is
not the admin: the harness writes that user and a token of theirs there with `psql` (CE cannot create
users through the API), adds them to the workspace, and revokes the token afterwards;
- network access to the hub, which the hub tools reach through the backend, and the
`embedding` cargo feature, without which `search_hub_scripts` has no index to search;
- the `http_trigger` cargo feature, which mounts the routes HTTP trigger drafts are checked
against;
- `RESOURCE_TYPE_EMBEDDINGS_CHECK_INTERVAL_SECS` set low (CI uses 2): each attempt seeds the
hub's resource types into its workspace and waits until the resource type search sees them,
which takes up to that interval (60 s by default).
A fixture's `jobs` become completed preview runs with the given id, path, args, logs and
outcome; `createdBy` is not reproduced. Its `resourceTypes` are created as workspace types,
alongside whatever types the instance already holds; a fixture that sets none gets the types
of an instance synced with the hub.
`runtime.userAnswers` scripts the user's side of `askUserQuestion` in a global case: one
answer per question, in the order they are asked, and a question past the last answer is
@@ -213,24 +238,12 @@ recorded call to that tool may pass the field at all (an explicit `null` counts
passing it). Use it for partial-update tools, where supplying a field the model could
not have read is itself the failure — e.g. `write_variable.value` on a secret variable.
Global (and flow) initial fixtures can seed `workspace.datatables` so the
`list_datatables`, `get_datatable_table_schema`, and `exec_datatable_sql` tools
return seeded data during evals. Each entry is
Global initial fixtures can seed `workspace.datatables`. Each entry is
`{ datatable_name, schemas: { <schema>: { <table>: { columns, rows? } } } }`,
plus `migrations_enabled: true` and `migrations: [{ timestamp, name, code_up, code_down?, applied? }]`
for a datatable that keeps its schema as migrations, which the migration tools then
read and write in memory.
SQL runs through a small in-memory engine (`datatableSqlEngine.ts`), not a real
database. Writes are **stateful within a case**: `CREATE`/`DROP`/`INSERT`/`UPDATE`/
`DELETE` mutate the seeded datatable in place, so a later `list_datatables`,
`get_datatable_table_schema`, `SELECT`, or `information_schema` query reflects them
— this is what stops a model from looping when it re-queries to verify a write.
The engine is best-effort: `SELECT` returns all rows of the referenced (or first)
table with no WHERE filtering/projection/joins, `WHERE` on UPDATE/DELETE supports
`col = value` predicates joined by `AND`, and anything unparseable is a no-op
success. So validate datatable cases through tool-use and SQL-argument assertions
(`requiredToolsUsed`, `stringIncludesAnyOf`) — not through exact returned row
values. An empty/absent `datatables` seed makes `list_datatables` return `[]`,
for a datatable that keeps its schema as migrations. Each one becomes a real datatable of
the case's workspace, backed by its scratch database, so the datatable and SQL tools see
exactly what a query would. An absent `datatables` seed leaves the workspace with none,
which is what the "no datatable configured" blocking cases rely on.
Set `WMILL_AI_EVAL_DISABLE_ACTIVE_EDITOR_CONTEXT=1` to run those cases with
@@ -1,36 +0,0 @@
import { describe, expect, it } from 'vitest'
import {
handleBenchmarkApiFetch,
hasBenchmarkApiHandler,
registerBenchmarkWorkspaceRunnables,
unregisterBenchmarkWorkspace
} from './mockBackend'
// A global eval run registers its workspace under a mkdtemp path, so the workspace id the
// frontend interpolates into the models URL carries slashes. A handler that assumed a single
// path segment silently fell through to the network, and every run took the offline fallback.
const WORKSPACE = '/tmp/wmill-frontend-global-benchmark-abc123'
const RESOURCE = 'f/evals/global/anthropic_main'
const URL_FOR = (workspace: string) =>
`http://benchmark.local/api/w/${workspace}/ai/proxy/models`
describe('benchmark /ai/proxy/models', () => {
it('serves the seeded listing for a workspace id that is a path', async () => {
registerBenchmarkWorkspaceRunnables(WORKSPACE, {
aiProviders: [
{ path: RESOURCE, kind: 'anthropic', models: ['claude-sonnet-5', 'claude-opus-5'] }
]
})
try {
expect(hasBenchmarkApiHandler(URL_FOR(WORKSPACE))).toBe(true)
const response = handleBenchmarkApiFetch(URL_FOR(WORKSPACE), {
headers: { 'X-Resource-Path': RESOURCE, 'X-Provider': 'anthropic' }
})
await expect(response.json()).resolves.toEqual({
data: [{ id: 'claude-sonnet-5' }, { id: 'claude-opus-5' }]
})
} finally {
unregisterBenchmarkWorkspace(WORKSPACE)
}
})
})
@@ -1,6 +1,3 @@
import { mkdtemp, rm } from "fs/promises";
import { tmpdir } from "os";
import { join } from "path";
import type { ChatCompletionMessageParam } from "openai/resources/chat/completions.mjs";
import type { AIProvider, OperatorSettings } from "$lib/gen/types.gen";
import {
@@ -29,6 +26,7 @@ import { appendPlanModeInstructions } from "../../../../../frontend/src/lib/comp
import type { Tool as ProductionTool } from "../../../../../frontend/src/lib/components/copilot/chat/shared";
import { createEvalPlanTools } from "./planModeTools";
import { UserDraft } from "../../../../../frontend/src/lib/userDraft.svelte";
import { DraftService } from "$lib/gen";
import {
createEvalArtifactHelpers,
type SeededArtifact,
@@ -42,11 +40,11 @@ import type { EvalCaseProfile, ModeRunContext } from "../../../../core/types";
import type { GlobalDraftState } from "../../../../core/validators";
import type { WindmillBackendSettings } from "../../../../core/windmillBackendSettings";
import {
registerBenchmarkWorkspaceRunnables,
seedBenchmarkDraft,
registerBenchmarkAiModels,
unregisterBenchmarkWorkspaceRunnables,
type BenchmarkWorkspaceRunnables,
} from "../../mockBackend";
import { createRealWorkspace } from "./realWorkspace";
import { runEval } from "../shared";
import type { TokenUsage, ToolCallDetail } from "../shared/types";
@@ -140,7 +138,6 @@ export interface GlobalEvalOptions {
userAnswers?: string[];
provider?: AIProvider;
backend: WindmillBackendSettings;
workspaceRoot?: string;
// Artifacts the session already holds when the run starts.
artifacts?: SeededArtifact[];
/** Tabs already open in the side panel, including any artifact version the reader pinned. */
@@ -153,18 +150,22 @@ export async function runGlobalEval(
apiKey: string,
options: GlobalEvalOptions,
): Promise<GlobalEvalResult> {
const workspaceRoot =
options.workspaceRoot ??
(await mkdtemp(join(tmpdir(), "wmill-frontend-global-benchmark-")));
const realWorkspace = await createRealWorkspace({
settings: options.backend,
caseId: options.runContext?.caseId ?? "global",
attempt: options.runContext?.attempt ?? 1,
fixtures: options.workspaceFixtures ?? {},
user: options.user,
profile: options.profile,
});
const workspaceRoot = realWorkspace.id;
clearGlobalDrafts(workspaceRoot);
registerBenchmarkWorkspaceRunnables(
registerBenchmarkAiModels(
workspaceRoot,
options.workspaceFixtures ?? {},
options.workspaceFixtures?.aiProviders ?? [],
);
seedLiveEditorDrafts(workspaceRoot, options.liveEditorDrafts ?? []);
// No deploy rules in the mock workspace, so they always pass. The role is read as a server
// that predates workspace roles describes it, from `operator_settings`.
// The eval workspace has no deploy rules, so they always pass.
const roleAccess =
options.profile &&
capabilitiesForRole({
@@ -187,6 +188,7 @@ export async function runGlobalEval(
let panel: EvalPreviewPanel | undefined;
try {
await seedLiveEditorDrafts(workspaceRoot, options.liveEditorDrafts ?? []);
const evalArtifacts = createEvalArtifactHelpers(options.artifacts);
// Only a session chat has a side panel, so only it gets one here. Seeded tabs would
// otherwise vanish without a word, and the case would measure an empty panel.
@@ -230,6 +232,9 @@ export async function runGlobalEval(
});
const helpers = {
...evalArtifacts.helpers,
// A standalone chat has no session: the tools that behave differently in one (a pipeline
// node is built into the pipeline draft there, not written as a script) read this.
sessionId: options.sessionChat ? evalArtifacts.helpers.sessionId : undefined,
access,
...(panel && { openArtifact: panel.openArtifact }),
};
@@ -275,7 +280,7 @@ export async function runGlobalEval(
model,
workspace: workspaceRoot,
provider: options.provider,
backend: options.backend,
backend: realWorkspace.backend,
caseId: options.runContext?.caseId,
attempt: options.runContext?.attempt,
},
@@ -299,9 +304,7 @@ export async function runGlobalEval(
clearLiveEditorDrafts(workspaceRoot, options.liveEditorDrafts ?? []);
seedOperatorSettings(workspaceRoot, undefined);
unregisterBenchmarkWorkspaceRunnables(workspaceRoot);
if (!options.workspaceRoot) {
await rm(workspaceRoot, { recursive: true, force: true });
}
await realWorkspace.dispose();
}
}
@@ -331,17 +334,22 @@ async function collectGlobalDraftState(
return { drafts: drafts as GlobalDraftState["drafts"] };
}
function seedLiveEditorDrafts(
async function seedLiveEditorDrafts(
workspace: string,
fixtures: GlobalLiveEditorDraftFixture[],
): void {
): Promise<void> {
for (const fixture of fixtures) {
const itemKind = LIVE_EDITOR_ITEM_KINDS[fixture.type];
const storagePath = fixture.storagePath ?? fixture.effectivePath ?? "";
if (fixture.value !== undefined) {
// Seed as a backend draft row, not an in-tab cell: a cell would shadow the
// model's DB-backed edit when the output is read back via listGlobalDrafts.
seedBenchmarkDraft(workspace, itemKind, storagePath, fixture.value);
await DraftService.updateDraft({
workspace,
kind: itemKind,
path: storagePath,
requestBody: { value: fixture.value, force: true },
});
}
UserDraft.setLiveEditorDraft({
workspace,
@@ -382,6 +390,10 @@ function seedOperatorSettings(
username: "",
color: "",
operator_settings: operatorSettings as OperatorSettings,
role: "operator",
capabilities: legacyOperatorCapabilities(
operatorSettings as OperatorSettings,
),
is_dev_workspace: false,
disabled: false,
});
@@ -0,0 +1,670 @@
// A real Windmill workspace per global-mode attempt. The session's tools run on the backend
// (`/api/w/<ws>/ai_session_tools/<name>`), so a case's fixture has to exist there: this creates
// the workspace, seeds it through the public API, and deletes it afterwards.
import { execFile } from "node:child_process";
import { randomUUID } from "node:crypto";
import { promisify } from "node:util";
import type { WindmillBackendSettings } from "../../../../core/windmillBackendSettings";
import type { EvalCaseProfile } from "../../../../core/types";
import {
syncedResourceTypes,
type BenchmarkDatatableSeed,
type BenchmarkWorkspaceRunnables,
} from "../../mockBackend";
import { WindmillBackendClient } from "../../windmillBackend";
import { buildWorkspaceId } from "../../workspaceId";
import type { GlobalUserFixture } from "./globalEvalRunner";
/** Workspace id -> the token its case runs as. Read by the adapter's fetch stub. */
const activeWorkspaces = new Map<string, string>();
let adminToken: string | undefined;
let baseUrl: string | undefined;
export function realWorkspaceRoute(
url: string,
): { url: string; token: string } | undefined {
if (!baseUrl || !adminToken || !url.startsWith("/api/")) return undefined;
const workspace = /^\/api\/w\/([^/?]+)/.exec(url)?.[1];
const token = workspace
? activeWorkspaces.get(decodeURIComponent(workspace))
: activeWorkspaces.size > 0
? adminToken
: undefined;
return token ? { url: `${baseUrl}${url}`, token } : undefined;
}
// Without the `postgresql` tag (a `native` worker group) datatable fixtures and SQL tools hang.
const DATATABLE_PG_URL =
process.env.WMILL_AI_EVAL_DATATABLE_PG_URL ??
"postgres://postgres:changeme@localhost:5432/postgres";
const ROOT_PG_RESOURCE = "$res:u/admin/ai_evals_pg_root";
const RESOURCE_TYPE_INDEX_WAIT_MS = 120_000;
const WORKSPACE_SLOT_WAIT_MS = 30 * 60_000;
/** The eval backend's own database, where a fixture user that is not the admin is created. */
const DATABASE_URL = process.env.WMILL_AI_EVAL_DATABASE_URL;
export interface RealWorkspace {
id: string;
/** Settings that make the model proxy reuse this workspace instead of creating another. */
backend: WindmillBackendSettings;
dispose(): Promise<void>;
}
export async function createRealWorkspace(input: {
settings: WindmillBackendSettings;
caseId: string;
attempt: number;
fixtures: BenchmarkWorkspaceRunnables;
user?: GlobalUserFixture;
profile?: EvalCaseProfile;
}): Promise<RealWorkspace> {
// Job ids are unique across the instance, and a case prompt can name a fixture job by its
// id: attempts that seed the same ids take turns, each holding them until it is deleted.
const releaseJobIds = await holdJobIds(
(input.fixtures.jobs ?? []).flatMap((job) => (job.id ? [job.id] : [])),
);
const client = new WindmillBackendClient(input.settings);
baseUrl = input.settings.baseUrl;
const id = buildWorkspaceId(input.caseId, input.attempt);
const api = apiFor(client, id);
try {
adminToken = await client.getToken();
await createWorkspace(api, id);
} catch (error) {
releaseJobIds();
throw error;
}
let datatableDb: string | undefined;
let userToken: string | undefined;
const dispose = async () => {
activeWorkspaces.delete(id);
if (input.user && DATABASE_URL) {
await psql(`DELETE FROM token WHERE label = ${literal(tokenLabel(id))}`).catch(
() => undefined,
);
}
const workers = (input.fixtures.workers ?? []).map((w) => literal(w.worker));
if (workers.length) {
await psql(`DELETE FROM worker_ping WHERE worker IN (${workers.join(", ")})`).catch(
() => undefined,
);
}
if (datatableDb) {
await sql(
api,
ROOT_PG_RESOURCE,
`DROP DATABASE IF EXISTS "${datatableDb}" WITH (FORCE)`,
).catch(() => undefined);
}
await api("DELETE", `/workspaces/delete/${id}`).catch(() => undefined);
releaseJobIds();
};
try {
activeWorkspaces.set(id, adminToken);
userToken = await seedUser(api, id, input.user, input.profile);
datatableDb = await seed(api, id, input.fixtures);
activeWorkspaces.set(id, userToken ?? adminToken);
return {
id,
backend: { ...input.settings, workspaceOverride: id },
dispose,
};
} catch (error) {
await dispose();
throw error;
}
}
/** Job id -> the turn of the last attempt that asked for it. */
const jobIdTurns = new Map<string, Promise<void>>();
async function holdJobIds(ids: string[]): Promise<() => void> {
let release!: () => void;
const turn = new Promise<void>((resolve) => (release = resolve));
const previous = ids.map((id) => jobIdTurns.get(id) ?? Promise.resolve());
for (const id of ids) jobIdTurns.set(id, turn);
await Promise.all(previous);
return release;
}
type Api = (method: string, path: string, body?: unknown) => Promise<any>;
function apiFor(client: WindmillBackendClient, workspace: string): Api {
return async (method, path, body) => {
const scoped = path.startsWith("/") ? path : `/w/${workspace}/${path}`;
const response = await client.request(scoped, {
method,
...(body instanceof FormData
? { body }
: body === undefined
? {}
: {
headers: { "Content-Type": "application/json" },
body: JSON.stringify(body),
}),
});
const text = await response.text();
if (!response.ok) {
throw new Error(`${method} ${scoped}: ${response.status} ${text}`);
}
try {
return JSON.parse(text);
} catch {
return text;
}
};
}
// CE caps the number of workspaces, so a run with more concurrent attempts than slots waits.
async function createWorkspace(api: Api, id: string): Promise<void> {
const deadline = Date.now() + WORKSPACE_SLOT_WAIT_MS;
for (;;) {
try {
await api("POST", "/workspaces/create", { id, name: id });
return;
} catch (error) {
if (
!String(error).includes("maximum number of workspaces") ||
Date.now() > deadline
) {
throw error;
}
await new Promise((resolve) => setTimeout(resolve, 5_000));
}
}
}
// The admin running the eval is `admin` in every workspace it creates. Any other identity is a
// real member, so the backend enforces its rights as it would for that user; CE cannot create
// users, so there the case runs as the admin, with the prompt alone describing the fixture user.
async function seedUser(
api: Api,
workspace: string,
user: GlobalUserFixture | undefined,
profile: EvalCaseProfile | undefined,
): Promise<string | undefined> {
const username = user?.username ?? "admin";
const writable = new Set(user?.folders ?? []);
const readable = new Set([...(user?.folders_read ?? []), ...writable]);
const asAdmin = username === "admin" && (user?.is_admin ?? true);
let token: string | undefined;
if (!asAdmin) {
// CE cannot create users through the API, so the user and a token of theirs are written
// straight into the instance database, as the backend's own tests do. The workspace
// membership still goes through the API.
const email = `${username}@ai-evals.windmill.dev`;
token = `aievals${randomUUID().replace(/-/g, "")}`;
await psql(
`INSERT INTO password (email, password_hash, login_type, super_admin, verified, name, username)
VALUES (${literal(email)}, 'ai-evals-no-login', 'password', false, true, ${literal(username)},
${literal(username)})
ON CONFLICT (email) DO UPDATE SET username = EXCLUDED.username;
INSERT INTO token (token_hash, token_prefix, token, email, label, super_admin)
VALUES (encode(sha256(${literal(token)}::bytea), 'hex'), ${literal(token.slice(0, 10))},
${literal(token)}, ${literal(email)}, ${literal(tokenLabel(workspace))}, false);`,
);
await api("POST", "workspaces/add_user", {
email,
is_admin: user?.is_admin ?? false,
operator: profile?.operator ?? false,
});
}
if (profile?.operator && profile.operatorSettings) {
await api("POST", "workspaces/operator_settings", profile.operatorSettings);
}
for (const folder of readable) {
await api("POST", "folders/create", {
name: folder,
...(asAdmin ? {} : { extra_perms: { [`u/${username}`]: writable.has(folder) } }),
});
}
return token;
}
/** Seeds every fixture kind; returns the scratch database a datatable fixture created. */
async function seed(
api: Api,
workspace: string,
fixtures: BenchmarkWorkspaceRunnables,
): Promise<string | undefined> {
// A fixture that sets no `resourceTypes` stands for an instance synced with the hub,
// which an eval backend usually is not; the types the instance already holds stay its own.
const held = new Set<string>(await api("GET", "resources/type/listnames"));
const resourceTypes = (fixtures.resourceTypes ?? syncedResourceTypes()).filter(
(rt) => !held.has(rt.name),
);
for (const rt of resourceTypes) {
await api("POST", "resources/type/create", {
name: rt.name,
schema: rt.schema ?? {},
description: rt.description ?? "",
});
}
// The resource type search reads an index the server rebuilds every
// RESOURCE_TYPE_EMBEDDINGS_CHECK_INTERVAL_SECS: a case must not start before it holds them.
const last = resourceTypes[resourceTypes.length - 1]?.name;
for (const deadline = Date.now() + RESOURCE_TYPE_INDEX_WAIT_MS; last; ) {
const found: { name: string }[] = await api(
"GET",
`embeddings/query_resource_types?text=${encodeURIComponent(last)}&limit=10`,
);
if (found.some((rt) => rt.name === last)) break;
if (Date.now() > deadline) {
throw new Error(`the resource type search never found the seeded "${last}"`);
}
await new Promise((resolve) => setTimeout(resolve, 2_000));
}
for (const group of fixtures.groups ?? []) {
await api("POST", "groups/create", { name: group.name, summary: group.summary });
// A fixture names members who are not users of the instance; a membership row needs none.
for (const member of group.members ?? []) {
await psql(
`INSERT INTO usr_to_group (workspace_id, usr, group_)
VALUES (${literal(workspace)}, ${literal(member)}, ${literal(group.name)})`,
"This fixture seeds group members",
);
}
}
for (const folder of fixtures.folders ?? []) {
await api("POST", "folders/create", {
name: folder.name,
summary: folder.summary,
owners: folder.owners ?? [],
extra_perms: folder.extra_perms ?? {},
});
}
for (const script of fixtures.scripts ?? []) {
await api("POST", "scripts/create", {
path: script.path,
summary: script.summary,
description: script.description ?? "",
content: script.content,
language: script.language,
schema: script.schema ?? {},
});
}
for (const flow of fixtures.flows ?? []) {
await api("POST", "flows/create", {
path: flow.path,
summary: flow.summary,
description: flow.description ?? "",
value: flow.value,
schema: flow.schema ?? {},
});
}
for (const app of fixtures.apps ?? []) {
await seedApp(api, app);
}
for (const variable of fixtures.variables ?? []) {
await api("POST", "variables/create", {
path: variable.path,
value: variable.value,
is_secret: variable.is_secret,
description: variable.description ?? "",
...(variable.labels ? { labels: variable.labels } : {}),
});
}
for (const resource of fixtures.resources ?? []) {
await api("POST", "resources/create", {
path: resource.path,
resource_type: resource.resource_type,
value: resource.value ?? {},
description: resource.description ?? "",
});
}
await seedAiProviders(api, fixtures.aiProviders ?? []);
for (const job of fixtures.jobs ?? []) {
await seedJob(api, workspace, job);
}
for (const schedule of fixtures.schedules ?? []) {
await api("POST", "schedules/create", {
path: schedule.path,
schedule: schedule.schedule,
timezone: "UTC",
script_path: schedule.script_path,
is_flow: schedule.is_flow ?? false,
args: {},
enabled: schedule.enabled ?? true,
summary: schedule.summary,
});
}
await seedWorkers(fixtures.workers ?? []);
if (fixtures.ducklakes?.length) {
await api("POST", "workspaces/edit_ducklake_config", {
settings: {
ducklakes: Object.fromEntries(
fixtures.ducklakes.map((name) => [
name,
{
catalog: { resource_type: "instance", resource_path: `${workspace}_${name}` },
storage: { storage: null, path: name },
},
]),
),
},
});
}
await seedDataMetrics(api, fixtures.dataMetrics ?? []);
const datatables = fixtures.datatables ?? [];
if (datatables.length > 1) {
throw new Error("A global fixture seeds at most one datatable.");
}
return datatables[0] ? await seedDatatable(api, workspace, datatables[0]) : undefined;
}
async function seedApp(
api: Api,
app: NonNullable<BenchmarkWorkspaceRunnables["apps"]>[number],
): Promise<void> {
const policy = { execution_mode: "publisher", triggerables_v2: {} };
if (app.rawApp === false) {
await api("POST", "apps/create", {
path: app.path,
summary: app.summary,
value: app.value,
policy,
});
return;
}
const form = new FormData();
form.append(
"app",
JSON.stringify({
path: app.path,
summary: app.summary,
value: { ...app.value, data: app.value.data ?? { tables: [] } },
policy,
}),
);
form.append("js", "");
form.append("css", "");
await api("POST", "apps/create_raw", form);
}
async function seedAiProviders(
api: Api,
providers: NonNullable<BenchmarkWorkspaceRunnables["aiProviders"]>,
): Promise<void> {
if (!providers.length) return;
const config: Record<string, unknown> = {};
let defaultModel: { model: string; provider: string } | undefined;
for (const provider of providers) {
await api("POST", "resources/create", {
path: provider.path,
resource_type: provider.kind,
value: {
api_key: "ai-evals-placeholder",
...(provider.base_url ? { base_url: provider.base_url } : {}),
},
});
const models = provider.configuredModels ?? provider.models ?? [];
config[provider.kind] = { resource_path: provider.path, models };
if (provider.isDefault && models[0]) {
defaultModel = { model: models[0], provider: provider.kind };
}
}
await api("POST", "workspaces/edit_copilot_config", {
providers: config,
...(defaultModel ? { default_model: defaultModel } : {}),
});
}
// A completed run with the fixture's id, path, args, logs and outcome. It is a preview job run
// by the admin, so `createdBy` and a `script` job kind are not reproduced.
async function seedJob(
api: Api,
workspace: string,
job: NonNullable<BenchmarkWorkspaceRunnables["jobs"]>[number],
): Promise<void> {
const args = job.args ?? {};
const id = job.id ?? randomUUID();
const state = job.state ?? "completed";
if (state === "suspended") {
// A flow parked on an approval step: its first step suspends, as the fixture flow's does.
await api("POST", `jobs/run/f/${job.scriptPath}?job_id=${id}`, args);
await waitFor(`run ${id} to suspend`, async () => {
const run = await api("GET", `jobs_u/get/${id}?no_logs=true`);
return (run.flow_status?.modules ?? []).some(
(module: { type?: string }) => module.type === "WaitingForEvents",
);
});
} else if (state === "queued") {
// Stays queued only on a tag no worker of the eval backend serves, as the fixture's does.
await api("POST", `jobs/run/preview?job_id=${id}`, {
content: "export async function main() {}",
language: "bun",
path: job.scriptPath,
args,
tag: job.tag,
});
} else if (state === "completed") {
const content = [
`export async function main(${Object.keys(args).join(", ")}) {`,
` console.log(${JSON.stringify(job.logs ?? "")});`,
job.success === false
? ` throw new Error(${JSON.stringify(lastLine(job.logs) ?? "Job failed")});`
: ` return ${JSON.stringify(job.result ?? null)};`,
`}`,
].join("\n");
await api("POST", `jobs/run_wait_result/preview?job_id=${id}`, {
content,
language: "bun",
path: job.scriptPath,
args,
}).catch((error) => {
// A failing fixture run answers with its error, which is the point.
if (job.success !== false) throw error;
});
} else {
throw new Error(`A ${state} fixture run cannot be seeded on a real backend.`);
}
// Who ran it, when and where cannot be asked of the API: the run is rewritten to match.
if (
job.createdBy === undefined &&
job.createdMinutesAgo === undefined &&
job.durationMs === undefined &&
job.tag === undefined &&
job.schedulePath === undefined &&
job.label === undefined
) {
return;
}
const createdAt =
job.createdMinutesAgo === undefined
? "created_at"
: `now() - interval '${Number(job.createdMinutesAgo)} minutes'`;
await psql(
`UPDATE v2_job SET created_at = ${createdAt},
created_by = COALESCE(${literal(job.createdBy)}, created_by),
tag = COALESCE(${literal(job.tag)}, tag),
labels = COALESCE(${job.label === undefined ? "NULL" : `ARRAY[${literal(job.label)}]`}, labels)
${job.schedulePath ? `, trigger = ${literal(job.schedulePath)}, trigger_kind = 'schedule'` : ""}
WHERE id = ${literal(id)} AND workspace_id = ${literal(workspace)};
UPDATE v2_job_queue q SET created_at = j.created_at, scheduled_for = j.created_at
FROM v2_job j WHERE q.id = j.id AND q.id = ${literal(id)};
UPDATE v2_job_completed c SET started_at = j.created_at,
duration_ms = COALESCE(${literal(job.durationMs)}, c.duration_ms),
completed_at = j.created_at + make_interval(secs => COALESCE(${literal(job.durationMs)}, c.duration_ms) / 1000.0)
FROM v2_job j WHERE c.id = j.id AND c.id = ${literal(id)};`,
"This fixture seeds who ran a run, when or where",
);
}
/** Fixture workers, listed beside the instance's own until the attempt ends. */
async function seedWorkers(
workers: NonNullable<BenchmarkWorkspaceRunnables["workers"]>,
): Promise<void> {
// ponytail: rows are instance-wide, so concurrent attempts of one case share them.
for (const worker of workers) {
await psql(
`INSERT INTO worker_ping (worker, worker_instance, ping_at, started_at, ip, jobs_executed,
custom_tags, worker_group)
VALUES (${literal(worker.worker)}, ${literal(worker.worker)},
now() - interval '${Number(worker.lastPingSecondsAgo ?? 0)} seconds', now(), 'ai-evals',
${Number(worker.jobs_executed ?? 0)},
ARRAY[${(worker.custom_tags ?? []).map(literal).join(", ")}]::varchar[],
${literal(worker.worker_group)})
ON CONFLICT (worker) DO UPDATE SET ping_at = EXCLUDED.ping_at`,
"This fixture seeds workers",
);
}
}
async function waitFor(what: string, done: () => Promise<boolean>): Promise<void> {
for (const deadline = Date.now() + 60_000; !(await done()); ) {
if (Date.now() > deadline) throw new Error(`timed out waiting for ${what}`);
await new Promise((resolve) => setTimeout(resolve, 500));
}
}
function lastLine(text: string | undefined): string | undefined {
return text?.trim().split("\n").pop();
}
// Metrics are catalogued from a DuckDB node's annotations when it is deployed.
async function seedDataMetrics(
api: Api,
metrics: NonNullable<BenchmarkWorkspaceRunnables["dataMetrics"]>,
): Promise<void> {
const byScript = new Map<string, typeof metrics>();
for (const metric of metrics) {
const key = `${metric.script_path}\n${metric.table_path}`;
byScript.set(key, [...(byScript.get(key) ?? []), metric]);
}
for (const [key, entries] of byScript) {
const [scriptPath, tablePath] = key.split("\n");
// `<lake>/<schema>.<table>`, as the catalog canonicalizes `ducklake://<lake>/<table>`.
const [lake, qualified] = tablePath.split("/");
const table = qualified.split(".").pop();
const lines = [`-- materialize ducklake://${lake}/${table}`];
for (const metric of entries) {
lines.push(
metric.kind === "measure"
? `-- measure ${metric.name} = ${metric.expr}${metric.filter ? ` where ${metric.filter}` : ""}`
: `-- dimension ${metric.name} = ${metric.expr}`,
);
}
lines.push("SELECT 1");
await api("POST", "scripts/create", {
path: scriptPath,
summary: "",
description: "",
content: lines.join("\n"),
language: "duckdb",
schema: {},
});
}
}
async function seedDatatable(
api: Api,
workspace: string,
seed: BenchmarkDatatableSeed,
): Promise<string> {
const root = new URL(DATATABLE_PG_URL);
const database = `${workspace.replace(/-/g, "_")}_dt`.slice(0, 63);
const resource = (dbname: string) => ({
host: root.hostname,
port: Number(root.port || 5432),
user: decodeURIComponent(root.username),
password: decodeURIComponent(root.password),
dbname,
sslmode: root.searchParams.get("sslmode") ?? "disable",
});
await api("POST", "resources/create", {
path: "u/admin/ai_evals_pg_root",
resource_type: "postgresql",
value: resource(root.pathname.slice(1) || "postgres"),
});
await sql(api, ROOT_PG_RESOURCE, `CREATE DATABASE "${database}"`);
await api("POST", "resources/create", {
path: "u/admin/ai_evals_datatable",
resource_type: "postgresql",
value: resource(database),
});
await api("POST", "workspaces/edit_datatable_config", {
settings: {
datatables: {
[seed.datatable_name]: {
database: { resource_type: "postgresql", resource_path: "u/admin/ai_evals_datatable" },
},
},
},
});
// A new datatable keeps its schema as migrations; a fixture says when this one does not.
if (!seed.migrations_enabled) {
await api("POST", `workspaces/disable_datatable_migrations/${seed.datatable_name}`);
}
const target = `datatable://${seed.datatable_name}`;
for (const migration of seed.migrations ?? []) {
await api("POST", `workspaces/upsert_datatable_migration/${seed.datatable_name}`, {
timestamp: migration.timestamp,
name: migration.name,
code_up: migration.code_up,
...(migration.code_down ? { code_down: migration.code_down } : {}),
});
if (migration.applied) {
await api(
"POST",
`workspaces/run_datatable_migrations/${seed.datatable_name}?only=${migration.timestamp}`,
);
}
}
const statements: string[] = [];
for (const [schema, tables] of Object.entries(seed.schemas)) {
statements.push(`CREATE SCHEMA IF NOT EXISTS "${schema}"`);
for (const [table, { columns, rows }] of Object.entries(tables)) {
const name = `"${schema}"."${table}"`;
statements.push(
`CREATE TABLE IF NOT EXISTS ${name} (${Object.entries(columns)
.map(([column, type]) => `"${column}" ${type}`)
.join(", ")})`,
);
for (const row of rows ?? []) {
const keys = Object.keys(row);
statements.push(
`INSERT INTO ${name} (${keys.map((k) => `"${k}"`).join(", ")}) VALUES (${keys
.map((k) => literal(row[k]))
.join(", ")})`,
);
}
}
}
await sql(api, target, statements.join(";\n"));
return database;
}
function literal(value: unknown): string {
if (value === null || value === undefined) return "NULL";
if (typeof value === "number" || typeof value === "boolean") return String(value);
const text = typeof value === "string" ? value : JSON.stringify(value);
return `'${text.replace(/'/g, "''")}'`;
}
async function sql(api: Api, database: string, content: string): Promise<void> {
const result = await api("POST", "jobs/run_wait_result/preview", {
content,
language: "postgresql",
args: { database },
});
if (result && typeof result === "object" && "error" in result) {
throw new Error(`seeding SQL failed: ${JSON.stringify(result.error)}`);
}
}
/** Each attempt's user token carries its workspace, so the attempt revokes exactly its own. */
function tokenLabel(workspace: string): string {
return `ai-evals ${workspace}`;
}
async function psql(
sql: string,
why = "This fixture's user is not the admin",
): Promise<void> {
if (!DATABASE_URL) {
throw new Error(
`${why}: set WMILL_AI_EVAL_DATABASE_URL to the eval backend's database URL so the harness can write it.`,
);
}
await promisify(execFile)("psql", [DATABASE_URL, "-v", "ON_ERROR_STOP=1", "-qc", sql]);
}
@@ -138,7 +138,7 @@ export async function runEval<THelpers, TOutput>(
// Accepts the run form exactly as the model prefilled it: there is nobody here to
// edit the arguments, so a case can assert what the model proposed but never how
// it reacts to the user changing something.
requestRunArgs: async (_toolId, form) => form.args,
requestRunArgs: async (_toolId, form) => ({ kind: "args", args: form.args }),
...(pendingAnswers && {
requestUserQuestion: async () => {
const answer = pendingAnswers.shift();
@@ -1,262 +0,0 @@
import { describe, expect, it } from 'bun:test'
import { applyDatatableSql, type BenchmarkDatatableSeed } from './datatableSqlEngine'
function makeDatatable(): BenchmarkDatatableSeed {
return {
datatable_name: 'main',
schemas: {
public: {
orders: {
columns: { id: 'int4', customer_id: 'int4', total: 'numeric', status: 'text' },
rows: [
{ id: 1, customer_id: 1, total: 42.5, status: 'shipped' },
{ id: 2, customer_id: 2, total: 19.99, status: 'pending' },
{ id: 3, customer_id: 1, total: 88, status: 'shipped' }
]
},
customers: {
columns: { id: 'int4', name: 'text' },
rows: [{ id: 1, name: 'Alice' }]
}
}
}
}
}
describe('SELECT', () => {
it('returns the referenced table rows', () => {
const dt = makeDatatable()
expect(applyDatatableSql(dt, 'SELECT id, name FROM customers').rows).toEqual([
{ id: 1, name: 'Alice' }
])
})
it('falls back to the first table when no known table is referenced', () => {
const dt = makeDatatable()
expect(applyDatatableSql(dt, 'select 1').rows).toHaveLength(3)
})
it('resolves a schema-qualified table', () => {
const dt = makeDatatable()
expect(applyDatatableSql(dt, 'SELECT * FROM public.customers').rows).toEqual([
{ id: 1, name: 'Alice' }
])
})
})
describe('CREATE TABLE', () => {
it('adds a table with parsed columns, skipping table constraints and FK clauses', () => {
const dt = makeDatatable()
const result = applyDatatableSql(
dt,
'CREATE TABLE public.refunds (\n order_id int4 NOT NULL REFERENCES public.orders(id),\n amount numeric(10,2),\n PRIMARY KEY (order_id)\n)'
)
expect(result.rows).toEqual([])
expect(dt.schemas.public.refunds).toEqual({
columns: { order_id: 'int4', amount: 'numeric(10,2)' },
rows: []
})
})
it('defaults an unqualified table to the public schema', () => {
const dt = makeDatatable()
applyDatatableSql(dt, 'CREATE TABLE notes (id int4, body text)')
expect(dt.schemas.public.notes.columns).toEqual({ id: 'int4', body: 'text' })
})
it('is a no-op for an existing table with IF NOT EXISTS', () => {
const dt = makeDatatable()
applyDatatableSql(dt, 'CREATE TABLE IF NOT EXISTS public.orders (x int4)')
expect(Object.keys(dt.schemas.public.orders.columns)).toContain('status')
})
})
describe('DROP TABLE', () => {
it('removes the table', () => {
const dt = makeDatatable()
applyDatatableSql(dt, 'DROP TABLE IF EXISTS public.customers')
expect(dt.schemas.public.customers).toBeUndefined()
})
})
describe('INSERT', () => {
it('appends a row using an explicit column list', () => {
const dt = makeDatatable()
applyDatatableSql(dt, "INSERT INTO customers (id, name) VALUES (2, 'Bob')")
expect(dt.schemas.public.customers.rows).toContainEqual({ id: 2, name: 'Bob' })
})
it('infers columns from the table when none are given, and appends multiple tuples', () => {
const dt = makeDatatable()
applyDatatableSql(dt, "INSERT INTO customers VALUES (2, 'Bob'), (3, 'Carol')")
expect(dt.schemas.public.customers.rows).toHaveLength(3)
})
it('returns the inserted rows when RETURNING is present', () => {
const dt = makeDatatable()
const result = applyDatatableSql(
dt,
"INSERT INTO customers (id, name) VALUES (2, 'Bob') RETURNING *"
)
expect(result.rows).toEqual([{ id: 2, name: 'Bob' }])
})
})
describe('UPDATE', () => {
it('updates only the rows matching an equality WHERE', () => {
const dt = makeDatatable()
const result = applyDatatableSql(
dt,
"UPDATE public.orders SET status = 'shipped' WHERE id = 2"
)
expect(result.rows).toEqual([])
expect(dt.schemas.public.orders.rows?.find((r) => r.id === 2)?.status).toBe('shipped')
expect(dt.schemas.public.orders.rows?.find((r) => r.id === 1)?.status).toBe('shipped')
})
it('strips a Postgres cast in the WHERE value', () => {
const dt = makeDatatable()
applyDatatableSql(dt, "UPDATE orders SET status = 'done' WHERE id = 2::int4")
expect(dt.schemas.public.orders.rows?.find((r) => r.id === 2)?.status).toBe('done')
})
it('matches multiple AND predicates including a numeric literal', () => {
const dt = makeDatatable()
applyDatatableSql(
dt,
"UPDATE orders SET status = 'done' WHERE customer_id = 2 AND total = 19.99"
)
expect(dt.schemas.public.orders.rows?.find((r) => r.id === 2)?.status).toBe('done')
expect(dt.schemas.public.orders.rows?.find((r) => r.id === 1)?.status).toBe('shipped')
})
it('updates every row when there is no WHERE', () => {
const dt = makeDatatable()
applyDatatableSql(dt, "UPDATE orders SET status = 'archived'")
expect(dt.schemas.public.orders.rows?.every((r) => r.status === 'archived')).toBe(true)
})
it('returns the affected rows when RETURNING is present', () => {
const dt = makeDatatable()
const result = applyDatatableSql(
dt,
"UPDATE orders SET status = 'shipped' WHERE id = 2 RETURNING *"
)
expect(result.rows).toHaveLength(1)
expect(result.rows[0]).toMatchObject({ id: 2, status: 'shipped' })
})
it('affects no rows when the WHERE clause cannot be parsed', () => {
const dt = makeDatatable()
applyDatatableSql(dt, "UPDATE orders SET status = 'x' WHERE total > 20")
expect(dt.schemas.public.orders.rows?.some((r) => r.status === 'x')).toBe(false)
})
})
describe('DELETE', () => {
it('removes only the matching rows', () => {
const dt = makeDatatable()
applyDatatableSql(dt, 'DELETE FROM orders WHERE id = 2')
expect(dt.schemas.public.orders.rows?.map((r) => r.id)).toEqual([1, 3])
})
it('returns the removed rows when RETURNING is present', () => {
const dt = makeDatatable()
const result = applyDatatableSql(dt, 'DELETE FROM orders WHERE id = 2 RETURNING *')
expect(result.rows).toEqual([{ id: 2, customer_id: 2, total: 19.99, status: 'pending' }])
})
})
describe('writes are reflected by later reads', () => {
it('UPDATE then SELECT sees the new value (the verify-loop fix)', () => {
const dt = makeDatatable()
applyDatatableSql(dt, "UPDATE orders SET status = 'shipped' WHERE id = 2")
const seen = applyDatatableSql(dt, 'SELECT * FROM orders').rows
expect(seen.find((r) => r.id === 2)?.status).toBe('shipped')
})
it('INSERT then SELECT sees the new row', () => {
const dt = makeDatatable()
applyDatatableSql(dt, "INSERT INTO customers (id, name) VALUES (9, 'Zed')")
const seen = applyDatatableSql(dt, 'SELECT * FROM customers').rows
expect(seen).toContainEqual({ id: 9, name: 'Zed' })
})
it('CREATE then SELECT on the new table returns its (empty) rows', () => {
const dt = makeDatatable()
applyDatatableSql(dt, 'CREATE TABLE public.refunds (order_id int4, amount numeric)')
expect(applyDatatableSql(dt, 'SELECT * FROM refunds').rows).toEqual([])
})
})
describe('system-catalog queries reflect the current tables/columns', () => {
it('lists current tables (including a freshly created one) via information_schema.tables', () => {
const dt = makeDatatable()
applyDatatableSql(dt, 'CREATE TABLE public.refunds (order_id int4)')
const rows = applyDatatableSql(
dt,
"SELECT table_name FROM information_schema.tables WHERE table_name = 'refunds'"
).rows
expect(rows.map((r) => r.table_name)).toContain('refunds')
})
it('does not list a dropped table', () => {
const dt = makeDatatable()
applyDatatableSql(dt, 'DROP TABLE public.customers')
const rows = applyDatatableSql(dt, 'SELECT table_name FROM information_schema.tables').rows
expect(rows.map((r) => r.table_name)).not.toContain('customers')
})
it('reports columns via information_schema.columns', () => {
const dt = makeDatatable()
const rows = applyDatatableSql(
dt,
"SELECT column_name FROM information_schema.columns WHERE table_name = 'orders'"
).rows
expect(rows.map((r) => r.column_name)).toContain('status')
})
})
describe('parser robustness (string/paren-aware splitting)', () => {
it('does not treat the word "returning" inside a string value as a RETURNING clause', () => {
const dt = makeDatatable()
const result = applyDatatableSql(
dt,
"INSERT INTO customers (id, name) VALUES (5, 'is returning soon')"
)
expect(result.rows).toEqual([])
expect(dt.schemas.public.customers.rows).toContainEqual({ id: 5, name: 'is returning soon' })
})
it('does not split on the word "where" inside a SET string value', () => {
const dt = makeDatatable()
applyDatatableSql(dt, "UPDATE orders SET status = 'ship where ordered' WHERE id = 2")
expect(dt.schemas.public.orders.rows?.find((r) => r.id === 2)?.status).toBe('ship where ordered')
expect(dt.schemas.public.orders.rows?.find((r) => r.id === 1)?.status).toBe('shipped')
})
it('keeps INSERT tuples intact when a value contains a function call', () => {
const dt = makeDatatable()
applyDatatableSql(dt, "INSERT INTO customers (id, name) VALUES (6, coalesce(NULL, 'x'))")
expect(dt.schemas.public.customers.rows).toHaveLength(2)
expect(dt.schemas.public.customers.rows?.[1]).toMatchObject({ id: 6 })
})
it('CREATE TABLE ignores a trailing semicolon-separated statement', () => {
const dt = makeDatatable()
applyDatatableSql(
dt,
'CREATE TABLE public.refunds (id int4, amount numeric); INSERT INTO refunds VALUES (1, 5)'
)
expect(dt.schemas.public.refunds.columns).toEqual({ id: 'int4', amount: 'numeric' })
expect(dt.schemas.public.refunds.rows).toEqual([])
})
})
describe('unparseable statements are a safe no-op', () => {
it('returns [] and does not throw', () => {
const dt = makeDatatable()
expect(applyDatatableSql(dt, 'VACUUM ANALYZE').rows).toEqual([])
expect(applyDatatableSql(dt, 'GRANT SELECT ON orders TO someone').rows).toEqual([])
})
})
@@ -1,552 +0,0 @@
/**
* A deliberately small, best-effort SQL engine for the benchmark datatable mock.
*
* This is NOT a real SQL implementation — it exists only so that writes a model
* issues during an eval (`CREATE TABLE`, `INSERT`, `UPDATE`, `DELETE`, `DROP`)
* become visible to its later reads (`list_datatables`, `get_datatable_table_schema`,
* `SELECT`). Without that, a model that re-queries to verify a write sees stale
* seed data, concludes the write failed, and loops until it exhausts its turns.
*
* It parses only the common statement shapes models produce. Anything it cannot
* parse is a no-op success (it never throws) — behavioral evals assert that the
* right statement was issued, not its exact data effects. Notable limits:
* - `SELECT` returns all rows of the referenced (or first) table — no WHERE
* filtering, projection, joins, or aggregation.
* - `WHERE` supports `col = value` predicates joined by `AND` only; an
* unparseable WHERE on UPDATE/DELETE affects zero rows (never the whole table).
*/
/** One seeded datatable table: its columns (col -> compact_type) and optional rows. */
export interface BenchmarkDatatableTableSeed {
columns: Record<string, string>
rows?: Record<string, unknown>[]
}
/** A seeded datatable: `datatable_name` plus a `schema -> table -> seed` map. */
export interface BenchmarkDatatableSeed {
datatable_name: string
schemas: {
[schema: string]: {
[table: string]: BenchmarkDatatableTableSeed
}
}
/** Whether the datatable keeps its schema as migrations. */
migrations_enabled?: boolean
migrations?: BenchmarkDatatableMigrationSeed[]
}
export interface BenchmarkDatatableMigrationSeed {
timestamp: number
name: string
code_up: string
code_down?: string
applied?: boolean
}
export interface DatatableSqlResult {
rows: Record<string, unknown>[]
}
const DEFAULT_SCHEMA = 'public'
type ParsedRef = { schema: string; table: string }
type Predicate = { column: string; value: unknown }
/**
* Apply one SQL statement to `datatable` IN PLACE and return the result rows.
* SELECT returns the referenced/first table's rows; a mutation returns its
* affected rows when it has a RETURNING clause, otherwise `[]`.
*/
export function applyDatatableSql(
datatable: BenchmarkDatatableSeed,
sql: string
): DatatableSqlResult {
const statement = stripTrailingSemicolon(sql.trim())
if (/^\s*(with|select)\b/i.test(statement)) {
return { rows: selectRows(datatable, statement) }
}
if (/^\s*create\s+table\b/i.test(statement)) {
return { rows: applyCreateTable(datatable, statement) }
}
if (/^\s*drop\s+table\b/i.test(statement)) {
return { rows: applyDropTable(datatable, statement) }
}
if (/^\s*insert\s+into\b/i.test(statement)) {
return { rows: applyInsert(datatable, statement) }
}
if (/^\s*update\b/i.test(statement)) {
return { rows: applyUpdate(datatable, statement) }
}
if (/^\s*delete\s+from\b/i.test(statement)) {
return { rows: applyDelete(datatable, statement) }
}
return { rows: [] }
}
// ============= Reads =============
function selectRows(
datatable: BenchmarkDatatableSeed,
sql: string
): Record<string, unknown>[] {
const fromRef = sql.match(/\bfrom\s+([a-zA-Z_"][\w."]*)/i)?.[1]
if (fromRef) {
const catalog = catalogRows(datatable, fromRef)
if (catalog) {
return catalog
}
}
const table = fromRef ? resolveTable(datatable, fromRef) : undefined
const seed = table ?? firstTable(datatable)
return seed?.rows ?? []
}
/**
* Synthesize rows for a system-catalog query so a model verifying a `CREATE`/`DROP`
* via `information_schema.tables` / `.columns` (or `pg_tables`) sees the current
* tables/columns instead of fallback data. WHERE is not applied, so the model gets
* the full set and finds (or no longer finds) the table it just changed.
* Returns `undefined` for non-catalog refs so normal table resolution proceeds.
*/
function catalogRows(
datatable: BenchmarkDatatableSeed,
ref: string
): Record<string, unknown>[] | undefined {
const normalized = ref.toLowerCase().replace(/"/g, '')
const name = normalized.split('.').pop()
const isCatalog = normalized.includes('information_schema.') || normalized.startsWith('pg_')
if (!isCatalog) {
return undefined
}
const tables = allTables(datatable)
if (name === 'tables' || name === 'pg_tables') {
return tables.map(({ schema, table }) => ({
table_schema: schema,
table_name: table,
schemaname: schema,
tablename: table
}))
}
if (name === 'columns') {
return tables.flatMap(({ schema, table, seed }) =>
Object.entries(seed.columns).map(([column, type]) => ({
table_schema: schema,
table_name: table,
column_name: column,
data_type: type
}))
)
}
return undefined
}
function allTables(
datatable: BenchmarkDatatableSeed
): { schema: string; table: string; seed: BenchmarkDatatableTableSeed }[] {
return Object.entries(datatable.schemas).flatMap(([schema, tables]) =>
Object.entries(tables).map(([table, seed]) => ({ schema, table, seed }))
)
}
// ============= DDL =============
function applyCreateTable(
datatable: BenchmarkDatatableSeed,
sql: string
): Record<string, unknown>[] {
const head = sql.match(
/^\s*create\s+table\s+(?:if\s+not\s+exists\s+)?([a-zA-Z_"][\w."]*)/i
)
// The first top-level paren group is the column-definition list; using it (rather
// than a greedy `(...)` capture) ignores any trailing `;`-separated statement.
const columnText = extractParenGroups(sql)[0]
if (!head || columnText === undefined) {
return []
}
const { schema, table } = parseRef(head[1])
const existing = datatable.schemas[schema]?.[table]
if (existing) {
return []
}
const columns: Record<string, string> = {}
for (const rawDef of splitTopLevel(columnText)) {
const def = rawDef.trim()
if (!def || isTableConstraint(def)) {
continue
}
const tokens = def.split(/\s+/)
const column = unquoteIdentifier(tokens[0])
if (!column) {
continue
}
columns[column] = tokens[1] ?? 'text'
}
if (!datatable.schemas[schema]) {
datatable.schemas[schema] = {}
}
datatable.schemas[schema][table] = { columns, rows: [] }
return []
}
function applyDropTable(
datatable: BenchmarkDatatableSeed,
sql: string
): Record<string, unknown>[] {
const match = sql.match(
/^\s*drop\s+table\s+(?:if\s+exists\s+)?([a-zA-Z_"][\w."]*)/i
)
if (!match) {
return []
}
const { schema, table } = parseRef(match[1])
if (datatable.schemas[schema]?.[table]) {
delete datatable.schemas[schema][table]
}
return []
}
// ============= DML =============
function applyInsert(
datatable: BenchmarkDatatableSeed,
sql: string
): Record<string, unknown>[] {
const { body, returning } = splitOffReturning(sql)
const match = body.match(
/^\s*insert\s+into\s+([a-zA-Z_"][\w."]*)\s*(?:\(([^)]*)\))?\s*values\s*([\s\S]+)$/i
)
if (!match) {
return []
}
const table = resolveTable(datatable, match[1])
if (!table) {
return []
}
const columns = match[2]
? splitTopLevel(match[2]).map((entry) => unquoteIdentifier(entry.trim()))
: Object.keys(table.columns)
const inserted: Record<string, unknown>[] = []
for (const tuple of extractParenGroups(match[3])) {
const values = splitTopLevel(tuple).map((entry) => parseValue(entry))
const row: Record<string, unknown> = {}
columns.forEach((column, index) => {
row[column] = values[index]
})
inserted.push(row)
}
table.rows ??= []
table.rows.push(...inserted)
return returning ? inserted : []
}
function applyUpdate(
datatable: BenchmarkDatatableSeed,
sql: string
): Record<string, unknown>[] {
const { body, returning } = splitOffReturning(sql)
const match = body.match(/^\s*update\s+([a-zA-Z_"][\w."]*)\s+set\s+([\s\S]+)$/i)
if (!match) {
return []
}
const table = resolveTable(datatable, match[1])
if (!table) {
return []
}
let assignmentText = match[2]
let whereText: string | undefined
const whereMatch = maskForClauseScan(assignmentText).match(/\swhere\s/i)
if (whereMatch && whereMatch.index !== undefined) {
whereText = assignmentText.slice(whereMatch.index + whereMatch[0].length)
assignmentText = assignmentText.slice(0, whereMatch.index)
}
const predicates = parsePredicates(whereText)
if (predicates === null) {
return []
}
const assignments: Record<string, unknown> = {}
for (const entry of splitTopLevel(assignmentText)) {
const pair = entry.match(/^\s*([a-zA-Z_"][\w."]*)\s*=\s*([\s\S]+?)\s*$/)
if (pair) {
assignments[lastIdentifier(pair[1])] = parseValue(pair[2])
}
}
const affected = (table.rows ?? []).filter((row) => rowMatches(row, predicates))
for (const row of affected) {
Object.assign(row, assignments)
}
return returning ? affected : []
}
function applyDelete(
datatable: BenchmarkDatatableSeed,
sql: string
): Record<string, unknown>[] {
const { body, returning } = splitOffReturning(sql)
const match = body.match(/^\s*delete\s+from\s+([a-zA-Z_"][\w."]*)\s*([\s\S]*)$/i)
if (!match) {
return []
}
const table = resolveTable(datatable, match[1])
if (!table) {
return []
}
const whereText = match[2].replace(/^\s*where\s+/i, '').trim() || undefined
const predicates = parsePredicates(whereText)
if (predicates === null) {
return []
}
const rows = table.rows ?? []
const removed = rows.filter((row) => rowMatches(row, predicates))
table.rows = rows.filter((row) => !rowMatches(row, predicates))
return returning ? removed : []
}
// ============= Parsing helpers =============
function resolveTable(
datatable: BenchmarkDatatableSeed,
ref: string
): BenchmarkDatatableTableSeed | undefined {
const { schema, table } = parseRef(ref)
const direct = datatable.schemas[schema]?.[table]
if (direct) {
return direct
}
// Bare table name: fall back to searching every schema for a matching table.
if (!ref.includes('.')) {
for (const tables of Object.values(datatable.schemas)) {
if (tables[table]) {
return tables[table]
}
}
}
return undefined
}
function firstTable(
datatable: BenchmarkDatatableSeed
): BenchmarkDatatableTableSeed | undefined {
for (const tables of Object.values(datatable.schemas)) {
for (const seed of Object.values(tables)) {
return seed
}
}
return undefined
}
function parseRef(ref: string): ParsedRef {
const parts = ref.split('.').map(unquoteIdentifier)
if (parts.length >= 2) {
return { schema: parts[parts.length - 2], table: parts[parts.length - 1] }
}
return { schema: DEFAULT_SCHEMA, table: parts[0] }
}
/** A WHERE clause with no parseable form returns `null`; absent WHERE returns `[]` (match all). */
function parsePredicates(whereText: string | undefined): Predicate[] | null {
if (whereText === undefined || whereText.trim() === '') {
return []
}
const predicates: Predicate[] = []
for (const part of whereText.split(/\s+and\s+/i)) {
const match = part.match(/^\s*([a-zA-Z_"][\w."]*)\s*=\s*([\s\S]+?)\s*$/)
if (!match) {
return null
}
predicates.push({ column: lastIdentifier(match[1]), value: parseValue(match[2]) })
}
return predicates
}
function rowMatches(row: Record<string, unknown>, predicates: Predicate[]): boolean {
return predicates.every((predicate) => looseEquals(row[predicate.column], predicate.value))
}
function looseEquals(left: unknown, right: unknown): boolean {
if (left === null || left === undefined) {
return right === null || right === undefined
}
if (typeof left === 'number' && typeof right === 'number') {
return left === right
}
return String(left) === String(right)
}
function parseValue(raw: string): unknown {
// Drop a trailing Postgres cast (e.g. `2::int4`) before interpreting the literal.
const token = raw.trim().replace(/::\s*[a-zA-Z_][\w]*(\([^)]*\))?\s*$/, '').trim()
const stringMatch = token.match(/^'([\s\S]*)'$/)
if (stringMatch) {
return stringMatch[1].replace(/''/g, "'")
}
if (/^-?\d+(\.\d+)?$/.test(token)) {
return Number(token)
}
if (/^true$/i.test(token)) {
return true
}
if (/^false$/i.test(token)) {
return false
}
if (/^null$/i.test(token)) {
return null
}
return token
}
function splitOffReturning(sql: string): { body: string; returning: boolean } {
const match = maskForClauseScan(sql).match(/\sreturning\s/i)
if (!match || match.index === undefined) {
return { body: sql, returning: false }
}
return { body: sql.slice(0, match.index), returning: true }
}
/**
* A same-length copy of `sql` with the contents of single-quoted strings and
* parenthesized groups blanked to spaces, so a top-level keyword scan
* (WHERE / RETURNING) cannot match inside a string literal or a subquery. Index
* positions in the result map 1:1 back onto the original.
*/
function maskForClauseScan(sql: string): string {
let masked = ''
let depth = 0
let inString = false
for (let i = 0; i < sql.length; i++) {
const char = sql[i]
if (inString) {
if (char === "'") {
if (sql[i + 1] === "'") {
masked += ' '
i++
continue
}
inString = false
}
masked += ' '
continue
}
if (char === "'") {
inString = true
masked += ' '
} else if (char === '(') {
depth++
masked += ' '
} else if (char === ')') {
depth = Math.max(0, depth - 1)
masked += ' '
} else {
masked += depth > 0 ? ' ' : char
}
}
return masked
}
/**
* Inner text of each top-level `( ... )` group in `input`, honoring nested parens
* (e.g. `now()`, `numeric(10,2)`) and single-quoted strings. Used for the CREATE
* column-definition group and INSERT value tuples.
*/
function extractParenGroups(input: string): string[] {
const groups: string[] = []
let depth = 0
let inString = false
let current = ''
for (let i = 0; i < input.length; i++) {
const char = input[i]
if (inString) {
current += char
if (char === "'") {
if (input[i + 1] === "'") {
current += input[++i]
} else {
inString = false
}
}
continue
}
if (char === "'") {
inString = true
current += char
} else if (char === '(') {
depth++
if (depth === 1) {
current = ''
} else {
current += char
}
} else if (char === ')') {
depth = Math.max(0, depth - 1)
if (depth === 0) {
groups.push(current)
current = ''
} else {
current += char
}
} else if (depth > 0) {
current += char
}
}
return groups
}
/** Split on commas that are not inside parentheses or single-quoted strings. */
function splitTopLevel(input: string): string[] {
const parts: string[] = []
let depth = 0
let inString = false
let current = ''
for (let i = 0; i < input.length; i++) {
const char = input[i]
if (inString) {
current += char
if (char === "'") {
if (input[i + 1] === "'") {
current += input[++i]
} else {
inString = false
}
}
continue
}
if (char === "'") {
inString = true
current += char
} else if (char === '(') {
depth++
current += char
} else if (char === ')') {
depth = Math.max(0, depth - 1)
current += char
} else if (char === ',' && depth === 0) {
parts.push(current)
current = ''
} else {
current += char
}
}
if (current.trim() !== '') {
parts.push(current)
}
return parts
}
function isTableConstraint(def: string): boolean {
return /^(primary\s+key|foreign\s+key|constraint|unique|check|exclude|like)\b/i.test(def)
}
function unquoteIdentifier(identifier: string): string {
const trimmed = identifier.trim()
const quoted = trimmed.match(/^"([\s\S]*)"$/)
return quoted ? quoted[1] : trimmed
}
/** For a qualified reference like `orders.id`, keep only the final identifier. */
function lastIdentifier(reference: string): string {
const parts = reference.split('.')
return unquoteIdentifier(parts[parts.length - 1])
}
function stripTrailingSemicolon(sql: string): string {
return sql.replace(/;\s*$/, '')
}
File diff suppressed because it is too large Load Diff
@@ -1,36 +0,0 @@
import { describe, expect, it } from 'bun:test'
import { handleBenchmarkApiFetch, hasBenchmarkApiHandler } from './mockBackend'
// `list_workers`'s client call is not faked, so its request really does leave as a relative
// `/api/...` url that node's fetch cannot resolve. Without this route the tool throws instead
// of answering.
describe('benchmark API fetch handlers', () => {
it('lists workers, the way list_workers reaches them', async () => {
const url = '/api/workers/list?per_page=100'
expect(hasBenchmarkApiHandler(url)).toBe(true)
const body = await handleBenchmarkApiFetch(url).json()
// The tool counts the list and maps over it, so it needs a bare array, not an envelope.
expect(Array.isArray(body)).toBe(true)
expect(body[0]).toMatchObject({ worker: expect.any(String), worker_group: expect.any(String) })
})
})
// A benchmark whose list calls an integration undocumented while its metadata endpoint
// hands back authored notes teaches the model the flag means nothing.
describe('benchmark hub integration list', () => {
it('flags exactly the integrations whose metadata carries authored notes', async () => {
const listed = (await handleBenchmarkApiFetch('/api/integrations/hub/list').json()) as Array<{
name: string
documented: boolean
}>
expect(listed.length).toBeGreaterThan(0)
for (const { name, documented } of listed) {
const res = handleBenchmarkApiFetch(`/api/integrations/hub/${name}/meta`)
const authored = res.status === 200 && !!((await res.json()) as { meta?: unknown }).meta
expect(authored).toBe(documented)
}
})
})
@@ -1,175 +0,0 @@
import { afterEach, beforeEach, describe, expect, it } from 'bun:test'
import {
getBenchmarkCompletedJobResultMaybe,
getBenchmarkDatatableSchema,
listBenchmarkDatatables,
registerBenchmarkWorkspaceRunnables,
resetBenchmarkMockBackend,
runBenchmarkDatatableSql,
type BenchmarkWorkspaceRunnables
} from './mockBackend'
const WORKSPACE = 'benchmark-datatable-ws'
// Mirrors the production `isDatatableNotConfiguredError` regex in
// datatableTools.ts. The schema mock's "not configured" message MUST match it,
// otherwise the not-configured mapping in get_datatable_table_schema is silently
// untested.
const NOT_CONFIGURED_RE = /datatable\s+\S+\s+not found/i
const SEED: BenchmarkWorkspaceRunnables = {
datatables: [
{
datatable_name: 'main',
schemas: {
public: {
orders: {
columns: { id: 'int', total: 'numeric' },
rows: [
{ id: 1, total: 10 },
{ id: 2, total: 20 }
]
},
customers: {
columns: { id: 'int', name: 'text' },
rows: [{ id: 1, name: 'alice' }]
}
}
}
}
]
}
beforeEach(() => resetBenchmarkMockBackend())
afterEach(() => resetBenchmarkMockBackend())
describe('listBenchmarkDatatables', () => {
it('returns null for a non-benchmark workspace (caller falls through to real backend)', () => {
expect(listBenchmarkDatatables('unregistered')).toBeNull()
})
it('returns [] for a registered workspace with no datatables seed', () => {
registerBenchmarkWorkspaceRunnables(WORKSPACE, {})
expect(listBenchmarkDatatables(WORKSPACE)).toEqual([])
})
it('projects seeded datatables to schema -> table names only (no columns)', () => {
registerBenchmarkWorkspaceRunnables(WORKSPACE, SEED)
expect(listBenchmarkDatatables(WORKSPACE)).toEqual([
{ datatable_name: 'main', schemas: { public: ['orders', 'customers'] } }
])
})
})
describe('getBenchmarkDatatableSchema', () => {
beforeEach(() => registerBenchmarkWorkspaceRunnables(WORKSPACE, SEED))
it('returns the columns for a seeded table', () => {
expect(
getBenchmarkDatatableSchema({
workspace: WORKSPACE,
datatableName: 'main',
schemaName: 'public',
tableName: 'orders'
})
).toEqual({
datatable_name: 'main',
schema_name: 'public',
table_name: 'orders',
columns: { id: 'int', total: 'numeric' }
})
})
it('throws a not-configured error matching the production regex for an unknown datatable', () => {
let error: Error | undefined
try {
getBenchmarkDatatableSchema({
workspace: WORKSPACE,
datatableName: 'ghost',
schemaName: 'public',
tableName: 'orders'
})
} catch (e) {
error = e as Error
}
expect(error).toBeDefined()
expect(error!.message).toMatch(NOT_CONFIGURED_RE)
})
it('throws a table-not-found error that does NOT match the datatable-not-configured regex', () => {
// The datatable IS configured; only the table is missing. Production maps
// this to a generic "error getting schema", not the blocking message.
let error: Error | undefined
try {
getBenchmarkDatatableSchema({
workspace: WORKSPACE,
datatableName: 'main',
schemaName: 'public',
tableName: 'ghost'
})
} catch (e) {
error = e as Error
}
expect(error).toBeDefined()
expect(error!.message).not.toMatch(NOT_CONFIGURED_RE)
})
})
describe('runBenchmarkDatatableSql + getBenchmarkCompletedJobResultMaybe', () => {
beforeEach(() => registerBenchmarkWorkspaceRunnables(WORKSPACE, SEED))
function exec(sql: string): { success: boolean; completed: boolean; result: unknown } {
const jobId = runBenchmarkDatatableSql({ workspace: WORKSPACE, datatableName: 'main', sql })
return getBenchmarkCompletedJobResultMaybe({ workspace: WORKSPACE, id: jobId })
}
it('returns the canned rows of the table named in a SELECT FROM clause', () => {
expect(exec('SELECT * FROM customers')).toEqual({
success: true,
completed: true,
result: [{ id: 1, name: 'alice' }]
})
})
it('falls back to the first seeded table when the SELECT references no known table', () => {
expect(exec('select 1').result).toEqual([
{ id: 1, total: 10 },
{ id: 2, total: 20 }
])
})
it('returns [] success for DDL and DML statements without RETURNING', () => {
expect(exec('CREATE TABLE foo (id int)').result).toEqual([])
expect(exec('INSERT INTO orders VALUES (3, 30)').result).toEqual([])
expect(exec('update orders set total = 0').result).toEqual([])
})
it('reflects a write in a later SELECT, isolated from the shared seed', () => {
exec('UPDATE orders SET total = 999 WHERE id = 1')
expect((exec('SELECT * FROM orders').result as Record<string, unknown>[])).toContainEqual({
id: 1,
total: 999
})
// Registration deep-clones the seed, so the shared SEED const stays pristine.
expect(SEED.datatables![0].schemas.public.orders.rows).toContainEqual({ id: 1, total: 10 })
})
it('reflects a CREATE in list_datatables and get_datatable_table_schema', () => {
exec('CREATE TABLE public.refunds (order_id int4, amount numeric)')
expect(listBenchmarkDatatables(WORKSPACE)?.[0].schemas.public).toContain('refunds')
expect(
getBenchmarkDatatableSchema({
workspace: WORKSPACE,
datatableName: 'main',
schemaName: 'public',
tableName: 'refunds'
}).columns
).toEqual({ order_id: 'int4', amount: 'numeric' })
})
it('throws for an unknown job id', () => {
expect(() =>
getBenchmarkCompletedJobResultMaybe({ workspace: WORKSPACE, id: 'does-not-exist' })
).toThrow()
})
})
@@ -1,116 +0,0 @@
import { afterEach, beforeEach, describe, expect, it } from 'bun:test'
import {
clearBenchmarkDrafts,
getBenchmarkDraftForUser,
getBenchmarkOwnDraft,
listBenchmarkDrafts,
resetBenchmarkMockBackend,
seedBenchmarkDraft,
updateBenchmarkDraft
} from './mockBackend'
const WORKSPACE = 'benchmark-drafts-ws'
// Drives the in-memory stand-in for the per-user draft backend (`DraftService`)
// that the global AI-chat eval round-trips its drafts through. Mirrors the
// production-unit-test mock in
// `frontend/src/lib/components/copilot/chat/global/core.test.ts`.
describe('mockBackend drafts', () => {
beforeEach(() => resetBenchmarkMockBackend())
afterEach(() => resetBenchmarkMockBackend())
it('round-trips a saved draft through update / get / list', () => {
const value = { summary: 'Greet a user', content: 'export async function main() {}' }
const res = updateBenchmarkDraft({
workspace: WORKSPACE,
kind: 'script',
path: 'f/evals/greet',
requestBody: { value }
})
expect(res.status).toBe('saved')
expect(getBenchmarkDraftForUser({ workspace: WORKSPACE, kind: 'script', path: 'f/evals/greet' }).value).toEqual(
value
)
const rows = listBenchmarkDrafts(WORKSPACE)
expect(rows).toHaveLength(1)
expect(rows[0]).toMatchObject({ kind: 'script', path: 'f/evals/greet', summary: 'Greet a user', draft_only: true })
})
it('treats a null value as a delete', () => {
updateBenchmarkDraft({
workspace: WORKSPACE,
kind: 'variable',
path: 'f/evals/token',
requestBody: { value: { summary: 'token' } }
})
updateBenchmarkDraft({
workspace: WORKSPACE,
kind: 'variable',
path: 'f/evals/token',
requestBody: { value: null }
})
expect(listBenchmarkDrafts(WORKSPACE)).toHaveLength(0)
expect(() => getBenchmarkDraftForUser({ workspace: WORKSPACE, kind: 'variable', path: 'f/evals/token' })).toThrow()
})
it('returns null from getOwnDraft when no draft exists', () => {
expect(
getBenchmarkOwnDraft({ workspace: WORKSPACE, kind: 'trigger_schedule', path: 'u/evals/missing' })
).toBeNull()
})
// The global chat hydrates drawer-kind drafts (schedule/trigger/resource/variable)
// through getOwnDraft — getDraftForUser rejects those kinds as private.
it('hydrates a saved drawer-kind draft through getOwnDraft', () => {
const value = { path: 'u/evals/nightly', schedule: '0 0 9 * * *' }
updateBenchmarkDraft({
workspace: WORKSPACE,
kind: 'trigger_schedule',
path: 'u/evals/nightly',
requestBody: { value }
})
expect(
getBenchmarkOwnDraft({ workspace: WORKSPACE, kind: 'trigger_schedule', path: 'u/evals/nightly' })?.value
).toEqual(value)
})
it('throws a 404-shaped error when no draft exists', () => {
try {
getBenchmarkDraftForUser({ workspace: WORKSPACE, kind: 'script', path: 'f/evals/missing' })
throw new Error('expected a throw')
} catch (e) {
expect((e as { status?: number }).status).toBe(404)
}
})
it('seeds a draft as a backend row that a later edit overwrites', () => {
seedBenchmarkDraft(WORKSPACE, 'script', 'f/evals/current', { content: 'seed' })
expect(getBenchmarkDraftForUser({ workspace: WORKSPACE, kind: 'script', path: 'f/evals/current' }).value).toEqual({
content: 'seed'
})
// A model edit persists the same path and must win over the seed.
updateBenchmarkDraft({
workspace: WORKSPACE,
kind: 'script',
path: 'f/evals/current',
requestBody: { value: { content: 'edited' } }
})
expect(getBenchmarkDraftForUser({ workspace: WORKSPACE, kind: 'script', path: 'f/evals/current' }).value).toEqual({
content: 'edited'
})
})
it('clears only the targeted workspace', () => {
seedBenchmarkDraft(WORKSPACE, 'script', 'f/a', { content: 'a' })
seedBenchmarkDraft('other-ws', 'script', 'f/b', { content: 'b' })
clearBenchmarkDrafts(WORKSPACE)
expect(listBenchmarkDrafts(WORKSPACE)).toHaveLength(0)
expect(listBenchmarkDrafts('other-ws')).toHaveLength(1)
})
})
+27 -509
View File
@@ -8,6 +8,7 @@ import { fileURLToPath } from 'url'
import { handleBenchmarkApiFetch, hasBenchmarkApiHandler } from './mockBackend'
import { resolveWindmillBackendSettings } from '../../core/windmillBackendSettings'
import { WindmillBackendClient } from './windmillBackend'
import { realWorkspaceRoute } from './core/global/realWorkspace'
const FRONTEND_DIR = fileURLToPath(new URL('../../../frontend/', import.meta.url))
@@ -30,6 +31,13 @@ globalThis.fetch = (async (input: unknown, init?: RequestInit) => {
if (typeof url === 'string' && hasBenchmarkApiHandler(url)) {
return handleBenchmarkApiFetch(url, init)
}
// A global-mode case runs against a real workspace: send its calls there, as its user.
const real = typeof url === 'string' ? realWorkspaceRoute(url) : undefined
if (real) {
const headers = new Headers(init?.headers)
headers.set('Authorization', `Bearer ${real.token}`)
return ORIGINAL_FETCH(real.url, { ...init, headers })
}
// The docs tools read the documentation bundled into the backend: hand them to the
// benchmark's real backend, logged in like the AI proxy calls.
if (typeof url === 'string' && url.startsWith('/api/docs/')) {
@@ -119,60 +127,21 @@ vi.mock('$lib/components/apps/components/display/dbtable/metadata', async () =>
vi.mock('$lib/gen', async () => {
const actual = await vi.importActual<any>('$lib/gen')
const {
getBenchmarkAppByPath,
getBenchmarkJob,
getBenchmarkCompletedJobResultMaybe,
listBenchmarkWorkers,
listBenchmarkSchedules,
getBenchmarkSchedule,
listBenchmarkFolders,
getBenchmarkFolder,
getBenchmarkGroup,
listBenchmarkGroups,
getBenchmarkWorkspaceSettingsDocument,
applyBenchmarkWorkspaceSettingsDocument,
getBenchmarkDatatableSchema,
getBenchmarkDatatableMigrationsStatus,
createBenchmarkDatatableMigration,
upsertBenchmarkDatatableMigration,
runBenchmarkDatatableMigrations,
rollbackBenchmarkDatatableMigration,
getBenchmarkDraftForUser,
getBenchmarkCompletedJob,
getBenchmarkFlowByPath,
getBenchmarkFlowAllResults,
getBenchmarkJobLogs,
getBenchmarkOwnDraft,
getBenchmarkScriptByHash,
getBenchmarkScriptByPath,
getBenchmarkAiConfig,
getBenchmarkResourceValue,
getBenchmarkVariableByPath,
hasBenchmarkWorkspace,
getBenchmarkResource,
listBenchmarkAiProviderResources,
listBenchmarkPlainResources,
listBenchmarkResourceTypeNames,
queryBenchmarkResourceTypes,
createBenchmarkResourceType,
listBenchmarkApps,
listBenchmarkDatatables,
listBenchmarkDataMetrics,
listBenchmarkDrafts,
listBenchmarkDucklakes,
listBenchmarkFlows,
listBenchmarkJobs,
listBenchmarkScripts,
listBenchmarkVariables,
createBenchmarkFolder,
createBenchmarkHttpTrigger,
createBenchmarkSchedule,
previewBenchmarkSchedule,
runBenchmarkDatatableSql,
runBenchmarkFlowByPath,
runBenchmarkFlowPreview,
runBenchmarkScriptByPath,
runBenchmarkScriptPreview,
updateBenchmarkDraft
runBenchmarkScriptPreview
} = await import('./mockBackend')
function wrapService<T extends object>(target: T, overrides: Record<string, unknown>): T {
@@ -188,104 +157,23 @@ vi.mock('$lib/gen', async () => {
return {
...actual,
DraftService: wrapService(actual.DraftService, {
updateDraft: async (data: {
workspace: string
kind: any
path: string
requestBody?: { value?: unknown }
}) =>
hasBenchmarkWorkspace(data.workspace)
? updateBenchmarkDraft(data)
: actual.DraftService.updateDraft(data),
getDraftForUser: async (data: { workspace: string; kind: any; path: string }) =>
hasBenchmarkWorkspace(data.workspace)
? getBenchmarkDraftForUser(data)
: actual.DraftService.getDraftForUser(data),
getOwnDraft: async (data: { workspace: string; kind: any; path: string }) =>
hasBenchmarkWorkspace(data.workspace)
? getBenchmarkOwnDraft(data)
: actual.DraftService.getOwnDraft(data),
listDrafts: async (data: { workspace: string }) =>
hasBenchmarkWorkspace(data.workspace)
? listBenchmarkDrafts(data.workspace)
: actual.DraftService.listDrafts(data)
}),
FolderService: wrapService(actual.FolderService, {
createFolder: async (data: { workspace: string; requestBody: { name: string } }) =>
hasBenchmarkWorkspace(data.workspace)
? createBenchmarkFolder(data.workspace, data.requestBody.name)
: actual.FolderService.createFolder(data),
listFolders: async (data: { workspace: string }) =>
hasBenchmarkWorkspace(data.workspace)
? listBenchmarkFolders(data.workspace)
: actual.FolderService.listFolders(data),
getFolder: async (data: { workspace: string; name: string }) => {
if (!hasBenchmarkWorkspace(data.workspace)) return actual.FolderService.getFolder(data)
const folder = getBenchmarkFolder(data.workspace, data.name)
if (!folder) throw new Error(`Folder "${data.name}" not found in benchmark workspace`)
return folder
}
}),
GroupService: wrapService(actual.GroupService, {
listGroups: async (data: { workspace: string }) =>
hasBenchmarkWorkspace(data.workspace)
? listBenchmarkGroups(data.workspace)
: actual.GroupService.listGroups(data),
getGroup: async (data: { workspace: string; name: string }) => {
if (!hasBenchmarkWorkspace(data.workspace)) return actual.GroupService.getGroup(data)
const group = getBenchmarkGroup(data.workspace, data.name)
if (!group) throw new Error(`Group "${data.name}" not found in benchmark workspace`)
return group
}
}),
ScriptService: wrapService(actual.ScriptService, {
listScripts: async (data: { workspace: string }) =>
hasBenchmarkWorkspace(data.workspace)
? (listBenchmarkScripts(data.workspace) ?? [])
: actual.ScriptService.listScripts(data),
existsScriptByPath: async (data: { workspace: string; path: string }) =>
hasBenchmarkWorkspace(data.workspace)
? Boolean(getBenchmarkScriptByPath(data.workspace, data.path))
: actual.ScriptService.existsScriptByPath(data),
getScriptByPath: async (data: { workspace: string; path: string; getDraft?: boolean }) => {
getScriptByPath: async (data: { workspace: string; path: string }) => {
if (hasBenchmarkWorkspace(data.workspace)) {
const script = getBenchmarkScriptByPath(data.workspace, data.path)
// `getDraft` mirrors production's overlay: the row plus the caller's
// draft and a `no_deployed` marker (draft-only item). The diff tool
// reads through this shape — without it every draft looks absent.
const draft = data.getDraft
? getBenchmarkOwnDraft({ workspace: data.workspace, kind: 'script', path: data.path })
: null
if (!script) {
if (data.getDraft && draft) {
return {
...(draft.value as Record<string, unknown>),
path: data.path,
draft: draft.value,
no_deployed: true
}
}
throw Object.assign(
new Error(`Script "${data.path}" not found in benchmark workspace`),
{ status: 404 }
)
}
return data.getDraft
? { ...script, draft: draft?.value ?? undefined, no_deployed: false }
: script
}
return actual.ScriptService.getScriptByPath(data)
},
getScriptByPathWithDraft: async (data: { workspace: string; path: string }) => {
if (hasBenchmarkWorkspace(data.workspace)) {
const script = getBenchmarkScriptByPath(data.workspace, data.path)
if (!script) {
throw new Error(`Script "${data.path}" not found in benchmark workspace`)
}
return script
}
return actual.ScriptService.getScriptByPathWithDraft(data)
return actual.ScriptService.getScriptByPath(data)
},
getScriptByHash: async (data: { workspace: string; hash: string }) => {
if (hasBenchmarkWorkspace(data.workspace)) {
@@ -303,56 +191,18 @@ vi.mock('$lib/gen', async () => {
hasBenchmarkWorkspace(data.workspace)
? (listBenchmarkFlows(data.workspace) ?? [])
: actual.FlowService.listFlows(data),
existsFlowByPath: async (data: { workspace: string; path: string }) =>
hasBenchmarkWorkspace(data.workspace)
? Boolean(getBenchmarkFlowByPath(data.workspace, data.path))
: actual.FlowService.existsFlowByPath(data),
getFlowByPath: async (data: { workspace: string; path: string; getDraft?: boolean }) => {
getFlowByPath: async (data: { workspace: string; path: string }) => {
if (hasBenchmarkWorkspace(data.workspace)) {
const flow = getBenchmarkFlowByPath(data.workspace, data.path)
// Mirror production's `getDraft` overlay (see getScriptByPath above).
const draft = data.getDraft
? getBenchmarkOwnDraft({ workspace: data.workspace, kind: 'flow', path: data.path })
: null
if (!flow) {
if (data.getDraft && draft) {
return {
...(draft.value as Record<string, unknown>),
path: data.path,
draft: draft.value,
no_deployed: true
}
}
throw Object.assign(
new Error(`Flow "${data.path}" not found in benchmark workspace`),
{ status: 404 }
)
}
return data.getDraft
? { ...flow, draft: draft?.value ?? undefined, no_deployed: false }
: flow
}
return actual.FlowService.getFlowByPath(data)
},
getFlowByPathWithDraft: async (data: { workspace: string; path: string }) => {
if (hasBenchmarkWorkspace(data.workspace)) {
const flow = getBenchmarkFlowByPath(data.workspace, data.path)
if (!flow) {
throw new Error(`Flow "${data.path}" not found in benchmark workspace`)
}
return flow
}
return actual.FlowService.getFlowByPathWithDraft(data)
},
getFlowLatestVersion: async (data: { workspace: string; path: string }) => {
if (hasBenchmarkWorkspace(data.workspace)) {
const flow = getBenchmarkFlowByPath(data.workspace, data.path)
if (!flow) {
throw new Error(`Flow "${data.path}" not found in benchmark workspace`)
}
return { id: 1 }
}
return actual.FlowService.getFlowLatestVersion(data)
return actual.FlowService.getFlowByPath(data)
}
}),
JobService: wrapService(actual.JobService, {
@@ -364,43 +214,16 @@ vi.mock('$lib/gen', async () => {
args?: Record<string, unknown>
path?: string
}
}) => {
if (!hasBenchmarkWorkspace(data.workspace)) {
return actual.JobService.runScriptPreview(data)
}
const requestBody = data.requestBody ?? {}
const database = requestBody.args?.database
// Datatable SQL runs as a `postgresql` preview against `datatable://<name>`.
// Execute it through the canned-SQL mock instead of linting it as a script.
if (
requestBody.language === 'postgresql' &&
typeof database === 'string' &&
database.startsWith('datatable://')
) {
return runBenchmarkDatatableSql({
workspace: data.workspace,
datatableName: database.slice('datatable://'.length),
sql: requestBody.content ?? ''
})
}
return runBenchmarkScriptPreview({ workspace: data.workspace, requestBody })
},
runScriptByPath: async (data: {
workspace: string
path: string
requestBody?: Record<string, unknown>
}) =>
hasBenchmarkWorkspace(data.workspace)
? runBenchmarkScriptByPath({
? runBenchmarkScriptPreview({
workspace: data.workspace,
path: data.path,
args: data.requestBody
requestBody: data.requestBody ?? {}
})
: actual.JobService.runScriptByPath(data),
: actual.JobService.runScriptPreview(data),
runFlowPreview: async (data: {
workspace: string
memoryId?: string
requestBody?: { path?: string; value?: { chat_input_enabled?: boolean }; args?: unknown }
requestBody?: { path?: string; args?: unknown }
}) =>
hasBenchmarkWorkspace(data.workspace)
? runBenchmarkFlowPreview(data)
@@ -426,24 +249,10 @@ vi.mock('$lib/gen', async () => {
return job
}
return actual.JobService.getJob(data)
},
getCompletedJobResultMaybe: async (data: { workspace: string; id: string }) =>
hasBenchmarkWorkspace(data.workspace)
? getBenchmarkCompletedJobResultMaybe({ workspace: data.workspace, id: data.id })
: actual.JobService.getCompletedJobResultMaybe(data),
listJobs: async (data: { workspace: string }) =>
hasBenchmarkWorkspace(data.workspace)
? (listBenchmarkJobs(data.workspace) ?? [])
: actual.JobService.listJobs(data),
getJobLogs: async (data: { workspace: string; id: string }) =>
hasBenchmarkWorkspace(data.workspace)
? getBenchmarkJobLogs(data.workspace, data.id)
: actual.JobService.getJobLogs(data),
getFlowAllResults: async (data: { workspace: string; id: string }) =>
hasBenchmarkWorkspace(data.workspace)
? getBenchmarkFlowAllResults(data.workspace, data.id)
: actual.JobService.getFlowAllResults(data)
}
}),
// A benchmark workspace holds no resources and no AI settings: answer as an empty
// workspace would, rather than send its temp-directory id to the real backend.
WorkspaceService: wrapService(actual.WorkspaceService, {
getWorkspaceSettingsDocument: async (data: { workspace: string }) =>
hasBenchmarkWorkspace(data.workspace)
@@ -454,77 +263,13 @@ vi.mock('$lib/gen', async () => {
? applyBenchmarkWorkspaceSettingsDocument(data)
: actual.WorkspaceService.applyWorkspaceSettingsDocument(data),
getCopilotInfo: async (data: { workspace: string }) =>
hasBenchmarkWorkspace(data.workspace)
? (getBenchmarkAiConfig(data.workspace) ?? {})
: actual.WorkspaceService.getCopilotInfo(data),
listDataTableTables: async (data: { workspace: string }) =>
hasBenchmarkWorkspace(data.workspace)
? (listBenchmarkDatatables(data.workspace) ?? [])
: actual.WorkspaceService.listDataTableTables(data),
listDucklakes: async (data: { workspace: string }) =>
hasBenchmarkWorkspace(data.workspace)
? (listBenchmarkDucklakes(data.workspace) ?? [])
: actual.WorkspaceService.listDucklakes(data),
getDataTableTableSchema: async (data: {
workspace: string
datatableName: string
schemaName: string
tableName: string
}) =>
hasBenchmarkWorkspace(data.workspace)
? getBenchmarkDatatableSchema({
workspace: data.workspace,
datatableName: data.datatableName,
schemaName: data.schemaName,
tableName: data.tableName
})
: actual.WorkspaceService.getDataTableTableSchema(data),
getDatatableMigrationsStatus: async (data: any) =>
hasBenchmarkWorkspace(data.workspace)
? getBenchmarkDatatableMigrationsStatus(data)
: actual.WorkspaceService.getDatatableMigrationsStatus(data),
createDatatableMigration: async (data: any) =>
hasBenchmarkWorkspace(data.workspace)
? createBenchmarkDatatableMigration(data)
: actual.WorkspaceService.createDatatableMigration(data),
upsertDatatableMigration: async (data: any) =>
hasBenchmarkWorkspace(data.workspace)
? upsertBenchmarkDatatableMigration(data)
: actual.WorkspaceService.upsertDatatableMigration(data),
runDatatableMigrations: async (data: any) =>
hasBenchmarkWorkspace(data.workspace)
? runBenchmarkDatatableMigrations(data)
: actual.WorkspaceService.runDatatableMigrations(data),
rollbackDatatableMigrations: async (data: any) =>
hasBenchmarkWorkspace(data.workspace)
? rollbackBenchmarkDatatableMigration(data)
: actual.WorkspaceService.rollbackDatatableMigrations(data)
}),
DataMetricService: wrapService(actual.DataMetricService, {
listDataMetrics: async (data: { workspace: string }) =>
hasBenchmarkWorkspace(data.workspace)
? { metrics: listBenchmarkDataMetrics(data.workspace) ?? [] }
: actual.DataMetricService.listDataMetrics(data)
hasBenchmarkWorkspace(data.workspace) ? {} : actual.WorkspaceService.getCopilotInfo(data)
}),
WorkerService: wrapService(actual.WorkerService, {
listWorkers: async (data: Record<string, unknown>) =>
listBenchmarkWorkers() ?? actual.WorkerService.listWorkers(data)
}),
ScheduleService: wrapService(actual.ScheduleService, {
existsSchedule: async (data: { workspace: string; path: string }) =>
hasBenchmarkWorkspace(data.workspace) ? false : actual.ScheduleService.existsSchedule(data),
listSchedules: async (data: { workspace: string }) =>
hasBenchmarkWorkspace(data.workspace)
? listBenchmarkSchedules(data.workspace)
: actual.ScheduleService.listSchedules(data),
getSchedule: async (data: { workspace: string; path: string }) => {
if (hasBenchmarkWorkspace(data.workspace)) {
const schedule = getBenchmarkSchedule(data.workspace, data.path)
if (schedule) return schedule
throw new Error(`Schedule "${data.path}" not found in benchmark workspace`)
}
return actual.ScheduleService.getSchedule(data)
},
previewSchedule: async (data: { requestBody?: Record<string, unknown> }) =>
previewBenchmarkSchedule(data),
createSchedule: async (data: { workspace: string; requestBody: Record<string, unknown> }) =>
@@ -533,254 +278,27 @@ vi.mock('$lib/gen', async () => {
: actual.ScheduleService.createSchedule(data)
}),
ResourceService: wrapService(actual.ResourceService, {
existsResource: async (data: { workspace: string; path: string }) =>
hasBenchmarkWorkspace(data.workspace)
? Boolean(getBenchmarkResourceValue(data.workspace, data.path))
: actual.ResourceService.existsResource(data),
listResource: async (data: { workspace: string; resourceType?: string }) => {
if (!hasBenchmarkWorkspace(data.workspace)) {
return actual.ResourceService.listResource(data)
}
const seeded = [
...(listBenchmarkAiProviderResources(data.workspace) ?? []),
...(listBenchmarkPlainResources(data.workspace) ?? [])
]
const wanted = data.resourceType?.split(',')
return wanted ? seeded.filter((r) => wanted.includes(r.resource_type)) : seeded
},
listResource: async (data: { workspace: string }) =>
hasBenchmarkWorkspace(data.workspace) ? [] : actual.ResourceService.listResource(data),
getResource: async (data: { workspace: string; path: string }) => {
if (hasBenchmarkWorkspace(data.workspace)) {
const resource = getBenchmarkResource(data.workspace, data.path)
if (!resource) {
throw new Error(`Resource "${data.path}" not found in benchmark workspace`)
}
return resource
throw new Error(`Resource "${data.path}" not found in benchmark workspace`)
}
return actual.ResourceService.getResource(data)
},
getResourceValue: async (data: { workspace: string; path: string }) => {
if (!hasBenchmarkWorkspace(data.workspace)) {
return actual.ResourceService.getResourceValue(data)
}
const value = getBenchmarkResourceValue(data.workspace, data.path)
if (!value) {
throw new Error(`Resource "${data.path}" not found in benchmark workspace`)
}
return value
},
queryResourceTypes: async (data: { workspace: string; text: string; limit?: number }) =>
hasBenchmarkWorkspace(data.workspace)
? queryBenchmarkResourceTypes(data.workspace, data.text, data.limit)
? queryBenchmarkResourceTypes(data.text, data.limit)
: actual.ResourceService.queryResourceTypes(data),
listResourceTypeNames: async (data: { workspace: string }) =>
listBenchmarkResourceTypeNames(data.workspace) ??
actual.ResourceService.listResourceTypeNames(data),
createResourceType: async (data: {
workspace: string
requestBody: { name: string; schema?: unknown; description?: string }
}) => {
if (!hasBenchmarkWorkspace(data.workspace)) {
return actual.ResourceService.createResourceType(data)
}
createBenchmarkResourceType(data.workspace, data.requestBody)
return `resource_type ${data.requestBody.name} created`
}
}),
VariableService: wrapService(actual.VariableService, {
existsVariable: async (data: { workspace: string; path: string }) =>
hasBenchmarkWorkspace(data.workspace)
? Boolean(getBenchmarkVariableByPath(data.workspace, data.path))
: actual.VariableService.existsVariable(data),
listVariable: async (data: { workspace: string }) =>
hasBenchmarkWorkspace(data.workspace)
? (listBenchmarkVariables(data.workspace) ?? [])
: actual.VariableService.listVariable(data),
getVariable: async (data: {
workspace: string
path: string
decryptSecret?: boolean
}) => {
if (hasBenchmarkWorkspace(data.workspace)) {
const variable = getBenchmarkVariableByPath(
data.workspace,
data.path,
data.decryptSecret ?? true
)
if (!variable) {
throw new Error(`Variable "${data.path}" not found in benchmark workspace`)
}
return variable
}
return actual.VariableService.getVariable(data)
}
}),
AppService: wrapService(actual.AppService, {
existsApp: async (data: { workspace: string; path: string }) =>
hasBenchmarkWorkspace(data.workspace)
? Boolean(getBenchmarkAppByPath(data.workspace, data.path))
: actual.AppService.existsApp(data),
listApps: async (data: { workspace: string }) =>
hasBenchmarkWorkspace(data.workspace)
? (listBenchmarkApps(data.workspace) ?? [])
: actual.AppService.listApps(data),
getAppByPath: async (data: {
workspace: string
path: string
getDraft?: boolean
rawApp?: boolean
}) => {
if (hasBenchmarkWorkspace(data.workspace)) {
const app = getBenchmarkAppByPath(data.workspace, data.path)
// Mirror production's `getDraft` overlay (see getScriptByPath above).
// Benchmark app drafts live under the raw_app kind.
const draft = data.getDraft
? getBenchmarkOwnDraft({ workspace: data.workspace, kind: 'raw_app', path: data.path })
: null
if (!app) {
if (data.getDraft && draft) {
return {
...(draft.value as Record<string, unknown>),
path: data.path,
raw_app: true,
draft: draft.value,
no_deployed: true
}
}
throw Object.assign(
new Error(`App "${data.path}" not found in benchmark workspace`),
{ status: 404 }
)
}
return data.getDraft
? { ...app, draft: draft?.value ?? undefined, no_deployed: false }
: app
}
return actual.AppService.getAppByPath(data)
}
actual.ResourceService.listResourceTypeNames(data)
}),
HttpTriggerService: wrapService(actual.HttpTriggerService, {
existsHttpTrigger: async (data: { workspace: string; path: string }) =>
hasBenchmarkWorkspace(data.workspace) ? false : actual.HttpTriggerService.existsHttpTrigger(data),
listHttpTriggers: async (data: { workspace: string }) =>
hasBenchmarkWorkspace(data.workspace) ? [] : actual.HttpTriggerService.listHttpTriggers(data),
getHttpTrigger: async (data: { workspace: string; path: string }) => {
if (hasBenchmarkWorkspace(data.workspace)) {
throw new Error(`HTTP trigger "${data.path}" not found in benchmark workspace`)
}
return actual.HttpTriggerService.getHttpTrigger(data)
},
createHttpTrigger: async (data: { workspace: string; requestBody: Record<string, unknown> }) =>
hasBenchmarkWorkspace(data.workspace)
? createBenchmarkHttpTrigger(data)
: actual.HttpTriggerService.createHttpTrigger(data)
}),
WebsocketTriggerService: wrapService(actual.WebsocketTriggerService, {
existsWebsocketTrigger: async (data: { workspace: string; path: string }) =>
hasBenchmarkWorkspace(data.workspace)
? false
: actual.WebsocketTriggerService.existsWebsocketTrigger(data),
listWebsocketTriggers: async (data: { workspace: string }) =>
hasBenchmarkWorkspace(data.workspace)
? []
: actual.WebsocketTriggerService.listWebsocketTriggers(data),
getWebsocketTrigger: async (data: { workspace: string; path: string }) => {
if (hasBenchmarkWorkspace(data.workspace)) {
throw new Error(`Websocket trigger "${data.path}" not found in benchmark workspace`)
}
return actual.WebsocketTriggerService.getWebsocketTrigger(data)
}
}),
KafkaTriggerService: wrapService(actual.KafkaTriggerService, {
existsKafkaTrigger: async (data: { workspace: string; path: string }) =>
hasBenchmarkWorkspace(data.workspace)
? false
: actual.KafkaTriggerService.existsKafkaTrigger(data),
listKafkaTriggers: async (data: { workspace: string }) =>
hasBenchmarkWorkspace(data.workspace) ? [] : actual.KafkaTriggerService.listKafkaTriggers(data),
getKafkaTrigger: async (data: { workspace: string; path: string }) => {
if (hasBenchmarkWorkspace(data.workspace)) {
throw new Error(`Kafka trigger "${data.path}" not found in benchmark workspace`)
}
return actual.KafkaTriggerService.getKafkaTrigger(data)
}
}),
NatsTriggerService: wrapService(actual.NatsTriggerService, {
existsNatsTrigger: async (data: { workspace: string; path: string }) =>
hasBenchmarkWorkspace(data.workspace) ? false : actual.NatsTriggerService.existsNatsTrigger(data),
listNatsTriggers: async (data: { workspace: string }) =>
hasBenchmarkWorkspace(data.workspace) ? [] : actual.NatsTriggerService.listNatsTriggers(data),
getNatsTrigger: async (data: { workspace: string; path: string }) => {
if (hasBenchmarkWorkspace(data.workspace)) {
throw new Error(`NATS trigger "${data.path}" not found in benchmark workspace`)
}
return actual.NatsTriggerService.getNatsTrigger(data)
}
}),
PostgresTriggerService: wrapService(actual.PostgresTriggerService, {
existsPostgresTrigger: async (data: { workspace: string; path: string }) =>
hasBenchmarkWorkspace(data.workspace)
? false
: actual.PostgresTriggerService.existsPostgresTrigger(data),
listPostgresTriggers: async (data: { workspace: string }) =>
hasBenchmarkWorkspace(data.workspace)
? []
: actual.PostgresTriggerService.listPostgresTriggers(data),
getPostgresTrigger: async (data: { workspace: string; path: string }) => {
if (hasBenchmarkWorkspace(data.workspace)) {
throw new Error(`Postgres trigger "${data.path}" not found in benchmark workspace`)
}
return actual.PostgresTriggerService.getPostgresTrigger(data)
}
}),
MqttTriggerService: wrapService(actual.MqttTriggerService, {
existsMqttTrigger: async (data: { workspace: string; path: string }) =>
hasBenchmarkWorkspace(data.workspace) ? false : actual.MqttTriggerService.existsMqttTrigger(data),
listMqttTriggers: async (data: { workspace: string }) =>
hasBenchmarkWorkspace(data.workspace) ? [] : actual.MqttTriggerService.listMqttTriggers(data),
getMqttTrigger: async (data: { workspace: string; path: string }) => {
if (hasBenchmarkWorkspace(data.workspace)) {
throw new Error(`MQTT trigger "${data.path}" not found in benchmark workspace`)
}
return actual.MqttTriggerService.getMqttTrigger(data)
}
}),
SqsTriggerService: wrapService(actual.SqsTriggerService, {
existsSqsTrigger: async (data: { workspace: string; path: string }) =>
hasBenchmarkWorkspace(data.workspace) ? false : actual.SqsTriggerService.existsSqsTrigger(data),
listSqsTriggers: async (data: { workspace: string }) =>
hasBenchmarkWorkspace(data.workspace) ? [] : actual.SqsTriggerService.listSqsTriggers(data),
getSqsTrigger: async (data: { workspace: string; path: string }) => {
if (hasBenchmarkWorkspace(data.workspace)) {
throw new Error(`SQS trigger "${data.path}" not found in benchmark workspace`)
}
return actual.SqsTriggerService.getSqsTrigger(data)
}
}),
GcpTriggerService: wrapService(actual.GcpTriggerService, {
existsGcpTrigger: async (data: { workspace: string; path: string }) =>
hasBenchmarkWorkspace(data.workspace) ? false : actual.GcpTriggerService.existsGcpTrigger(data),
listGcpTriggers: async (data: { workspace: string }) =>
hasBenchmarkWorkspace(data.workspace) ? [] : actual.GcpTriggerService.listGcpTriggers(data),
getGcpTrigger: async (data: { workspace: string; path: string }) => {
if (hasBenchmarkWorkspace(data.workspace)) {
throw new Error(`GCP trigger "${data.path}" not found in benchmark workspace`)
}
return actual.GcpTriggerService.getGcpTrigger(data)
}
}),
AzureTriggerService: wrapService(actual.AzureTriggerService, {
existsAzureTrigger: async (data: { workspace: string; path: string }) =>
hasBenchmarkWorkspace(data.workspace)
? false
: actual.AzureTriggerService.existsAzureTrigger(data),
listAzureTriggers: async (data: { workspace: string }) =>
hasBenchmarkWorkspace(data.workspace) ? [] : actual.AzureTriggerService.listAzureTriggers(data),
getAzureTrigger: async (data: { workspace: string; path: string }) => {
if (hasBenchmarkWorkspace(data.workspace)) {
throw new Error(`Azure trigger "${data.path}" not found in benchmark workspace`)
}
return actual.AzureTriggerService.getAzureTrigger(data)
}
})
}
})
+3 -1
View File
@@ -33,7 +33,9 @@ export function createGlobalModeRunner(
): ModeRunner<GlobalInitialFixture, GlobalDraftState, GlobalDraftState> {
return {
mode: "global",
concurrency: 3,
// Each attempt holds a workspace of its own, and CE allows two besides `admins`; a third
// attempt would wait for a slot, and the wait would count toward its duration.
concurrency: 2,
judgeThreshold: 80,
async loadInitial(path) {
return path ? await loadGlobalInitialFixture(path) : undefined;
+78
View File
@@ -14806,6 +14806,84 @@ paths:
type: boolean
description: more buckets matched than were returned, so summing them under-reports
/w/{workspace}/ai_session_tools/{name}:
post:
summary: run one of the AI session's tools as the calling user
description: The in-app chat's entry to the session toolset the MCP server also serves. Refused for job tokens and scoped tokens.
operationId: callAiSessionTool
tags:
- ai
parameters:
- $ref: "#/components/parameters/WorkspaceId"
- name: name
in: path
required: true
schema:
type: string
requestBody:
required: true
content:
application/json:
schema:
type: object
properties:
args:
description: the tool's arguments, as the model sent them
context:
type: object
description: browser-held state the tool reads
additionalProperties: true
properties:
live_editors:
type: array
items:
type: object
required:
- kind
- storage_path
properties:
kind:
type: string
storage_path:
type: string
effective_path:
type: string
cells:
type: array
items:
type: object
required:
- kind
- storage_path
- value
properties:
kind:
type: string
storage_path:
type: string
value: {}
responses:
"200":
description: what the model reads, and what the chat applies
content:
application/json:
schema:
type: object
required:
- text
- is_error
- effects
properties:
text:
type: string
is_error:
type: boolean
effects:
type: array
items:
type: object
additionalProperties: true
/w/{workspace}/ai/sessions/list:
get:
summary: list the calling user's AI session backups in the workspace object storage
@@ -0,0 +1,445 @@
//! The provider config of AI agent and AI decision steps, checked against the workspace's
//! AI provider resources before a flow is written.
//!
//! The catalog of those resources and the models each serves is the chat's own
//! (`chat/flow/aiAgentProviderCatalog.ts`): its prompt needs it anyway, and it lists each
//! resource's models through the provider. The chat sends it when a call needs it. An MCP
//! client gets one built here from the resources and the AI settings alone, without model
//! listings, so it can rule out a malformed or unknown provider but never a model id.
use serde::Deserialize;
use serde_json::{json, Value};
use super::chat::Effect;
use super::{array, truthy, Ctx, Surface};
#[derive(Deserialize, Clone, Default)]
#[serde(rename_all = "camelCase")]
pub(crate) struct Catalog {
#[serde(default)]
options: Vec<ProviderOption>,
/// `options` holds every AI provider resource of the workspace: one missing from it is
/// not a resource at all.
#[serde(default)]
resources_are_complete: bool,
}
#[derive(Deserialize, Clone)]
#[serde(rename_all = "camelCase")]
struct ProviderOption {
kind: String,
resource_ref: String,
models: Models,
/// The models came from the endpoint's own listing, not a fallback.
models_are_live: bool,
/// The resource points at a gateway that may serve ids its listing omits.
custom_endpoint: bool,
}
#[derive(Deserialize, Clone, Default)]
struct Models {
ids: Vec<String>,
complete: bool,
}
/// The kinds that serve decision models only, which an AI agent step cannot run on.
const DECISION_ONLY_KINDS: &[&str] = &["typesafe", "cloudflare"];
/// The model of OpenAI's Decisions API, the one kind both step types run on.
const OPENAI_DECISION_MODEL: &str = "gpt-6-luna";
const DECISION_PROVIDER_KINDS: &[&str] = &["typesafe", "cloudflare", "openai"];
const MAX_PROMPTED_MODELS: usize = 25;
const PROVIDER_SHAPE: &str = r#"{ "type": "static", "value": { "kind": "<provider kind>", "resource": "$res:<resource path>", "model": "<model id>" } }"#;
/// Every AI agent and AI decision module of a module tree, agent tools included, in the
/// order the editor's `forEachAiAgentModule` visits them.
fn for_each_ai_module<'a>(modules: &'a Value, f: &mut impl FnMut(&'a Value, &'a Value)) {
for module in array(Some(modules)) {
let Some(value) = module.get("value") else {
continue;
};
match value.get("type").and_then(Value::as_str) {
Some("aiagent") => {
f(module, value);
if let Some(tools) = value.get("tools") {
for_each_ai_module(tools, f);
}
}
Some("forloopflow" | "whileloopflow") => {
if let Some(inner) = value.get("modules") {
for_each_ai_module(inner, f);
}
}
Some("aidecision") => f(module, value),
Some(kind @ ("branchone" | "branchall")) => {
if kind == "branchone" {
if let Some(default) = value.get("default") {
for_each_ai_module(default, f);
}
}
for branch in array(value.get("branches")) {
if let Some(inner) = branch.get("modules") {
for_each_ai_module(inner, f);
}
}
}
_ => {}
}
}
}
/// A step's own static provider: what the catalog checks.
fn static_provider(value: &Value) -> Option<&Value> {
if truthy(value.get("agent")) {
return None;
}
let transform = value.get("input_transforms")?.get("provider")?;
(transform.get("type").and_then(Value::as_str) == Some("static")).then_some(transform)
}
/// The `$res:` references the steps' providers use, or `None` when no step states a provider
/// of its own and the catalog is not needed.
fn provider_refs(modules: &Value) -> Option<Vec<String>> {
let mut needed = false;
let mut refs: Vec<String> = Vec::new();
for_each_ai_module(modules, &mut |_, value| {
let Some(transform) = static_provider(value) else {
return;
};
needed = true;
if let Some(resource) = transform
.get("value")
.and_then(|v| v.get("resource"))
.and_then(Value::as_str)
{
if !refs.iter().any(|r| r == resource) {
refs.push(resource.to_string());
}
}
});
needed.then_some(refs)
}
/// What the catalog for `modules` is: none needed, the one to check against, or none yet
/// because the chat has to send it first.
pub(super) enum CatalogFor {
NotNeeded,
Ready(Catalog),
Requested,
}
pub(super) async fn catalog_for(ctx: &Ctx<'_>, modules: &Value) -> CatalogFor {
let Some(resource_refs) = provider_refs(modules) else {
return CatalogFor::NotNeeded;
};
match ctx.surface {
Surface::Chat => match ctx.chat.and_then(|chat| chat.ai_provider_catalog.clone()) {
Some(catalog) => CatalogFor::Ready(catalog),
None => {
ctx.effect(Effect::NeedsAiProviderCatalog { resource_refs });
CatalogFor::Requested
}
},
Surface::Mcp => CatalogFor::Ready(listed_catalog(ctx).await),
}
}
/// The resource types an AI agent or AI decision step's provider can be, as the editor lists
/// them (`AI_PROVIDERS` and `DECISION_AI_PROVIDERS` in `copilot/lib.ts`).
const AI_RESOURCE_TYPES: &[&str] = &[
"openai",
"anthropic",
"googleai",
"azure_openai",
"azure_foundry",
"mistral",
"deepseek",
"groq",
"openrouter",
"togetherai",
"aws_bedrock",
"customai",
"typesafe",
"cloudflare",
];
const MAX_PROVIDER_RESOURCES: usize = 8;
/// The workspace's AI provider resources, without model listings: each one's models are the
/// ones its AI settings configure, which can never rule an id out.
async fn listed_catalog(ctx: &Ctx<'_>) -> Catalog {
let route = ctx.ws(&format!(
"resources/list?resource_type={}",
AI_RESOURCE_TYPES.join(",")
));
let (resources, listed_all) = match ctx.get(&route).await {
Ok(resources) => (resources, true),
Err(_) => (json!([]), false),
};
let config = ctx
.get(&ctx.ws("workspaces/get_copilot_info"))
.await
.unwrap_or(Value::Null);
let providers = config.get("providers").and_then(Value::as_object);
let configured = |path: &str| {
providers
.into_iter()
.flatten()
.find(|(_, p)| p.get("resource_path").and_then(Value::as_str) == Some(path))
.map(|(kind, p)| (kind.clone(), p))
};
let default_kind = config
.get("default_model")
.and_then(|m| m.get("provider"))
.and_then(Value::as_str);
let mut candidates: Vec<(String, String)> = array(Some(&resources))
.iter()
.filter_map(|r| {
let kind = r.get("resource_type")?.as_str()?;
let path = r.get("path")?.as_str()?;
AI_RESOURCE_TYPES
.contains(&kind)
.then(|| (kind.to_string(), path.to_string()))
})
.collect();
let rank = |(kind, path): &(String, String)| match configured(path) {
None => 2,
Some(_) if Some(kind.as_str()) == default_kind => 0,
Some(_) => 1,
};
candidates.sort_by(|a, b| rank(a).cmp(&rank(b)).then_with(|| a.1.cmp(&b.1)));
let complete = listed_all && candidates.len() <= MAX_PROVIDER_RESOURCES;
let options = candidates
.into_iter()
.take(MAX_PROVIDER_RESOURCES)
.map(|(kind, path)| {
let ids = configured(&path)
.map(|(_, p)| {
array(p.get("models"))
.iter()
.filter_map(|m| m.as_str().map(String::from))
.collect()
})
.unwrap_or_default();
ProviderOption {
kind,
resource_ref: format!("$res:{path}"),
models: Models { ids, complete: false },
models_are_live: false,
custom_endpoint: true,
}
})
.collect();
Catalog { options, resources_are_complete: complete }
}
struct Issue {
message: String,
blocking: bool,
/// The catalog established it rather than failing to rule it out.
settled: bool,
}
fn blocking(message: String) -> Option<Issue> {
Some(Issue { message, blocking: true, settled: false })
}
fn rules_out_other_models(option: &ProviderOption) -> bool {
option.models_are_live && option.models.complete && !option.custom_endpoint
}
fn check_provider(value: Option<&Value>, catalog: &Catalog, step_type: &str) -> Option<Issue> {
let provider = match value {
Some(Value::String(reference)) => {
return blocking(format!(
"provider must be an object {PROVIDER_SHAPE}, not the resource reference string {}",
json!(reference)
))
}
Some(Value::Object(provider)) => provider,
_ => return blocking(format!("provider must be an object {PROVIDER_SHAPE}")),
};
let text = |key: &str| {
provider
.get(key)
.and_then(Value::as_str)
.unwrap_or_default()
};
let (kind, resource, model) = (text("kind"), text("resource"), text("model"));
if kind.is_empty() {
return blocking(format!(
"provider.kind is missing. Expected {PROVIDER_SHAPE}"
));
}
if step_type == "aidecision" && !DECISION_PROVIDER_KINDS.contains(&kind) {
return blocking(format!("an AI decision step runs on a decision model: provider.kind must be \"typesafe\", \"cloudflare\" or \"openai\", not \"{kind}\""));
}
if step_type != "aidecision" && DECISION_ONLY_KINDS.contains(&kind) {
return blocking(format!("\"{kind}\" serves decision models, which answer typed questions rather than messages: use an \"aidecision\" step rather than an AI agent"));
}
if !resource.starts_with("$res:") {
return blocking(format!(
"provider.resource must be a \"$res:<resource path>\" reference. Expected {PROVIDER_SHAPE}"
));
}
if model.is_empty() {
return blocking(format!(
"provider.model is missing. Expected {PROVIDER_SHAPE}"
));
}
if catalog.options.is_empty() {
return catalog.resources_are_complete.then(|| Issue {
message: format!("this workspace has no AI provider resources, so \"{resource}\" cannot resolve and the step cannot run. Tell the user to create an AI provider resource (Anthropic, OpenAI, ...) and rewrite the step against it."),
blocking: false,
settled: true,
});
}
let Some(found) = catalog.options.iter().find(|o| o.resource_ref == resource) else {
return Some(Issue {
message: format!(
"provider.resource \"{resource}\" is not one of this workspace's AI provider resources"
),
blocking: catalog.resources_are_complete,
settled: false,
});
};
if found.kind != kind {
return blocking(format!(
"provider.kind \"{kind}\" does not match \"{resource}\", which is a `{}` resource",
found.kind
));
}
// An OpenAI resource lists its chat models, which says nothing about its decision model.
if step_type == "aidecision" && kind == "openai" {
return (model != OPENAI_DECISION_MODEL).then(|| Issue {
message: format!("model \"{model}\" is not the model OpenAI's Decisions API is known to serve, \"{OPENAI_DECISION_MODEL}\""),
blocking: false,
settled: false,
});
}
if found.models_are_live && !found.models.ids.iter().any(|id| id == model) {
return Some(Issue {
message: format!("model \"{model}\" is not in the model listing of \"{resource}\""),
blocking: rules_out_other_models(found),
settled: false,
});
}
None
}
fn describe_option(option: &ProviderOption) -> String {
let shown = &option.models.ids[..option.models.ids.len().min(MAX_PROMPTED_MODELS)];
let more = if option.models.ids.len() > shown.len() {
format!(", ... ({} more)", option.models.ids.len() - shown.len())
} else {
String::new()
};
let caveat = if !option.models_are_live {
" (the endpoint would not list its models, so these are a guess — confirm the id with the user)"
} else if !rules_out_other_models(option) {
" (the endpoint may also serve model names this list does not show)"
} else {
""
};
let models = if shown.is_empty() {
"none listed".to_string()
} else {
format!("{}{more}{caveat}", shown.join(", "))
};
format!(
"- `{}` (kind `{}`) — models: {models}",
option.resource_ref, option.kind
)
}
/// Refuse a provider config that would fail at run time, and report in `warnings` what the
/// catalog could not establish, so an incomplete catalog never rejects a provider that works.
pub(super) fn validate(
modules: &Value,
catalog: Option<&Catalog>,
warnings: &mut Vec<String>,
) -> Result<(), String> {
let empty = Catalog::default();
let catalog = catalog.unwrap_or(&empty);
let mut errors = Vec::new();
for_each_ai_module(modules, &mut |module, value| {
let Some(transform) = static_provider(value) else {
return;
};
let step_type = value
.get("type")
.and_then(Value::as_str)
.unwrap_or_default();
let Some(issue) = check_provider(transform.get("value"), catalog, step_type) else {
return;
};
let id = module.get("id").and_then(Value::as_str).unwrap_or_default();
if issue.blocking {
errors.push(format!("Step \"{id}\": {}", issue.message));
} else if issue.settled {
warnings.push(format!("Step \"{id}\": {}", issue.message));
} else {
warnings.push(format!("Step \"{id}\": {}, which could not be ruled out. Confirm it with the user if the step fails to run.", issue.message));
}
});
if errors.is_empty() {
return Ok(());
}
let hint = if catalog.options.is_empty() {
String::new()
} else {
format!(
"\nAI provider resources in this workspace:\n{}",
catalog
.options
.iter()
.map(describe_option)
.collect::<Vec<_>>()
.join("\n")
)
};
Err(format!(
"Invalid AI agent provider configuration:\n{}{hint}",
errors.join("\n")
))
}
/// The tool-result suffix for provider findings that did not block the write.
pub(super) fn format_warnings(warnings: &[String]) -> String {
if warnings.is_empty() {
return String::new();
}
format!("\n\nAI agent provider warnings:\n{}", warnings.join("\n"))
}
#[cfg(test)]
mod tests {
use super::*;
fn agent(model: &str) -> Value {
json!([{ "id": "a", "value": { "type": "aiagent", "input_transforms": { "provider": {
"type": "static",
"value": { "kind": "openai", "resource": "$res:u/x/oai", "model": model }
} } } }])
}
fn catalog(models_are_live: bool) -> Catalog {
serde_json::from_value(json!({
"resourcesAreComplete": true,
"options": [{
"kind": "openai", "resourceRef": "$res:u/x/oai", "modelsAreLive": models_are_live,
"customEndpoint": false, "models": { "ids": ["gpt-a"], "complete": true }
}]
}))
.unwrap()
}
#[test]
fn only_a_live_complete_listing_rules_out_a_model() {
let mut warnings = vec![];
assert!(validate(&agent("gpt-z"), Some(&catalog(true)), &mut warnings).is_err());
assert!(validate(&agent("gpt-z"), Some(&catalog(false)), &mut warnings).is_ok());
assert!(validate(&agent("gpt-z"), None, &mut warnings).is_ok());
assert!(validate(&agent("gpt-a"), Some(&catalog(true)), &mut warnings).is_ok());
assert!(warnings.is_empty());
}
}
@@ -7,17 +7,16 @@ use std::sync::LazyLock;
use serde_json::{json, Map, Value};
use sha2::{Digest, Sha256};
use super::runs::finish_run;
use super::runs::{finish_run, run_tool, start_run, Resolved, RunTarget};
use super::workspace::{
api_message, app_source_to_draft_value, deployed_item_exists, find_and_replace, get_deployed,
get_draft, item_exists, normalize_app_draft, path_enc, persist_draft, Persist,
get_draft, item_exists, normalize_app_draft, persist_draft, Persist,
};
use super::{
bool_arg, js_head, js_len, matches_glob, opt_str_arg, pretty, str_arg, truthy, Ctx, ToolResult,
bool_arg, js_head, js_len, matches_glob, opt_str_arg, pretty, put, str_arg, truthy, Ctx,
ToolResult,
};
// `init_app` is dispatched below but stays out of this list until `gen/app_templates.json`
// is emitted by the generator.
pub(super) const TOOLS: &[&str] = &[
"init_app",
"read_app_file",
@@ -214,23 +213,28 @@ async fn load_app(ctx: &Ctx<'_>, path: &str) -> Result<Loaded, String> {
Ok(Loaded { value: app_source_to_draft_value(&app, Some(&app)), last_sync })
}
/// What the backend requires of a draft from an operator with builder rights: sandboxed,
/// never in viewer mode, and running as nobody but its builder.
async fn conform_builder_policy(ctx: &Ctx<'_>, value: &mut Value) {
/// Whether the caller's role composes apps out of deployed runnables. They can save no inline
/// runnable.
async fn builds_apps(ctx: &Ctx<'_>) -> bool {
if !ctx.auth.is_operator {
return;
return false;
}
let Ok(memberships) = ctx.get("/workspaces/users").await else {
return;
return false;
};
let builder = memberships
memberships
.get("workspaces")
.and_then(Value::as_array)
.and_then(|list| list.iter().find(|w| text_of(w, "id") == Some(ctx.w_id)))
.and_then(|w| w.get("operator_settings"))
.and_then(|s| s.get("builder_apps"))
== Some(&Value::Bool(true));
if !builder {
.and_then(|w| w.get("capabilities"))
.and_then(|c| text_of(c, "apps"))
== Some("compose")
}
/// What the backend requires of a draft from a role that composes apps: sandboxed,
/// never in viewer mode, and running as nobody but its builder.
async fn conform_builder_policy(ctx: &Ctx<'_>, value: &mut Value) {
if !builds_apps(ctx).await {
return;
}
let policy = object_mut(value, "policy");
@@ -272,11 +276,13 @@ async fn save_app(
let mut out = Map::new();
out.insert("success".into(), json!(true));
out.insert("message".into(), json!(message));
out.insert(
"preview_url".into(),
json!(ctx.page_url(&format!(
put(
&mut out,
"preview_url",
ctx.link(&format!(
"/apps_raw/edit/{path}?view=preview&nomenubar=true"
))),
))
.map(Value::from),
);
if let Some(warning) = warning {
out.insert("warning".into(), json!(warning));
@@ -390,9 +396,6 @@ fn app_table_ref(datatable: &str, schema: Option<&str>, reference: &str) -> Stri
async fn init_app(ctx: &Ctx<'_>, args: &Value) -> ToolResult {
let path = str_arg(args, "path")?;
let framework = str_arg(args, "framework")?;
let template = APP_TEMPLATES["templates"]
.get(framework)
.ok_or_else(|| format!("Invalid framework \"{framework}\""))?;
if get_draft(ctx, "raw_app", path).await?.is_some() {
return Err(format!(
@@ -405,13 +408,28 @@ async fn init_app(ctx: &Ctx<'_>, args: &Value) -> ToolResult {
));
}
ctx.status(format!(
"Saving app \"{path}\" draft ({framework} template)…"
));
// A builder can save no inline runnable, so they start from a static page instead of the tour.
let builder = builds_apps(ctx).await;
let template = APP_TEMPLATES[if builder {
"builder_templates"
} else {
"templates"
}]
.get(framework)
.ok_or_else(|| format!("Invalid framework \"{framework}\""))?;
let starters = &APP_TEMPLATES["starter_runnables"];
let mut value = Map::new();
if let Some(summary) = opt_str_arg(args, "summary") {
value.insert("summary".into(), json!(summary));
}
value.insert("files".into(), template.clone());
value.insert("runnables".into(), starters.clone());
value.insert(
"runnables".into(),
if builder { json!({}) } else { starters.clone() },
);
value.insert("policy".into(), json!({ "sandbox": true }));
let data = args.get("data").filter(|d| truthy(Some(d)));
let schema = data.and_then(|d| text_of(d, "schema"));
@@ -449,9 +467,23 @@ async fn init_app(ctx: &Ctx<'_>, args: &Value) -> ToolResult {
Some(schema) => format!(" Its data config names schema \"{schema}\" of data table \"{datatable}\" without creating it: if it does not exist yet, create it before the tables that go in it."),
None => String::new(),
};
let start = if builder {
"The template is a static page with no runnables: replace its UI, and add only deployed scripts and flows, by path.".to_string()
} else {
format!("The template is a demo tour with starter runnables {}: replace its UI and delete the runnables the app does not need (delete_app_runnable).", starter_keys.join(", "))
};
// The chat opens the app in its own panel; an MCP client only has the link.
let open_preview = if ctx.link("/").is_some() {
" Open preview_url in a browser the user can see from this conversation if you have one, and otherwise give them the link: its live preview follows every later edit."
} else {
""
};
let message = format!(
"Initialized a per-user draft app \"{path}\" from the {framework} template (saved server-side, not a deployed workspace item). The template is a demo tour with starter runnables {}: replace its UI and delete the runnables the app does not need (delete_app_runnable). Use write_app_file / write_app_runnable to evolve it.{data_note} Open preview_url in a browser the user can see from this conversation if you have one, and otherwise give them the link: its live preview follows every later edit.",
starter_keys.join(", ")
"Initialized a per-user draft app \"{path}\" from the {framework} template (saved server-side, not a deployed workspace item). {start} Use write_app_file / write_app_runnable to evolve it.{data_note}{open_preview}"
);
ctx.status_after_save(
format!("Saved app \"{path}\" draft ({framework})"),
Some("Saved as draft"),
);
save_app(ctx, path, value, None, message, None).await
}
@@ -507,6 +539,7 @@ async fn read_app_file(ctx: &Ctx<'_>, args: &Value) -> ToolResult {
inline_runnable_content(&value, key, extension, path)?.0
}
};
ctx.status(format!("Read {}", target.file_path()));
let int = |name: &str| args.get(name).and_then(Value::as_i64);
Ok(format_file_read(content, int("offset"), int("limit")))
}
@@ -565,7 +598,7 @@ fn search_files(
query: &str,
file_glob: Option<&str>,
max_matches: i64,
) -> String {
) -> (String, usize, usize) {
let needle = query.to_lowercase();
files.sort_by(|a, b| locale_compare(&a.0, &b.0));
if let Some(glob) = file_glob {
@@ -601,7 +634,7 @@ fn search_files(
}
if total == 0 {
return format!(
let text = format!(
"No matches. Try a broader or differently-spelled term{}.",
if file_glob.is_some() {
", or drop the file_glob"
@@ -609,6 +642,7 @@ fn search_files(
""
}
);
return (text, 0, 0);
}
let plural = |n: usize, one: &str, many: &str| {
@@ -665,7 +699,7 @@ fn search_files(
.to_string(),
);
}
out.join("\n")
(out.join("\n"), total, file_count)
}
async fn search_app(ctx: &Ctx<'_>, args: &Value) -> ToolResult {
@@ -681,12 +715,18 @@ async fn search_app(ctx: &Ctx<'_>, args: &Value) -> ToolResult {
.unwrap_or(SEARCH_DEFAULT_MAX_MATCHES)
.min(SEARCH_MAX_MATCHES_CEILING);
let file_glob = opt_str_arg(args, "file_glob").filter(|glob| !glob.is_empty());
Ok(search_files(
searchable_files(&value),
query,
file_glob,
max_matches,
))
let (text, total, files) =
search_files(searchable_files(&value), query, file_glob, max_matches);
ctx.status(if total == 0 {
format!("No matches for \"{query}\"")
} else {
format!(
"Found {total} match{} in {files} file{}",
if total == 1 { "" } else { "es" },
if files == 1 { "" } else { "s" }
)
});
Ok(text)
}
// ---------------------------------------------------------------------------
@@ -702,6 +742,16 @@ async fn write_app_file(ctx: &Ctx<'_>, args: &Value) -> ToolResult {
assert_not_generated(&file_path)?;
let Loaded { mut value, last_sync } = load_app(ctx, path).await?;
let before = value
.get("files")
.and_then(|files| text_of(files, &file_path))
.unwrap_or_default()
.to_string();
ctx.file_diff(&before, content, &file_path, &file_path);
ctx.status_after_save(
format!("Updated {file_path} in app \"{path}\""),
Some("Saved as draft"),
);
object_mut(&mut value, "files").insert(file_path.clone(), json!(content));
let message = format!("Updated draft app \"{path}\" with frontend file \"{file_path}\".");
save_app(ctx, path, value, last_sync.as_deref(), message, None).await
@@ -715,14 +765,21 @@ async fn delete_app_file(ctx: &Ctx<'_>, args: &Value) -> ToolResult {
assert_not_generated(&file_path)?;
let Loaded { mut value, last_sync } = load_app(ctx, path).await?;
if object_mut(&mut value, "files")
.shift_remove(&file_path)
.is_none()
{
let Some(removed) = object_mut(&mut value, "files").shift_remove(&file_path) else {
return Err(format!(
"Frontend file \"{file_path}\" not found in app \"{path}\"."
));
}
};
ctx.file_diff(
removed.as_str().unwrap_or_default(),
"",
&file_path,
&file_path,
);
ctx.status_after_save(
format!("Removed {file_path} from app \"{path}\""),
Some("Saved as draft"),
);
let message = format!("Removed \"{file_path}\" from draft app \"{path}\".");
save_app(ctx, path, value, last_sync.as_deref(), message, None).await
}
@@ -748,11 +805,14 @@ async fn patch_app_file(ctx: &Ctx<'_>, args: &Value) -> ToolResult {
format!("Frontend file \"{file_path}\" not found in app \"{path}\".")
})?;
let updated = find_and_replace(current, old, new, replace_all, &label)?;
ctx.file_diff(current, &updated, file_path, file_path);
object_mut(&mut value, "files").insert(file_path.clone(), json!(updated));
}
Target::Backend { key, extension, .. } => {
let (current, runnable) = inline_runnable_content(&value, key, extension, path)?;
let updated = find_and_replace(current, old, new, replace_all, &label)?;
let file_path = target.file_path();
ctx.file_diff(current, &updated, &file_path, &file_path);
let language = runnable
.get("inlineScript")
.and_then(|s| s.get("language"))
@@ -764,6 +824,10 @@ async fn patch_app_file(ctx: &Ctx<'_>, args: &Value) -> ToolResult {
object_mut(&mut value, "runnables").insert(key.clone(), runnable);
}
}
ctx.status_after_save(
format!("Patched {} in app \"{path}\"", target.file_path()),
Some("Saved as draft"),
);
let message = format!(
"Patched \"{}\" in draft app \"{path}\".",
target.file_path()
@@ -782,7 +846,11 @@ fn build_persisted_runnable(input: &Value, existing: Option<&Value>) -> Result<V
let name = str_arg(input, "name")?;
let kind = str_arg(input, "type")?;
let existing = existing.and_then(Value::as_object);
let fields = match input.get("staticInputs").and_then(Value::as_object) {
let static_inputs = input.get("staticInputs").and_then(Value::as_object);
// An agent runnable takes every input from the app: a static one inherited from the kind it
// was converted from would be pinned by the deployed policy over every viewer's message.
let fields = match static_inputs {
_ if kind == "agent" => json!({}),
Some(static_inputs) => Value::Object(
static_inputs
.iter()
@@ -828,9 +896,16 @@ fn build_persisted_runnable(input: &Value, existing: Option<&Value>) -> Result<V
return Ok(Value::Object(out));
}
let path = text_of(input, "path")
.filter(|p| !p.is_empty())
.ok_or("path is required when runnable type is \"script\", \"flow\", or \"hubscript\".")?;
let path = text_of(input, "path").filter(|p| !p.is_empty()).ok_or(
"path is required when runnable type is \"script\", \"flow\", \"hubscript\" or \"agent\".",
)?;
// A static input is pinned by the deployed policy: every viewer's turn would send the same one.
if kind == "agent" && static_inputs.is_some_and(|inputs| !inputs.is_empty()) {
return Err(
"An \"agent\" runnable takes no staticInputs: each turn's inputs come from the app."
.to_string(),
);
}
let same_target = existing.is_some_and(|e| {
matches!(
e.get("type").and_then(Value::as_str),
@@ -852,6 +927,45 @@ fn build_persisted_runnable(input: &Value, existing: Option<&Value>) -> Result<V
Ok(Value::Object(out))
}
/// An inline runnable's code; none for one that names a workspace or hub item.
fn inline_code(runnable: Option<&Value>) -> Option<&str> {
let runnable = runnable?;
matches!(text_of(runnable, "type"), Some("inline" | "runnableByName")).then(|| {
runnable
.get("inlineScript")
.and_then(|s| text_of(s, "content"))
.unwrap_or_default()
})
}
/// The card's diff of a runnable's code. A runnable that references an item has no code
/// of its own to diff.
fn runnable_diff(ctx: &Ctx<'_>, key: &str, before: Option<&Value>, after: Option<&Value>) {
let (before_code, after_code) = (inline_code(before), inline_code(after));
if (before_code.is_none() && after_code.is_none()) || (after.is_some() && after_code.is_none())
{
return;
}
let file = |runnable: Option<&Value>| {
format!(
"backend/{key}/main.{}",
runnable.map_or("ts", inline_extension)
)
};
let after_file = file(after.or(before));
let before_file = if before_code.is_none() {
after_file.clone()
} else {
file(before)
};
ctx.file_diff(
before_code.unwrap_or_default(),
after_code.unwrap_or_default(),
&after_file,
&before_file,
);
}
/// Whether the script or flow a path runnable names is deployed. None when it names
/// neither, or the probe failed: the answer only annotates a write.
async fn path_target_deployed(ctx: &Ctx<'_>, runnable: &Value) -> Option<bool> {
@@ -871,6 +985,11 @@ async fn write_app_runnable(ctx: &Ctx<'_>, args: &Value) -> ToolResult {
let Loaded { mut value, last_sync } = load_app(ctx, path).await?;
let existing = value.get("runnables").and_then(|r| r.get(key));
let persisted = build_persisted_runnable(input, existing)?;
runnable_diff(ctx, key, existing, Some(&persisted));
ctx.status_after_save(
format!("Updated runnable \"{key}\" in app \"{path}\""),
Some("Saved as draft"),
);
let warning = match path_target_deployed(ctx, &persisted).await {
Some(false) => Some(format!(
"This runnable points at an item that is NOT deployed (\"{key}\" → {} \"{}\"), so it fails at runtime — a path runnable runs the deployed item, never a draft. Offer to deploy just that item with deploy_workspace_item; the app itself does not need deploying, since the preview runs its draft.",
@@ -889,14 +1008,16 @@ async fn delete_app_runnable(ctx: &Ctx<'_>, args: &Value) -> ToolResult {
let path = str_arg(args, "path")?;
let key = str_arg(args, "key")?;
let Loaded { mut value, last_sync } = load_app(ctx, path).await?;
if object_mut(&mut value, "runnables")
.shift_remove(key)
.is_none()
{
let Some(removed) = object_mut(&mut value, "runnables").shift_remove(key) else {
return Err(format!(
"Backend runnable \"{key}\" not found in app \"{path}\"."
));
}
};
runnable_diff(ctx, key, Some(&removed), None);
ctx.status_after_save(
format!("Removed runnable \"{key}\" from app \"{path}\""),
Some("Saved as draft"),
);
recompute_policy(&mut value);
let message = format!("Removed runnable \"{key}\" from draft app \"{path}\".");
save_app(ctx, path, value, last_sync.as_deref(), message, None).await
@@ -907,7 +1028,12 @@ async fn delete_app_runnable(ctx: &Ctx<'_>, args: &Value) -> ToolResult {
// ---------------------------------------------------------------------------
/// The `execute_component` body that runs a runnable the way the editor preview does.
fn execute_component_body(app_path: &str, key: &str, runnable: &Value, args: &Value) -> Value {
pub(super) fn execute_component_body(
app_path: &str,
key: &str,
runnable: &Value,
args: &Value,
) -> Value {
let fields = entries(runnable, "fields");
let mut test_args = args.as_object().cloned().unwrap_or_default();
let mut static_fields = Map::new();
@@ -998,9 +1124,7 @@ fn execute_component_body(app_path: &str, key: &str, runnable: &Value, args: &Va
Value::Object(body)
}
async fn test_run_app_runnable(ctx: &Ctx<'_>, args: &Value) -> ToolResult {
let path = str_arg(args, "path")?;
let key = str_arg(args, "key")?;
async fn resolve_app_runnable(ctx: &Ctx<'_>, path: &str, key: &str) -> Result<Resolved, String> {
let value = load_app(ctx, path).await?.value;
let runnables = entries(&value, "runnables");
let Some(runnable) = runnables.get(key).filter(|r| truthy(Some(r))) else {
@@ -1014,35 +1138,64 @@ async fn test_run_app_runnable(ctx: &Ctx<'_>, args: &Value) -> ToolResult {
}
));
};
let body = execute_component_body(
path,
key,
runnable,
args.get("args").unwrap_or(&Value::Null),
);
let id = ctx
.request(
"POST",
&ctx.ws(&format!("apps_u/execute_component/{}", path_enc(path))),
Some(body),
)
.await
.map_err(|e| {
let message = api_message(&e);
let message = if js_len(message) > MAX_TOOL_ERROR_LENGTH {
format!("{}... (truncated)", js_head(message, MAX_TOOL_ERROR_LENGTH))
} else {
message.to_string()
};
format!("Failed to execute test run: {message}")
})?;
let id = id.trim().trim_matches('"').to_string();
// A path runnable aimed at a flow queues a flow job, whose steps are worth a look.
let is_flow = text_of(runnable, "runType") == Some("flow");
let label = format!("{path} / {key}");
finish_run(ctx, Ok(id), &label, is_flow, "test", args).await
Ok(Resolved {
target: RunTarget::AppRunnable {
app_path: path.to_string(),
key: key.to_string(),
runnable: runnable.clone(),
},
schema: None,
code: None,
lang: None,
summary: None,
runnable_kind: if is_flow { "flow" } else { "script" },
start_message: format!("Running backend runnable \"{key}\" of app \"{path}\"..."),
conversational: false,
module: None,
})
}
async fn test_run_app_runnable(ctx: &Ctx<'_>, args: &Value) -> ToolResult {
let path = str_arg(args, "path")?;
let key = str_arg(args, "key")?;
run_tool(
ctx,
args,
resolve_app_runnable(ctx, path, key),
async |resolved: Resolved| {
let run_args = args
.get("args")
.and_then(Value::as_object)
.cloned()
.unwrap_or_default();
let id = start_run(ctx, &resolved.target, run_args, &[])
.await
.map_err(|message| {
let message = if js_len(&message) > MAX_TOOL_ERROR_LENGTH {
format!(
"{}... (truncated)",
js_head(&message, MAX_TOOL_ERROR_LENGTH)
)
} else {
message
};
format!("Failed to execute test run: {message}")
})?;
let label = format!("{path} / {key}");
finish_run(
ctx,
Ok(id),
&label,
resolved.runnable_kind == "flow",
"test",
args,
)
.await
},
)
.await
}
#[cfg(test)]
@@ -1094,15 +1247,15 @@ mod tests {
("/App.tsx".to_string(), "foo".to_string()),
];
assert_eq!(
search_files(files.clone(), "foo", None, 100),
search_files(files.clone(), "foo", None, 100).0,
"3 matches in 2 files\n/App.tsx\n 1: foo\n/b.ts\n 1: one\n 2: Foo()\n 3: foo()\n 4: four\n 5: five"
);
assert_eq!(
search_files(files.clone(), "foo", Some("*.ts"), 1),
search_files(files.clone(), "foo", Some("*.ts"), 1).0,
"2 matches in 1 file (showing the first 1; narrow with file_glob or a more specific query)\n/b.ts\n 1: one\n 2: Foo()\n 3: foo()\n 4: four"
);
assert_eq!(
search_files(files, "nope", Some("*.ts"), 100),
search_files(files, "nope", Some("*.ts"), 100).0,
"No matches. Try a broader or differently-spelled term, or drop the file_glob."
);
}
@@ -1137,6 +1290,19 @@ mod tests {
.unwrap();
assert_eq!(retargeted["schema"], json!({}));
let agent = build_persisted_runnable(
&json!({ "name": "n", "type": "agent", "path": "u/a/agent" }),
Some(&existing),
)
.unwrap();
assert_eq!(agent["runType"], "agent");
assert_eq!(agent["fields"], json!({}));
assert!(build_persisted_runnable(
&json!({ "name": "n", "type": "agent", "path": "u/a/agent", "staticInputs": { "m": 1 } }),
None,
)
.is_err());
let inline = build_persisted_runnable(
&json!({
"name": "n", "type": "inline", "staticInputs": { "y": 2 },
@@ -0,0 +1,308 @@
//! What the in-app chat sends beside a tool call, and what it gets back to apply in the
//! browser. MCP has neither: its tools read and write the server alone.
use axum::{extract::Path, routing::post, Extension, Json, Router};
use serde::{Deserialize, Serialize};
use serde_json::{Map, Value};
use windmill_api_auth::is_effectively_unscoped;
use windmill_common::error::{Error, JsonResult};
use super::{call_session_tool, CallRefused, Ctx, SessionAccess, Surface, ToolOutcome};
use crate::db::ApiAuthed;
#[derive(Clone)]
struct BaseInternalUrl(String);
pub(crate) fn workspaced_service(base_internal_url: String) -> Router {
Router::new()
.route("/{name}", post(call_tool))
.layer(Extension(BaseInternalUrl(base_internal_url)))
}
#[derive(Deserialize)]
struct ToolCall {
#[serde(default)]
args: Value,
#[serde(default)]
context: ChatContext,
}
#[derive(Serialize)]
struct ToolResponse {
#[serde(flatten)]
outcome: ToolOutcome,
effects: Vec<Effect>,
}
async fn call_tool(
authed: ApiAuthed,
Extension(BaseInternalUrl(base_internal_url)): Extension<BaseInternalUrl>,
Path((w_id, name)): Path<(String, String)>,
Json(call): Json<ToolCall>,
) -> JsonResult<ToolResponse> {
// A tool calls the API with a JWT carrying its caller's full rights, so a token
// minted for something narrower (a job, a scope) must not reach it.
if authed.job_id.is_some() || !is_effectively_unscoped(authed.scopes.as_deref()) {
return Err(Error::PermissionDenied(
"The AI session tools are only reachable with an unscoped user token".to_string(),
));
}
let mut ctx = Ctx::new(&authed, &w_id, &base_internal_url, SessionAccess::Full);
ctx.surface = Surface::Chat;
ctx.chat = Some(&call.context);
let outcome =
call_session_tool(&ctx, &name, call.args)
.await
.map_err(|refused| match refused {
CallRefused::UnknownTool(message)
| CallRefused::NotAllowed(message)
| CallRefused::NotAvailable(message) => Error::BadRequest(message),
})?;
let effects = ctx.effects.into_inner().unwrap();
Ok(Json(ToolResponse { outcome, effects }))
}
/// Browser-held state a tool needs and the server cannot see.
#[derive(Deserialize, Default)]
pub(crate) struct ChatContext {
/// The editor open on each draft kind, at most one per kind.
#[serde(default)]
pub live_editors: Vec<LiveEditor>,
/// The draft cells the browser holds: an open editor's, and any whose latest edit is
/// not on the server yet. They are newer than the server's row.
#[serde(default)]
pub cells: Vec<Cell>,
/// Run only the check that comes before the chat's confirmation card: what a
/// confirmed call would refuse, and what it would change.
#[serde(default)]
pub check_only: bool,
/// Whether the chat's user left the skill a call reads switched on.
#[serde(default)]
pub skill_enabled: Option<bool>,
/// The folders the session works in, which a listing defaults to.
#[serde(default)]
pub scope_roots: Option<Vec<String>>,
/// The call comes from an AI session, which has a preview panel and builds pipeline
/// nodes in the folder's pipeline draft.
#[serde(default)]
pub session: bool,
/// The workspace's AI provider resources and the models each serves, sent once a call
/// asked for it with `needs_ai_provider_catalog`.
#[serde(default)]
pub ai_provider_catalog: Option<super::ai_providers::Catalog>,
/// A run tool called in two phases around the chat's own argument form: `prepare`
/// resolves what will run, `start` queues it with the arguments the form settled on.
#[serde(default)]
pub run: Option<RunPhase>,
}
#[derive(Deserialize)]
#[serde(tag = "phase", rename_all = "snake_case")]
pub(crate) enum RunPhase {
Prepare,
Start {
/// What `prepare` resolved, so the run is the version the form was shown for;
/// absent for a tool with no form, which resolves and starts in one call.
#[serde(default)]
target: Option<super::runs::RunTarget>,
#[serde(default)]
args: Map<String, Value>,
/// An agent step's form rows, each fed from the run's arguments.
#[serde(default)]
form_keys: Vec<String>,
},
}
#[derive(Deserialize)]
pub(crate) struct LiveEditor {
/// The draft kind (`script`, `flow`, `raw_app`, `trigger_schedule`, …).
pub kind: String,
/// The key the draft is stored under: `u/{user}/draft_{uuid}` for a new item.
pub storage_path: String,
/// The path the editor shows, which a rename leaves the storage key behind.
pub effective_path: Option<String>,
}
#[derive(Deserialize)]
pub(crate) struct Cell {
pub kind: String,
pub storage_path: String,
pub value: Value,
}
/// Something a call changed that the chat applies in the browser.
#[derive(Serialize)]
#[serde(tag = "type", rename_all = "snake_case")]
pub(crate) enum Effect {
/// Save `value` as the draft at `storage_path`, through the editor's own draft store.
/// `last_sync` is the server row the value was built on, when it was read from the
/// server rather than from a cell: the save detects a conflict against it.
DraftWrite {
kind: String,
storage_path: String,
value: Value,
last_sync: Option<String>,
force: bool,
},
/// Delete the draft at `storage_path`. `keep_live_editor` leaves an open editor's
/// copy in place, detached from the server row.
DraftDelete {
kind: String,
storage_path: String,
last_sync: Option<String>,
keep_live_editor: bool,
},
/// `get_run` read one job, which the card renders the run from.
InspectedRun { job_id: String },
/// What the call found that only its card shows, not the model.
Facts(serde_json::Map<String, Value>),
/// A patch to the tool's card, in the order the call reached it. One carrying
/// `after_save` applies only once the chat saved the call's draft writes.
Status(serde_json::Map<String, Value>),
/// The call wrote nothing: it needs the AI provider catalog, which the chat builds, and
/// is to be made again with it. `resource_refs` are the ones the flow references.
NeedsAiProviderCatalog { resource_refs: Vec<String> },
/// The code a write changed, for the card's diff: of a script in `language`, or of an
/// app file at `file_path` (`before_file_path` when its language changed).
CodeDiff {
before: String,
after: String,
language: Option<String>,
file_path: Option<String>,
before_file_path: Option<String>,
},
}
impl ChatContext {
fn live_editor(&self, kind: &str) -> Option<&LiveEditor> {
self.live_editors.iter().find(|e| e.kind == kind)
}
/// The key a draft named `path` is stored under: an open editor's storage key when
/// `path` is either of its names, else `path` itself.
pub fn storage_path<'a>(&'a self, kind: &str, path: &'a str) -> &'a str {
match self.live_editor(kind) {
Some(editor)
if path == editor.storage_path
|| editor.effective_path.as_deref() == Some(path) =>
{
&editor.storage_path
}
_ => path,
}
}
/// The path a draft stored at `storage_path` is shown under, and whether an editor is
/// open on it.
pub fn display_path<'a>(&'a self, kind: &str, storage_path: &'a str) -> (&'a str, bool) {
match self.live_editor(kind) {
Some(editor) if editor.storage_path == storage_path => (
editor
.effective_path
.as_deref()
.filter(|p| !p.is_empty())
.unwrap_or(storage_path),
true,
),
_ => (storage_path, false),
}
}
pub fn cell(&self, kind: &str, storage_path: &str) -> Option<&Value> {
self.cells
.iter()
.find(|c| c.kind == kind && c.storage_path == storage_path)
.map(|c| &c.value)
}
}
impl super::Ctx<'_> {
/// Patch the chat's card for this call.
pub(super) fn status(&self, content: impl Into<String>) {
let mut patch = serde_json::Map::new();
patch.insert("content".into(), Value::String(content.into()));
self.effect(Effect::Status(patch));
}
/// The card's label while a draft write builds and saves.
pub(super) fn saving_draft(&self, item_type: &str, path: &str) {
self.status(format!("Saving {item_type} \"{path}\" as a draft…"));
}
/// `status`, applied only once the chat saved the call's draft writes and deletes.
pub(super) fn status_after_save(&self, content: impl Into<String>, result: Option<&str>) {
let mut patch = serde_json::Map::new();
patch.insert("content".into(), Value::String(content.into()));
if let Some(result) = result {
patch.insert("result".into(), Value::String(result.to_string()));
}
patch.insert("after_save".into(), Value::Bool(true));
self.effect(Effect::Status(patch));
}
pub(super) fn code_diff(&self, before: &str, after: &str, language: Option<&str>) {
self.effect(Effect::CodeDiff {
before: before.to_string(),
after: after.to_string(),
language: language.map(String::from),
file_path: None,
before_file_path: None,
});
}
/// `code_diff` of an app file, whose language its path gives.
pub(super) fn file_diff(
&self,
before: &str,
after: &str,
file_path: &str,
before_file_path: &str,
) {
self.effect(Effect::CodeDiff {
before: before.to_string(),
after: after.to_string(),
language: None,
file_path: Some(file_path.to_string()),
before_file_path: (before_file_path != file_path).then(|| before_file_path.to_string()),
});
}
/// Whether the call comes from an AI session rather than the side-panel chat.
pub(super) fn in_session(&self) -> bool {
self.chat.is_some_and(|chat| chat.session)
}
/// Hand the card facts the model does not read.
pub(super) fn facts(&self, facts: Value) {
if let Value::Object(facts) = facts {
self.effect(Effect::Facts(facts));
}
}
/// Whether the call is the chat's check before its confirmation card.
pub(super) fn check_only(&self) -> bool {
self.chat.is_some_and(|chat| chat.check_only)
}
/// Mark the card failed with `message`, and answer it as the tool's result.
pub(super) fn failed(&self, message: String) -> String {
let mut patch = serde_json::Map::new();
patch.insert("content".into(), Value::String(message.clone()));
patch.insert("error".into(), Value::String(message.clone()));
self.effect(Effect::Status(patch));
message
}
/// The key a draft named `path` is stored under; `path` itself outside the chat.
pub(super) fn storage_path<'s>(&'s self, kind: &str, path: &'s str) -> &'s str {
self.chat.map_or(path, |chat| chat.storage_path(kind, path))
}
/// The path a draft stored at `storage_path` is shown under, and whether an editor is
/// open on it.
pub(super) fn display_path<'s>(&'s self, kind: &str, storage_path: &'s str) -> (&'s str, bool) {
self.chat.map_or((storage_path, false), |chat| {
chat.display_path(kind, storage_path)
})
}
}
@@ -5,7 +5,10 @@ use std::sync::LazyLock;
use regex::Regex;
use serde_json::{json, Map, Value};
use super::runs::{backgrounded_summary, run_script_preview, wait_budget, wait_job};
use super::runs::{
backgrounded_summary, run_script_preview, run_tool, start_run, wait_budget, wait_job, Resolved,
RunTarget,
};
use super::workspace::{api_message, path_enc};
use super::{array, enc, js_str, non_null, opt_str_arg, pretty, str_arg, truthy, Ctx, ToolResult};
@@ -128,6 +131,9 @@ async fn list_datatables(ctx: &Ctx<'_>, args: &Value) -> ToolResult {
if truthy(d.get("error")) {
out.insert("error".into(), d["error"].clone());
}
if truthy(d.get("migrations_enabled")) {
out.insert("migrations_enabled".into(), Value::Bool(true));
}
if truthy(d.get("permissioned")) {
for key in ["usable_roles", "default_role"] {
if let Some(v) = d.get(key) {
@@ -221,30 +227,33 @@ async fn exec_datatable_sql(ctx: &Ctx<'_>, args: &Value) -> ToolResult {
let name = str_arg(args, "datatable_name")?;
let sql = str_arg(args, "sql")?;
let role = role_arg(args)?;
let failed = |e: String| format!("Failed to execute test run: {e}");
let database = match role {
None => format!("datatable://{name}"),
// A stored name holding a `?` is read whole by the server, so a role cannot follow it.
Some(role) if name.contains('?') => {
return Err(failed(format!("Data table '{name}' has a '?' in its name, so it can only be used here as its default role. Rename it to connect as role '{role}'.")));
}
Some(role) => format!("datatable://{name}?role={role}"),
let resolved = async {
Ok(Resolved {
target: RunTarget::DatatableSql {
datatable: name.to_string(),
sql: sql.to_string(),
role: role.map(String::from),
},
schema: None,
code: None,
lang: None,
summary: None,
runnable_kind: "script",
start_message: format!("Executing SQL on \"{name}\"..."),
conversational: false,
module: None,
})
};
let id = run_script_preview(
ctx,
None,
"postgresql",
sql,
json!({ "database": database }),
)
run_tool(ctx, args, resolved, async |resolved: Resolved| {
let id = start_run(ctx, &resolved.target, Map::new(), &[])
.await
.map_err(|e| format!("Failed to execute test run: {e}"))?;
match wait_job(ctx, &id, wait_budget(args), false).await {
Some(job) => Ok(format_datatable_sql_completion(&job, name)),
None => Ok(backgrounded_summary(&id, &format!("SQL · {name}"))),
}
})
.await
.map_err(failed)?;
match wait_job(ctx, &id, wait_budget(args), false).await {
Some(job) => Ok(format_datatable_sql_completion(&job, name)),
None => Ok(backgrounded_summary(&id, &format!("SQL · {name}"))),
}
}
fn format_datatable_sql_completion(job: &Value, datatable_name: &str) -> String {
@@ -309,9 +318,13 @@ async fn migrations_status(ctx: &Ctx<'_>, datatable_name: &str) -> Result<Value,
}
/// `POST` without a body: applying and rolling back take their target from the query.
/// The migration routes run each migration as a job and wait for it, `TIMEOUT_WAIT_RESULT`
/// (600 s by default) apiece, so a run of several outlasts any per-request budget.
const MIGRATION_RUN_TIMEOUT: std::time::Duration = std::time::Duration::from_secs(3600);
async fn post_migrations(ctx: &Ctx<'_>, path: &str) -> Result<Value, String> {
let text = ctx
.request("POST", path, None)
.request_within("POST", path, None, MIGRATION_RUN_TIMEOUT)
.await
.map_err(|e| api_message(&e).to_string())?;
serde_json::from_str(&text).map_err(|e| format!("unexpected response: {e}"))
@@ -449,9 +462,12 @@ async fn list_datatable_migrations(ctx: &Ctx<'_>, args: &Value) -> ToolResult {
let version = version_arg(args)?;
let status = match migrations_status(ctx, datatable_name).await {
Ok(status) => status,
Err(e) => return Ok(format!("Error listing migrations: {e}")),
Err(e) => return Ok(ctx.failed(format!("Error listing migrations: {}", api_message(&e)))),
};
if !truthy(status.get("enabled")) {
ctx.status(format!(
"Migrations are not enabled on \"{datatable_name}\""
));
return Ok(format!("Migrations are not enabled on datatable \"{datatable_name}\": its schema is changed with plain SQL and there is no history to list. Enabling them is a workspace admin's decision, made in Workspace settings → Data Tables."));
}
let migrations = array(status.get("migrations"));
@@ -474,16 +490,24 @@ async fn list_datatable_migrations(ctx: &Ctx<'_>, args: &Value) -> ToolResult {
if truthy(status.get("error")) {
out.insert("error".into(), status["error"].clone());
}
ctx.status(format!(
"Listed {} migration(s) of \"{datatable_name}\"",
migrations.len()
));
return Ok(pretty(&Value::Object(out)));
};
let Some(migration) = migrations
.iter()
.find(|m| m["timestamp"].as_i64() == Some(version))
else {
return Ok(format!(
return Ok(ctx.failed(format!(
"No migration {version} on datatable \"{datatable_name}\""
));
)));
};
ctx.status(format!(
"Read migration {}",
migration_label(&version.into(), &migration["name"])
));
let mut out = Map::new();
out.insert("version".into(), version.into());
out.insert("name".into(), migration["name"].clone());
@@ -605,11 +629,12 @@ async fn create_datatable_migration(ctx: &Ctx<'_>, args: &Value) -> ToolResult {
}
let version = match save_migration(ctx, datatable_name, version, definition).await {
Ok(Ok(version)) => version,
Ok(Err(message)) => return Ok(message),
Err(e) => return Ok(format!("Error saving migration: {}", api_message(&e))),
Ok(Err(message)) => return Ok(ctx.failed(message)),
Err(e) => return Ok(ctx.failed(format!("Error saving migration: {}", api_message(&e)))),
};
let label = format!("{version} ({name})");
if args.get("apply") == Some(&Value::Bool(false)) {
ctx.status(format!("Saved migration {label} as pending"));
return Ok(format!(
"Saved migration {label} as pending. It is not applied yet."
));
@@ -619,19 +644,27 @@ async fn create_datatable_migration(ctx: &Ctx<'_>, args: &Value) -> ToolResult {
let own_scope = (!scope.is_empty()).then_some(scope.as_slice());
let earlier = match earlier_pending(ctx, datatable_name, version, own_scope).await {
Ok(earlier) => earlier,
Err(e) => return Ok(format!("Saved migration {label} as pending, but could not check what else is pending, so it was not applied: {e}")),
Err(e) => return Ok(ctx.failed(format!("Saved migration {label} as pending, but could not check what else is pending, so it was not applied: {}", api_message(&e)))),
};
if !earlier.is_empty() {
ctx.status(format!(
"Saved migration {label}; {} earlier migration(s) still pending",
earlier.len()
));
return Ok(format!("Saved migration {label} as pending without applying it: earlier migrations are not applied yet ({}) and it may depend on them. Tell the user, then apply them all in order with run_datatable_migrations.", earlier.join(", ")));
}
ctx.status(format!("Applying migration {label}..."));
let run = format!(
"{}?only={version}",
migrations_path(ctx, "run_datatable_migrations", datatable_name)
);
Ok(match post_migrations(ctx, &run).await {
Ok(_) => format!("Applied migration {label} on datatable \"{datatable_name}\"."),
Err(e) => format!("Migration {label} is saved but failed to apply: {e}. It stays pending and blocks later migrations. If nothing of it was applied, fix it by calling this tool again with version {version}; otherwise, or to drop it, tell the user, who can edit or delete it in Workspace settings → Data Tables."),
Ok(_) => {
ctx.status(format!("Applied migration {label}"));
format!("Applied migration {label} on datatable \"{datatable_name}\".")
}
Err(e) => ctx.failed(format!("Migration {label} is saved but failed to apply: {}. It stays pending and blocks later migrations. If nothing of it was applied, fix it by calling this tool again with version {version}; otherwise, or to drop it, tell the user, who can edit or delete it in Workspace settings → Data Tables.", api_message(&e))),
})
}
@@ -642,8 +675,13 @@ async fn run_datatable_migrations(ctx: &Ctx<'_>, args: &Value) -> ToolResult {
Ok(result) => {
let applied = array(result.get("applied"));
if applied.is_empty() {
ctx.status("No pending migration to apply");
format!("No pending migration to apply on datatable \"{datatable_name}\".")
} else {
ctx.status(format!(
"Applied {} migration(s) on \"{datatable_name}\"",
applied.len()
));
format!(
"Applied {} migration(s) on datatable \"{datatable_name}\": {}.",
applied.len(),
@@ -651,7 +689,7 @@ async fn run_datatable_migrations(ctx: &Ctx<'_>, args: &Value) -> ToolResult {
)
}
}
Err(e) => format!("Error applying migrations: {e}. Migrations before the failing one may have been applied: list them to see."),
Err(e) => ctx.failed(format!("Error applying migrations: {}. Migrations before the failing one may have been applied: list them to see.", api_message(&e))),
})
}
@@ -663,12 +701,16 @@ async fn rollback_datatable_migration(ctx: &Ctx<'_>, args: &Value) -> ToolResult
migrations_path(ctx, "rollback_datatable_migrations", datatable_name)
);
Ok(match post_migrations(ctx, &path).await {
Ok(result) if array(result.get("rolled_back")).is_empty() => format!("Migration {version} is not applied on datatable \"{datatable_name}\", so there is nothing to roll back."),
Ok(result) => format!(
"Rolled back migration {} on datatable \"{datatable_name}\". It is pending again.",
labels(result.get("rolled_back"))
),
Err(e) => format!("Error rolling back: {e}"),
Ok(result) if array(result.get("rolled_back")).is_empty() => {
ctx.status(format!("Migration {version} is not applied"));
format!("Migration {version} is not applied on datatable \"{datatable_name}\", so there is nothing to roll back.")
}
Ok(result) => {
let label = labels(result.get("rolled_back"));
ctx.status(format!("Rolled back migration {label}"));
format!("Rolled back migration {label} on datatable \"{datatable_name}\". It is pending again.")
}
Err(e) => ctx.failed(format!("Error rolling back: {}", api_message(&e))),
})
}
@@ -14,7 +14,7 @@ use super::workspace::{
};
use super::{
array, bool_arg, enc, is_not_found, js_str, matches_glob, non_null, opt_str_arg, pretty,
str_arg, truthy, Ctx, ToolResult,
str_arg, truthy, Ctx, Surface, ToolResult,
};
pub(super) const TOOLS: &[&str] = &["deploy_workspace_item", "delete_workspace_item", "diff"];
@@ -299,7 +299,7 @@ async fn undeployed_runnable_targets(ctx: &Ctx<'_>, runnables: &Map<String, Valu
} else {
kind
};
let (Some(target_type @ ("script" | "flow")), Some(path)) = (
let (Some(target_type @ ("script" | "flow" | "agent")), Some(path)) = (
target_type,
runnable
.get("path")
@@ -309,13 +309,30 @@ async fn undeployed_runnable_targets(ctx: &Ctx<'_>, runnables: &Map<String, Valu
continue;
};
// Advisory: a failed probe must not block the deploy it annotates.
if let Ok(false) = deployed_item_exists(ctx, target_type, None, path).await {
let deployed = if target_type == "agent" {
agent_exists(ctx, path).await
} else {
deployed_item_exists(ctx, target_type, None, path).await
};
if let Ok(false) = deployed {
targets.push(format!("\"{key}\" → {target_type} \"{path}\""));
}
}
targets
}
/// Whether a saved agent, an `ai_agent` resource, is at `path`: another resource runs nothing.
async fn agent_exists(ctx: &Ctx<'_>, path: &str) -> Result<bool, String> {
match ctx
.get(&ctx.ws(&format!("resources/get/{}", path_enc(path))))
.await
{
Ok(resource) => Ok(resource["resource_type"] == "ai_agent"),
Err(e) if is_not_found(&e) => Ok(false),
Err(e) => Err(e),
}
}
async fn guest_access_is_live(ctx: &Ctx<'_>) -> bool {
let Ok(usage) = ctx.get(&ctx.ws("workspaces/guest_usage")).await else {
return false;
@@ -574,7 +591,7 @@ async fn deploy_item(ctx: &Ctx<'_>, args: &Value) -> ToolResult {
}
// The baseline keeps a draft saved since the read above from being deleted undeployed.
delete_draft(ctx, &kind, path, draft.created_at.as_deref()).await?;
delete_draft(ctx, &kind, path, draft.created_at.as_deref(), true).await?;
let note = note.map(|note| format!(" {note}")).unwrap_or_default();
Ok(mutation_result(
@@ -582,7 +599,7 @@ async fn deploy_item(ctx: &Ctx<'_>, args: &Value) -> ToolResult {
item_type,
path,
trigger_kind,
deployed_page(item_type, path, value).map(|page| ctx.page_url(&page)),
deployed_page(item_type, path, value).and_then(|page| ctx.link(&page)),
))
}
@@ -597,17 +614,28 @@ async fn delete_item(ctx: &Ctx<'_>, args: &Value) -> ToolResult {
if item_type == "trigger" && trigger_kind.is_none() {
return Err("trigger_kind is required when deleting a trigger.".to_string());
}
// Undoing a draft is a discard, not a delete: say so before the delete route 404s.
if !deployed_item_exists(ctx, item_type, trigger_kind, path).await? {
let draft = get_draft_item(ctx, item_type, path, trigger_kind).await?;
return Err(if draft.is_some() {
format!(
"No deployed {item_type} at \"{path}\" — it only exists as a draft, so there is nothing to delete \
from the workspace. Call discard_local_draft with the same arguments to remove the draft."
)
} else {
format!("No {item_type} at \"{path}\": neither a deployed item nor a draft. Nothing to delete.")
});
// Undoing a draft is a discard, not a delete: say so before the delete route 404s. The
// chat asks this before its confirmation card, which a refused delete never shows.
if ctx.surface == Surface::Mcp || ctx.check_only() {
if !deployed_item_exists(ctx, item_type, trigger_kind, path).await? {
let draft = get_draft_item(ctx, item_type, path, trigger_kind).await?;
let refusal = if draft.is_some() {
format!(
"No deployed {item_type} at \"{path}\" — it only exists as a draft, so there is nothing to delete \
from the workspace. Call discard_local_draft with the same arguments to remove the draft."
)
} else {
format!("No {item_type} at \"{path}\": neither a deployed item nor a draft. Nothing to delete.")
};
if ctx.check_only() {
ctx.facts(json!({ "refused": refusal }));
return Ok(refusal);
}
return Err(refusal);
}
if ctx.check_only() {
return Ok(String::new());
}
}
let path_in_url = path_enc(path);
@@ -626,8 +654,9 @@ async fn delete_item(ctx: &Ctx<'_>, args: &Value) -> ToolResult {
.await?;
}
if let Some(kind) = draft_kind(item_type, trigger_kind) {
delete_draft(ctx, &kind, path, None).await?;
delete_draft(ctx, &kind, path, None, false).await?;
}
ctx.status_after_save(format!("Deleted {item_type} \"{path}\""), Some("Deleted"));
Ok(mutation_result(
format!(
"Deleted {item_type} \"{path}\" from the workspace. Any matching draft was also cleared."
@@ -9,6 +9,7 @@ use std::collections::{HashMap, HashSet};
use serde_json::{json, Map, Value};
use windmill_types::flows::FlowModule;
use super::ai_providers::{self, CatalogFor};
use super::workspace::{
compact_flow, empty_schema, find_and_replace, finish_draft_write, load_flow, persist_draft,
write_base, Persist, FLOW_VALUE_SETTINGS_KEYS,
@@ -69,7 +70,6 @@ const RESERVED_TOOL_NAMES: &[&str] = &[
"Trigger",
"__wm_web_search",
];
const PROVIDER_SHAPE: &str = r#"{ "type": "static", "value": { "kind": "<provider kind>", "resource": "$res:<resource path>", "model": "<model id>" } }"#;
fn module_id(module: &Value) -> &str {
module.get("id").and_then(Value::as_str).unwrap_or_default()
@@ -410,45 +410,6 @@ fn tool_name_error(name: &str, siblings: &[&str]) -> Option<String> {
(siblings.iter().filter(|n| **n == name).count() > 1).then(|| "Duplicate tool name".into())
}
/// A provider config that cannot run whatever the workspace holds.
fn provider_shape_error(provider: Option<&Value>, step_type: &str) -> Option<String> {
let provider = match provider {
Some(Value::String(reference)) => {
return Some(format!(
"provider must be an object {PROVIDER_SHAPE}, not the resource reference string {}",
json!(reference)
))
}
Some(Value::Object(provider)) => provider,
_ => return Some(format!("provider must be an object {PROVIDER_SHAPE}")),
};
let text = |key: &str| {
provider
.get(key)
.and_then(Value::as_str)
.unwrap_or_default()
};
let kind = text("kind");
if kind.is_empty() {
return Some(format!(
"provider.kind is missing. Expected {PROVIDER_SHAPE}"
));
}
let decision_only = ["typesafe", "cloudflare"].contains(&kind);
if step_type == "aidecision" && !decision_only && kind != "openai" {
return Some(format!("an AI decision step runs on a decision model: provider.kind must be \"typesafe\", \"cloudflare\" or \"openai\", not \"{kind}\""));
}
if step_type != "aidecision" && decision_only {
return Some(format!("\"{kind}\" serves decision models, which answer typed questions rather than messages: use an \"aidecision\" step rather than an AI agent"));
}
if !text("resource").starts_with("$res:") {
return Some(format!("provider.resource must be a \"$res:<resource path>\" reference. Expected {PROVIDER_SHAPE}"));
}
text("model")
.is_empty()
.then(|| format!("provider.model is missing. Expected {PROVIDER_SHAPE}"))
}
fn quoted_list(ids: &[String]) -> String {
ids.iter()
.map(|id| format!("\"{id}\""))
@@ -465,7 +426,6 @@ fn validate_modules(modules: &mut Value) -> Result<HashSet<String>, String> {
let mut ids = Vec::new();
let mut providerless = Vec::new();
let mut tool_names = Vec::new();
let mut providers = Vec::new();
walk(Some(modules), false, &mut |module, in_tool| {
let id = module_id(module);
// A tool is not a flow module: the agent that owns it checks its shape.
@@ -517,13 +477,6 @@ fn validate_modules(modules: &mut Value) -> Result<HashSet<String>, String> {
));
}
}
let is_static = |p: &&Value| p.get("type").and_then(Value::as_str) == Some("static");
if let Some(provider) = provider.filter(|_| !linked).filter(is_static) {
if let Some(error) = provider_shape_error(provider.get("value"), kind) {
providers.push(format!("Step \"{id}\": {error}"));
}
}
});
if !shape_errors.is_empty() {
@@ -551,12 +504,6 @@ fn validate_modules(modules: &mut Value) -> Result<HashSet<String>, String> {
tool_names.join("; ")
));
}
if !providers.is_empty() {
return Err(format!(
"Invalid AI agent provider configuration:\n{}",
providers.join("\n")
));
}
Ok(unique)
}
@@ -722,8 +669,12 @@ fn validate_notes(raw: Option<&Value>, module_ids: &HashSet<String>) -> Result<V
/// Checks a compact flow view (`modules, schema, preprocessor_module, failure_module,
/// groups, notes` and the top-level settings) and returns it with every structural key
/// present, `null` when unset.
fn validate_editable(raw: Value) -> Result<Map<String, Value>, String> {
/// present, `null` when unset. Provider findings that do not block go to `warnings`.
fn validate_editable(
raw: Value,
catalog: Option<&ai_providers::Catalog>,
warnings: &mut Vec<String>,
) -> Result<Map<String, Value>, String> {
let Value::Object(mut flow) = raw else {
return Err("Flow JSON must be an object".into());
};
@@ -744,8 +695,9 @@ fn validate_editable(raw: Value) -> Result<Map<String, Value>, String> {
}
validate_settings(&flow)?;
let mut modules = flow.remove("modules").unwrap_or(Value::Null);
let mut modules = flow.shift_remove("modules").unwrap_or(Value::Null);
let module_ids = validate_modules(&mut modules)?;
ai_providers::validate(&modules, catalog, warnings)?;
let schema = match flow.get("schema") {
None | Some(Value::Null) => Value::Null,
Some(schema @ Value::Object(_)) => schema.clone(),
@@ -776,12 +728,12 @@ fn validate_editable(raw: Value) -> Result<Map<String, Value>, String> {
editable.insert("modules".into(), modules);
editable.insert("schema".into(), schema);
for (key, _) in SPECIAL_MODULES {
editable.insert(key.into(), flow.remove(key).unwrap_or(Value::Null));
editable.insert(key.into(), flow.shift_remove(key).unwrap_or(Value::Null));
}
editable.insert("groups".into(), groups);
editable.insert("notes".into(), notes);
for key in FLOW_VALUE_SETTINGS_KEYS {
if let Some(setting) = flow.remove(*key) {
if let Some(setting) = flow.shift_remove(*key) {
editable.insert((*key).into(), setting);
}
}
@@ -804,6 +756,7 @@ struct FlowWrite<'a> {
async fn write_flow_draft(ctx: &Ctx<'_>, write: FlowWrite<'_>) -> ToolResult {
let FlowWrite { path, mut value, .. } = write;
ctx.saving_draft("flow", path);
let base = write_base(ctx, "flow", path, None).await?;
let existed = base.base.is_some();
if let Some(groups) = write.groups {
@@ -910,7 +863,13 @@ async fn write_flow(ctx: &Ctx<'_>, args: &Value) -> ToolResult {
for key in STRUCTURAL_KEYS {
raw.insert((*key).into(), json_arg(args, key)?);
}
let mut editable = validate_editable(Value::Object(raw))?;
let catalog = match ai_providers::catalog_for(ctx, &raw["modules"]).await {
CatalogFor::Requested => return Ok(String::new()),
CatalogFor::Ready(catalog) => Some(catalog),
CatalogFor::NotNeeded => None,
};
let mut warnings = Vec::new();
let mut editable = validate_editable(Value::Object(raw), catalog.as_ref(), &mut warnings)?;
// A placeholder keeps the body the current flow has for that module, so a rewrite
// does not have to send code it is not changing.
@@ -924,9 +883,9 @@ async fn write_flow(ctx: &Ctx<'_>, args: &Value) -> ToolResult {
let schema = non_null(editable.get("schema")).cloned();
let groups = non_null(editable.get("groups")).cloned();
editable.remove("schema");
editable.shift_remove("schema");
editable.retain(|_, field| !field.is_null());
write_flow_draft(
let written = write_flow_draft(
ctx,
FlowWrite {
path,
@@ -940,7 +899,18 @@ async fn write_flow(ctx: &Ctx<'_>, args: &Value) -> ToolResult {
force: args.get("override").and_then(Value::as_bool) == Some(true),
},
)
.await
.await?;
Ok(with_provider_warnings(ctx, written, &warnings))
}
/// The provider findings that did not block the write, after the result. The chat appends
/// them itself when its own save fails and replaces this text.
fn with_provider_warnings(ctx: &Ctx<'_>, written: String, warnings: &[String]) -> String {
let suffix = ai_providers::format_warnings(warnings);
if !suffix.is_empty() {
ctx.facts(json!({ "text_suffix": suffix }));
}
format!("{written}{suffix}")
}
async fn patch_flow_json(ctx: &Ctx<'_>, args: &Value) -> ToolResult {
@@ -959,10 +929,16 @@ async fn patch_flow_json(ctx: &Ctx<'_>, args: &Value) -> ToolResult {
)?;
let parsed: Value = serde_json::from_str(&updated)
.map_err(|e| format!("Invalid JSON after replacement: {e}"))?;
let editable = validate_editable(parsed)?;
let catalog = match ai_providers::catalog_for(ctx, &parsed["modules"]).await {
CatalogFor::Requested => return Ok(String::new()),
CatalogFor::Ready(catalog) => Some(catalog),
CatalogFor::NotNeeded => None,
};
let mut warnings = Vec::new();
let editable = validate_editable(parsed, catalog.as_ref(), &mut warnings)?;
let mut value = apply_editable(base.flow.get("value"), &editable, &scripts);
finalize_inline_scripts(&mut value)?;
write_flow_draft(
let written = write_flow_draft(
ctx,
FlowWrite {
path,
@@ -976,14 +952,38 @@ async fn patch_flow_json(ctx: &Ctx<'_>, args: &Value) -> ToolResult {
force: false,
},
)
.await
.await?;
Ok(with_provider_warnings(ctx, written, &warnings))
}
/// A step as the chat's card names it: its id, and its summary when it has one.
fn step_name(flow: Option<&Value>, module_id: &str) -> String {
let mut flow = flow.and_then(Value::as_object).cloned().unwrap_or_default();
let mut summary = None;
walk_flow(&mut flow, &mut |module, _| {
if summary.is_none() && self::module_id(module) == module_id {
summary = module
.get("summary")
.and_then(Value::as_str)
.map(String::from);
}
});
match summary.filter(|s| !s.is_empty()) {
Some(summary) => format!("{module_id} \"{summary}\""),
None => module_id.to_string(),
}
}
async fn read_flow_module_code(ctx: &Ctx<'_>, args: &Value) -> ToolResult {
let (path, module_id) = (str_arg(args, "path")?, str_arg(args, "module_id")?);
let base = load_flow(ctx, path).await?;
let (_, mut scripts) = compact_flow(base.flow.get("value").unwrap_or(&Value::Null), None);
scripts.remove(module_id).ok_or_else(|| format!("Module \"{module_id}\" is not an inline rawscript in flow \"{path}\". (Path runnables, branches, and loops have no inline script content; use patch_flow_json to inspect them.)"))
let code = scripts.remove(module_id).ok_or_else(|| format!("Module \"{module_id}\" is not an inline rawscript in flow \"{path}\". (Path runnables, branches, and loops have no inline script content; use patch_flow_json to inspect them.)"))?;
ctx.status(format!(
"Read code of step {}",
step_name(base.flow.get("value"), module_id)
));
Ok(code)
}
async fn set_flow_module_code(ctx: &Ctx<'_>, args: &Value) -> ToolResult {
@@ -999,6 +999,14 @@ async fn set_flow_module_code(ctx: &Ctx<'_>, args: &Value) -> ToolResult {
}
scripts.insert(module_id.to_string(), code.to_string());
let editable = compact.as_object().cloned().unwrap_or_default();
// Several code edits of one flow read as identical rows under the generic flow label.
ctx.status_after_save(
format!(
"Updated code of step {}",
step_name(base.flow.get("value"), module_id)
),
None,
);
write_flow_draft(
ctx,
FlowWrite {
@@ -1111,7 +1119,8 @@ mod tests {
.to_string()
.replace(",\"same_worker\":true", "")
.replace("\"id\":\"b\"", "\"id\":\"c\"");
let editable = validate_editable(serde_json::from_str(&patched).unwrap()).unwrap();
let editable =
validate_editable(serde_json::from_str(&patched).unwrap(), None, &mut vec![]).unwrap();
let mut applied = apply_editable(Some(&value), &editable, &scripts);
finalize_inline_scripts(&mut applied).unwrap();
assert_eq!(content(&applied, "/0"), "A");
@@ -1124,7 +1133,7 @@ mod tests {
#[test]
fn validation_reports_what_the_editor_would_refuse() {
let check = |flow: Value| validate_editable(flow).unwrap_err();
let check = |flow: Value| validate_editable(flow, None, &mut vec![]).unwrap_err();
assert_eq!(
check(json!({ "modules": [raw("a", ""), raw("a", "")] })),
"Duplicate module IDs found in flow"
@@ -1159,7 +1168,7 @@ mod tests {
assert!(check(agent(json!([tool("search docs")]))).starts_with(
"Invalid AI agent tool name(s): agent \"agent\" tool \"t\" is named \"search docs\" - Tool name must only contain letters, numbers and underscores. "
));
validate_editable(agent(json!([tool("search_docs")]))).unwrap();
validate_editable(agent(json!([tool("search_docs")])), None, &mut vec![]).unwrap();
}
#[test]
@@ -0,0 +1,99 @@
{
"ai": "AI",
"ai21": "AI21",
"amqp": "AMQP",
"api": "API",
"aws": "AWS",
"cms": "CMS",
"crm": "CRM",
"db": "DB",
"ftp": "FTP",
"gcp": "GCP",
"gpg": "GPG",
"hr": "HR",
"http": "HTTP",
"id": "ID",
"ifs": "IFS",
"json": "JSON",
"jwt": "JWT",
"ldap": "LDAP",
"mcp": "MCP",
"mqtt": "MQTT",
"ms": "Microsoft",
"nats": "NATS",
"oauth": "OAuth",
"odk": "ODK",
"oidc": "OIDC",
"rss": "RSS",
"s3": "S3",
"sdk": "SDK",
"smtp": "SMTP",
"sql": "SQL",
"ssh": "SSH",
"url": "URL",
"ynab": "YNAB",
"abstractapi": "AbstractAPI",
"arcgis": "ArcGIS",
"assemblyai": "AssemblyAI",
"bigquery": "BigQuery",
"chromadb": "ChromaDB",
"circleci": "CircleCI",
"clickhouse": "ClickHouse",
"clickup": "ClickUp",
"cockroachdb": "CockroachDB",
"comapeo": "CoMapeo",
"convertkit": "ConvertKit",
"currencyapi": "CurrencyAPI",
"customai": "CustomAI",
"datocms": "DatoCMS",
"dbt": "dbt",
"deepl": "DeepL",
"deepseek": "DeepSeek",
"digitalocean": "DigitalOcean",
"docspring": "DocSpring",
"docusign": "DocuSign",
"edgedb": "EdgeDB",
"faunadb": "FaunaDB",
"gcloud": "Google Cloud",
"ghostcms": "Ghost CMS",
"gitbook": "GitBook",
"github": "GitHub",
"gitlab": "GitLab",
"googleai": "GoogleAI",
"graphql": "GraphQL",
"groqai": "GroqAI",
"hubspot": "HubSpot",
"ipinfo": "IPinfo",
"kobotoolbox": "KoboToolbox",
"leonardoai": "LeonardoAI",
"linkedin": "LinkedIn",
"lumaai": "LumaAI",
"mailerlite": "MailerLite",
"mongodb": "MongoDB",
"mysql": "MySQL",
"netbox": "NetBox",
"netsuite": "NetSuite",
"newsapi": "NewsAPI",
"neondb": "NeonDB",
"nocodb": "NocoDB",
"openai": "OpenAI",
"openrouter": "OpenRouter",
"oracledb": "OracleDB",
"pagerduty": "PagerDuty",
"pandadoc": "PandaDoc",
"paypal": "PayPal",
"planetscale": "PlanetScale",
"postgresql": "PostgreSQL",
"readme": "ReadMe",
"rest": "REST",
"quickbooks": "QuickBooks",
"sendgrid": "SendGrid",
"servicenow": "ServiceNow",
"signoz": "SigNoz",
"surrealdb": "SurrealDB",
"togetherai": "TogetherAI",
"webscrapingai": "WebScrapingAI",
"weatherapi": "WeatherAPI",
"whatsapp": "WhatsApp",
"woocommerce": "WooCommerce"
}
@@ -6,7 +6,8 @@ use std::sync::LazyLock;
use serde::Deserialize;
use serde_json::Value;
use super::{opt_str_arg, str_arg, Ctx, ToolResult};
use super::workspace::api_message;
use super::{opt_str_arg, str_arg, Ctx, Surface, ToolResult};
pub(super) const TOOLS: &[&str] = &["get_instructions", "read_skill"];
@@ -57,6 +58,8 @@ const TRUNCATED: &str = "… [truncated]";
pub(super) struct Skill {
pub path: String,
/// Listed in the MCP instructions; the chat builds its own prompt.
#[cfg_attr(not(feature = "mcp"), allow(dead_code))]
pub description: String,
}
@@ -103,11 +106,19 @@ async fn read_skill(ctx: &Ctx<'_>, args: &Value) -> ToolResult {
// reads any resource holding a string `content`.
let listed = match list_skills(ctx).await {
Ok(skills) => skills.iter().any(|skill| skill.path == path),
Err(e) => return Ok(read_failed(path, &e)),
Err(e) => return Ok(read_failed(ctx, path, api_message(&e))),
};
if !listed {
// The chat's user can switch a skill off for that chat, a setting their browser holds.
let switched_off = ctx
.chat
.is_some_and(|chat| !chat.skill_enabled.unwrap_or(true));
if !listed || switched_off {
let (available, listing) = match ctx.surface {
Surface::Chat => ("available to this chat", "the system prompt"),
Surface::Mcp => ("available here", "the instructions"),
};
return Ok(format!(
"\"{path}\" is not one of the skills available here. Only the paths listed under \"Skills\" in the instructions can be read."
"\"{path}\" is not one of the skills {available}. Only the paths listed under \"Skills\" in {listing} can be read."
));
}
let value = match ctx
@@ -115,10 +126,11 @@ async fn read_skill(ctx: &Ctx<'_>, args: &Value) -> ToolResult {
.await
{
Ok(value) => value,
Err(e) => return Ok(read_failed(path, &e)),
Err(e) => return Ok(read_failed(ctx, path, api_message(&e))),
};
let Some(body) = value.get("content").and_then(Value::as_str) else {
return Ok(read_failed(
ctx,
path,
&format!("resource {path} has no string \"content\" — is it an {SKILL_RESOURCE_TYPE}?"),
));
@@ -135,8 +147,12 @@ async fn read_skill(ctx: &Ctx<'_>, args: &Value) -> ToolResult {
Ok(format!("Skill: {path}\n\nInstructions:\n{body}"))
}
fn read_failed(path: &str, message: &str) -> String {
fn read_failed(ctx: &Ctx<'_>, path: &str, message: &str) -> String {
let listing = match ctx.surface {
Surface::Chat => "the system prompt",
Surface::Mcp => "the instructions",
};
format!(
"Failed to read skill \"{path}\": {message}. Check the path against the Skills list in the instructions."
"Failed to read skill \"{path}\": {message}. Check the path against the Skills list in {listing}."
)
}
@@ -121,7 +121,7 @@ async fn appendix(ctx: &Ctx<'_>) -> String {
}
/// The whole prompt, for a connection that reaches one workspace.
pub(in crate::mcp) async fn session_instructions(ctx: &Ctx<'_>) -> String {
pub(crate) async fn session_instructions(ctx: &Ctx<'_>) -> String {
let mut prompt = PARTS
.base
.replace("{{folder_guidance}}", &folder_guidance(ctx.auth))
@@ -132,7 +132,7 @@ pub(in crate::mcp) async fn session_instructions(ctx: &Ctx<'_>) -> String {
/// The prompt without anything specific to a workspace, for a connection that reaches
/// several: usernames are per workspace, so paths read `u/<username>/...`.
pub(in crate::mcp) fn generic_session_instructions() -> String {
pub(crate) fn generic_session_instructions() -> String {
PARTS
.base
.replace("{{folder_guidance}}", "")
@@ -140,7 +140,7 @@ pub(in crate::mcp) fn generic_session_instructions() -> String {
}
/// What `generic_session_instructions` leaves out, for one workspace.
pub(in crate::mcp) async fn workspace_context(ctx: &Ctx<'_>) -> String {
pub(crate) async fn workspace_context(ctx: &Ctx<'_>) -> String {
let username = ctx.auth.username();
format!(
"Workspace `{}`. Your username here is \"{username}\", so your personal paths are \
@@ -44,7 +44,7 @@ static SCHEDULE_SCHEMA: LazyLock<Value> = LazyLock::new(|| {
let mut schema = tool_schema("write_schedule");
let advanced: Value = serde_json::from_str(&SCHEMAS.schedule).unwrap_or_default();
if let Some(properties) = schema.get_mut("properties").and_then(Value::as_object_mut) {
properties.remove("advanced");
properties.shift_remove("advanced");
if let Some(advanced) = advanced.get("properties").and_then(Value::as_object) {
properties.extend(advanced.clone());
}
@@ -282,7 +282,7 @@ fn into_object(value: Option<Value>) -> Map<String, Value> {
fn merge_config(base: Option<Value>, config: Value, path: &str) -> Map<String, Value> {
let mut draft = into_object(base);
for key in ["workspace_id", "edited_by", "edited_at", "email", "error"] {
draft.remove(key);
draft.shift_remove(key);
}
draft.extend(into_object(Some(config)));
draft.insert("path".into(), json!(path));
@@ -454,6 +454,7 @@ async fn write_draft(
from_deployed: impl FnOnce(Value) -> Value,
build: impl FnOnce(Option<Value>) -> Result<Value, String>,
) -> ToolResult {
ctx.saving_draft(item_type, path);
let base = write_base(ctx, item_type, path, trigger_kind).await?;
let existed = base.base.is_some();
let from_draft = base.from_draft;
@@ -480,7 +481,7 @@ async fn write_draft(
fn take_override(parsed: &mut Value) -> bool {
parsed
.as_object_mut()
.and_then(|o| o.remove("override"))
.and_then(|o| o.shift_remove("override"))
.and_then(|v| v.as_bool())
.unwrap_or(false)
}
@@ -1,13 +1,17 @@
//! Lookups that touch no draft: hub, docs, npm, resource types, workers, folders.
use std::collections::HashSet;
use std::collections::{HashMap, HashSet};
use std::sync::LazyLock;
use futures::future::join_all;
use serde_json::{json, Map, Value};
use windmill_common::utils::HTTP_CLIENT;
use super::chat::Effect;
use super::workspace::api_message;
use super::{
array, enc, is_not_found, opt_str_arg, pretty, str_arg, truthy, Ctx, ToolResult, SCHEMAS,
array, enc, is_not_found, js_head, js_len, opt_str_arg, pretty, str_arg, truthy, Ctx,
ToolResult, SCHEMAS,
};
pub(super) const TOOLS: &[&str] = &[
@@ -28,7 +32,7 @@ pub(super) async fn call(ctx: &Ctx<'_>, name: &str, args: &Value) -> ToolResult
match name {
"search_hub_scripts" => search_hub_scripts(ctx, args).await,
"get_hub_integration" => get_hub_integration(ctx, args).await,
"search_npm_packages" => search_npm_packages(args).await,
"search_npm_packages" => search_npm_packages(ctx, args).await,
"search_docs" => search_docs(ctx, args).await,
"read_docs_page" => read_docs_page(ctx, args).await,
"search_resource_types" => search_resource_types(ctx, args).await,
@@ -90,12 +94,37 @@ async fn hub_integrations(ctx: &Ctx<'_>) -> Vec<Value> {
}
}
/// The frontend's `RESOURCE_TYPE_WORDS`: how a slug word reads where capitalizing gets it
/// wrong, some into several words (`gcloud` -> `Google Cloud`).
static DISPLAY_WORDS: LazyLock<HashMap<String, String>> = LazyLock::new(|| {
serde_json::from_str(include_str!("gen/display_words.json"))
.expect("gen/display_words.json is valid")
});
/// What the frontend's `integrationDisplayName` calls a slug the hub names nothing.
fn slug_display_name(slug: &str) -> String {
slug.split(['_', '-'])
.filter(|w| !w.is_empty())
.map(|w| {
DISPLAY_WORDS
.get(&w.to_ascii_lowercase())
.cloned()
.unwrap_or_else(|| w.to_string())
})
.collect::<Vec<_>>()
.join(" ")
}
/// A token matches a whole word of the slug or display name, or extends one from four
/// characters on: a bare substring test makes `for` hit sales*for*ce.
fn token_matches_integration(token: &str, integration: &Value) -> bool {
let name = integration.get("name").and_then(Value::as_str);
let display_name = integration.get("display_name").and_then(Value::as_str);
let words: Vec<String> = [name, display_name]
let display_name = integration
.get("display_name")
.and_then(Value::as_str)
.map(str::to_string)
.or_else(|| name.map(slug_display_name));
let words: Vec<String> = [name, display_name.as_deref()]
.into_iter()
.flatten()
.flat_map(words)
@@ -254,8 +283,9 @@ async fn get_hub_integration(ctx: &Ctx<'_>, args: &Value) -> ToolResult {
Ok(doc) => doc,
Err(e) => {
// Only a 404 means the integration is absent. Reporting a timeout the same
// way would teach the model that a real integration does not exist.
let error = if is_not_found(&e) {
// way would teach the model that a real integration does not exist. No slug holds
// a `/`, and the route refuses one as malformed rather than absent.
let error = if is_not_found(&e) || integration.contains('/') {
format!("No hub metadata for \"{integration}\".")
} else {
format!("Could not reach the hub for \"{integration}\"; it may still exist. Read its scripts instead, or try again.")
@@ -347,49 +377,183 @@ async fn get_hub_integration(ctx: &Ctx<'_>, args: &Value) -> ToolResult {
// npm
// ---------------------------------------------------------------------------
const NPM_SCORE_THRESHOLD: f64 = 1000.0;
/// The session caps a package's readme and types at 1% of its model's context window
/// each. The caller's model is unknown here, so this is 1% of the window the session
/// assumes for an unknown model (128000).
/// The registry's best match is kept, and the runner-up only when it scores this share of it:
/// the scores have no fixed scale (a descriptive query's right answer can score a few hundred,
/// a package's own name thousands), so a match is judged against the best one.
const NPM_RUNNER_UP_SHARE: f64 = 0.6;
/// A package's readme and types are each capped at 1% of the model's context window. An MCP
/// client's model is unknown, so this is 1% of the window assumed for an unknown model (128000).
const NPM_TEXT_LIMIT: usize = 1280;
fn take_chars(text: &str, limit: usize) -> String {
text.chars().take(limit).collect()
}
async fn fetch(url: &str) -> Result<reqwest::Response, reqwest::Error> {
HTTP_CLIENT.get(url).send().await
}
/// The package's declared entry declaration file. The session walks every `.d.ts` of the
/// package in the browser and keeps the same number of characters of the result.
async fn npm_package_types(name: &str, version: &str) -> Option<String> {
let base = format!("https://unpkg.com/{name}@{version}");
let manifest: Value = fetch(&format!("{base}/package.json"))
.await
.ok()?
.error_for_status()
.ok()?
.json()
.await
.ok()?;
let entry = ["types", "typings"]
.iter()
.find_map(|key| manifest.get(key)?.as_str())
.unwrap_or("index.d.ts")
.trim_start_matches("./");
let types = fetch(&format!("{base}/{entry}"))
.await
.ok()?
.error_for_status()
.ok()?
.text()
.await
.ok()?;
Some(format!("// /node_modules/{name}/{entry}\n{types}"))
/// Where a package's files come from: the public CDN, or the workspace's npm proxy, which
/// is the authoritative source once the instance names a registry of its own. Each falls
/// back to the other.
struct NpmSources<'a> {
ctx: &'a Ctx<'a>,
proxy_first: bool,
}
async fn npm_search(query: &str) -> Result<Vec<Value>, String> {
impl NpmSources<'_> {
async fn new<'a>(ctx: &'a Ctx<'a>) -> NpmSources<'a> {
let proxy_first = ctx
.get(&ctx.ws("npm_proxy/config"))
.await
.is_ok_and(|config| truthy(config.get("registry_configured")));
NpmSources { ctx, proxy_first }
}
async fn json(&self, cdn: String, proxy: String) -> Option<Value> {
let via_cdn = || async {
let response = fetch(&cdn).await.ok()?.error_for_status().ok()?;
response.json::<Value>().await.ok()
};
let via_proxy = || async { self.ctx.get(&self.ctx.ws(&proxy)).await.ok() };
if self.proxy_first {
match via_proxy().await {
Some(v) => Some(v),
None => via_cdn().await,
}
} else {
match via_cdn().await {
Some(v) => Some(v),
None => via_proxy().await,
}
}
}
async fn text(&self, cdn: String, proxy: String) -> Option<String> {
let via_cdn = || async {
let response = fetch(&cdn).await.ok()?.error_for_status().ok()?;
response.text().await.ok()
};
let via_proxy = || async {
self.ctx
.request("GET", &self.ctx.ws(&proxy), None)
.await
.ok()
};
if self.proxy_first {
match via_proxy().await {
Some(v) => Some(v),
None => via_cdn().await,
}
} else {
match via_cdn().await {
Some(v) => Some(v),
None => via_proxy().await,
}
}
}
/// `tag` as an exact version; one with two dots already is one.
async fn version(&self, module: &str, tag: &str) -> Option<String> {
if tag.split('.').count() >= 3 {
return Some(tag.to_string());
}
let resolved = self
.json(
format!("https://data.jsdelivr.com/v1/package/resolve/npm/{module}@{tag}"),
format!("npm_proxy/resolve/{}?tag={}", enc(module), enc(tag)),
)
.await?;
resolved.get("version")?.as_str().map(String::from)
}
/// The `.d.ts` files of `module` at `version`, in the order its file tree lists them.
async fn dts_files(&self, module: &str, version: &str) -> Option<Vec<String>> {
let tree = self
.json(
format!("https://data.jsdelivr.com/v1/package/npm/{module}@{version}/flat"),
format!("npm_proxy/filetree/{}/{}", enc(module), enc(version)),
)
.await?;
Some(
array(tree.get("files"))
.iter()
.filter_map(|f| f.get("name")?.as_str())
.filter(|name| name.ends_with(".d.ts"))
.map(String::from)
.collect(),
)
}
}
/// `@bla/foo` → `bla__foo`, the DefinitelyTyped name of a scoped package.
fn dt_name(name: &str) -> String {
match name.strip_prefix('@') {
Some(scoped) if scoped.contains('/') => scoped.replacen('/', "__", 1),
_ => name.to_string(),
}
}
/// Every declaration file of the package, or of its `@types` package when it ships none,
/// each headed by the path it would have under `node_modules`.
/// Files fetched at once, as the browser's type acquisition did.
const NPM_TYPES_CONCURRENCY: usize = 6;
/// The package's declarations joined in order, fetched only until they reach `limit`
/// characters: only that head is returned, and `@types/lodash` alone has hundreds of files.
async fn npm_package_types(
sources: &NpmSources<'_>,
name: &str,
version: &str,
limit: usize,
) -> String {
let mut module = name.to_string();
let mut prefix = format!("/node_modules/{name}");
let mut version = match sources.version(name, version).await {
Some(version) => version,
None => return String::new(),
};
let mut files = sources.dts_files(name, &version).await.unwrap_or_default();
if files.is_empty() {
module = format!("@types/{}", dt_name(name));
prefix = format!("/node_modules/@types/{}", dt_name(name));
let Some(dt_version) = sources.version(&module, "latest").await else {
return String::new();
};
version = dt_version;
files = sources
.dts_files(&module, &version)
.await
.unwrap_or_default();
}
let mut parts: Vec<String> = Vec::new();
for batch in files.chunks(NPM_TYPES_CONCURRENCY) {
let contents = join_all(batch.iter().map(|file| {
sources.text(
format!("https://cdn.jsdelivr.net/npm/{module}@{version}{file}"),
format!("npm_proxy/file/{}/{}{file}", enc(&module), enc(&version)),
)
}))
.await;
parts.extend(
batch
.iter()
.zip(contents)
.filter_map(|(file, code)| Some(format!("// {prefix}{file}\n{}", code?))),
);
if js_len(&parts.join("\n\n")) >= limit {
break;
}
}
parts.join("\n\n")
}
fn confident_npm_matches(objects: &[Value]) -> Vec<&Value> {
let score = |o: &Value| o.get("searchScore").and_then(Value::as_f64).unwrap_or(0.0);
let best = objects.iter().map(score).fold(0.0, f64::max);
objects
.iter()
.filter(|o| score(o) >= best * NPM_RUNNER_UP_SHARE)
.collect()
}
async fn npm_search(ctx: &Ctx<'_>, query: &str, limit: usize) -> Result<Vec<Value>, String> {
let data: Value = fetch(&format!(
"https://registry.npmjs.org/-/v1/search?text={}&size=2",
enc(query)
@@ -399,41 +563,36 @@ async fn npm_search(query: &str) -> Result<Vec<Value>, String> {
.json()
.await
.map_err(|e| e.to_string())?;
let packages = data
let objects = data
.get("objects")
.and_then(Value::as_array)
.ok_or("no `objects` in the registry answer")?
.iter()
.filter(|o| {
o.get("searchScore").and_then(Value::as_f64).unwrap_or(0.0) >= NPM_SCORE_THRESHOLD
})
.filter_map(|o| {
let package = o.get("package")?;
Some((
package.get("name")?.as_str()?,
package.get("version").and_then(Value::as_str),
))
});
.ok_or("no `objects` in the registry answer")?;
let packages = confident_npm_matches(objects).into_iter().filter_map(|o| {
let package = o.get("package")?;
Some((
package.get("name")?.as_str()?,
package.get("version").and_then(Value::as_str),
))
});
let sources = &NpmSources::new(ctx).await;
Ok(join_all(packages.map(|(name, version)| async move {
let documentation = match fetch(&format!("https://unpkg.com/{name}/readme.md")).await {
Ok(response) => response.text().await.unwrap_or_default(),
Err(_) => String::new(),
};
let types = npm_package_types(name, version.unwrap_or("latest"))
.await
.unwrap_or_default();
let types = npm_package_types(sources, name, version.unwrap_or("latest"), limit).await;
json!({
"package": name,
"documentation": take_chars(&documentation, NPM_TEXT_LIMIT),
"types": take_chars(&types, NPM_TEXT_LIMIT),
"documentation": js_head(&documentation, limit),
"types": js_head(&types, limit),
})
}))
.await)
}
async fn search_npm_packages(args: &Value) -> ToolResult {
async fn search_npm_packages(ctx: &Ctx<'_>, args: &Value) -> ToolResult {
let query = str_arg(args, "query")?;
Ok(match npm_search(query).await {
Ok(match npm_search(ctx, query, NPM_TEXT_LIMIT).await {
Ok(results) => Value::Array(results).to_string(),
Err(e) => {
tracing::warn!("npm package search failed: {e}");
@@ -454,7 +613,10 @@ fn docs_text(result: Result<Value, String>, action: &str) -> String {
.and_then(Value::as_str)
.unwrap_or_default()
.to_string(),
Err(e) => format!("Failed to {action}: {e}, pursuing with the user request..."),
Err(e) => format!(
"Failed to {action}: {}, pursuing with the user request...",
api_message(&e)
),
}
}
@@ -466,6 +628,14 @@ async fn search_docs(ctx: &Ctx<'_>, args: &Value) -> ToolResult {
);
}
let result = ctx.get(&format!("/docs/search?query={}", enc(query))).await;
if let Ok(response) = &result {
let mut facts = Map::new();
facts.insert(
"result_count".into(),
array(response.get("results")).len().into(),
);
ctx.effect(Effect::Facts(facts));
}
Ok(docs_text(result, "search documentation"))
}
@@ -661,7 +831,7 @@ async fn create_folder(ctx: &Ctx<'_>, args: &Value) -> ToolResult {
"success": true,
"message": format!("Created folder `f/{name}`. You can now write items to `f/{name}/<name>`.")
}),
Err(error) => json!({ "success": false, "error": error }),
Err(error) => json!({ "success": false, "error": api_message(&error) }),
}
.to_string(),
)
@@ -671,6 +841,25 @@ async fn create_folder(ctx: &Ctx<'_>, args: &Value) -> ToolResult {
mod tests {
use super::*;
#[test]
fn npm_matches_are_judged_against_the_best_one() {
let names = |scores: &[f64]| {
let objects: Vec<Value> = scores
.iter()
.enumerate()
.map(|(i, s)| json!({ "searchScore": s, "package": { "name": i.to_string() } }))
.collect();
confident_npm_matches(&objects)
.iter()
.map(|o| o["package"]["name"].as_str().unwrap().to_string())
.collect::<Vec<_>>()
};
// A descriptive query's right answer scores a few hundred; a weak runner-up is dropped.
assert_eq!(names(&[815.0, 454.0]), ["0"]);
assert_eq!(names(&[419.0, 384.0]), ["0", "1"]);
assert!(names(&[]).is_empty());
}
#[test]
fn suggestions_rank_by_matched_words() {
let integrations = vec![
@@ -690,6 +879,12 @@ mod tests {
["salesforce"]
);
assert_eq!(suggest_hub_integrations(&integrations, "s3 bucket"), ["s3"]);
// With no display name from the hub, the slug reads as the frontend labels it.
let unnamed = vec![json!({ "name": "gcloud", "display_name": null })];
assert_eq!(
suggest_hub_integrations(&unnamed, "deploy to google cloud"),
["gcloud"]
);
}
#[test]
@@ -1,30 +1,46 @@
//! The Windmill MCP server's toolset (`/api/mcp`, tokens with `mcp:all`): an AI session's toolset, for a client
//! outside the Windmill UI.
//! The AI session's GLOBAL toolset, run on the server for both of its callers: the
//! in-app chat (`POST /w/{workspace}/ai_session_tools/{name}`) and the Windmill MCP
//! server (`/api/mcp`, tokens with `mcp:all`).
//!
//! Everything a model reads here — tool names, descriptions, input schemas, the
//! instructions, the `get_instructions` texts — is generated from the session's own
//! definitions into `gen/` (`npm run gen:mcp-session` in `frontend/`). What a tool
//! *does* is written here, mirroring its session counterpart in
//! `frontend/src/lib/components/copilot/chat/global/core.ts`: a change to one side's
//! behaviour belongs on the other.
//! *does* is written here once; the chat keeps only what needs a browser (cards,
//! confirmations, the run form, the open editor).
//!
//! A tool acts through the API as the caller, the way the session's does, so every
//! permission check is the API's own. It calls the server's router in-process rather
//! than over a socket.
//! A tool acts through the API as the caller, so every permission check is the API's
//! own. It calls the server's router in-process rather than over a socket.
use std::collections::HashMap;
use std::sync::LazyLock;
use serde::Deserialize;
use serde_json::{json, Map, Value};
use windmill_mcp::server::{
BackendResult, CallToolResult, ContentBlock, ErrorData, MetaObject, SessionAccess, Tool,
ToolAnnotations,
};
use serde::{Deserialize, Serialize};
use serde_json::{Map, Value};
use windmill_common::auth::create_jwt_token;
use crate::db::ApiAuthed;
use super::utils::{mint_caller_jwt, send_as_caller};
pub(crate) mod chat;
/// What a caller may do with the toolset. An MCP token sets it from its scopes; the chat
/// is always `Full`, its own gating running before a tool is called.
#[derive(Clone, Copy, Debug, PartialEq, Eq)]
#[cfg_attr(not(feature = "mcp"), allow(dead_code))]
pub(crate) enum SessionAccess {
ReadOnly,
ReadAndRun,
Full,
}
/// Which caller a tool runs for, at the few spots where the two genuinely differ.
#[derive(Clone, Copy, Debug, PartialEq, Eq)]
pub(crate) enum Surface {
Mcp,
Chat,
}
/// The largest tool result the MCP runner passes on whole; it truncates beyond it.
pub(crate) const MAX_TOOL_RESULT_BYTES: usize = 87_500;
/// Each running server's fully layered router, by its `base_internal_url`. Keyed rather
/// than global because the backend tests run one server per test in a single process,
@@ -42,11 +58,13 @@ pub(crate) fn register_router(base_internal_url: &str, router: Option<axum::Rout
};
}
mod ai_providers;
mod apps;
mod data;
mod deploy;
mod flows;
mod guidance;
#[cfg(feature = "mcp")]
mod instructions;
mod items;
mod misc;
@@ -55,13 +73,15 @@ mod settings;
mod views;
mod workspace;
pub(super) use instructions::{
#[cfg(feature = "mcp")]
pub(crate) use instructions::{
generic_session_instructions, session_instructions, workspace_context,
};
#[derive(Deserialize)]
struct ToolDef {
name: String,
#[cfg_attr(not(feature = "mcp"), allow(dead_code))]
description: String,
parameters: Map<String, Value>,
read_only: bool,
@@ -108,6 +128,7 @@ const EDITOR_ONLY: &[&str] = &[
/// live datatable, run code, cancel a job, delete work, or replace a draft's content, which
/// discarding the draft does not restore. Stands in for the session's confirmation card. The
/// ChatGPT and Claude directories reject such a tool without this hint.
#[cfg(any(test, feature = "mcp"))]
const DESTRUCTIVE: &[&str] = &[
"write_script",
"edit_script",
@@ -144,15 +165,11 @@ const DESTRUCTIVE: &[&str] = &[
"delete_app_runnable",
];
/// The destructive tools a card may call, each from a button the user presses. Every
/// other destructive tool is hidden from cards: a card acts as the user with no one
/// confirming.
const CARD_ACTIONS: &[&str] = &["deploy_workspace_item", "cancel_job"];
/// Tools that reach a system outside Windmill on the workspace's behalf: running code can
/// reach any system the workspace can, and `get_db_schema` connects to a resource's
/// database, and `search_npm_packages` queries the npm registry, which Windmill does not
/// control. Reading Windmill's own hub and docs does not count.
#[cfg(any(test, feature = "mcp"))]
const OPEN_WORLD: &[&str] = &[
"search_npm_packages",
"run_script",
@@ -166,23 +183,6 @@ const OPEN_WORLD: &[&str] = &[
"get_db_schema",
];
/// A tool's display name, which Claude shows in its tool list and approval prompts.
fn tool_title(name: &str) -> String {
let words: Vec<String> = name
.split('_')
.map(|word| match word {
"db" | "sql" | "json" => word.to_uppercase(),
_ => word.to_string(),
})
.collect();
let title = words.join(" ");
let mut chars = title.chars();
chars
.next()
.map(|first| first.to_uppercase().chain(chars).collect())
.unwrap_or_default()
}
/// Tools the session counts as reads that still start a job. A read-only token runs
/// nothing on the tools server, so it runs nothing here either.
const RUNS_A_JOB: &[&str] = &["get_db_schema"];
@@ -202,6 +202,7 @@ fn allowed(def: &ToolDef, access: SessionAccess) -> bool {
/// Each group's tools and its dispatcher, tried in order.
macro_rules! groups {
($($group:ident),*) => {
#[cfg(any(test, feature = "mcp"))]
fn implemented() -> impl Iterator<Item = &'static str> {
std::iter::empty()$(.chain($group::TOOLS.iter().copied()))*
}
@@ -216,92 +217,53 @@ macro_rules! groups {
}
groups!(guidance, workspace, items, misc, flows, apps, runs, deploy, data, settings);
/// Claude clients keep an MCP result in context only up to a token budget well under the
/// authoring references `get_instructions` returns, unless the tool raises it. Above the
/// runner's own cap on a result, so that cap is the one that applies.
fn result_size_meta() -> MetaObject {
let mut meta = Map::new();
meta.insert("anthropic/maxResultSizeChars".to_string(), 100_000.into());
MetaObject(meta)
/// A tool as a client lists it.
#[cfg(feature = "mcp")]
pub(crate) struct ToolInfo {
pub name: &'static str,
pub description: &'static str,
pub parameters: &'static Map<String, Value>,
pub read_only: bool,
pub destructive: bool,
pub open_world: bool,
}
/// The MCP Apps view a client that renders them draws these tools' results in. It draws
/// what the tool put in its result's `_meta` (`Ctx::view`), and acts by calling tools.
/// Hosts cache a view by its URI: bump the version when a change breaks the `_meta` shape
/// the deployed card reads.
pub(super) const CARD_URI: &str = "ui://windmill/card-v1";
const CARD_HTML: &str = include_str!("card.html");
/// The `ui://` views the session's tools render in, as `(uri, html)`.
pub(super) fn session_views() -> &'static [(&'static str, &'static str)] {
&[(CARD_URI, CARD_HTML)]
}
pub(super) fn session_tools(access: SessionAccess) -> Vec<Tool> {
/// The tools implemented here that `access` may call.
#[cfg(feature = "mcp")]
pub(crate) fn session_tools(access: SessionAccess) -> impl Iterator<Item = ToolInfo> {
TOOL_DEFS
.iter()
.filter(|def| implemented().any(|name| name == def.name))
.filter(|def| allowed(def, access))
.filter(move |def| allowed(def, access))
.map(|def| {
let name = def.name.as_str();
let mut meta = result_size_meta();
let destructive = DESTRUCTIVE.contains(&name);
if destructive && !CARD_ACTIONS.contains(&name) {
meta.0
.insert("ui".into(), json!({ "visibility": ["model"] }));
ToolInfo {
name,
description: &def.description,
parameters: &def.parameters,
read_only: def.read_only && !RUNS_A_JOB.contains(&name),
destructive: DESTRUCTIVE.contains(&name),
open_world: OPEN_WORLD.contains(&name),
}
let title = tool_title(name);
Tool::new(
def.name.clone(),
def.description.clone(),
def.parameters.clone(),
)
.with_title(title.clone())
.with_annotations(
ToolAnnotations::with_title(title)
.read_only(def.read_only && !RUNS_A_JOB.contains(&name))
.destructive(destructive)
.open_world(OPEN_WORLD.contains(&name)),
)
.with_meta(meta)
})
.chain(
views::definitions()
.into_iter()
.filter(|(name, ..)| views::allowed(name, access))
.map(|(name, description, schema)| {
let mut meta = result_size_meta();
meta.0
.insert("ui".into(), json!({ "resourceUri": CARD_URI }));
let Value::Object(schema) = schema else {
unreachable!("a view tool's schema is an object")
};
let title = tool_title(name);
Tool::new(name, description, schema)
.with_title(title.clone())
.with_annotations(
ToolAnnotations::with_title(title)
.read_only(true)
.destructive(false)
.open_world(false),
)
.with_meta(meta)
}),
)
.collect()
}
/// What a tool hands back: the text the model reads, or the message of a failed call.
pub(super) type ToolResult = Result<String, String>;
/// The caller, and the API reached as them.
pub(super) struct Ctx<'a> {
pub(crate) struct Ctx<'a> {
pub auth: &'a ApiAuthed,
pub w_id: &'a str,
pub base_internal_url: &'a str,
pub access: SessionAccess,
/// The scheme and host the MCP client reached, when known.
pub origin: Option<&'a str>,
pub surface: Surface,
/// What the chat sends beside a call; `None` for MCP.
pub chat: Option<&'a chat::ChatContext>,
/// What the call changed that the chat has to apply: its editors, cards and drafts.
pub effects: std::sync::Mutex<Vec<chat::Effect>>,
/// Minted once per tool call: every JWT the API sees is cached by the auth layer, and a
/// tool can make dozens of calls, which would evict other users' cached sessions.
pub jwt: tokio::sync::OnceCell<String>,
@@ -310,6 +272,38 @@ pub(super) struct Ctx<'a> {
pub view: std::sync::Mutex<Option<(&'static str, Value)>>,
}
impl<'a> Ctx<'a> {
pub(crate) fn new(
auth: &'a ApiAuthed,
w_id: &'a str,
base_internal_url: &'a str,
access: SessionAccess,
) -> Self {
Ctx {
auth,
w_id,
base_internal_url,
access,
origin: None,
surface: Surface::Mcp,
chat: None,
effects: Default::default(),
jwt: Default::default(),
view: Default::default(),
}
}
/// Hand the chat something to apply. A no-op for MCP, which has no editor.
pub(super) fn effect(&self, effect: chat::Effect) {
if self.surface == Surface::Chat {
self.effects.lock().unwrap().push(effect);
}
}
}
/// The total timeout of the HTTP client the in-process router replaces.
const REQUEST_TIMEOUT: std::time::Duration = std::time::Duration::from_secs(30);
impl Ctx<'_> {
/// Set what the card draws of this call, replacing what an inner step set.
pub fn set_view(&self, key: &'static str, value: Value) {
@@ -318,6 +312,20 @@ impl Ctx<'_> {
/// Call the API as the caller. `path` follows `/api`; a non-2xx answer is the error.
pub async fn request(&self, method: &str, path: &str, body: Option<Value>) -> ToolResult {
self.request_within(method, path, body, REQUEST_TIMEOUT)
.await
}
/// `request` for a route that waits on jobs of its own, like the datatable migration
/// routes: each job is bounded server-side, so the default budget would cut a long one
/// off midway.
pub async fn request_within(
&self,
method: &str,
path: &str,
body: Option<Value>,
timeout: std::time::Duration,
) -> ToolResult {
let url = guarded_url(self.base_internal_url, self.w_id, method, path, self.access)?;
// No route scope: this server only admits `mcp:all`, several routes (hub, embeddings)
// map to none, and `guarded_url` is what keeps a call on its route. The tag fence
@@ -337,17 +345,18 @@ impl Ctx<'_> {
.get(self.base_internal_url)
.cloned();
let (status, text) = match router {
Some(router) => self.call_router(router, method, &url, body, scopes).await?,
None => {
let response = send_as_caller(method, &url, self.w_id, self.auth, body, scopes)
.await
.map_err(|e| e.message.to_string())?;
let status = response.status();
(status, response.text().await.map_err(|e| e.to_string())?)
Some(router) => {
self.call_router(router, method, &url, body, scopes, timeout)
.await?
}
None => self.call_http(method, &url, body, scopes, timeout).await?,
};
if status.is_success() {
Ok(text)
} else if text.trim().is_empty() {
// What a browser's request reports for an answer with no body.
let reason = status.canonical_reason().unwrap_or_default();
Err(format!("{}: {reason}", status.as_u16()))
} else {
Err(format!("{}: {}", status.as_u16(), text))
}
@@ -362,6 +371,7 @@ impl Ctx<'_> {
url: &str,
body: Option<Value>,
scopes: Option<Vec<String>>,
timeout: std::time::Duration,
) -> Result<(http::StatusCode, String), String> {
use tower::ServiceExt;
let parsed = reqwest::Url::parse(url).map_err(|e| e.to_string())?;
@@ -376,7 +386,7 @@ impl Ctx<'_> {
.jwt
.get_or_try_init(|| mint_caller_jwt(self.w_id, self.auth, scopes))
.await
.map_err(|e| e.message.to_string())?;
.map_err(|e| e.to_string())?;
let mut request = http::Request::builder()
.method(method)
.uri(uri)
@@ -389,8 +399,7 @@ impl Ctx<'_> {
None => axum::body::Body::empty(),
};
let request = request.body(body).map_err(|e| e.to_string())?;
// The total timeout of the HTTP client this replaces.
tokio::time::timeout(std::time::Duration::from_secs(30), async {
tokio::time::timeout(timeout, async {
let response = router.oneshot(request).await.map_err(|e| e.to_string())?;
let status = response.status();
let bytes = axum::body::to_bytes(response.into_body(), usize::MAX)
@@ -402,6 +411,36 @@ impl Ctx<'_> {
.map_err(|_| format!("Failed to execute request: {method} {url} timed out"))?
}
/// `request` over HTTP, for a server whose router is not registered.
async fn call_http(
&self,
method: &str,
url: &str,
body: Option<Value>,
scopes: Option<Vec<String>>,
timeout: std::time::Duration,
) -> Result<(http::StatusCode, String), String> {
let method = http::Method::from_bytes(method.as_bytes()).map_err(|e| e.to_string())?;
let token = self
.jwt
.get_or_try_init(|| mint_caller_jwt(self.w_id, self.auth, scopes))
.await
.map_err(|e| e.to_string())?;
let mut request = crate::HTTP_CLIENT
.request(method, url)
.header("Authorization", format!("Bearer {token}"))
.timeout(timeout);
if let Some(body) = body {
request = request.json(&body);
}
let response = request
.send()
.await
.map_err(|e| format!("Failed to execute request: {e}"))?;
let status = response.status();
Ok((status, response.text().await.map_err(|e| e.to_string())?))
}
pub async fn get(&self, path: &str) -> Result<Value, String> {
let text = self.request("GET", path, None).await?;
serde_json::from_str(&text).map_err(|e| format!("unexpected response from {path}: {e}"))
@@ -424,6 +463,12 @@ impl Ctx<'_> {
format!("/w/{}/{}", self.w_id, rest)
}
/// `page_url` for an MCP client only: the chat shows what a tool made or ran in its own
/// cards and panel instead.
pub fn link(&self, page: &str) -> Option<String> {
(self.surface == Surface::Mcp).then(|| self.page_url(page))
}
/// A Windmill page in this workspace, as a link the user can open: an external client
/// has no preview panel, so a link is how it shows what a tool made or ran.
pub fn page_url(&self, page: &str) -> String {
@@ -637,57 +682,92 @@ pub(super) fn matches_glob(subject: &str, glob: &str) -> bool {
regex::Regex::new(&pattern).is_ok_and(|re| re.is_match(subject))
}
/// A generated session tool, refused when the token's access does not cover it.
async fn call_generated_tool(ctx: &Ctx<'_>, name: &str, args: &Value) -> BackendResult<ToolResult> {
/// A JWT that acts as `api_authed`, carrying `scopes`; `None` is the caller's full rights.
pub(crate) async fn mint_caller_jwt(
workspace_id: &str,
api_authed: &ApiAuthed,
scopes: Option<Vec<String>>,
) -> windmill_common::error::Result<String> {
// Carry the caller's job provenance into the JWT: a job's WM_TOKEN is capped at
// workspace admin (GHSA-hfh4-cx4h-3fcr), and dropping `job_id` here would re-mint an
// uncapped token that satisfies require_super_admin / require_devops_role.
// A browser caller stays a session: runs, trigger history and schedules attribute a
// session's actions to the UI, and anything else to the API (`webhook`).
create_jwt_token(
windmill_common::db::Authed::from(api_authed.clone()),
workspace_id,
3600,
api_authed.job_id,
api_authed.is_session_token.then(|| "session".to_string()),
None,
scopes,
)
.await
}
/// Why a call was refused before the tool ran.
pub(crate) enum CallRefused {
UnknownTool(String),
NotAllowed(String),
NotAvailable(String),
}
/// What the model reads, and whether the call failed.
#[derive(Serialize)]
pub(crate) struct ToolOutcome {
pub text: String,
pub is_error: bool,
}
/// The card tools only MCP Apps clients draw; the chat draws its own cards.
#[cfg(feature = "mcp")]
pub(crate) use views::{allowed as view_allowed, definitions as view_definitions};
pub(crate) async fn call_session_tool(
ctx: &Ctx<'_>,
name: &str,
args: Value,
) -> Result<ToolOutcome, CallRefused> {
if ctx.surface == Surface::Mcp && views::TOOLS.contains(&name) {
return Ok(outcome(views::call(ctx, name, &args).await));
}
let result = call_generated_tool(ctx, name, &args).await?;
Ok(outcome(result))
}
/// A generated session tool, refused when the caller's access does not cover it.
async fn call_generated_tool(
ctx: &Ctx<'_>,
name: &str,
args: &Value,
) -> Result<ToolResult, CallRefused> {
let Some(def) = TOOL_DEFS.iter().find(|def| def.name == name) else {
return Err(ErrorData::invalid_params(
format!("Unknown tool '{name}'"),
None,
));
return Err(CallRefused::UnknownTool(format!("Unknown tool '{name}'")));
};
if !allowed(def, ctx.access) {
let token = match ctx.access {
SessionAccess::ReadAndRun => "can only read and run deployed scripts and flows",
_ => "is read-only",
};
return Err(ErrorData::invalid_request(
format!("Tool '{name}' changes or runs something, and this token {token}"),
None,
));
return Err(CallRefused::NotAllowed(format!(
"Tool '{name}' changes or runs something, and this token {token}"
)));
}
let Some(result) = dispatch(ctx, name, args).await else {
return Err(ErrorData::invalid_params(
format!("Tool '{name}' is not available on this server"),
None,
));
};
Ok(result)
dispatch(ctx, name, args).await.ok_or_else(|| {
CallRefused::NotAvailable(format!("Tool '{name}' is not available on this server"))
})
}
pub(super) async fn call_session_tool(
ctx: &Ctx<'_>,
name: &str,
args: Value,
) -> BackendResult<CallToolResult> {
let result = if views::TOOLS.contains(&name) {
views::call(ctx, name, &args).await
} else {
call_generated_tool(ctx, name, &args).await?
};
windmill_common::feature_usage::log_feature_usage("mcp_workspace", "tool_call", name);
let mut result = match result {
Ok(text) => CallToolResult::success(vec![ContentBlock::text(text)]),
fn outcome(result: ToolResult) -> ToolOutcome {
match result {
Ok(text) => ToolOutcome { text, is_error: false },
// Without the status code `Ctx` prefixes an API failure with, as the session reads it.
Err(message) => {
let body = workspace::api_message(&message);
let text = if body.is_empty() { &message } else { body };
CallToolResult::error(vec![ContentBlock::text(text.to_string())])
ToolOutcome { text: text.to_string(), is_error: true }
}
};
if let Some((key, view)) = ctx.view.lock().unwrap().take() {
result.meta = Some(MetaObject(Map::from_iter([(key.to_string(), view)])));
}
Ok(result)
}
#[cfg(test)]
@@ -728,38 +808,6 @@ mod tests {
assert!(url("POST", "/w/ws/variables/create", ReadOnly).is_err());
}
#[test]
fn no_card_can_call_a_destructive_tool_but_deploy_and_cancel() {
for tool in session_tools(SessionAccess::Full) {
let name = tool.name.as_ref();
if DESTRUCTIVE.contains(&name) {
let ui = tool.meta.as_ref().and_then(|m| m.0.get("ui"));
let model_only = ui == Some(&json!({ "visibility": ["model"] }));
assert_eq!(
model_only,
!matches!(name, "deploy_workspace_item" | "cancel_job"),
"{name}"
);
}
}
}
/// The ChatGPT directory wants every hint explicit, and Claude's a title on every tool.
#[test]
fn every_session_tool_has_a_title_and_explicit_hints() {
for tool in session_tools(SessionAccess::Full) {
let a = tool.annotations.as_ref().expect("annotations");
assert!(tool.title.is_some() && a.title.is_some(), "{}", tool.name);
assert!(
a.read_only_hint.is_some()
&& a.destructive_hint.is_some()
&& a.open_world_hint.is_some(),
"{}",
tool.name
);
}
}
#[test]
fn every_session_tool_is_implemented_or_editor_only() {
let implemented: Vec<&str> = implemented().collect();
@@ -0,0 +1,120 @@
{
"//": "How a run's arguments are settled, which the chat does in the browser ($lib/components/job_args.ts, secretArgUtils.ts) and MCP on the server (runs.rs). Both test suites read this file, so a change on one side only fails. Each expectation applies its step to `args` alone. A minted variable is named <minted>.",
"cases": [
{
"name": "undeclared arguments are dropped and named",
"schema": { "properties": { "a": { "type": "string" } } },
"args": { "a": "x", "b": 1, "constructor": 2 },
"declared": { "args": { "a": "x" }, "undeclared": ["b", "constructor"] }
},
{
"name": "a schema without properties declares nothing",
"schema": {},
"args": { "a": "x" },
"declared": { "args": {}, "undeclared": ["a"] }
},
{
"name": "a disabled field takes its default, reported only when a different value was proposed",
"schema": {
"properties": {
"locked": { "type": "string", "disabled": true, "default": "fixed" },
"same": { "type": "object", "disabled": true, "default": { "k": [1, 2] } },
"absent": { "type": "number", "disabled": true, "default": 3 },
"no_default": { "type": "string", "disabled": true },
"free": { "type": "string", "default": "d" }
}
},
"args": { "locked": "mine", "same": { "k": [1, 2] }, "no_default": "kept", "free": "mine" },
"disabled_defaults": {
"args": {
"locked": "fixed",
"same": { "k": [1, 2] },
"absent": 3,
"no_default": "kept",
"free": "mine"
},
"reset": ["locked"]
}
},
{
"name": "secrets are minted at every depth, and references and empty fields are left alone",
"schema": {
"properties": {
"token": { "type": "string", "password": true },
"creds": { "type": "object", "password": true, "properties": { "user": { "type": "string" } } },
"nested": { "type": "object", "properties": { "inner": { "type": "string", "password": true } } },
"rows": {
"type": "array",
"items": { "type": "object", "properties": { "key": { "type": "string", "password": true } } }
},
"variant": {
"type": "object",
"oneOf": [
{ "properties": { "label": { "type": "string" } } },
{ "properties": { "pin": { "type": "string", "password": true } } }
]
},
"named": { "type": "string", "password": true },
"named_object": { "type": "object", "password": true },
"blank": { "type": "string", "password": true },
"nothing": { "type": "string", "password": true },
"plain": { "type": "string" }
}
},
"args": {
"token": "hunter2",
"creds": { "user": "ada" },
"nested": { "inner": "deep" },
"rows": [{ "key": "k1" }, { "key": "$var:f/team/k2" }],
"variant": { "pin": "1234" },
"named": "$var:f/team/api_token",
"named_object": "$var:f/team/creds",
"blank": "",
"nothing": null,
"plain": "kept"
},
"secrets": {
"args": {
"token": "$var:<minted>",
"creds": "$jsonvar:<minted>",
"nested": { "inner": "$var:<minted>" },
"rows": [{ "key": "$var:<minted>" }, { "key": "$var:f/team/k2" }],
"variant": { "pin": "$var:<minted>" },
"named": "$var:f/team/api_token",
"named_object": "$jsonvar:f/team/creds",
"blank": "",
"nothing": null,
"plain": "kept"
},
"stored": ["1234", "deep", "hunter2", "k1", "{\"user\":\"ada\"}"]
},
"redacted": {
"token": "<hidden>",
"creds": "<hidden>",
"nested": { "inner": "<hidden>" },
"rows": [{ "key": "<hidden>" }, { "key": "$var:f/team/k2" }],
"variant": { "pin": "<hidden>" },
"named": "$var:f/team/api_token",
"named_object": "$var:f/team/creds",
"blank": "",
"plain": "kept"
}
},
{
"name": "file bytes are replaced by their size",
"schema": {
"properties": {
"doc": { "type": "string", "contentEncoding": "base64" },
"docs": { "type": "array", "items": { "type": "string", "contentEncoding": "base64" } },
"name": { "type": "string" }
}
},
"args": {
"doc": "QUJDREVGR0hJSktMTU5PUFFSU1RVVldYWVo=",
"docs": ["QUJD", "QUJDREVGR0hJSktMTU5PUFFSU1RVVldYWVpBQkNERUZHSElKS0xNTk9QUVJTVFVWV1hZWg=="],
"name": "report.pdf"
},
"redacted": { "doc": "<file: 1 KB>", "docs": ["<file: 1 KB>", "<file: 1 KB>"], "name": "report.pdf" }
}
]
}
@@ -2,8 +2,9 @@
use serde_json::{json, Map, Value};
use super::chat::Effect;
use super::workspace::api_message;
use super::{array, js_head, js_len, js_str, str_arg, Ctx, ToolResult};
use super::{array, js_head, js_len, js_str, str_arg, Ctx, Surface, ToolResult};
pub(super) const TOOLS: &[&str] = &["get_workspace_settings", "set_workspace_settings"];
@@ -65,37 +66,73 @@ fn parse_settings_yaml(text: &str) -> Result<Value, String> {
}
}
/// The dry run stands in for the session's confirmation card: it reports a no-op or an
/// invalid document before anything is written.
/// The dry run reports a no-op or an invalid document before anything is written: for MCP
/// it stands in for the session's confirmation card, and the chat runs it alone, as a
/// check-only call, before showing that card.
async fn set_workspace_settings(ctx: &Ctx<'_>, args: &Value) -> ToolResult {
let route = ctx.ws("workspaces/settings_document");
let refuse = |label: String, text: String| {
ctx.facts(json!({ "refused": label }));
Ok(text)
};
let document = match parse_settings_yaml(str_arg(args, "yaml")?) {
Ok(document) => document,
Err(e) => return Ok(format!("Nothing was changed: {e}")),
Err(e) if ctx.surface == Surface::Chat && !ctx.check_only() => return Err(e),
Err(e) => {
return refuse(
format!("Invalid settings: {e}"),
format!("Nothing was changed: {e}"),
)
}
};
let dry_run = json!({ "document": document, "replace": false, "dry_run": true });
let dry_run = match ctx.post(&route, dry_run).await {
Ok(result) => result,
Err(e) => return Ok(format!("Nothing was changed: {}", api_message(&e))),
};
if array(dry_run.get("changes")).is_empty() {
let warnings: Vec<String> = array(dry_run.get("warnings")).iter().map(js_str).collect();
let warnings = if warnings.is_empty() {
String::new()
} else {
format!(" Warnings: {}", warnings.join(" "))
// The chat's own check ran before its card, and the user confirmed what it showed.
if ctx.surface == Surface::Mcp || ctx.check_only() {
let dry_run = json!({ "document": document, "replace": false, "dry_run": true });
let dry_run = match ctx.post(&route, dry_run).await {
Ok(result) => result,
Err(e) => {
let e = api_message(&e);
return refuse(
format!("Invalid settings: {e}"),
format!("Nothing was changed: {e}"),
);
}
};
return Ok(format!(
"The settings already hold these values: nothing to change.{warnings}"
));
if array(dry_run.get("changes")).is_empty() {
let warnings: Vec<String> = array(dry_run.get("warnings")).iter().map(js_str).collect();
let warnings = if warnings.is_empty() {
String::new()
} else {
format!(" Warnings: {}", warnings.join(" "))
};
return refuse(
"No settings change".to_string(),
format!("The settings already hold these values: nothing to change.{warnings}"),
);
}
if ctx.check_only() {
ctx.facts(json!({ "changes": dry_run["changes"] }));
return Ok(String::new());
}
}
let applied = match ctx
.post(&route, json!({ "document": document, "replace": false }))
.await
{
Ok(result) => result,
Err(e) => return Ok(json!({ "success": false, "error": api_message(&e) }).to_string()),
Err(e) => {
let error = api_message(&e);
let mut failed = Map::new();
failed.insert(
"content".into(),
json!("Failed to change workspace settings"),
);
failed.insert("error".into(), json!(error));
ctx.effect(Effect::Status(failed));
return Ok(json!({ "success": false, "error": error }).to_string());
}
};
ctx.facts(json!({ "changes": applied["changes"] }));
let keys: Vec<String> = array(applied.get("changes"))
.iter()
.map(|c| js_str(c.get("key").unwrap_or(&Value::Null)))
@@ -105,7 +142,11 @@ async fn set_workspace_settings(ctx: &Ctx<'_>, args: &Value) -> ToolResult {
out.insert(
"message".into(),
json!(format!(
"These settings are now applied: {}.",
"{} now applied: {}.",
match ctx.surface {
Surface::Chat => "The user approved and these settings are",
Surface::Mcp => "These settings are",
},
keys.join(", ")
)),
);
@@ -3,7 +3,9 @@
//! the other tools, which draw nothing, and calls one of these when it means the user to
//! see a result: a host draws a card for every call of a tool that has one.
use serde_json::{json, Value};
#[cfg(feature = "mcp")]
use serde_json::json;
use serde_json::Value;
use super::{deploy, runs, workspace, Ctx, SessionAccess, ToolResult};
@@ -11,7 +13,7 @@ pub(super) const TOOLS: &[&str] = &["show_run", "show_runs", "show_deploy_review
/// The review card's one action deploys, which only a full token may do: such a token is
/// neither offered the tool nor answered when it calls it by name.
pub(super) fn allowed(name: &str, access: SessionAccess) -> bool {
pub(crate) fn allowed(name: &str, access: SessionAccess) -> bool {
name != "show_deploy_review" || access == SessionAccess::Full
}
@@ -32,7 +34,8 @@ pub(super) async fn call(ctx: &Ctx<'_>, name: &str, args: &Value) -> ToolResult
/// Each tool as `(name, description, input schema)`. All only read: the review card's
/// Deploy button calls `deploy_workspace_item`, which the token's own access gates.
pub(super) fn definitions() -> [(&'static str, &'static str, Value); 4] {
#[cfg(feature = "mcp")]
pub(crate) fn definitions() -> [(&'static str, &'static str, Value); 4] {
[
(
"show_run",
@@ -4,9 +4,10 @@ use std::collections::HashMap;
use serde_json::{json, Map, Value};
use super::chat::Effect;
use super::{
array, bool_arg, enc, is_not_found, js_len, js_str, non_null, opt_str_arg, pretty, put,
str_arg, truthy, Ctx, ToolResult,
str_arg, truthy, Ctx, Surface, ToolResult,
};
pub(super) const TOOLS: &[&str] = &[
@@ -294,11 +295,17 @@ pub(super) struct Draft {
pub created_at: Option<String>,
}
/// The caller's own draft named `path`. For the chat, a cell the browser holds is newer
/// than the server's row and wins, the way the editor reads it.
pub(super) async fn get_draft(
ctx: &Ctx<'_>,
kind: &str,
path: &str,
) -> Result<Option<Draft>, String> {
let path = ctx.storage_path(kind, path);
if let Some(value) = ctx.chat.and_then(|chat| chat.cell(kind, path)) {
return Ok(Some(Draft { value: value.clone(), created_at: None }));
}
let row = ctx
.get(&ctx.ws(&format!("drafts/get_own/{kind}/{}", path_enc(path))))
.await?;
@@ -312,6 +319,9 @@ pub(super) async fn get_draft(
Ok(Some(Draft { value, created_at }))
}
/// Save the caller's draft, or delete it with `None`. False on a conflict with a newer
/// row. The chat saves through its editor's own draft store instead, after the call: it
/// gets the write as an effect, and reports a conflict or failure itself.
pub(super) async fn save_draft(
ctx: &Ctx<'_>,
kind: &str,
@@ -320,6 +330,25 @@ pub(super) async fn save_draft(
last_sync: Option<&str>,
force: bool,
) -> Result<bool, String> {
if ctx.surface == Surface::Chat {
let storage_path = ctx.storage_path(kind, path).to_string();
ctx.effect(match value {
Some(value) => Effect::DraftWrite {
kind: kind.to_string(),
storage_path,
value: value.clone(),
last_sync: last_sync.map(String::from),
force,
},
None => Effect::DraftDelete {
kind: kind.to_string(),
storage_path,
last_sync: last_sync.map(String::from),
keep_live_editor: false,
},
});
return Ok(true);
}
let mut body = Map::new();
body.insert("value".into(), value.cloned().unwrap_or(Value::Null));
if !force {
@@ -335,12 +364,24 @@ pub(super) async fn save_draft(
Ok(response.get("status").and_then(Value::as_str) != Some("conflict"))
}
/// Delete the caller's draft. `keep_live_editor` leaves an editor open on it with its
/// copy, detached from the row, as a deploy does.
pub(super) async fn delete_draft(
ctx: &Ctx<'_>,
kind: &str,
path: &str,
last_sync: Option<&str>,
keep_live_editor: bool,
) -> Result<(), String> {
if ctx.surface == Surface::Chat {
ctx.effect(Effect::DraftDelete {
kind: kind.to_string(),
storage_path: ctx.storage_path(kind, path).to_string(),
last_sync: last_sync.map(String::from),
keep_live_editor,
});
return Ok(());
}
match save_draft(ctx, kind, path, None, last_sync, false).await {
Ok(true) => Ok(()),
Ok(false) => Err(format!(
@@ -364,6 +405,7 @@ pub(super) async fn list_drafts(ctx: &Ctx<'_>) -> Result<Vec<Value>, String> {
let Some((item_type, trigger_kind)) = item_type_of_kind(kind) else {
continue;
};
let (path, live) = ctx.display_path(kind, path);
let mut item = Map::new();
item.insert("type".into(), json!(item_type));
item.insert("path".into(), json!(path));
@@ -380,8 +422,31 @@ pub(super) async fn list_drafts(ctx: &Ctx<'_>) -> Result<Vec<Value>, String> {
if kind == "raw_app" {
item.insert("rawApp".into(), json!(true));
}
if live {
item.insert("isLiveDraft".into(), json!(true));
}
drafts.insert(item_key(item_type, path, trigger_kind), Value::Object(item));
}
// The browser's cells carry edits the rows may not have yet, and win.
for cell in ctx
.chat
.map(|chat| chat.cells.as_slice())
.unwrap_or_default()
{
let Some(Value::Object(mut item)) =
shown_draft_item(ctx, &cell.kind, &cell.storage_path, &cell.value)
else {
continue;
};
item.shift_remove("value");
item.shift_remove("schema");
let text = |key: &str| item.get(key).and_then(Value::as_str).map(String::from);
let (Some(item_type), Some(path)) = (text("type"), text("path")) else {
continue;
};
let key = item_key(&item_type, &path, text("triggerKind").as_deref());
drafts.insert(key, Value::Object(item));
}
Ok(drafts.into_iter().map(|(_, item)| item).collect())
}
@@ -540,7 +605,24 @@ pub(super) async fn get_draft_item(
let Some(draft) = get_draft(ctx, &kind, path).await? else {
return Ok(None);
};
Ok(draft_item(&kind, path, &draft.value).map(|item| (item, draft)))
Ok(shown_draft_item(ctx, &kind, path, &draft.value).map(|item| (item, draft)))
}
/// `draft_item` under the path the user sees: an open editor's own name for its draft,
/// flagged as live.
pub(super) fn shown_draft_item(
ctx: &Ctx<'_>,
kind: &str,
path: &str,
draft: &Value,
) -> Option<Value> {
let (shown, live) = ctx.display_path(kind, ctx.storage_path(kind, path));
let mut item = draft_item(kind, shown, draft)?;
if live {
item.as_object_mut()?
.insert("isLiveDraft".into(), json!(true));
}
Some(item)
}
fn stash_inline_script(
@@ -766,6 +848,7 @@ pub(super) fn summarize_app_value(value: &Value) -> Value {
_ if !by_path => "inline",
Some("flow") => "flow",
Some("hubscript") => "hubscript",
Some("agent") => "agent",
_ => "script",
};
out.insert("type".into(), json!(run_type));
@@ -869,7 +952,7 @@ pub(super) fn serialize_for_read(item: &Value, execution_mode: Option<Value>) ->
let mut value = flow.get("value").cloned().unwrap_or_else(|| json!({}));
if let (Some(groups), Some(value)) = (flow.get("groups"), value.as_object_mut()) {
if groups.is_null() {
value.remove("groups");
value.shift_remove("groups");
} else {
value.insert("groups".into(), groups.clone());
}
@@ -1013,7 +1096,7 @@ pub(super) async fn persist_draft(
) -> Result<Persist, String> {
let kind = draft_kind(item_type, trigger_kind)
.ok_or_else(|| format!("Unsupported draft type \"{item_type}\"."))?;
let item = draft_item(&kind, path, value)
let item = shown_draft_item(ctx, &kind, path, value)
.ok_or_else(|| format!("Could not synthesize {item_type} draft \"{path}\"."))?;
let failure = match save_draft(ctx, &kind, path, Some(value), last_sync, force).await {
Ok(true) => return Ok(Persist::Saved(item)),
@@ -1057,8 +1140,9 @@ pub(super) fn finish_draft_write(ctx: &Ctx<'_>, item: &Value, existed: bool) ->
json!(format!("{verb} {item_type} \"{path}\" as a per-user draft (saved server-side, visible only to this user — not a deployed workspace item). It was not deployed.")),
);
let value = item.get("value").unwrap_or(&Value::Null);
// The chat opens the draft in its own panel; an MCP client only has the link.
if let Some(page) = draft_page(item_type, path, value) {
out.insert("url".into(), json!(ctx.page_url(&page)));
put(&mut out, "url", ctx.link(&page).map(Value::from));
}
// A flow is not echoed back: the model just sent it. A variable comes back redacted.
match item_type {
@@ -1246,7 +1330,13 @@ pub(super) fn unified_patch(
fn deployed_item(item_type: &str, trigger_kind: Option<&str>, row: &Value) -> Value {
let mut item = Map::new();
item.insert("type".into(), json!(item_type));
put(&mut item, "triggerKind", trigger_kind.map(Value::from));
put(
&mut item,
"triggerKind",
trigger_kind
.filter(|_| item_type == "trigger")
.map(Value::from),
);
put(&mut item, "path", row.get("path").cloned());
let mut is_draft = false;
match item_type {
@@ -1313,8 +1403,29 @@ async fn list_workspace_items(ctx: &Ctx<'_>, args: &Value) -> ToolResult {
let limit = args.get("limit").and_then(Value::as_u64).unwrap_or(50);
let page = args.get("page").and_then(Value::as_u64);
let prefix = opt_str_arg(args, "path_prefix");
// Scoped, the session's folders are the default, one listing each; an explicit
// path_prefix, "" included, overrides them.
let scope_roots: &[String] = match prefix {
None => ctx
.chat
.and_then(|chat| chat.scope_roots.as_deref())
.unwrap_or_default(),
Some(_) => &[],
};
let prefixes: Vec<Option<String>> = if scope_roots.is_empty() {
vec![prefix.filter(|p| !p.is_empty()).map(str::to_string)]
} else {
scope_roots.iter().map(|root| Some(format!("{root}/"))).collect()
};
let per_page = limit.min(MAX_LIST_LIMIT);
let page_number = page.unwrap_or(1).max(1);
// Several folders read as one listing, folder after folder, so `limit` stays one cap per
// type (per trigger kind for triggers): page N is rows (N-1)*limit..N*limit of each
// concatenation, which takes every folder's first N*limit rows.
let window = (prefixes.len() > 1)
.then(|| (page_number - 1) * per_page..page_number * per_page);
let mut query = format!("per_page={}", limit.min(MAX_LIST_LIMIT));
let mut query = format!("per_page={per_page}");
if let Some(page) = page {
query.push_str(&format!("&page={page}"));
}
@@ -1338,40 +1449,54 @@ async fn list_workspace_items(ctx: &Ctx<'_>, args: &Value) -> ToolResult {
}
}
if let Some(prefix) = prefix {
query.push_str(&format!("&path_start={}", enc(prefix)));
}
let mut by_key = Map::new();
for item_type in ITEM_TYPES.iter().filter(|t| types.contains(t)) {
let kinds: Vec<Option<&str>> = if *item_type == "trigger" {
TRIGGER_KINDS.iter().map(|kind| Some(*kind)).collect()
} else {
vec![None]
};
let extra = if matches!(*item_type, "script" | "flow") {
"&include_draft_only=true&without_description=true"
} else {
""
};
for trigger_kind in kinds {
let route = ctx.ws(&format!(
"{}/list?{query}{extra}",
service(item_type, trigger_kind)?
));
let rows = match ctx.get(&route).await {
Ok(rows) => rows,
// A trigger kind this build does not serve answers 404: skip that kind
// alone. Any other failure is real.
Err(e) if trigger_kind.is_some() && is_not_found(&e) => continue,
Err(e) => return Err(e),
// Each type's index into its concatenation across folders, per trigger kind for triggers.
let mut index_by_kind: HashMap<String, u64> = HashMap::new();
for prefix in &prefixes {
for item_type in ITEM_TYPES.iter().filter(|t| types.contains(t)) {
let kinds: Vec<Option<&str>> = if *item_type == "trigger" {
TRIGGER_KINDS.iter().map(|kind| Some(*kind)).collect()
} else {
vec![None]
};
for row in array(Some(&rows)) {
let path = row.get("path").and_then(Value::as_str).unwrap_or_default();
by_key.insert(
item_key(item_type, path, trigger_kind),
deployed_item(item_type, trigger_kind, row),
);
let extra = if matches!(*item_type, "script" | "flow") {
"&include_draft_only=true&without_description=true"
} else {
""
};
for trigger_kind in kinds {
let mut query = match &window {
Some(window) => format!("per_page={}", window.end),
None => query.clone(),
};
if let Some(prefix) = prefix {
query.push_str(&format!("&path_start={}", enc(prefix)));
}
let route = ctx.ws(&format!(
"{}/list?{query}{extra}",
service(item_type, trigger_kind)?
));
let rows = match ctx.get(&route).await {
Ok(rows) => rows,
// A trigger kind this build does not serve answers 404: skip that kind
// alone. Any other failure is real.
Err(e) if trigger_kind.is_some() && is_not_found(&e) => continue,
Err(e) => return Err(e),
};
let index = index_by_kind
.entry(format!("{item_type}:{}", trigger_kind.unwrap_or_default()))
.or_default();
for row in array(Some(&rows)) {
*index += 1;
if window.as_ref().is_some_and(|w| !w.contains(&(*index - 1))) {
continue;
}
let path = row.get("path").and_then(Value::as_str).unwrap_or_default();
by_key.insert(
item_key(item_type, path, trigger_kind),
deployed_item(item_type, trigger_kind, row),
);
}
}
}
}
@@ -1380,6 +1505,7 @@ async fn list_workspace_items(ctx: &Ctx<'_>, args: &Value) -> ToolResult {
// pages never repeat a page-1 item as its draft twin.
if page.unwrap_or(1) == 1 {
let mut count_by_type: HashMap<String, u64> = HashMap::new();
let narrowing: Vec<&str> = prefixes.iter().flatten().map(String::as_str).collect();
for draft in list_drafts(ctx).await? {
let text = |key: &str| draft.get(key).and_then(Value::as_str);
let (item_type, path) = (
@@ -1391,11 +1517,13 @@ async fn list_workspace_items(ctx: &Ctx<'_>, args: &Value) -> ToolResult {
}
// A draft's staged name is often not where it is stored, and this pass is
// the only source of draft-only rows under a prefix: either name may match.
if let Some(prefix) = prefix {
let staged = text("draftPath").is_some_and(|p| p.starts_with(prefix));
if !path.starts_with(prefix) && !staged {
continue;
}
let staged = text("draftPath");
if !narrowing.is_empty()
&& !narrowing.iter().any(|prefix| {
path.starts_with(prefix) || staged.is_some_and(|p| p.starts_with(prefix))
})
{
continue;
}
let count = count_by_type.entry(item_type.to_string()).or_default();
if *count >= limit {
@@ -1425,7 +1553,15 @@ async fn list_workspace_items(ctx: &Ctx<'_>, args: &Value) -> ToolResult {
})
})
.collect();
Ok(pretty(&Value::Array(results)))
let listed = pretty(&Value::Array(results));
if scope_roots.is_empty() {
return Ok(listed);
}
let folders: Vec<&str> = prefixes.iter().flatten().map(String::as_str).collect();
Ok(format!(
"Items under {}, where this session works. Pass path_prefix \"\" for the whole workspace.\n{listed}",
folders.join(", ")
))
}
/// The card of one workspace item: what `read_workspace_item` returns, under a one-line
@@ -1663,8 +1799,28 @@ async fn write_script_draft(
content: &str,
force: bool,
) -> ToolResult {
// A session builds pipeline nodes into the folder's pipeline draft; a script draft
// written here never reaches the pipeline graph, and saving the pipeline drops it.
if ctx.in_session() && windmill_common::assets::parse_pipeline_annotations(content).in_pipeline
{
let folder = path
.strip_prefix("f/")
.and_then(|rest| rest.split_once('/'))
.map_or("<folder>", |(folder, _)| folder);
return Err(format!(
"\"{path}\" is a data pipeline node (its source starts with the pipeline annotation). In a session, pipeline nodes are not script drafts: call open_preview(kind=\"pipeline\", path=\"{folder}\") if that folder's pipeline editor is not open yet, then use build_pipeline_node (new node) or edit_pipeline_node (existing node). Nothing was written."
));
}
ctx.saving_draft("script", path);
let base = write_base(ctx, "script", path, None).await?;
let existed = base.base.is_some();
let before = base
.base
.as_ref()
.and_then(|b| b.get("content"))
.and_then(Value::as_str)
.unwrap_or_default()
.to_string();
let mut draft = match base.base {
Some(Value::Object(mut script)) => {
if !base.from_draft {
@@ -1697,17 +1853,21 @@ async fn write_script_draft(
};
// Inferred into the schema the base carries: its properties are the only copy of
// what no parser emits (`password`, enums, formats, titles). Code that does not
// parse keeps the schema as it was.
let previous = draft
.get("schema")
.filter(|schema| truthy(schema.get("properties")))
.cloned()
.unwrap_or_else(empty_schema);
let code = if content.is_empty() { " " } else { content };
// A language this build cannot parse, or code that does not parse yet, keeps the
// schema the draft had.
let schema = infer_script_schema(language, code, Some(&previous), None).unwrap_or(previous);
draft.insert("schema".into(), schema);
// parse keeps the schema as it was. The chat infers with the editor's own parsers
// before saving, so its schema matches what the editor shows.
if ctx.surface == Surface::Mcp {
let previous = draft
.get("schema")
.filter(|schema| truthy(schema.get("properties")))
.cloned()
.unwrap_or_else(empty_schema);
let code = if content.is_empty() { " " } else { content };
// A language this build cannot parse, or code that does not parse yet, keeps
// the schema the draft had.
let schema = infer_script_schema(language, code, Some(&previous), None).unwrap_or(previous);
draft.insert("schema".into(), schema);
}
ctx.code_diff(&before, content, Some(language));
let persisted = persist_draft(
ctx,
"script",
@@ -1755,7 +1915,11 @@ async fn discard_local_draft(ctx: &Ctx<'_>, args: &Value) -> ToolResult {
let not_found = || format!("No draft found for {item_type} \"{path}\".");
let kind = draft_kind(item_type, trigger_kind).ok_or_else(not_found)?;
let draft = get_draft(ctx, &kind, path).await?.ok_or_else(not_found)?;
delete_draft(ctx, &kind, path, draft.created_at.as_deref()).await?;
delete_draft(ctx, &kind, path, draft.created_at.as_deref(), false).await?;
ctx.status_after_save(
format!("Discarded {item_type} \"{path}\" draft"),
Some("Draft discarded"),
);
let mut out = Map::new();
out.insert("success".into(), json!(true));
out.insert(
@@ -1826,18 +1990,18 @@ async fn rebase_draft(ctx: &Ctx<'_>, args: &Value) -> ToolResult {
.as_ref()
.map_or_else(|| "undefined".to_string(), js_str);
if base.is_some() && base == head {
let message = format!(
"Draft \"{path}\" is already based on the latest deployed version ({head_text})."
);
ctx.status(message.clone());
let mut out = Map::new();
out.insert("success".into(), json!(true));
out.insert("alreadyLatest".into(), json!(true));
put(&mut out, head_key, head);
out.insert(
"message".into(),
json!(format!(
"Draft \"{path}\" is already based on the latest deployed version ({head_text})."
)),
);
out.insert("message".into(), json!(message));
return Ok(Value::Object(out).to_string());
}
ctx.status(format!("Rebasing draft \"{path}\" onto latest..."));
// Captured before the discard: this diff is the only record of what to replay. A
// base version that can no longer be fetched diffs against nothing, so the whole
@@ -1928,7 +2092,7 @@ async fn rebase_draft(ctx: &Ctx<'_>, args: &Value) -> ToolResult {
let result = pretty(&Value::Object(out));
// The diff is the only copy of the draft once it is discarded: a result the server
// would truncate must keep the draft instead.
if result.len() > windmill_mcp::server::MAX_TOOL_RESULT_BYTES {
if ctx.surface == Surface::Mcp && result.len() > super::MAX_TOOL_RESULT_BYTES {
let editor = match item_type {
"script" => "/scripts/edit",
"flow" => "/flows/edit",
@@ -1940,7 +2104,8 @@ async fn rebase_draft(ctx: &Ctx<'_>, args: &Value) -> ToolResult {
));
}
let kind = draft_kind(item_type, None).ok_or_else(no_draft)?;
delete_draft(ctx, &kind, path, draft.created_at.as_deref()).await?;
delete_draft(ctx, &kind, path, draft.created_at.as_deref(), false).await?;
ctx.status_after_save(format!("Discarded stale draft \"{path}\""), Some("Rebased"));
Ok(result)
}
+8 -5
View File
@@ -208,12 +208,13 @@ mod variables;
pub mod volumes_ee;
mod volumes_oss;
pub mod webhook_util;
mod workspace_settings_document;
mod workspaces;
#[cfg(feature = "private")]
pub mod workspaces_ee;
mod workspaces_export;
mod workspace_settings_document;
mod ai_session_tools;
#[cfg(feature = "mcp")]
mod mcp_tools;
@@ -780,6 +781,10 @@ pub async fn run_server(
Router::new()
})
.nest("/ai", ai::workspaced_service())
.nest(
"/ai_session_tools",
ai_session_tools::chat::workspaced_service(_base_internal_url.clone()),
)
.nest("/npm_proxy", windmill_api_npm_proxy::workspaced_service())
.nest(
"/path_autocomplete",
@@ -1287,8 +1292,7 @@ pub async fn run_server(
if let Some(name) = name.as_ref() {
tracing::info!("server starting for name={name}");
}
#[cfg(feature = "mcp")]
mcp::register_router(&_base_internal_url, Some(app.clone()));
ai_session_tools::register_router(&_base_internal_url, Some(app.clone()));
let listener = listener.tap_io(move |tcp_stream| {
let _ = tcp_stream.set_nodelay(!server_mode);
@@ -1328,8 +1332,7 @@ pub async fn run_server(
});
let served = server.await;
#[cfg(feature = "mcp")]
mcp::register_router(&_base_internal_url, None);
ai_session_tools::register_router(&_base_internal_url, None);
served?;
let forks = &windmill_api_workspaces::workspaces::BACKGROUND_FORKS;
forks.close();
+3 -10
View File
@@ -83,15 +83,7 @@ impl WindmillBackend {
w_id: &'a str,
access: SessionAccess,
) -> session::Ctx<'a> {
session::Ctx {
auth,
w_id,
base_internal_url: &self.base_internal_url,
access,
origin: None,
jwt: Default::default(),
view: Default::default(),
}
session::Ctx::new(auth, w_id, &self.base_internal_url, session::access(access))
}
pub fn new(
@@ -507,7 +499,8 @@ impl McpBackend for WindmillBackend {
access: SessionAccess,
origin: Option<&str>,
) -> BackendResult<CallToolResult> {
let ctx = session::Ctx { origin, ..self.session_ctx_with(auth, workspace_id, access) };
let mut ctx = self.session_ctx_with(auth, workspace_id, access);
ctx.origin = origin;
session::call_session_tool(&ctx, name, args).await
}
}
-1
View File
@@ -16,4 +16,3 @@ pub use core::{
extract_and_store_workspace_id, extract_workspace_from_token, list_tools_service,
reject_token_query_param, setup_mcp_server, toolset_service, McpEndpoint,
};
pub(crate) use session::register_router;
+175
View File
@@ -0,0 +1,175 @@
//! The AI session's toolset (`crate::ai_session_tools`) as MCP serves it.
use serde_json::{json, Map, Value};
use windmill_mcp::server::{
BackendResult, CallToolResult, ContentBlock, ErrorData, MetaObject, SessionAccess, Tool,
ToolAnnotations,
};
use crate::ai_session_tools::{self as tools, CallRefused};
pub(super) use tools::{
generic_session_instructions, session_instructions, workspace_context, Ctx,
};
pub(super) fn access(access: SessionAccess) -> tools::SessionAccess {
match access {
SessionAccess::ReadOnly => tools::SessionAccess::ReadOnly,
SessionAccess::ReadAndRun => tools::SessionAccess::ReadAndRun,
SessionAccess::Full => tools::SessionAccess::Full,
}
}
/// Claude clients keep an MCP result in context only up to a token budget well under the
/// authoring references `get_instructions` returns, unless the tool raises it. Above the
/// runner's own cap on a result, so that cap is the one that applies.
fn result_size_meta() -> MetaObject {
let mut meta = Map::new();
meta.insert("anthropic/maxResultSizeChars".to_string(), 100_000.into());
MetaObject(meta)
}
/// The destructive tools a card may call, each from a button the user presses. Every
/// other destructive tool is hidden from cards: a card acts as the user with no one
/// confirming.
const CARD_ACTIONS: &[&str] = &["deploy_workspace_item", "cancel_job"];
/// A tool's display name, which Claude shows in its tool list and approval prompts.
fn tool_title(name: &str) -> String {
let words: Vec<String> = name
.split('_')
.map(|word| match word {
"db" | "sql" | "json" => word.to_uppercase(),
_ => word.to_string(),
})
.collect();
let title = words.join(" ");
let mut chars = title.chars();
chars
.next()
.map(|first| first.to_uppercase().chain(chars).collect())
.unwrap_or_default()
}
/// The MCP Apps view a client that renders them draws these tools' results in. It draws
/// what the tool put in its result's `_meta` (`Ctx::view`), and acts by calling tools.
/// Hosts cache a view by its URI: bump the version when a change breaks the `_meta` shape
/// the deployed card reads.
pub(super) const CARD_URI: &str = "ui://windmill/card-v1";
const CARD_HTML: &str = include_str!("card.html");
/// The `ui://` views the session's tools render in, as `(uri, html)`.
pub(super) fn session_views() -> &'static [(&'static str, &'static str)] {
&[(CARD_URI, CARD_HTML)]
}
pub(super) fn session_tools(session_access: SessionAccess) -> Vec<Tool> {
let access = access(session_access);
tools::session_tools(access)
.map(|tool| {
let mut meta = result_size_meta();
if tool.destructive && !CARD_ACTIONS.contains(&tool.name) {
meta.0
.insert("ui".into(), json!({ "visibility": ["model"] }));
}
let title = tool_title(tool.name);
Tool::new(tool.name, tool.description, tool.parameters.clone())
.with_title(title.clone())
.with_annotations(
ToolAnnotations::with_title(title)
.read_only(tool.read_only)
.destructive(tool.destructive)
.open_world(tool.open_world),
)
.with_meta(meta)
})
.chain(
tools::view_definitions()
.into_iter()
.filter(|(name, ..)| tools::view_allowed(name, access))
.map(|(name, description, schema)| {
let mut meta = result_size_meta();
meta.0
.insert("ui".into(), json!({ "resourceUri": CARD_URI }));
let Value::Object(schema) = schema else {
unreachable!("a view tool's schema is an object")
};
let title = tool_title(name);
Tool::new(name, description, schema)
.with_title(title.clone())
.with_annotations(
ToolAnnotations::with_title(title)
.read_only(true)
.destructive(false)
.open_world(false),
)
.with_meta(meta)
}),
)
.collect()
}
pub(super) async fn call_session_tool(
ctx: &Ctx<'_>,
name: &str,
args: Value,
) -> BackendResult<CallToolResult> {
let outcome =
tools::call_session_tool(ctx, name, args)
.await
.map_err(|refused| match refused {
CallRefused::NotAllowed(message) => ErrorData::invalid_request(message, None),
CallRefused::UnknownTool(message) | CallRefused::NotAvailable(message) => {
ErrorData::invalid_params(message, None)
}
})?;
windmill_common::feature_usage::log_feature_usage("mcp_workspace", "tool_call", name);
let content = vec![ContentBlock::text(outcome.text)];
let mut result = if outcome.is_error {
CallToolResult::error(content)
} else {
CallToolResult::success(content)
};
if let Some((key, view)) = ctx.view.lock().unwrap().take() {
result.meta = Some(MetaObject(Map::from_iter([(key.to_string(), view)])));
}
Ok(result)
}
#[cfg(test)]
mod tests {
use super::*;
#[test]
fn no_card_can_call_a_destructive_tool_but_deploy_and_cancel() {
for tool in session_tools(SessionAccess::Full) {
let name = tool.name.as_ref();
let destructive = tool.annotations.as_ref().and_then(|a| a.destructive_hint);
if destructive == Some(true) {
let ui = tool.meta.as_ref().and_then(|m| m.0.get("ui"));
let model_only = ui == Some(&json!({ "visibility": ["model"] }));
assert_eq!(
model_only,
!matches!(name, "deploy_workspace_item" | "cancel_job"),
"{name}"
);
}
}
}
/// The ChatGPT directory wants every hint explicit, and Claude's a title on every tool.
#[test]
fn every_session_tool_has_a_title_and_explicit_hints() {
for tool in session_tools(SessionAccess::Full) {
let a = tool.annotations.as_ref().expect("annotations");
assert!(tool.title.is_some() && a.title.is_some(), "{}", tool.name);
assert!(
a.read_only_hint.is_some()
&& a.destructive_hint.is_some()
&& a.open_world_hint.is_some(),
"{}",
tool.name
);
}
}
}
+5 -28
View File
@@ -9,8 +9,7 @@ use axum::body::{to_bytes, Body};
use axum::response::Response;
use serde_json::Value;
use sql_builder::prelude::*;
use windmill_common::auth::create_jwt_token;
use windmill_common::db::{Authed, UserDB};
use windmill_common::db::UserDB;
use windmill_common::error::Error;
use windmill_common::runnable_ref::RunnableRef;
use windmill_common::scripts::{get_full_hub_script_by_path, Schema};
@@ -214,9 +213,7 @@ fn items_query(
} else {
let conds: Vec<String> = segments
.iter()
.map(|s| {
"o.path LIKE ? ESCAPE '\\'".bind(&format!("%/{}", escape_like(s)))
})
.map(|s| "o.path LIKE ? ESCAPE '\\'".bind(&format!("%/{}", escape_like(s))))
.collect();
sqlb.and_where(format!("({})", conds.join(" OR ")));
}
@@ -711,28 +708,6 @@ pub async fn create_http_request(
send_as_caller(method, url, workspace_id, api_authed, body_json, scopes).await
}
/// A JWT that acts as `api_authed`, carrying `scopes`; `None` is the caller's full rights.
pub(crate) async fn mint_caller_jwt(
workspace_id: &str,
api_authed: &ApiAuthed,
scopes: Option<Vec<String>>,
) -> BackendResult<String> {
// Carry the caller's job provenance into the JWT: a job's WM_TOKEN is capped at
// workspace admin (GHSA-hfh4-cx4h-3fcr), and dropping `job_id` here would re-mint an
// uncapped token that satisfies require_super_admin / require_devops_role.
create_jwt_token(
Authed::from(api_authed.clone()),
workspace_id,
3600,
api_authed.job_id,
None,
None,
scopes,
)
.await
.map_err(|e| ErrorData::internal_error(e.to_string(), None))
}
/// Call the API as `api_authed` with a JWT carrying `scopes`; `None` is the caller's
/// full rights, for a server whose own code picks every route it calls.
pub async fn send_as_caller(
@@ -758,7 +733,9 @@ pub async fn send_as_caller(
}
};
let token = mint_caller_jwt(workspace_id, api_authed, scopes).await?;
let token = crate::ai_session_tools::mint_caller_jwt(workspace_id, api_authed, scopes)
.await
.map_err(|e| ErrorData::internal_error(e.to_string(), None))?;
request_builder = request_builder.header("Authorization", format!("Bearer {}", token));
// Add body if present
@@ -1,141 +0,0 @@
import { beforeEach, describe, expect, it, vi } from 'vitest'
const { statusMock, createMock, upsertMock, runMock } = vi.hoisted(() => ({
upsertMock: vi.fn(),
statusMock: vi.fn(),
createMock: vi.fn(),
runMock: vi.fn()
}))
vi.mock('./shared', () => ({
createToolDef: (_schema: unknown, name: string, description: string) => ({
type: 'function',
function: { name, description, parameters: {} }
})
}))
vi.mock('$lib/gen', () => ({
WorkspaceService: {
getDatatableMigrationsStatus: statusMock,
createDatatableMigration: createMock,
upsertDatatableMigration: upsertMock,
runDatatableMigrations: runMock
}
}))
import { getDatatableMigrationTools } from './datatableMigrationTools'
function create(args: Record<string, unknown>) {
const tool = getDatatableMigrationTools().find(
(entry) => entry.def.function.name === 'create_datatable_migration'
)!
return tool.fn({
args: {
datatable_name: 'main',
name: 'add_refunds',
code_up: 'BEGIN;\nCREATE TABLE refunds ();\nEND;',
...args
},
workspace: 'test-workspace',
helpers: {},
toolCallbacks: { setToolStatus: vi.fn(), removeToolStatus: vi.fn() } as any,
toolId: 'tool-1'
})
}
const applied = { timestamp: 10, name: 'init', code_up: '', status: 'ran' }
const created = { timestamp: 30, name: 'add_refunds', code_up: '', status: 'not_run' }
beforeEach(() => {
for (const mock of [statusMock, createMock, upsertMock, runMock]) mock.mockReset()
createMock.mockResolvedValue({ timestamp: 30 })
statusMock.mockResolvedValue({ enabled: true, migrations: [applied, created] })
runMock.mockResolvedValue({ applied: [{ version: 30, name: 'add_refunds' }] })
})
describe('create_datatable_migration', () => {
it('applies only the migration it created', async () => {
const result = await create({})
expect(runMock).toHaveBeenCalledWith({
workspace: 'test-workspace',
datatableName: 'main',
only: 30
})
expect(result).toContain('Applied migration 30 (add_refunds)')
})
it('leaves it pending when an earlier migration is not applied', async () => {
statusMock.mockResolvedValue({
enabled: true,
migrations: [
applied,
{ timestamp: 20, name: 'earlier', code_up: '', status: 'not_run' },
created
]
})
const result = await create({})
expect(runMock).not.toHaveBeenCalled()
expect(result).toContain('20 (earlier)')
})
it('keeps a migration that fails to apply and rewrites it on request', async () => {
runMock.mockRejectedValueOnce({ body: 'relation "refunds" already exists' })
const failed = await create({})
expect(failed).toContain('relation "refunds" already exists')
expect(failed).toContain('with version 30')
statusMock.mockResolvedValue({
enabled: true,
migrations: [applied, { ...created, code_down: 'DROP TABLE refunds;' }]
})
await create({ version: 30, code_up: 'BEGIN;\nCREATE TABLE refunds2 ();\nEND;' })
expect(createMock).toHaveBeenCalledTimes(1)
expect(upsertMock).toHaveBeenCalledWith({
workspace: 'test-workspace',
datatableName: 'main',
requestBody: {
timestamp: 30,
code_down: 'DROP TABLE refunds;',
name: 'add_refunds',
code_up: 'BEGIN;\nCREATE TABLE refunds2 ();\nEND;'
}
})
expect(runMock).toHaveBeenLastCalledWith({
workspace: 'test-workspace',
datatableName: 'main',
only: 30
})
})
it('refuses to rewrite a version that is not pending', async () => {
const result = await create({ version: 10 })
expect(upsertMock).not.toHaveBeenCalled()
expect(runMock).not.toHaveBeenCalled()
expect(result).toContain('cannot be rewritten')
})
it('does not call unreadable earlier migrations pending', async () => {
statusMock.mockResolvedValue({
enabled: true,
migrations: [
{ ...applied, status: 'unknown' },
{ ...created, status: 'unknown' }
],
error: 'connection refused'
})
const result = await create({})
expect(runMock).not.toHaveBeenCalled()
expect(result).toContain('connection refused')
expect(result).not.toContain('10 (init)')
})
it('declares the role above the SQL of both directions', async () => {
await create({ role: 'app_rw', code_down: 'DROP TABLE refunds;', apply: false })
expect(createMock.mock.calls[0][0].requestBody).toEqual({
name: 'add_refunds',
code_up: '-- role app_rw\nBEGIN;\nCREATE TABLE refunds ();\nEND;',
code_down: '-- role app_rw\nDROP TABLE refunds;'
})
expect(runMock).not.toHaveBeenCalled()
})
})
@@ -1,14 +1,7 @@
import { z } from 'zod'
import { WorkspaceService } from '$lib/gen'
import { withMigrationRole } from '$lib/components/datatableMigrationRole'
import {
migrationScope,
parseMigrationSchemas,
scopesOverlap,
withMigrationSchemas
} from '$lib/components/datatableMigrationSchemas'
import { createToolDef, type ToolCallbacks } from './shared'
import { createToolDef } from './shared'
import { NONE, RUN_PREVIEW, type SessionTool } from './sessionCapabilities'
import { runServerTool } from './global/backendSessionTools'
/**
* Tools over a data table's versioned SQL migrations (`datatable_migrations.rs`).
@@ -23,21 +16,6 @@ const memo = <T>(factory: () => T): (() => T) => {
return () => (cached ??= factory())
}
function apiErrorMessage(e: unknown): string {
const err = e as { body?: unknown; message?: unknown } | undefined
const body = err?.body
if (typeof body === 'string' && body !== '') return body
if (typeof err?.message === 'string') return err.message
return String(e)
}
function fail(toolCallbacks: ToolCallbacks, toolId: string, message: string): string {
toolCallbacks.setToolStatus(toolId, { content: message, error: message })
return message
}
const migrationLabel = (m: { version: number; name: string }) => `${m.version} (${m.name})`
const datatableNameSchema = () =>
z.string().describe('The datatable the migrations belong to, e.g. "main".')
@@ -146,66 +124,12 @@ export function getDatatableMigrationTools(): SessionTool<{}>[] {
requires: NONE,
def: getListMigrationsToolDef(),
planModeSafe: true,
fn: async ({ args, workspace, toolId, toolCallbacks }) => {
const { datatable_name, version } = getListMigrationsSchema().parse(args)
toolCallbacks.setToolStatus(toolId, {
fn: async (ctx) => {
const { datatable_name } = getListMigrationsSchema().parse(ctx.args)
ctx.toolCallbacks.setToolStatus(ctx.toolId, {
content: `Listing migrations of "${datatable_name}"...`
})
try {
const status = await WorkspaceService.getDatatableMigrationsStatus({
workspace,
datatableName: datatable_name
})
if (!status.enabled) {
toolCallbacks.setToolStatus(toolId, {
content: `Migrations are not enabled on "${datatable_name}"`
})
return `Migrations are not enabled on datatable "${datatable_name}": its schema is changed with plain SQL and there is no history to list. Enabling them is a workspace admin's decision, made in Workspace settings → Data Tables.`
}
if (version !== undefined) {
const migration = status.migrations.find((m) => m.timestamp === version)
if (!migration) {
return fail(
toolCallbacks,
toolId,
`No migration ${version} on datatable "${datatable_name}"`
)
}
toolCallbacks.setToolStatus(toolId, {
content: `Read migration ${migrationLabel({ version, name: migration.name })}`
})
return JSON.stringify(
{
version,
name: migration.name,
status: migrationStatus(migration.status),
code_up: migration.code_up,
...(migration.code_down != null && { code_down: migration.code_down })
},
null,
2
)
}
toolCallbacks.setToolStatus(toolId, {
content: `Listed ${status.migrations.length} migration(s) of "${datatable_name}"`
})
return JSON.stringify(
{
migrations: status.migrations.map((m) => ({
version: m.timestamp,
name: m.name,
status: migrationStatus(m.status),
schemas: m.schemas ?? 'whole database'
})),
// Set when the database could not be read, which is why every status is unknown.
...(status.error && { error: status.error })
},
null,
2
)
} catch (e) {
return fail(toolCallbacks, toolId, `Error listing migrations: ${apiErrorMessage(e)}`)
}
return (await runServerTool('list_datatable_migrations', ctx)).text
}
},
{
@@ -218,124 +142,12 @@ export function getDatatableMigrationTools(): SessionTool<{}>[] {
`${args?.version === undefined ? 'Create' : 'Rewrite'}${args?.apply === false ? '' : ' and apply'} data table migration`,
queuedLabel: (args) => `Write migration "${args?.name ?? ''}"`,
showDetails: true,
fn: async ({ args, workspace, toolId, toolCallbacks }) => {
const parsed = getCreateMigrationSchema().parse(args)
const datatableName = parsed.datatable_name
toolCallbacks.setToolStatus(toolId, {
content: `Saving migration "${parsed.name}" on "${datatableName}"...`
fn: async (ctx) => {
const parsed = getCreateMigrationSchema().parse(ctx.args)
ctx.toolCallbacks.setToolStatus(ctx.toolId, {
content: `Saving migration "${parsed.name}" on "${parsed.datatable_name}"...`
})
// The role is an annotation in the SQL, read from above everything else, and a rollback
// has to run as the role that created what it drops.
// No inference here: the model reads the SQL it wrote better than a pattern does, so
// the scope is what it declares, and the whole database when it declares none.
const declared = parseMigrationSchemas(parsed.code_up)
const scope = parsed.schemas ?? (declared.kind === 'schemas' ? declared.schemas : [])
const definition = {
name: parsed.name,
code_up: withMigrationSchemas(
withRole(parsed.code_up, parsed.role),
scope.map((s) => s.toLowerCase())
),
...(parsed.code_down !== undefined && {
code_down: withRole(parsed.code_down, parsed.role)
})
}
let version: number
try {
if (parsed.version === undefined) {
const created = await WorkspaceService.createDatatableMigration({
workspace,
datatableName,
requestBody: definition
})
version = created.timestamp
} else {
version = parsed.version
// The server inserts at a version it does not know, which would wedge a new
// migration into the middle of the history, and stores a down left out as none.
const { migrations } = await WorkspaceService.getDatatableMigrationsStatus({
workspace,
datatableName
})
const existing = migrations.find((m) => m.timestamp === version)
if (existing?.status !== 'not_run') {
return fail(
toolCallbacks,
toolId,
`Migration ${version} is not a pending migration of datatable "${datatableName}", so it cannot be rewritten. Omit \`version\` to add a new migration.`
)
}
await WorkspaceService.upsertDatatableMigration({
workspace,
datatableName,
requestBody: {
timestamp: version,
...(existing.code_down != null && { code_down: existing.code_down }),
...definition
}
})
}
} catch (e) {
return fail(toolCallbacks, toolId, `Error saving migration: ${apiErrorMessage(e)}`)
}
const label = migrationLabel({ version, name: parsed.name })
if (parsed.apply === false) {
toolCallbacks.setToolStatus(toolId, { content: `Saved migration ${label} as pending` })
return `Saved migration ${label} as pending. It is not applied yet.`
}
// Applying it alone would run it ahead of migrations it may depend on, and applying
// those with it would run SQL the user was never shown.
let earlierPending: string[]
try {
const status = await WorkspaceService.getDatatableMigrationsStatus({
workspace,
datatableName
})
const ownScope = migrationScope(definition.code_up)
const earlier = status.migrations.filter(
(m) => m.timestamp < version && scopesOverlap(m.schemas, ownScope)
)
// Unreadable is not pending: reporting it so would send the user to apply
// migrations that already ran.
if (status.error || earlier.some((m) => m.status === 'unknown')) {
throw new Error(status.error ?? 'the applied migrations could not be read')
}
earlierPending = earlier
.filter((m) => m.status === 'not_run')
.map((m) => migrationLabel({ version: m.timestamp, name: m.name }))
} catch (e) {
return fail(
toolCallbacks,
toolId,
`Saved migration ${label} as pending, but could not check what else is pending, so it was not applied: ${apiErrorMessage(e)}`
)
}
if (earlierPending.length > 0) {
toolCallbacks.setToolStatus(toolId, {
content: `Saved migration ${label}; ${earlierPending.length} earlier migration(s) still pending`
})
return `Saved migration ${label} as pending without applying it: earlier migrations are not applied yet (${earlierPending.join(', ')}) and it may depend on them. Tell the user, then apply them all in order with run_datatable_migrations.`
}
toolCallbacks.setToolStatus(toolId, { content: `Applying migration ${label}...` })
try {
await WorkspaceService.runDatatableMigrations({
workspace,
datatableName,
only: version
})
} catch (e) {
// Kept whatever failed: part of it may be applied, or all of it with only the record
// missing, and the definition with its down is then the one trace of that.
return fail(
toolCallbacks,
toolId,
`Migration ${label} is saved but failed to apply: ${apiErrorMessage(e)}. It stays pending and blocks later migrations. If nothing of it was applied, fix it by calling this tool again with version ${version}; otherwise, or to drop it, tell the user, who can edit or delete it in Workspace settings → Data Tables.`
)
}
toolCallbacks.setToolStatus(toolId, { content: `Applied migration ${label}` })
return `Applied migration ${label} on datatable "${datatableName}".`
return (await runServerTool('create_datatable_migration', ctx)).text
}
},
{
@@ -346,32 +158,12 @@ export function getDatatableMigrationTools(): SessionTool<{}>[] {
confirmationMessage: 'Apply pending data table migrations',
queuedLabel: (args) => `Apply migrations on "${args?.datatable_name ?? 'datatable'}"`,
showDetails: true,
fn: async ({ args, workspace, toolId, toolCallbacks }) => {
const { datatable_name } = getRunMigrationsSchema().parse(args)
toolCallbacks.setToolStatus(toolId, {
fn: async (ctx) => {
const { datatable_name } = getRunMigrationsSchema().parse(ctx.args)
ctx.toolCallbacks.setToolStatus(ctx.toolId, {
content: `Applying migrations on "${datatable_name}"...`
})
try {
const { applied } = await WorkspaceService.runDatatableMigrations({
workspace,
datatableName: datatable_name
})
if (applied.length === 0) {
toolCallbacks.setToolStatus(toolId, { content: 'No pending migration to apply' })
return `No pending migration to apply on datatable "${datatable_name}".`
}
toolCallbacks.setToolStatus(toolId, {
content: `Applied ${applied.length} migration(s) on "${datatable_name}"`
})
return `Applied ${applied.length} migration(s) on datatable "${datatable_name}": ${applied.map(migrationLabel).join(', ')}.`
} catch (e) {
// Those before the failing one are applied: the list is what says where it stopped.
return fail(
toolCallbacks,
toolId,
`Error applying migrations: ${apiErrorMessage(e)}. Migrations before the failing one may have been applied: list them to see.`
)
}
return (await runServerTool('run_datatable_migrations', ctx)).text
}
},
{
@@ -382,36 +174,13 @@ export function getDatatableMigrationTools(): SessionTool<{}>[] {
confirmationMessage: (args) => `Roll back data table migration ${args?.version ?? ''}`,
queuedLabel: (args) => `Roll back migration ${args?.version ?? ''}`,
showDetails: true,
fn: async ({ args, workspace, toolId, toolCallbacks }) => {
const { datatable_name, version } = getRollbackMigrationSchema().parse(args)
toolCallbacks.setToolStatus(toolId, {
fn: async (ctx) => {
const { datatable_name, version } = getRollbackMigrationSchema().parse(ctx.args)
ctx.toolCallbacks.setToolStatus(ctx.toolId, {
content: `Rolling back migration ${version} on "${datatable_name}"...`
})
try {
const { rolled_back } = await WorkspaceService.rollbackDatatableMigrations({
workspace,
datatableName: datatable_name,
only: version
})
if (rolled_back.length === 0) {
toolCallbacks.setToolStatus(toolId, { content: `Migration ${version} is not applied` })
return `Migration ${version} is not applied on datatable "${datatable_name}", so there is nothing to roll back.`
}
const label = rolled_back.map(migrationLabel).join(', ')
toolCallbacks.setToolStatus(toolId, { content: `Rolled back migration ${label}` })
return `Rolled back migration ${label} on datatable "${datatable_name}". It is pending again.`
} catch (e) {
return fail(toolCallbacks, toolId, `Error rolling back: ${apiErrorMessage(e)}`)
}
return (await runServerTool('rollback_datatable_migration', ctx)).text
}
}
]
}
function migrationStatus(status: 'ran' | 'not_run' | 'unknown'): 'applied' | 'pending' | 'unknown' {
return status === 'ran' ? 'applied' : status === 'not_run' ? 'pending' : 'unknown'
}
function withRole(sql: string, role: string | undefined): string {
return role === undefined ? sql : withMigrationRole(sql, role)
}
@@ -1,226 +0,0 @@
import { beforeEach, describe, expect, it, vi } from 'vitest'
const { listMock, schemaMock, runScriptMock, executeTestRunMock } = vi.hoisted(() => ({
listMock: vi.fn(),
schemaMock: vi.fn(),
runScriptMock: vi.fn(),
executeTestRunMock: vi.fn()
}))
vi.mock('./shared', () => ({
createToolDef: (_schema: unknown, name: string, description: string) => ({
type: 'function',
function: { name, description, parameters: {} }
}),
executeTestRun: executeTestRunMock
}))
vi.mock('$lib/gen', () => ({
WorkspaceService: {
listDataTableTables: listMock,
getDataTableTableSchema: schemaMock
}
}))
vi.mock('$lib/components/jobs/utils', () => ({
runScript: runScriptMock
}))
// exec_datatable_sql now routes through executeTestRun; stub it to exercise the
// jobStarter (so runScript args can be asserted) and then hand a completed job to
// the tool's formatCompletion — which owns the datatable-specific result shaping.
// The detach/tray orchestration itself is covered by shared.test.ts.
function stubExecuteTestRunWithJob(job: { success: boolean; result: unknown }) {
runScriptMock.mockResolvedValue('job-123')
executeTestRunMock.mockImplementation(async (config: any) => {
await config.jobStarter()
const { llmText, card } = config.formatCompletion(job)
config.toolCallbacks.setToolStatus(config.toolId, card)
return llmText
})
}
import { getDatatableTools } from './datatableTools'
function createToolCallbacks() {
return {
setToolStatus: vi.fn(),
removeToolStatus: vi.fn()
}
}
function getTool(name: string) {
const tool = getDatatableTools().find((entry) => entry.def.function.name === name)
if (!tool) {
throw new Error(`${name} tool not found`)
}
return tool
}
function run(name: string, args: Record<string, unknown> = {}) {
return getTool(name).fn({
args,
workspace: 'test-workspace',
helpers: {},
toolCallbacks: createToolCallbacks(),
toolId: `tool-${name}`
})
}
beforeEach(() => {
listMock.mockReset()
schemaMock.mockReset()
runScriptMock.mockReset()
executeTestRunMock.mockReset()
})
describe('list_datatables', () => {
it('aggregates the table count across schemas and returns metadata verbatim', async () => {
const metadata = [
{ datatable_name: 'main', schemas: { public: ['users', 'orders'], analytics: ['events'] } },
{ datatable_name: 'warehouse', schemas: { public: ['facts'] } }
]
listMock.mockResolvedValue(metadata)
const tool = getTool('list_datatables')
const callbacks = createToolCallbacks()
const result = await tool.fn({
args: {},
workspace: 'test-workspace',
helpers: {},
toolCallbacks: callbacks,
toolId: 'tool-list'
})
expect(listMock).toHaveBeenCalledWith({ workspace: 'test-workspace' })
expect(JSON.parse(result)).toEqual(metadata)
// 2 + 1 + 1 = 4 tables across 2 datatables
expect(callbacks.setToolStatus).toHaveBeenCalledWith('tool-list', {
content: 'Listed 2 datatable(s) with 4 table(s)'
})
})
it('explains that configuring a datatable is a blocking prerequisite when none exist', async () => {
listMock.mockResolvedValue([])
const result = await run('list_datatables')
expect(result).toContain('No datatables are configured in this workspace')
expect(result).toContain('Workspace settings → Data Tables')
expect(result).toContain('blocked')
expect(result).toContain('Do not call exec_datatable_sql')
})
it('surfaces backend errors as a readable message', async () => {
listMock.mockRejectedValue(new Error('boom'))
const result = await run('list_datatables')
expect(result).toContain('Error listing datatables: boom')
})
})
describe('get_datatable_table_schema', () => {
it('returns the columns for one table', async () => {
schemaMock.mockResolvedValue({
datatable_name: 'main',
schema_name: 'public',
table_name: 'users',
columns: { id: 'int4', email: 'text' }
})
const result = await run('get_datatable_table_schema', {
datatable_name: 'main',
schema_name: 'public',
table_name: 'users'
})
expect(schemaMock).toHaveBeenCalledWith({
workspace: 'test-workspace',
datatableName: 'main',
schemaName: 'public',
tableName: 'users'
})
expect(JSON.parse(result)).toEqual({
datatable_name: 'main',
schema_name: 'public',
table_name: 'users',
columns: { id: 'int4', email: 'text' }
})
})
})
describe('exec_datatable_sql', () => {
it('requires confirmation', () => {
expect(getTool('exec_datatable_sql').requiresConfirmation).toBe(true)
})
it('starts the query as a postgresql job scoped to the datatable', async () => {
stubExecuteTestRunWithJob({ success: true, result: [] })
await run('exec_datatable_sql', { datatable_name: 'main', sql: 'SELECT * FROM t' })
expect(runScriptMock).toHaveBeenCalledWith({
workspace: 'test-workspace',
requestBody: {
language: 'postgresql',
content: 'SELECT * FROM t',
args: { database: 'datatable://main' }
}
})
})
it('returns all rows when the result is at or below the cap', async () => {
const rows = Array.from({ length: 100 }, (_, i) => ({ id: i }))
stubExecuteTestRunWithJob({ success: true, result: rows })
const parsed = JSON.parse(
await run('exec_datatable_sql', { datatable_name: 'main', sql: 'SELECT * FROM t' })
)
expect(parsed.success).toBe(true)
expect(parsed.rowCount).toBe(100)
expect(parsed.result).toHaveLength(100)
expect(parsed.note).toBeUndefined()
})
it('truncates results above the cap and reports the full count', async () => {
const rows = Array.from({ length: 150 }, (_, i) => ({ id: i }))
stubExecuteTestRunWithJob({ success: true, result: rows })
const parsed = JSON.parse(
await run('exec_datatable_sql', { datatable_name: 'main', sql: 'SELECT 1' })
)
expect(parsed.rowCount).toBe(150)
expect(parsed.result).toHaveLength(100)
expect(parsed.note).toBe('Showing first 100 of 150 rows')
})
it('treats a non-array result (e.g. DDL) as zero rows', async () => {
stubExecuteTestRunWithJob({ success: true, result: undefined })
const parsed = JSON.parse(
await run('exec_datatable_sql', {
datatable_name: 'main',
sql: 'CREATE TABLE t (id serial primary key)'
})
)
expect(parsed).toEqual({ success: true, rowCount: 0, result: [] })
})
it('returns a failure object when the SQL job errors', async () => {
stubExecuteTestRunWithJob({ success: false, result: { error: { message: 'syntax error' } } })
const parsed = JSON.parse(
await run('exec_datatable_sql', { datatable_name: 'main', sql: 'SLECT' })
)
expect(parsed).toEqual({ success: false, error: 'syntax error' })
})
it('turns the backend "datatable not found" error into an actionable, blocking message', async () => {
stubExecuteTestRunWithJob({
success: false,
result: { error: { message: 'Internal: datatable main not found @workspaces.rs:565:20' } }
})
const parsed = JSON.parse(
await run('exec_datatable_sql', { datatable_name: 'main', sql: 'CREATE TABLE t (id int)' })
)
expect(parsed.success).toBe(false)
expect(parsed.error).toContain('not configured in this workspace')
expect(parsed.error).toContain('Workspace settings → Data Tables')
expect(parsed.error).toContain('do not retry')
// The raw internal error is not surfaced to the model.
expect(parsed.error).not.toContain('workspaces.rs')
})
})
@@ -1,8 +1,6 @@
import { z } from 'zod'
import { WorkspaceService, type CompletedJob } from '$lib/gen'
import type { CompletedJob } from '$lib/gen'
import type { DataTableTables } from '$lib/gen/types.gen'
import { runScript } from '$lib/components/jobs/utils'
import { datatableReference } from '$lib/components/dbTypes'
import {
createToolDef,
executeTestRun,
@@ -10,6 +8,7 @@ import {
type ChatJobResultFormat
} from './shared'
import { NONE, RUN_PREVIEW, type SessionTool } from './sessionCapabilities'
import { callSessionTool, runOnServer } from './global/backendSessionTools'
/**
* Workspace-scoped datatable tools, with no app whitelist and no creation policy.
@@ -30,41 +29,6 @@ const memo = <T>(factory: () => T): (() => T) => {
return () => (cached ??= factory())
}
// ============= Pure workspace-scoped operations =============
/** List the datatables configured in the workspace, with their schema/table names: all of them as
* their default role, or only `datatableName`, as `role` when one is given. */
export async function listDatatables(
workspace: string,
datatableName?: string,
role?: string
): Promise<DataTableTables[]> {
if (datatableName === undefined) return await WorkspaceService.listDataTableTables({ workspace })
return await WorkspaceService.listDataTableTables({
workspace,
datatableName,
...(role !== undefined && { roleFor: datatableName, role })
})
}
/** Get the columns (column_name -> compact_type) of one datatable table. */
export async function getDatatableColumns(
workspace: string,
datatableName: string,
schemaName: string,
tableName: string,
role?: string
): Promise<Record<string, string>> {
const schema = await WorkspaceService.getDataTableTableSchema({
workspace,
datatableName,
schemaName,
tableName,
role
})
return schema.columns
}
// ============= Error helpers =============
/**
@@ -254,22 +218,16 @@ export function getDatatableTools(): SessionTool<{}>[] {
planModeSafe: true,
fn: async ({ args, workspace, toolId, toolCallbacks }) => {
toolCallbacks.setToolStatus(toolId, { content: 'Listing datatables...' })
try {
const parsedArgs = getListDatatablesSchema().parse(args ?? {})
if (parsedArgs.role !== undefined && parsedArgs.datatable_name === undefined) {
throw new Error('`role` needs `datatable_name`, the datatable it is a role of')
}
const metadata = await listDatatables(
workspace,
parsedArgs.datatable_name,
parsedArgs.role
)
if (metadata.length === 0) {
toolCallbacks.setToolStatus(toolId, {
content: 'No datatables configured — set one up in workspace settings'
})
return NO_DATATABLES_CONFIGURED_MESSAGE
}
getListDatatablesSchema().parse(args ?? {})
const { text } = await runOnServer('list_datatables', workspace, args ?? {})
if (text === NO_DATATABLES_CONFIGURED_MESSAGE) {
toolCallbacks.setToolStatus(toolId, {
content: 'No datatables configured — set one up in workspace settings'
})
} else if (text.startsWith('Error listing datatables: ')) {
toolCallbacks.setToolStatus(toolId, { content: text, error: text })
} else {
const metadata: DataTableTables[] = JSON.parse(text)
const totalTables = metadata.reduce(
(acc, datatable) =>
acc +
@@ -279,24 +237,8 @@ export function getDatatableTools(): SessionTool<{}>[] {
toolCallbacks.setToolStatus(toolId, {
content: `Listed ${metadata.length} datatable(s) with ${totalTables} table(s)`
})
// Only what the model acts on: the roles it may pass, not the creation privileges
// the manager's UI gates on.
return JSON.stringify(
metadata.map((d) => ({
datatable_name: d.datatable_name,
schemas: d.schemas,
...(d.error && { error: d.error }),
...(d.migrations_enabled && { migrations_enabled: true }),
...(d.permissioned && { usable_roles: d.usable_roles, default_role: d.default_role })
})),
null,
2
)
} catch (e) {
const errorMsg = `Error listing datatables: ${e instanceof Error ? e.message : String(e)}`
toolCallbacks.setToolStatus(toolId, { content: errorMsg, error: errorMsg })
return errorMsg
}
return text
}
},
{
@@ -308,39 +250,21 @@ export function getDatatableTools(): SessionTool<{}>[] {
toolCallbacks.setToolStatus(toolId, {
content: `Getting schema for ${parsedArgs.datatable_name}.${parsedArgs.schema_name}.${parsedArgs.table_name}...`
})
try {
const columns = await getDatatableColumns(
workspace,
parsedArgs.datatable_name,
parsedArgs.schema_name,
parsedArgs.table_name,
parsedArgs.role
)
toolCallbacks.setToolStatus(toolId, {
content: `Retrieved schema for ${parsedArgs.schema_name}.${parsedArgs.table_name}`
})
return JSON.stringify(
{
datatable_name: parsedArgs.datatable_name,
schema_name: parsedArgs.schema_name,
table_name: parsedArgs.table_name,
columns
},
null,
2
)
} catch (e) {
const raw = e instanceof Error ? e.message : String(e)
const errorMsg = isDatatableNotConfiguredError(raw)
? datatableNotConfiguredMessage(parsedArgs.datatable_name)
: `Error getting table schema: ${raw}`
toolCallbacks.setToolStatus(toolId, { content: errorMsg, error: errorMsg })
return errorMsg
}
const { text } = await runOnServer('get_datatable_table_schema', workspace, args)
const failed =
text.startsWith('Error getting table schema: ') ||
text === datatableNotConfiguredMessage(parsedArgs.datatable_name)
toolCallbacks.setToolStatus(
toolId,
failed
? { content: text, error: text }
: { content: `Retrieved schema for ${parsedArgs.schema_name}.${parsedArgs.table_name}` }
)
return text
}
},
{
// `runScript` below goes through /jobs/run/preview, which jobs.rs refuses operators.
// The query runs as a /jobs/run/preview job, which jobs.rs refuses operators.
requires: RUN_PREVIEW,
def: getExecDatatableSqlToolDef(),
requiresConfirmation: true,
@@ -355,15 +279,13 @@ export function getDatatableTools(): SessionTool<{}>[] {
// tray (and honors `background`) instead of blocking the chat turn,
// while keeping the datatable-specific result/error shaping.
return executeTestRun({
jobStarter: () =>
runScript({
workspace,
requestBody: {
language: 'postgresql',
content: parsedArgs.sql,
args: { database: datatableReference(name, parsedArgs.role) }
}
}),
jobStarter: async () => {
const response = await callSessionTool(workspace, 'exec_datatable_sql', args, {
run: { phase: 'start' }
})
if (response.is_error) throw new Error(response.text)
return response.text
},
workspace,
toolCallbacks,
toolId,
@@ -1,122 +0,0 @@
import { beforeEach, describe, expect, it, vi } from 'vitest'
const { listMock, listMetricsMock } = vi.hoisted(() => ({
listMock: vi.fn(),
listMetricsMock: vi.fn()
}))
vi.mock('./shared', () => ({
createToolDef: (_schema: unknown, name: string, description: string) => ({
type: 'function',
function: { name, description, parameters: {} }
})
}))
vi.mock('$lib/gen', () => ({
WorkspaceService: { listDucklakes: listMock },
DataMetricService: { listDataMetrics: listMetricsMock }
}))
import { getDucklakeTools } from './ducklakeTools'
function createToolCallbacks() {
return { setToolStatus: vi.fn(), removeToolStatus: vi.fn() }
}
function run(name: string, args: Record<string, unknown> = {}) {
const tool = getDucklakeTools().find((entry) => entry.def.function.name === name)
if (!tool) throw new Error(`${name} tool not found`)
return tool.fn({
args,
workspace: 'test-workspace',
helpers: {},
toolCallbacks: createToolCallbacks(),
toolId: `tool-${name}`
})
}
beforeEach(() => {
listMock.mockReset()
listMetricsMock.mockReset()
})
describe('list_ducklakes', () => {
it('returns the configured catalog names', async () => {
listMock.mockResolvedValue(['main', 'analytics'])
const result = await run('list_ducklakes')
expect(listMock).toHaveBeenCalledWith({ workspace: 'test-workspace' })
expect(JSON.parse(result)).toEqual({ ducklakes: ['main', 'analytics'] })
})
it('explains the storage prerequisite with role-appropriate steps when none exist', async () => {
listMock.mockResolvedValue([])
const result = await run('list_ducklakes')
expect(result).toContain('No DuckLake catalogs are configured')
expect(result).toContain('Workspace settings → Object Storage')
expect(result).toContain('ask a workspace admin')
// Drafting is not blocked: the message must say the scripts can still be drafted.
expect(result).toContain('still draft the pipeline scripts')
})
})
describe('list_data_metrics', () => {
it('forwards the filters and returns the declarations', async () => {
listMetricsMock.mockResolvedValue({
metrics: [
{
script_path: 'f/analytics/rev',
table_path: 'main/main.orders',
kind: 'measure',
name: 'revenue',
expr: 'sum(amount)',
filter: 'not is_test'
}
]
})
const result = await run('list_data_metrics', {
table: 'ducklake://main/main.orders',
path_prefix: 'f/analytics',
limit: 50
})
expect(listMetricsMock).toHaveBeenCalledWith({
workspace: 'test-workspace',
table: 'ducklake://main/main.orders',
pathPrefix: 'f/analytics',
perPage: 50
})
expect(JSON.parse(result).metrics[0]).toMatchObject({ name: 'revenue', expr: 'sum(amount)' })
})
it('never reports an empty result as proof that nothing is declared', async () => {
listMetricsMock.mockResolvedValue({ metrics: [] })
const result = await run('list_data_metrics', {})
// Unreadable declarations are omitted, not flagged, so absence is unprovable.
expect(result).toContain('does not establish')
expect(result).toContain('cannot read')
})
// The server matches a lake-less name against nothing, so the readability hedge
// would confirm "nothing is declared" for a filter worth retrying. The scheme is
// optional on the way in, so the retry it names must not carry it back.
it.each(['orders', 'ducklake://orders'])(
'blames the missing lake, not readability, for table %s',
async (table) => {
listMetricsMock.mockResolvedValue({ metrics: [] })
const result = await run('list_data_metrics', { table })
expect(result).toContain('`<lake>/orders`')
expect(result).not.toContain('cannot read')
}
)
it('warns that more declarations exist when the page is cut short', async () => {
listMetricsMock.mockResolvedValue({
// A cursor only comes back on a full page, never with an empty one.
metrics: [{ table_path: 't', kind: 'measure', name: 'n', script_path: 's' }],
next_cursor: { table_path: 't', kind: 'measure', name: 'n', script_path: 's' }
})
const result = await run('list_data_metrics', {})
// Without this the model reads a partial page as "no such measure" and
// re-derives a number that disagrees with the declared one.
expect(result).toContain('rather than concluding a measure is undeclared')
})
})
@@ -1,7 +1,7 @@
import { z } from 'zod'
import { DataMetricService, WorkspaceService } from '$lib/gen'
import { createToolDef } from './shared'
import { NONE, type SessionTool } from './sessionCapabilities'
import { runOnServer } from './global/backendSessionTools'
/**
* Workspace-scoped DuckLake tools, the pipeline counterpart to `list_datatables`
@@ -17,11 +17,6 @@ import { NONE, type SessionTool } from './sessionCapabilities'
* set.
*/
/** List the names of the DuckLake catalogs configured in the workspace. */
export async function listDucklakes(workspace: string): Promise<string[]> {
return await WorkspaceService.listDucklakes({ workspace })
}
const NO_DUCKLAKES_CONFIGURED_MESSAGE =
'No DuckLake catalogs are configured in this workspace. A data pipeline that materializes DuckLake tables or reads/writes S3 assets cannot run until object storage and a DuckLake catalog are set up. ' +
'You can still draft the pipeline scripts, but tell the user how to enable it by role: a workspace admin adds object storage under Workspace settings → Object Storage (S3/Azure/GCS), then a DuckLake catalog on top of it; a user without admin rights should ask a workspace admin. ' +
@@ -59,16 +54,6 @@ const listDataMetricsToolDef = createToolDef(
'List the measures and dimensions declared on DuckLake tables (from `// measure` / `// dimension` annotations in deployed scripts). Call this before writing any aggregate query over a DuckLake table: a declared measure is the canonical definition of that number, and reproducing it yourself silently disagrees with it (a `revenue` measure typically excludes refunds or test rows). Use each returned `expr` verbatim, and when a measure has a `filter` write it as `expr FILTER (WHERE filter)` so measures with different predicates share one GROUP BY. Only declarations whose producing script you can read are returned, so what comes back is never proof of what exists: if the number you need is not here you may still write your own aggregate, but say that you found no declared measure for it rather than implying none exists.'
)
// The endpoint drops declarations whose producing script the caller cannot read
// (token scope + RLS on `script`), so an empty result means "none declared" or
// "none readable by you" and the tool cannot tell which.
const NO_DATA_METRICS_NOTE =
'Nothing matched. That does not establish that nothing is declared: declarations whose producing script you cannot read are omitted from this list, not flagged. You may write your own aggregate, but tell the user you found no declared measure you can read rather than stating none is declared.'
// Well under the endpoint's 1000 cap: a full page is pretty-printed into the
// chat context, and 1000 declarations would cost tens of thousands of tokens.
const DEFAULT_DATA_METRICS_LIMIT = 200
/** The workspace DuckLake tools, for registration in global mode. */
export function getDucklakeTools(): SessionTool<{}>[] {
return [
@@ -78,59 +63,35 @@ export function getDucklakeTools(): SessionTool<{}>[] {
planModeSafe: true,
showDetails: true,
fn: async ({ args, workspace, toolId, toolCallbacks }) => {
const parsed = listDataMetricsSchema.parse(args)
listDataMetricsSchema.parse(args)
toolCallbacks.setToolStatus(toolId, { content: 'Listing declared measures...' })
const limit = parsed.limit ?? DEFAULT_DATA_METRICS_LIMIT
const { metrics, next_cursor } = await DataMetricService.listDataMetrics({
workspace,
table: parsed.table,
pathPrefix: parsed.path_prefix,
perPage: limit
})
// `canonical_table_path` expands a value only once it contains a `/`, so a bare
// table name matches nothing — a retryable filter, not a permissions outcome.
const bareTable = parsed.table?.replace('ducklake://', '')
const emptyNote =
bareTable && !bareTable.includes('/')
? `Nothing matched \`${parsed.table}\`: a table filter must name its lake, as \`<lake>/${bareTable}\`. Re-call with the lake before concluding anything about what is declared.`
: NO_DATA_METRICS_NOTE
const note = next_cursor
? `More declarations exist beyond the first ${limit}. Re-call with table/path_prefix to target what you are looking for, or with a higher limit (max 1000) for the rest of the list, rather than concluding a measure is undeclared.`
: metrics.length === 0
? emptyNote
: undefined
const result = JSON.stringify({ metrics, ...(note ? { note } : {}) }, null, 2)
const { text } = await runOnServer('list_data_metrics', workspace, args)
toolCallbacks.setToolStatus(toolId, {
content: `Listed ${metrics.length} declared measure(s)/dimension(s)`,
result
content: `Listed ${JSON.parse(text).metrics.length} declared measure(s)/dimension(s)`,
result: text
})
return result
return text
}
},
{
requires: NONE,
def: listDucklakesToolDef,
planModeSafe: true,
fn: async ({ workspace, toolId, toolCallbacks }) => {
fn: async ({ args, workspace, toolId, toolCallbacks }) => {
toolCallbacks.setToolStatus(toolId, { content: 'Listing DuckLake catalogs...' })
try {
const ducklakes = await listDucklakes(workspace)
if (ducklakes.length === 0) {
toolCallbacks.setToolStatus(toolId, {
content:
'No DuckLake configured — set up object storage + DuckLake in workspace settings'
})
return NO_DUCKLAKES_CONFIGURED_MESSAGE
}
const { text } = await runOnServer('list_ducklakes', workspace, args)
if (text === NO_DUCKLAKES_CONFIGURED_MESSAGE) {
toolCallbacks.setToolStatus(toolId, {
content: `Listed ${ducklakes.length} DuckLake catalog(s)`
content: 'No DuckLake configured — set up object storage + DuckLake in workspace settings'
})
} else if (text.startsWith('Error listing DuckLake catalogs: ')) {
toolCallbacks.setToolStatus(toolId, { content: text, error: text })
} else {
toolCallbacks.setToolStatus(toolId, {
content: `Listed ${JSON.parse(text).ducklakes.length} DuckLake catalog(s)`
})
return JSON.stringify({ ducklakes }, null, 2)
} catch (e) {
const errorMsg = `Error listing DuckLake catalogs: ${e instanceof Error ? e.message : String(e)}`
toolCallbacks.setToolStatus(toolId, { content: errorMsg, error: errorMsg })
return errorMsg
}
return text
}
}
]
@@ -118,6 +118,15 @@ export async function getAiAgentProviderCatalogFor(
if (!needsCatalog) {
return undefined
}
return getAiAgentProviderCatalogForRefs(workspace, resourceRefs)
}
/** `getAiAgentProviderCatalogFor` for modules whose steps state a provider and reference
* `resourceRefs`, as the server reports them when a flow write needs the catalog. */
export async function getAiAgentProviderCatalogForRefs(
workspace: string | undefined,
resourceRefs: string[]
): Promise<AiAgentProviderCatalog> {
const catalog = await getAiAgentProviderCatalog(workspace)
const unknown = resourceRefs.some(
(ref) => !catalog.options.some((option) => option.resourceRef === ref)
@@ -307,7 +307,7 @@ export function validateFlowSchema(rawSchema: unknown): Record<string, any> | nu
return rawSchema as Record<string, any>
}
function validateOptionalFlowModule(rawModule: unknown, fieldName: string): FlowModule | null {
export function validateOptionalFlowModule(rawModule: unknown, fieldName: string): FlowModule | null {
if (rawModule == null) return null
const result = flowModuleSchema.safeParse(rawModule)
@@ -0,0 +1,170 @@
/**
* The session tools that run on the server (`backend/windmill-api/src/ai_session_tools`),
* the same code the Windmill MCP server serves. The chat sends what only the browser
* holds (its open editors and unsaved draft cells) and applies what the call changed
* through its own draft store, so an open editor, its auto-save toggle and its conflict
* dialog behave as when the browser wrote the draft itself.
*/
import { AiService } from '$lib/gen'
import type { ToolCallbacks, ToolDisplayMessage } from '../shared'
import { UserDraft, type UserDraftItemKind } from '$lib/userDraft.svelte'
import {
deleteDraftAt,
GLOBAL_DRAFT_KINDS,
persistDraftAt,
type DraftPersistResult
} from './userDraftAdapter'
export type SessionToolEffect =
| {
type: 'draft_write'
kind: UserDraftItemKind
storage_path: string
value: unknown
last_sync?: string | null
force: boolean
}
| {
type: 'draft_delete'
kind: UserDraftItemKind
storage_path: string
last_sync?: string | null
keep_live_editor: boolean
}
| { type: 'inspected_run'; job_id: string }
| ({ type: 'facts' } & Record<string, unknown>)
| ({ type: 'status'; after_save?: boolean } & Partial<ToolDisplayMessage>)
| {
type: 'code_diff'
before: string
after: string
language?: string | null
file_path?: string | null
before_file_path?: string | null
}
| { type: 'needs_ai_provider_catalog'; resource_refs: string[] }
export type SessionToolResponse = {
text: string
is_error: boolean
effects: SessionToolEffect[]
}
/** What the browser holds that a tool reads: the editor open on each draft kind, and
* every draft cell, which is newer than the server's row. */
export function sessionToolContext(workspace: string) {
const live_editors = GLOBAL_DRAFT_KINDS.flatMap((kind) => {
const editor = UserDraft.getLiveEditorDraft(kind, { workspace })
return editor
? [{ kind, storage_path: editor.storagePath, effective_path: editor.effectivePath }]
: []
})
const cells = UserDraft.list({ workspace, itemKinds: [...GLOBAL_DRAFT_KINDS] }).map((entry) => ({
kind: entry.itemKind,
storage_path: entry.path,
value: entry.value
}))
return { live_editors, cells }
}
export async function callSessionTool(
workspace: string,
name: string,
args: unknown,
context: Record<string, unknown> = {}
): Promise<SessionToolResponse> {
const response = await AiService.callAiSessionTool({
workspace,
name,
requestBody: { args: args ?? {}, context: { ...sessionToolContext(workspace), ...context } }
})
return response as SessionToolResponse
}
/** Run a session tool on the server. A failed call throws its message, as the browser's own
* request did. */
export async function runOnServer(
name: string,
workspace: string,
args: unknown,
context?: Record<string, unknown>
): Promise<SessionToolResponse> {
const response = await callSessionTool(workspace, name, args, context)
if (response.is_error) throw new Error(response.text)
return response
}
/** `runOnServer` for a tool whose card the server narrates: its status patches are applied
* in the order the call reached them. */
export async function runServerTool(
name: string,
ctx: {
args: unknown
workspace: string
toolId: string
toolCallbacks: Pick<ToolCallbacks, 'setToolStatus'>
},
context?: Record<string, unknown>
): Promise<SessionToolResponse> {
const response = await callSessionTool(ctx.workspace, name, ctx.args, context)
applyStatusEffects(response.effects, ctx)
if (response.is_error) throw new Error(response.text)
return response
}
export function applyStatusEffects(
effects: SessionToolEffect[],
ctx: { toolId: string; toolCallbacks: Pick<ToolCallbacks, 'setToolStatus'> }
): void {
for (const effect of effects) {
if (effect.type === 'status') {
const { type: _, after_save: __, ...patch } = effect
ctx.toolCallbacks.setToolStatus(ctx.toolId, patch)
}
}
}
/** A failed delete, as the browser's own delete would have thrown it. */
export type DraftDeleteFailure = { storagePath: string; message: string }
export type AppliedEffects = {
/** One per `draft_write`, in order. */
writes: DraftPersistResult[]
deleteFailure?: DraftDeleteFailure
}
/** Apply a call's effects in order. A failed delete stops there, as a throw would have. */
export async function applySessionToolEffects(
workspace: string,
effects: SessionToolEffect[],
deletePath: (storagePath: string) => string = (storagePath) => storagePath
): Promise<AppliedEffects> {
const applied: AppliedEffects = { writes: [] }
for (const effect of effects) {
if (effect.type === 'draft_write') {
applied.writes.push(
await persistDraftAt(workspace, effect.kind, effect.storage_path, effect.value, {
force: effect.force,
lastSync: effect.last_sync
})
)
} else if (effect.type === 'draft_delete') {
try {
await deleteDraftAt(
workspace,
effect.kind,
effect.storage_path,
deletePath(effect.storage_path),
{ preserveLiveDraft: effect.keep_live_editor, lastSync: effect.last_sync }
)
} catch (e) {
applied.deleteFailure = {
storagePath: effect.storage_path,
message: e instanceof Error ? e.message : String(e)
}
return applied
}
}
}
return applied
}
File diff suppressed because it is too large Load Diff
File diff suppressed because it is too large Load Diff
@@ -1,218 +0,0 @@
import { describe, expect, it } from 'vitest'
import { buildFlowTree, shapeFlowRunTree, type FlowResultEntry } from './flowRunTree'
let uuidCounter = 0
function entry(partial: Partial<FlowResultEntry>): FlowResultEntry {
uuidCounter++
return {
job_id: `00000000-0000-0000-0000-${String(uuidCounter).padStart(12, '0')}`,
label: partial.step_path ? `Step ${partial.step_path}` : 'Flow',
kind: 'script',
depth: 1,
sibling_index: 1,
sibling_count: 1,
status: 'success',
success: true,
result_prefix: '"ok"',
result_length: 4,
...partial
}
}
function root(partial: Partial<FlowResultEntry> = {}): FlowResultEntry {
return entry({ kind: 'flow', depth: 0, step_path: null, label: 'Flow', ...partial })
}
/** Root flow with steps a, b where b is a subflow containing c. */
function nestedEntries(): FlowResultEntry[] {
return [
root(),
entry({ step_path: 'a', flow_step_id: 'a' }),
entry({ step_path: 'b', flow_step_id: 'b', kind: 'flow', label: 'Step b (subflow)' }),
entry({
step_path: 'b/c',
flow_step_id: 'c',
depth: 2,
status: 'failure',
success: false,
result_prefix: '{"error":"boom"}',
result_length: 16
})
]
}
/** Root flow with loop step l of `n` iterations; `failed` are 1-based indices. */
function loopEntries(n: number, failed: number[]): FlowResultEntry[] {
const entries = [root()]
for (let i = 1; i <= n; i++) {
entries.push(
entry({
step_path: 'l',
flow_step_id: 'l',
kind: 'flow',
label: `Step l forloop (iteration ${i}/${n})`,
parent_module_type: 'forloopflow',
sibling_index: i,
sibling_count: n,
status: failed.includes(i) ? 'failure' : 'success',
success: !failed.includes(i)
})
)
}
return entries
}
describe('buildFlowTree', () => {
it('rebuilds nesting from the depth-first flat list', () => {
const tree = buildFlowTree(nestedEntries())!
expect(tree.children.map((c) => c.entry.flow_step_id)).toEqual(['a', 'b'])
expect(tree.children[1].children.map((c) => c.entry.flow_step_id)).toEqual(['c'])
})
})
describe('shapeFlowRunTree', () => {
it('renders subflow steps nested under their parent step', () => {
const rendered = JSON.parse(shapeFlowRunTree({ entries: nestedEntries() }))
expect(rendered.run.status).toBe('success')
const stepB = rendered.steps[1]
expect(stepB.step).toBe('b')
expect(stepB.steps[0].step).toBe('b/c')
expect(stepB.steps[0].status).toBe('failure')
expect(stepB.steps[0].result).toBe('{"error":"boom"}')
})
it('collapses loop iterations to a tally with capped failed iterations plus the last', () => {
const rendered = JSON.parse(shapeFlowRunTree({ entries: loopEntries(50, [3, 7, 21, 30]) }))
const loop = rendered.steps[0]
expect(loop.type).toBe('forloopflow')
expect(loop.iterations).toBe(50)
expect(loop.ok).toBe(46)
expect(loop.failed_iterations).toEqual([3, 7, 21, 30])
// 3 failed shown (cap) + the last iteration
expect(loop.iterations_shown.map((i: any) => i.iteration)).toEqual([3, 7, 21, 50])
expect(loop.iterations_elided).toBe(46)
})
it('renders retried steps as attempts of one step, not loop iterations', () => {
const entries = [root()]
for (let i = 1; i <= 3; i++) {
entries.push(
entry({
step_path: 'a',
flow_step_id: 'a',
label: `Step a (attempt ${i}/3)`,
parent_module_type: 'rawscript',
sibling_index: i,
sibling_count: 3,
status: i < 3 ? 'failure' : 'success',
success: i === 3
})
)
}
const step = JSON.parse(shapeFlowRunTree({ entries })).steps[0]
expect(step.iterations).toBeUndefined()
expect(step.attempts).toBe(3)
expect(step.status).toBe('success')
expect(step.label).toBe('Step a (attempt 3/3)')
expect(step.previous_attempts.map((a: any) => a.status)).toEqual(['failure', 'failure'])
})
it('counts running iterations as unfinished and skipped ones as skipped', () => {
const entries = loopEntries(4, [])
entries[2].status = 'running'
entries[2].success = undefined
entries[2].result_prefix = undefined
entries[2].result_length = undefined
entries[3].status = 'skipped'
const loop = JSON.parse(shapeFlowRunTree({ entries })).steps[0]
expect(loop.ok).toBe(2)
expect(loop.unfinished).toBe(1)
expect(loop.skipped).toBe(1)
expect(loop.failed_iterations).toBeUndefined()
})
it('reports the truncated total size and surfaces the enclosing flow of a step job', () => {
const entries = nestedEntries()
entries[1].result_prefix = 'x'.repeat(700)
entries[1].result_length = 5000
const rendered = JSON.parse(shapeFlowRunTree({ enclosing_job: 'enclosing-uuid', entries }))
expect(rendered.note).toContain('enclosing-uuid')
expect(rendered.steps[0].result_total_chars).toBe(5000)
})
it('notes when the server truncated the tree', () => {
const rendered = JSON.parse(shapeFlowRunTree({ entries: nestedEntries(), truncated: true }))
expect(rendered.note).toContain('more jobs')
})
it('compares result sizes in code points and never splits a surrogate pair', () => {
const entries = [root()]
// 400 astral chars: 800 UTF-16 units but 400 code points. The 700-unit
// head budget keeps only 350 of them — comparing in UTF-16 units
// (400 < 700) used to hide that truncation entirely.
entries.push(
entry({
step_path: 'a',
flow_step_id: 'a',
status: 'failure',
success: false,
result_prefix: '🦄'.repeat(400),
result_length: 400
})
)
const step = JSON.parse(shapeFlowRunTree({ entries })).steps[0]
expect(step.result_total_chars).toBe(400)
// the cut must never leave a lone high surrogate at the tail
expect(step.result.charCodeAt(step.result.length - 1)).toBeGreaterThan(0xdbff)
})
it('shrinks result heads to fit the total budget instead of overflowing', () => {
const entries = [root()]
for (let i = 1; i <= 60; i++) {
entries.push(
entry({
step_path: `s${i}`,
flow_step_id: `s${i}`,
status: i === 1 ? 'failure' : 'success',
success: i !== 1,
result_prefix: 'y'.repeat(700),
result_length: 700
})
)
}
const rendered = shapeFlowRunTree({ entries })
expect(rendered.length).toBeLessThanOrEqual(21000)
const parsed = JSON.parse(rendered)
// failure detail outlives success detail: the failed step keeps a result
// head while succeeded steps are reduced to their size
expect(parsed.steps[0].status).toBe('failure')
expect(parsed.steps[0].result.length).toBeGreaterThan(0)
expect(parsed.steps[1].result).toBeUndefined()
expect(parsed.steps[1].result_total_chars).toBe(700)
})
it('spends the tree budget on the tree, not on the run payloads carried with it', () => {
const entries = [root()]
for (let i = 1; i <= 12; i++) {
entries.push(
entry({
step_path: `s${i}`,
flow_step_id: `s${i}`,
status: 'failure',
success: false,
result_prefix: 'y'.repeat(700),
result_length: 700
})
)
}
const logs = 'L'.repeat(12000)
const rendered = shapeFlowRunTree({ entries }, { status: 'failure', logs })
// Counting the overrides against the budget would shrink this tree away
// even though the tree itself fits: the model asked for the payloads.
expect(rendered.length).toBeGreaterThan(20000)
const parsed = JSON.parse(rendered)
expect(parsed.run.logs).toBe(logs)
expect(parsed.steps[0].result.length).toBe(700)
})
})
@@ -1,519 +0,0 @@
import { JobService, type Job, type GetFlowAllResultsResponse } from '$lib/gen'
import { isWindmillTooBigObject } from '$lib/components/job_args'
/**
* Model-facing view of a run for the global chat's get_run tool: the job's own
* summary, args, result and logs, plus — when the run has steps — its execution
* tree. The backend endpoint (get_flow_all_results) enumerates every job of the
* tree with per-entry truncated results; this module shapes that flat list into
* a compact per-step tree the model can read in one tool result. Step addresses
* ('b/c', 'b[12]/c') are resolved server-side by the same endpoint for
* full-result drill-down.
*
* `summarizeRun` also backs list_runs' per-job summary, so it lives here rather
* than in core.ts: get_run needs it, and importing it back would be circular.
*/
export type FlowResultEntry = GetFlowAllResultsResponse['entries'][number]
/** Per-entry result budget requested from the server for the tree view. */
export const TREE_RESULT_HEAD_CHARS = 700
/** Cap on a full payload handed to the model: a drilled step result, and the
* run's own args, result and logs. */
export const STEP_RESULT_MAX_CHARS = 12000
/** Cap on the whole rendered tree; heads shrink progressively to fit. */
const TREE_TOTAL_BUDGET_CHARS = 20000
/** Failed loop iterations shown with a result head (the rest are indices only). */
const MAX_FAILED_ITERATIONS_SHOWN = 3
/** Head-shrink ladder tried in order until the rendered tree fits the budget.
* `successHead` caps results of succeeded steps separately so failure detail
* survives longest. */
const SHRINK_LADDER = [
{ head: TREE_RESULT_HEAD_CHARS, successHead: TREE_RESULT_HEAD_CHARS },
{ head: 250, successHead: 250 },
{ head: 250, successHead: 0 },
{ head: 100, successHead: 0 }
]
export interface FlowTreeNode {
entry: FlowResultEntry
children: FlowTreeNode[]
}
/** Rebuild the parent/child tree from the server's depth-first flat list. */
export function buildFlowTree(entries: FlowResultEntry[]): FlowTreeNode | undefined {
let root: FlowTreeNode | undefined
const stack: FlowTreeNode[] = []
for (const entry of entries) {
const node: FlowTreeNode = { entry, children: [] }
while (stack.length && stack[stack.length - 1].entry.depth >= entry.depth) {
stack.pop()
}
const parent = stack[stack.length - 1]
if (parent) {
parent.children.push(node)
} else if (!root) {
root = node
}
stack.push(node)
}
return root
}
interface ChildGroup {
stepId: string
nodes: FlowTreeNode[]
}
/** Group sibling jobs by flow step id (loop iterations / branches of one step
* form one group). Keyed rather than run-length so ordering quirks between
* parallel iterations can't split a step into two groups. */
function groupChildren(children: FlowTreeNode[]): ChildGroup[] {
const groups = new Map<string, ChildGroup>()
for (const child of children) {
const stepId = child.entry.flow_step_id ?? `job:${child.entry.job_id}`
let group = groups.get(stepId)
if (!group) {
group = { stepId, nodes: [] }
groups.set(stepId, group)
}
group.nodes.push(child)
}
return Array.from(groups.values())
}
interface ShapeOpts {
head: number
successHead: number
}
/** Postgres LEFT()/length() count code points while JS .length/.slice count
* UTF-16 units — compare and cut in code points so astral characters neither
* hide a truncation nor get split through a surrogate pair. */
function countCodePoints(s: string): number {
let n = 0
for (const _ of s) n++
return n
}
function sliceCodePointSafe(s: string, maxUnits: number): string {
const cut = s.slice(0, maxUnits)
const last = cut.charCodeAt(cut.length - 1)
// drop a trailing lone high surrogate
return last >= 0xd800 && last <= 0xdbff ? cut.slice(0, -1) : cut
}
function sliceCodePointSafeEnd(s: string, maxUnits: number): string {
const cut = s.slice(-maxUnits)
const first = cut.charCodeAt(0)
// drop a leading lone low surrogate
return first >= 0xdc00 && first <= 0xdfff ? cut.slice(1) : cut
}
function shapeResult(
entry: FlowResultEntry,
opts: ShapeOpts
): { result?: string; result_total_chars?: number } {
if (entry.result_prefix === undefined || entry.result_prefix === null) return {}
const budget = entry.success ? opts.successHead : opts.head
if (budget <= 0) return { result_total_chars: entry.result_length ?? undefined }
const head = sliceCodePointSafe(entry.result_prefix, budget)
const total = entry.result_length ?? countCodePoints(entry.result_prefix)
return {
result: head,
...(total > countCodePoints(head) ? { result_total_chars: total } : {})
}
}
function shapeStep(node: FlowTreeNode, opts: ShapeOpts, iteration?: number): Record<string, any> {
const entry = node.entry
const shaped: Record<string, any> = {
...(entry.step_path ? { step: entry.step_path } : {}),
...(iteration !== undefined ? { iteration } : {}),
label: entry.label,
job_id: entry.job_id,
status: entry.status,
...(entry.duration_ms !== undefined && entry.duration_ms !== null
? { duration_ms: entry.duration_ms }
: {}),
...shapeResult(entry, opts)
}
const childSteps = shapeChildren(node.children, opts)
if (childSteps.length > 0) {
shaped.steps = childSteps
}
return shaped
}
/** Module types whose sibling jobs are iterations/branches; sibling jobs of
* any other step are retry attempts of that step. */
const FAN_OUT_MODULE_TYPES = new Set([
'forloopflow',
'whileloopflow',
'branchall',
'branchone',
'aiagent'
])
function shapeGroup(group: ChildGroup, opts: ShapeOpts): Record<string, any> {
if (group.nodes.length === 1 && group.nodes[0].entry.sibling_count <= 1) {
return shapeStep(group.nodes[0], opts)
}
const first = group.nodes[0].entry
if (first.parent_module_type === 'branchall') {
return {
step: first.step_path ?? group.stepId,
type: 'branchall',
branches: group.nodes.map((n) => shapeStep(n, opts))
}
}
const byIndex = [...group.nodes].sort((a, b) => a.entry.sibling_index - b.entry.sibling_index)
if (first.parent_module_type && !FAN_OUT_MODULE_TYPES.has(first.parent_module_type)) {
// Retried step: siblings are attempts of the same step, the last one is
// the final outcome — render it as the step, keeping earlier attempts as
// status-only references.
const shaped = shapeStep(byIndex[byIndex.length - 1], opts)
shaped.attempts = byIndex.length
shaped.previous_attempts = byIndex.slice(0, -1).map((n) => ({
attempt: n.entry.sibling_index,
status: n.entry.status,
job_id: n.entry.job_id
}))
return shaped
}
// Loop-like fan-out (forloopflow, whileloopflow, aiagent actions, or any
// other multi-job step): tally statuses, show failed iterations (capped) and
// the latest one, elide the rest.
const ok = byIndex.filter((n) => n.entry.status === 'success').length
const failedNodes = byIndex.filter(
(n) => n.entry.status === 'failure' || n.entry.status === 'canceled'
)
const skipped = byIndex.filter((n) => n.entry.status === 'skipped').length
// skipped is terminal — only running/queued/suspended count as unfinished
const unfinished = byIndex.length - ok - failedNodes.length - skipped
const shown = failedNodes.slice(0, MAX_FAILED_ITERATIONS_SHOWN)
const last = byIndex[byIndex.length - 1]
if (!shown.includes(last)) {
shown.push(last)
}
return {
step: first.step_path ?? group.stepId,
type: first.parent_module_type || 'loop',
iterations: byIndex.length,
ok,
...(failedNodes.length > 0
? { failed_iterations: failedNodes.map((n) => n.entry.sibling_index) }
: {}),
...(skipped > 0 ? { skipped } : {}),
...(unfinished > 0 ? { unfinished } : {}),
iterations_shown: shown.map((n) => shapeStep(n, opts, n.entry.sibling_index)),
...(byIndex.length > shown.length ? { iterations_elided: byIndex.length - shown.length } : {})
}
}
function shapeChildren(children: FlowTreeNode[], opts: ShapeOpts): Record<string, any>[] {
return groupChildren(children).map((g) => shapeGroup(g, opts))
}
function renderTree(
root: FlowTreeNode,
rootJobNote: string | undefined,
opts: ShapeOpts,
runOverrides?: Record<string, unknown>
): Record<string, any> {
const run = shapeStep(root, opts)
const steps = run.steps
delete run.steps
// The backend labels every depth-0 job "Flow"; correct that for the graceful
// non-flow case so the model doesn't mistake a plain script run for a flow.
if (!steps && root.entry.kind !== 'flow' && root.entry.kind !== 'flowpreview') {
run.label = `Job (${root.entry.kind})`
}
if (runOverrides) {
// A replacement result supersedes the tree entry's head, and the stale
// truncation marker must go with it. Without one the head stays: it is a
// real server-side prefix of a payload `getJob` would only hand back as a
// WINDMILL_TOO_BIG placeholder.
if ('result' in runOverrides) {
delete run.result
delete run.result_total_chars
}
Object.assign(run, runOverrides)
}
return {
...(rootJobNote ? { note: rootJobNote } : {}),
run,
...(steps
? {
steps,
hint: `Step results are truncated. Call get_run again with step="<step>" (e.g. "b/c", or "b[12]" for one loop iteration) for a step's result in full (up to ${STEP_RESULT_MAX_CHARS} chars), or with id set to a step's job_id for that step's own args, result and logs.`
}
: {})
}
}
/** Render the whole tree, shrinking result heads until it fits the budget. */
export function shapeFlowRunTree(
response: GetFlowAllResultsResponse,
runOverrides?: Record<string, unknown>
): string {
const root = buildFlowTree(response.entries)
if (!root) {
// The tree is the optional half: what the caller already read of the job
// still answers the question, so don't drop it with the missing tree.
return runOverrides
? JSON.stringify({ run: runOverrides }, null, 1)
: 'No jobs found for this run.'
}
const notes = [
...(response.enclosing_job
? [
`This job is a step of a larger flow run — its enclosing run is ${response.enclosing_job}; pass that id to see more of the tree.`
]
: []),
...(response.truncated
? [
`The run has more jobs than the server returns — this tree only covers the first ${response.entries.length} (depth-first), so tallies may undercount.`
]
: []),
...(response.scope_filtered
? [`Your token is tag-scoped: steps running on other tags are omitted from this tree.`]
: [])
]
const rootJobNote = notes.length > 0 ? notes.join(' ') : undefined
// The budget caps the tree, not the run's own args/result/logs — those are
// capped on their own and the model asked for them, so they don't count here
// and the last-resort slice keeps room for them (`run` precedes `steps`).
const overrideChars = runOverrides ? JSON.stringify(runOverrides, null, 1).length : 0
let rendered = ''
for (const opts of SHRINK_LADDER) {
rendered = JSON.stringify(renderTree(root, rootJobNote, opts, runOverrides), null, 1)
if (rendered.length - overrideChars <= TREE_TOTAL_BUDGET_CHARS) {
return rendered
}
}
return (
rendered.slice(0, TREE_TOTAL_BUDGET_CHARS + overrideChars) +
`\n… (tree truncated at ${TREE_TOTAL_BUDGET_CHARS} chars — drill into specific steps with the step parameter)`
)
}
// Compact metadata for one run. The raw Job carries args/result/logs/raw_code
// which can be huge — this is only what's needed to identify a run, so list_runs
// can return one entry per job.
export function summarizeRun(job: Job): Record<string, unknown> {
const base = {
id: job.id,
job_kind: job.job_kind,
path: job.script_path,
created_by: job.created_by,
created_at: job.created_at,
started_at: job.started_at,
schedule_path: job.schedule_path,
is_flow_step: job.is_flow_step,
tag: job.tag,
worker: job.worker
}
if ('success' in job) {
// CompletedJob. `success` is true for a skipped job too, so is_skipped has
// to be read first or a skipped step reports as a successful one.
return {
...base,
status: job.canceled
? 'canceled'
: job.is_skipped
? 'skipped'
: job.success
? 'success'
: 'failure',
duration_ms: job.duration_ms
}
}
// QueuedJob. A running job with suspends outstanding is parked — on an approval
// step or on a parallelism slot — not working, and `running` alone hides that.
return {
...base,
status: job.running ? (job.suspend ? 'suspended' : 'running') : 'queued'
}
}
/** Cap a payload to STEP_RESULT_MAX_CHARS. `tail` keeps the end instead of the
* start — for logs, where the failure is at the bottom.
*
* Counts and cuts without materialising the string: logs arrive whole and
* unbounded, and spreading one into an array of code points costs ~9x its size
* (a 10MB log measured +90MB). Cutting in UTF-16 units keeps at most the budget
* in code points, never more, so an astral-heavy payload is trimmed slightly
* shorter than advertised rather than overshooting. */
function cap(text: string, tail = false): string {
// UTF-16 length is never below the code-point count, so anything passing this
// is already under the cap and needs no counting pass at all.
if (text.length <= STEP_RESULT_MAX_CHARS) return text
const total = countCodePoints(text)
if (total <= STEP_RESULT_MAX_CHARS) return text
const note = `… (truncated: ${total} chars total)`
return tail
? `${note}\n${sliceCodePointSafeEnd(text, STEP_RESULT_MAX_CHARS)}`
: `${sliceCodePointSafe(text, STEP_RESULT_MAX_CHARS)}\n${note}`
}
/** Told apart from an empty log so the model doesn't report "no logs" for a run
* whose logs it simply failed to read. */
const LOGS_UNREADABLE = 'Logs could not be read for this run.'
/** `getJob` swaps a payload over ~90KB for a marker rather than sending it
* (`get_job_query!` in backend/windmill-api/src/jobs.rs), and the two fields use
* different ones: a result becomes the bare string, args become exactly
* `{reason: <marker>}`. Each field matches only its own form, so a payload that
* merely carries that string in a `reason` of its own stays the run's value. */
const TOO_BIG_RESULT = 'WINDMILL_TOO_BIG'
function stringify(value: unknown): string {
return typeof value === 'string' ? value : JSON.stringify(value, null, 1)
}
/** An elided args/result payload is reported, never fetched around: the
* endpoints that return those whole take no length parameter, so recovering a
* usable head would mean pulling the entire payload (up to MAX_RESULT_SIZE_MB,
* 500 by default) into the tab to keep 12k of it. The flag is the whole report —
* the result keeps the head the step tree already carries beside it. */
function shapeRunArgs(job: Job): Record<string, unknown> {
if (!job.args) return {}
return isWindmillTooBigObject(job.args)
? { args_truncated: true }
: { args: cap(stringify(job.args)) }
}
function shapeRunResult(job: Job): Record<string, unknown> {
if (!('result' in job) || job.result === undefined) return {}
// No `result` key when elided, so the tree's `result_prefix` — a real
// server-side head of the same payload — stands in its place unoverridden.
return job.result === TOO_BIG_RESULT
? { result_truncated: true }
: { result: cap(stringify(job.result)) }
}
/** Why a run ended badly. Kept out of summarizeRun, which list_runs pays per job. */
function diagnoseRun(job: Job): Record<string, unknown> {
return {
...(job.canceled_by ? { canceled_by: job.canceled_by } : {}),
...(job.canceled_reason ? { canceled_reason: job.canceled_reason } : {}),
...(job.mem_peak ? { mem_peak_kb: job.mem_peak } : {})
}
}
/** What get_run answers with: the model's payload, and — when the call addresses one
* job — that job's id, which the card renders the run from. The two are separate
* audiences: `text` is capped for the model, `jobId` is how the user gets the whole
* thing. An address resolving to several jobs (a loop's `b`), an unfinished step or an
* unknown one carries no id, and the call renders as an ordinary tool row. */
export type RunInspection = { text: string; jobId?: string }
/** Entry point of the get_run tool. Without `step`: the run's summary, args,
* result and logs, plus the per-step tree when the run has steps. With `step`:
* that step's full (capped) result, resolved server-side. */
export async function getRun(workspace: string, id: string, step?: string): Promise<RunInspection> {
if (!step) {
// Only the job read is load-bearing: logs and the step tree each answer
// part of the question, so neither failing should cost the model the rest.
const [job, logs, results] = await Promise.all([
JobService.getJob({ workspace, id, noLogs: true, noCode: true }),
// The dedicated endpoint rather than the job's own `logs` field, which holds too
// little to serve a tail — RunScriptCard's fetch has the mechanism. Only this
// job's own logs either way: a flow's are its orchestration lines, not its steps'.
JobService.getJobLogs({
workspace,
id,
// Suppress the "to remove ansi colors, use: sed ..." hint the backend
// otherwise prepends — noise for the model, and not actual stripping.
removeAnsiWarnings: true
}).catch(() => LOGS_UNREADABLE),
// `.catch` alone would leave the tree half-guarded: like the log read, this
// one can fail by resolving `undefined` rather than rejecting, and reading
// `.entries` off that throws — losing the job and logs already in hand.
JobService.getFlowAllResults({ workspace, id, maxResultLen: TREE_RESULT_HEAD_CHARS })
.catch(() => undefined)
.then((r) => r ?? ({ entries: [] } as GetFlowAllResultsResponse))
])
const { id: _id, ...summary } = summarizeRun(job)
const payloads = { ...shapeRunArgs(job), ...shapeRunResult(job) }
// A read that fails without rejecting still arrives here: the generated client
// resolves `undefined` when it cannot read the body (a truncated response, a
// dropped connection). Logs are the one payload where empty is a real answer,
// so that has to be told apart from an empty log rather than reported as one.
const shapedLogs =
typeof logs !== 'string' || logs === LOGS_UNREADABLE
? LOGS_UNREADABLE
: logs.trim()
? cap(logs, true)
: 'No logs for this run.'
return {
text: shapeFlowRunTree(results, {
...summary,
...diagnoseRun(job),
// A successful read always carries the job itself as the root entry, so
// no entries means the read failed — and nothing else would name the run.
...(results.entries.length === 0 ? { job_id: id, steps_unavailable: true } : {}),
...payloads,
logs: shapedLogs
}),
jobId: id
}
}
return getStepResult(workspace, id, step)
}
/** One step's result in full, addressed by step path. The server resolves the
* address directly (a few indexed lookups, no tree enumeration) and returns the
* single job as an entry. */
async function getStepResult(workspace: string, id: string, step: string): Promise<RunInspection> {
const response = await JobService.getFlowAllResults({
workspace,
id,
maxResultLen: STEP_RESULT_MAX_CHARS,
step
})
if (response.step_error) {
return {
text:
response.step_error +
(response.scope_filtered
? ' (Steps running on tags outside your token’s scope are hidden.)'
: '')
}
}
const entry = response.entries[0]
if (!entry) {
return { text: 'No jobs found for this run.' }
}
if (entry.status === 'running' || entry.status === 'queued' || entry.status === 'suspended') {
return {
text: `Step "${step}" (job ${entry.job_id}) has not completed yet — status: ${entry.status}.`
}
}
// Every completed step is a job of its own, so the card renders it from source —
// including a skipped one, whose inputs and logs are all there is to see.
if (entry.result_prefix === undefined || entry.result_prefix === null) {
return {
text: `Step "${step}" (job ${entry.job_id}, ${entry.status}) has no recorded result.`,
jobId: entry.job_id
}
}
const total = entry.result_length ?? countCodePoints(entry.result_prefix)
const capped =
total > countCodePoints(entry.result_prefix)
? entry.result_prefix + `\n… (result truncated: ${total} chars total)`
: entry.result_prefix
return {
text: `Step "${step}" (job ${entry.job_id}, ${entry.status}) result:\n${capped}`,
jobId: entry.job_id
}
}
@@ -1,5 +1,5 @@
/**
* Generates what the Windmill MCP server's toolset (backend/windmill-api/src/mcp/session)
* Generates what the Windmill MCP server's toolset (backend/windmill-api/src/ai_session_tools)
* shows a model, from the same definitions an AI session uses: tool names, descriptions and
* input schemas, the system prompt, and the `get_instructions` texts. The server implements
* the tools in Rust; everything the model reads comes from here, so the two cannot drift
@@ -19,10 +19,15 @@ import {
prepareGlobalSystemMessage
} from './core'
import type { ScriptLang } from '$lib/gen'
import { FRAMEWORK_TEMPLATES, STARTER_RUNNABLES } from '$lib/components/raw_apps/templates'
import {
BUILDER_FRAMEWORK_TEMPLATES,
FRAMEWORK_TEMPLATES,
STARTER_RUNNABLES
} from '$lib/components/raw_apps/templates'
import { RESOURCE_TYPE_WORDS } from '$lib/components/resourceTypeDisplay'
const OUT_DIR = fileURLToPath(
new URL('../../../../../../../backend/windmill-api/src/mcp/session/gen/', import.meta.url)
new URL('../../../../../../../backend/windmill-api/src/ai_session_tools/gen/', import.meta.url)
)
// Sentences that describe the chat's run form, which an MCP client does not have. Each
@@ -260,7 +265,10 @@ async function generate(): Promise<Record<string, string>> {
const tools = toolDefs()
const files: Record<string, string> = {
'tools.json': JSON.stringify(tools, null, '\t') + '\n',
'prompt.json': JSON.stringify(promptParts(), null, '\t') + '\n'
'prompt.json': JSON.stringify(promptParts(), null, '\t') + '\n',
// How a slug's words read, for matching a query against a hub integration the hub
// gives no display name.
'display_words.json': JSON.stringify(RESOURCE_TYPE_WORDS, null, '\t') + '\n'
}
const languages: (ScriptLang | undefined)[] = [
undefined,
@@ -284,10 +292,15 @@ async function generate(): Promise<Record<string, string>> {
index[subject][language ?? ''] = id
}
}
// What `init_app` seeds a new raw app with.
// What `init_app` seeds a new raw app with; an operator with builder rights starts from
// a static page, since they can save no inline runnable.
files['app_templates.json'] =
JSON.stringify(
{ templates: FRAMEWORK_TEMPLATES, starter_runnables: STARTER_RUNNABLES },
{
templates: FRAMEWORK_TEMPLATES,
builder_templates: BUILDER_FRAMEWORK_TEMPLATES,
starter_runnables: STARTER_RUNNABLES
},
null,
'\t'
) + '\n'
@@ -0,0 +1,97 @@
/**
* The GLOBAL toolset's versions of tools the other chat modes share: the same definition,
* run on the server like every other GLOBAL tool.
*/
import type { SessionTool } from '../sessionCapabilities'
import { createSearchHubScriptsTool, searchHubScriptsSchema } from '../shared'
import { createDbSchemaTool } from '../script/core'
import { readDocsPageTool, searchDocsTool } from '../docs/core'
import { runOnServer } from './backendSessionTools'
export const searchHubScriptsServerTool: SessionTool<{}> = {
...createSearchHubScriptsTool(false),
fn: async ({ args, workspace, toolId, toolCallbacks }) => {
const { query, integration } = searchHubScriptsSchema.parse(args)
const subject = query ? `"${query}"` : `the ${integration} integration`
if (query || integration) {
toolCallbacks.setToolStatus(toolId, { content: `Searching hub scripts for ${subject}...` })
}
const { text } = await runOnServer('search_hub_scripts', workspace, args)
if (query || integration) {
const found = JSON.parse(text).results.length
toolCallbacks.setToolStatus(toolId, {
content:
found === 0
? `No hub script found for ${subject}`
: `Found ${found} hub script${found === 1 ? '' : 's'} for ${subject}`
})
}
return text
}
}
export const searchDocsServerTool: SessionTool<{}> = {
...searchDocsTool,
fn: async ({ args, workspace, toolId, toolCallbacks }) => {
const query = typeof args?.query === 'string' ? args.query.trim() : ''
toolCallbacks.setToolStatus(toolId, {
content: query ? `Searching documentation for "${query}"...` : 'Searching documentation...'
})
const { text, effects } = await runOnServer('search_docs', workspace, args)
if (text.startsWith('Failed to search documentation:')) {
toolCallbacks.setToolStatus(toolId, {
content: 'Error searching documentation',
error: 'Error searching documentation'
})
} else if (query) {
const count = Number(
effects.flatMap((e) => (e.type === 'facts' ? [e.result_count] : []))[0] ?? 0
)
toolCallbacks.setToolStatus(toolId, {
content: count > 0 ? `Found ${count} matching page(s)` : 'No matching pages found'
})
}
return text
}
}
export const readDocsPageServerTool: SessionTool<{}> = {
...readDocsPageTool,
fn: async ({ args, workspace, toolId, toolCallbacks }) => {
const url = typeof args?.url === 'string' ? args.url : ''
const section =
typeof args?.section === 'string' && args.section.trim() ? args.section : undefined
toolCallbacks.setToolStatus(toolId, {
content: section ? `Reading docs section "${section}"...` : 'Reading documentation page...'
})
const { text } = await runOnServer('read_docs_page', workspace, args)
if (text.startsWith('Failed to read documentation page:')) {
toolCallbacks.setToolStatus(toolId, {
content: 'Error reading documentation page',
error: 'Error reading documentation page'
})
} else if (url.trim()) {
toolCallbacks.setToolStatus(toolId, { content: 'Read documentation page' })
}
return text
}
}
export function dbSchemaServerTool(description: string): SessionTool<{}> {
return {
...createDbSchemaTool<{}>({ description, updateEditorCache: false }),
fn: async ({ args, workspace, toolId, toolCallbacks }) => {
if (!args.resourcePath) {
throw new Error('Database path not provided')
}
toolCallbacks.setToolStatus(toolId, {
content: 'Getting database schema for ' + args.resourcePath + '...'
})
const { text } = await runOnServer('get_db_schema', workspace, args)
toolCallbacks.setToolStatus(toolId, {
content: 'Retrieved database schema for ' + args.resourcePath
})
return text
}
}
}
@@ -36,7 +36,7 @@ const TRIGGER_KIND_BY_DRAFT_KIND = Object.fromEntries(
])
) as Partial<Record<UserDraftItemKind, TriggerKind>>
const GLOBAL_DRAFT_KINDS = [
export const GLOBAL_DRAFT_KINDS = [
'script',
'flow',
'raw_app',
@@ -377,22 +377,6 @@ async function fetchBackendDraftValue(
return resp.value ?? undefined
}
// Draft VALUE for a write merge: cell-if-present (the user's freshest in-tab
// edits) else the current user's backend draft.
export async function readGlobalDraftValue<V>(
workspace: string,
type: WorkspaceItemType,
path: string,
triggerKind?: TriggerKind
): Promise<V | undefined> {
const itemKind = itemKindFor(type, triggerKind)
if (!itemKind) return undefined
const storagePath = resolveDraftStoragePath(workspace, itemKind, path)
const cell = UserDraft.get<V>(itemKind, storagePath, { workspace })
if (cell !== undefined) return cell
return (await fetchBackendDraftValue(workspace, itemKind, storagePath)) as V | undefined
}
// `itemKind` + `storagePath` are the canonical identity of the persisted draft
// (NOT item.path, which is the friendly display path). Callers use them to record
// the chat's modified-items mask.
@@ -427,6 +411,22 @@ export async function persistGlobalDraft(
const itemKind = itemKindFor(type, opts.triggerKind)
if (!itemKind) throw new Error(`Unsupported draft type "${type}".`)
const storagePath = resolveDraftStoragePath(workspace, itemKind, path)
return persistDraftAt(workspace, itemKind, storagePath, value, opts)
}
/** `persistGlobalDraft` at a resolved storage key. `lastSync` is the server row the value
* was built on when it was read from the server rather than from a cell: the save detects
* a conflict against it, as a read through `fetchBackendDraftValue` records. */
export async function persistDraftAt(
workspace: string,
itemKind: UserDraftItemKind,
storagePath: string,
value: unknown,
opts: { force?: boolean; lastSync?: string | null } = {}
): Promise<DraftPersistResult> {
if (opts.lastSync) {
UserDraftDbSyncer.recordRemoteSync({ workspace, itemKind, path: storagePath }, opts.lastSync)
}
UserDraft.seed(itemKind, storagePath, value, { workspace })
await UserDraftDbSyncer.save({
workspace,
@@ -442,7 +442,7 @@ export async function persistGlobalDraft(
displayPath,
isLiveDraft
)
if (!item) throw new Error(`Could not synthesize ${type} draft "${path}".`)
if (!item) throw new Error(`Could not synthesize ${itemKind} draft "${storagePath}".`)
// A failed save (network/5xx) is recorded in the syncer's failure map, not
// thrown — so check it before reporting success, else a write tool would tell
// the chat "saved" while the DB-backed source of truth was never updated.
@@ -593,6 +593,20 @@ export async function deleteGlobalDraft(
const itemKind = itemKindFor(type, triggerKind)
if (!itemKind) return
const storagePath = resolveDraftStoragePath(workspace, itemKind, path)
await deleteDraftAt(workspace, itemKind, storagePath, path, options)
}
/** `deleteGlobalDraft` at a resolved storage key; `path` names the draft in its errors. */
export async function deleteDraftAt(
workspace: string,
itemKind: UserDraftItemKind,
storagePath: string,
path: string,
options: DeleteGlobalDraftOptions & { lastSync?: string | null } = {}
): Promise<void> {
if (options.lastSync) {
UserDraftDbSyncer.recordRemoteSync({ workspace, itemKind, path: storagePath }, options.lastSync)
}
const liveDraft = UserDraft.getLiveEditorDraft(itemKind, { workspace })
if (options.preserveLiveDraft && liveDraft?.storagePath === storagePath) {
UserDraft.remove(itemKind, storagePath, { workspace })
@@ -1,14 +1,13 @@
import { z } from 'zod'
import YAML from 'yaml'
import { ApiError, WorkspaceService, type ApplyWorkspaceSettingsDocumentResponse } from '$lib/gen'
import type { ApplyWorkspaceSettingsDocumentResponse } from '$lib/gen'
import { createToolDef, type ToolCodeDiff } from '../shared'
import type { SessionTool, SessionToolPolicy } from '../sessionCapabilities'
import { toStableYaml } from './draftDiff'
import { runOnServer, runServerTool, type SessionToolEffect } from './backendSessionTools'
// Both endpoints call `require_admin`.
const ADMIN: SessionToolPolicy = ['admin']
const MAX_READ_CHARS = 20_000
type SettingsChange = ApplyWorkspaceSettingsDocumentResponse['changes'][number]
@@ -27,21 +26,6 @@ const setWorkspaceSettingsSchema = z.object({
)
})
function apiErrorText(e: unknown): string {
if (e instanceof ApiError) {
return typeof e.body === 'string' ? e.body : JSON.stringify(e.body)
}
return e instanceof Error ? e.message : String(e)
}
function parseSettingsYaml(text: string): Record<string, unknown> {
const doc = YAML.parse(text)
if (doc === null || typeof doc !== 'object' || Array.isArray(doc)) {
throw new Error('The yaml must be a mapping of setting keys to values.')
}
return doc
}
// An unset value is left out rather than written as null, so a side with nothing set is empty.
function changesSide(changes: SettingsChange[], side: 'before' | 'after'): string {
const set = changes.filter((c) => c[side] != null)
@@ -72,23 +56,7 @@ export const workspaceSettingsTools: SessionTool<{}>[] = [
showDetails: true,
fn: async ({ args, workspace, toolId, toolCallbacks }) => {
const parsed = getWorkspaceSettingsSchema.parse(args)
const doc = (await WorkspaceService.getWorkspaceSettingsDocument({
workspace,
secrets: 'redacted'
})) as Record<string, unknown>
let picked = doc
let note = ''
if (parsed.keys?.length) {
picked = Object.fromEntries(parsed.keys.filter((k) => k in doc).map((k) => [k, doc[k]]))
const unset = parsed.keys.filter((k) => !(k in doc))
if (unset.length) {
note = `\n# Not in the document (unset, or not a setting): ${unset.join(', ')}`
}
}
let text = YAML.stringify(picked, { aliasDuplicateObjects: false, lineWidth: 0 }) + note
if (text.length > MAX_READ_CHARS) {
text = `${text.slice(0, MAX_READ_CHARS)}\n… truncated at ${MAX_READ_CHARS} chars; read the keys you need with \`keys\`.`
}
const { text } = await runOnServer('get_workspace_settings', workspace, args)
toolCallbacks.setToolStatus(toolId, {
content: parsed.keys?.length
? `Read workspace settings: ${parsed.keys.join(', ')}`
@@ -110,27 +78,15 @@ export const workspaceSettingsTools: SessionTool<{}>[] = [
cancellationMessage: 'The user declined: no setting was changed.',
showDetails: true,
validateBeforeConfirmation: async ({ args, workspace }) => {
let changes: SettingsChange[]
let warnings: string[] | undefined
try {
const document = parseSettingsYaml(setWorkspaceSettingsSchema.parse(args).yaml)
;({ changes, warnings } = await WorkspaceService.applyWorkspaceSettingsDocument({
workspace,
requestBody: { document, replace: false, dry_run: true }
}))
} catch (e) {
const error = apiErrorText(e)
return { label: `Invalid settings: ${error}`, result: `Nothing was changed: ${error}` }
setWorkspaceSettingsSchema.parse(args)
const { text, effects } = await runOnServer('set_workspace_settings', workspace, args, {
check_only: true
})
const facts = settingsFacts(effects)
if (facts.refused) {
return { label: facts.refused, result: text }
}
if (changes.length === 0) {
return {
label: 'No settings change',
result: `The settings already hold these values: nothing to change.${
warnings?.length ? ` Warnings: ${warnings.join(' ')}` : ''
}`
}
}
dryRuns.set(args, changes)
dryRuns.set(args, facts.changes ?? [])
return undefined
},
onConfirmationRequested: ({ args, toolCallbacks, toolId }) => {
@@ -140,31 +96,33 @@ export const workspaceSettingsTools: SessionTool<{}>[] = [
}
},
fn: async ({ args, workspace, toolId, toolCallbacks }) => {
const document = parseSettingsYaml(setWorkspaceSettingsSchema.parse(args).yaml)
setWorkspaceSettingsSchema.parse(args)
toolCallbacks.setToolStatus(toolId, { content: 'Changing workspace settings...' })
try {
const { changes, warnings } = await WorkspaceService.applyWorkspaceSettingsDocument({
workspace,
requestBody: { document, replace: false }
})
const keys = changes.map((c) => c.key)
const { text, effects } = await runServerTool('set_workspace_settings', {
args,
workspace,
toolId,
toolCallbacks
})
const { changes } = settingsFacts(effects)
if (changes) {
toolCallbacks.setToolStatus(toolId, {
content: `Changed workspace settings: ${keys.join(', ')}`,
content: `Changed workspace settings: ${changes.map((c) => c.key).join(', ')}`,
codeDiff: changesDiff(changes)
})
return JSON.stringify({
success: true,
message: `The user approved and these settings are now applied: ${keys.join(', ')}.`,
...(warnings?.length ? { warnings } : {})
})
} catch (e) {
const error = apiErrorText(e)
toolCallbacks.setToolStatus(toolId, {
content: 'Failed to change workspace settings',
error
})
return JSON.stringify({ success: false, error })
}
return text
}
}
]
function settingsFacts(effects: SessionToolEffect[]): {
refused?: string
changes?: SettingsChange[]
} {
const facts = effects.find((e) => e.type === 'facts') as Record<string, unknown> | undefined
return {
refused: facts?.refused as string | undefined,
changes: facts?.changes as SettingsChange[] | undefined
}
}
@@ -32,8 +32,10 @@ import { getDbSchemas } from '$lib/components/apps/components/display/dbtable/me
import { scriptLangToEditorLang } from '$lib/scripts'
import { getScriptPrompt, getWorkflowAsCodePrompt } from '$system_prompts'
// Score threshold for npm packages search filtering
const SCORE_THRESHOLD = 1000
// The registry's best match is kept, and the runner-up only when it scores this share of it:
// the scores have no fixed scale (a descriptive query's right answer can score a few hundred, a
// package's own name thousands), so a match is judged against the best one.
const NPM_RUNNER_UP_SHARE = 0.6
// percentage of the context window for documentation of npm packages
const DOCS_CONTEXT_PERCENTAGE = 1
// percentage of the context window for types of npm packages
@@ -612,8 +614,9 @@ export async function searchExternalIntegrationResources(args: { query: string }
`https://registry.npmjs.org/-/v1/search?text=${encodeURIComponent(args.query)}&size=2`
)
const data = await result.json()
const best = Math.max(0, ...data.objects.map((r: PackageSearchQuery) => r.searchScore))
const filtered = data.objects.filter(
(r: PackageSearchQuery) => r.searchScore >= SCORE_THRESHOLD
(r: PackageSearchQuery) => r.searchScore >= best * NPM_RUNNER_UP_SHARE
)
const model = getCurrentModel()
@@ -2251,152 +2251,6 @@ describe('getHubIntegrationTool', () => {
expect(getHubIntegrationTool.planModeSafe).toBe(true)
expect(createSearchHubScriptsTool().planModeSafe).toBe(true)
})
const doc = {
app: 'confluence',
display_name: 'Confluence',
// Authored knowledge, but nobody has asserted whether the script set was pruned.
curated: null,
metadata_source: 'curated',
meta: { gotchas: ['Auth is Basic with an API token, not the password'] },
derived: {
api_hosts: [{ host: 'api.atlassian.com', count: 12 }],
style: 'fetch',
languages: { bun: 14 },
script_counts: { total: 14, by_kind: { script: 13 } },
top_scripts: [
{ path: 'hub/1/confluence/create_page', summary: 'Create page', language: 'bun' }
]
},
resource_types: [{ name: 'confluence', schema: { type: 'object' } }]
}
// Hand-validated provider knowledge and facts inferred from script bodies must
// stay under separate keys, or the model will report guesses as verified.
it('keeps authored notes apart from what was inferred from the scripts', async () => {
const { IntegrationService } = await import('$lib/gen')
Object.assign(IntegrationService, { getHubIntegrationMeta: vi.fn(async () => doc) })
const { getHubIntegrationTool } = await import('./shared')
const parsed = JSON.parse(
await getHubIntegrationTool.fn({
args: { integration: 'confluence' },
toolId: 't1',
toolCallbacks: { setToolStatus: vi.fn() }
} as any)
)
expect(parsed.verified_provider_notes).toEqual(doc.meta)
expect(parsed.scripts_note).toBeUndefined()
expect(parsed.observed_from_scripts).toEqual({
api_hosts: ['api.atlassian.com'],
style: 'fetch',
languages: ['bun'],
script_counts: { total: 14, by_kind: { script: 13 } }
})
})
// The hub repo owns this payload, so a hub older than this client can send a
// subset. Reading it must degrade to less content, never to a tool error.
it('returns what an older hub sent instead of failing on the missing sections', async () => {
const { IntegrationService } = await import('$lib/gen')
Object.assign(IntegrationService, {
getHubIntegrationMeta: vi.fn(async () => ({
app: 'stripe',
display_name: 'Stripe',
curated: false
}))
})
const { getHubIntegrationTool } = await import('./shared')
const parsed = JSON.parse(
await getHubIntegrationTool.fn({
args: { integration: 'stripe' },
toolId: 't1',
toolCallbacks: { setToolStatus: vi.fn() }
} as any)
)
expect(parsed.integration).toBe('stripe')
expect(parsed.observed_from_scripts).toBeUndefined()
expect(parsed.resource_types).toEqual([])
expect(parsed.example_scripts).toEqual([])
})
// `curated` has three states and only `true` is a licence to call the scripts good
// examples. `false` and the unassessed `null` must both stay silent, or the 212
// integrations nobody has looked at get characterised anyway.
it.each([
[true, true],
[false, false],
[null, false]
])('speaks about the script set only when curated is true (%s)', async (curated, expected) => {
const { IntegrationService } = await import('$lib/gen')
Object.assign(IntegrationService, {
getHubIntegrationMeta: vi.fn(async () => ({ ...doc, curated }))
})
const { getHubIntegrationTool } = await import('./shared')
const parsed = JSON.parse(
await getHubIntegrationTool.fn({
args: { integration: 'confluence' },
toolId: 't1',
toolCallbacks: { setToolStatus: vi.fn() }
} as any)
)
expect(!!parsed.scripts_note).toBe(expected)
})
// A hub that times out has said nothing about whether the integration exists, and
// reporting it as absent would stick for the rest of the conversation.
it('does not report a transient hub failure as a missing integration', async () => {
const { IntegrationService } = await import('$lib/gen')
Object.assign(IntegrationService, {
getHubIntegrationMeta: vi.fn(async () => {
throw Object.assign(new Error('Service Unavailable'), { status: 503 })
}),
listHubIntegrations: vi.fn(async () => [{ name: 'confluence' }])
})
const { getHubIntegrationTool, clearHubIntegrationsCache } = await import('./shared')
clearHubIntegrationsCache()
const parsed = JSON.parse(
await getHubIntegrationTool.fn({
args: { integration: 'confluence' },
toolId: 't1',
toolCallbacks: { setToolStatus: vi.fn() }
} as any)
)
expect(parsed.error).toContain('Could not reach the hub')
expect(parsed.error).not.toContain('No hub metadata')
})
// A hub with no such integration and one too old to serve the endpoint both 404;
// neither may surface as a tool error, since the model can still read scripts.
it('suggests real slugs instead of failing when the integration is unknown', async () => {
const { IntegrationService } = await import('$lib/gen')
Object.assign(IntegrationService, {
getHubIntegrationMeta: vi.fn(async () => {
throw Object.assign(new Error('Not Found'), { status: 404 })
}),
listHubIntegrations: vi.fn(async () => [{ name: 'stripe' }, { name: 'slack' }])
})
const { getHubIntegrationTool, clearHubIntegrationsCache } = await import('./shared')
clearHubIntegrationsCache()
const parsed = JSON.parse(
await getHubIntegrationTool.fn({
args: { integration: 'stripe_billing' },
toolId: 't1',
toolCallbacks: { setToolStatus: vi.fn() }
} as any)
)
expect(parsed.error).toContain('stripe_billing')
expect(parsed.suggested_integrations).toEqual(['stripe'])
})
})
describe('processToolCall confirmation hooks', () => {
@@ -78,6 +78,7 @@ import {
import type { editor as meditor } from 'monaco-editor'
import { pendingFolderInstructions, type FolderInstructionsContext } from './folderInstructions'
import type { WebSearchSource } from './webSearchResult'
import { runOnServer } from './global/backendSessionTools'
// Prettify function for code arguments - extracts and formats code from JSON
function prettifyCodeArguments(content: string): string {
@@ -1704,7 +1705,7 @@ function hasOptionalProperties(schema: Record<string, any> | undefined): boolean
return false
}
const searchHubScriptsSchema = z.object({
export const searchHubScriptsSchema = z.object({
query: z
.string()
.optional()
@@ -1861,41 +1862,6 @@ const getHubIntegrationToolDef = createToolDef(
'Read how one integration works before writing code against it: the resource type it takes, its auth fields, and its most-used scripts as examples. For an integration a search marked `documented` it also returns provider knowledge checked against the live API: pagination, enums, error codes and gotchas.'
)
/** Enough to show the integration's idiom; search_hub_scripts is the way to find
* a specific one. */
const MAX_INTEGRATION_EXAMPLES = 5
/** How the authored notes were checked — method, per-surface confidence, the instance
* used — is provenance for a human reviewing the hub, and up to a third of the
* document. The verdict and any fields it flags as shaky are the parts a caller can
* act on, so those stay. */
function withoutProvenance(meta: unknown): unknown {
if (typeof meta !== 'object' || meta === null || Array.isArray(meta)) {
return meta
}
const { validation, ...rest } = meta as Record<string, unknown>
const kept = Object.fromEntries(
Object.entries((validation as Record<string, unknown> | undefined) ?? {}).filter(
([key]) => key === 'status' || key === 'low_confidence_fields'
)
)
return Object.keys(kept).length ? { ...rest, validation: kept } : rest
}
/** The hub keeps a resource type's schema in a text column and hands it back as a
* JSON string, so parse it rather than passing an escaped blob to the model.
* Anything already structured goes through untouched. */
function parseResourceTypeSchema(schema: unknown): unknown {
if (typeof schema !== 'string') {
return schema
}
try {
return JSON.parse(schema)
} catch {
return schema
}
}
export const HUB_SYNC_INSTRUCTIONS =
'Tell the user: no resource of a missing type can be created until a superadmin syncs resource types from the hub, with "Sync resource types with Hub" in the Add resource dialog of the Resources page.'
@@ -1904,76 +1870,20 @@ export const getHubIntegrationTool = {
def: getHubIntegrationToolDef,
// Reads one hub document over a GET and writes nothing, so planning may use it.
planModeSafe: true,
fn: async ({ args, toolId, toolCallbacks }) => {
fn: async ({ args, workspace, toolId, toolCallbacks }) => {
const { integration } = getHubIntegrationSchema.parse(args)
toolCallbacks.setToolStatus(toolId, { content: `Reading the ${integration} integration...` })
let doc: Awaited<ReturnType<typeof IntegrationService.getHubIntegrationMeta>>
try {
doc = await IntegrationService.getHubIntegrationMeta({ app: integration })
} catch (err) {
// Only a 404 means the integration is absent — an unknown slug, or a hub
// predating the endpoint. Reporting a timeout the same way would teach the
// model that a real integration does not exist for the rest of the chat.
const absent = (err as { status?: number } | undefined)?.status === 404
const label = absent
? `No hub integration named ${integration}`
: `Could not reach the hub for ${integration}`
toolCallbacks.setToolStatus(toolId, { content: label })
const suggested = await suggestHubIntegrations(integration)
return JSON.stringify({
error: absent
? `No hub metadata for "${integration}".`
: `Could not reach the hub for "${integration}"; it may still exist. Read its scripts instead, or try again.`,
suggested_integrations: suggested
})
}
toolCallbacks.setToolStatus(toolId, { content: `Read the ${doc.display_name} integration` })
// The hub owns this response's shape and can be older than this client, so
// every section is read as optional: a hub that sends less should return less,
// not fail the call.
const derived = doc.derived
return JSON.stringify({
integration: doc.app,
display_name: doc.display_name,
...(doc.description ? { description: doc.description } : {}),
...(doc.docs_url ? { docs_url: doc.docs_url } : {}),
// Authored provider knowledge and facts inferred from the scripts stay
// separate: only the former was checked against the live API.
...(doc.meta ? { verified_provider_notes: withoutProvenance(doc.meta) } : {}),
...(derived
? {
observed_from_scripts: {
api_hosts: derived.api_hosts?.map((h) => h.host) ?? [],
style: derived.style,
languages: Object.keys(derived.languages ?? {}),
script_counts: derived.script_counts
}
}
: {}),
// `curated` is three-state: only an integration's own meta.json asserts it, and
// anything else means nobody has assessed the script set. Speak only for a
// positive assertion, so an unassessed integration is never characterised
// either way; `script_counts` below is the factual signal for the rest.
...(doc.curated === true
? {
scripts_note:
'These scripts were pruned to idiomatic actions rather than generated one per API endpoint, so they are worth following as style examples.'
}
: {}),
resource_types: (doc.resource_types ?? []).map((rt) => ({
name: rt.name,
...(rt.description ? { description: rt.description } : {}),
schema: parseResourceTypeSchema(rt.schema)
})),
example_scripts: (derived?.top_scripts ?? []).slice(0, MAX_INTEGRATION_EXAMPLES).map((s) => ({
path: s.path,
summary: s.summary,
...(s.description ? { description: s.description } : {}),
...(s.language ? { language: s.language } : {})
}))
const { text } = await runOnServer('get_hub_integration', workspace, args)
const doc = JSON.parse(text)
toolCallbacks.setToolStatus(toolId, {
content:
doc.error === undefined
? `Read the ${doc.display_name} integration`
: doc.error.startsWith('No hub metadata')
? `No hub integration named ${integration}`
: `Could not reach the hub for ${integration}`
})
return text
}
} satisfies SessionTool<{}>
@@ -2692,6 +2602,18 @@ async function loadDeployedScriptForFlowStep(
}
}
/** What a test run of an AI decision step supplies: the state. The provider and questions
* are the step's own. */
export function decisionStepRunSchema(): Record<string, any> {
return {
$schema: 'https://json-schema.org/draft/2020-12/schema',
type: 'object',
properties: { state: AI_DECISION_SCHEMA.properties.state },
required: ['state'],
order: ['state']
}
}
export async function resolveFlowStepRun({
flowValue,
stepId,
@@ -2833,14 +2755,7 @@ export async function resolveFlowStepRun({
// Run alone as a one-step flow preview, as the editor's step test does: the inputs given here
// override the step's own, so a state that reads an earlier step can be passed directly.
runnableKind: 'flow',
// What a run supplies: the state. The provider and questions are the step's own.
schema: {
$schema: 'https://json-schema.org/draft/2020-12/schema',
type: 'object',
properties: { state: AI_DECISION_SCHEMA.properties.state },
required: ['state'],
order: ['state']
},
schema: decisionStepRunSchema(),
startMessage: `Starting test run of AI decision step "${stepId}"...`,
startJob: async (args) =>
JobService.runFlowPreview({
@@ -78,17 +78,15 @@ export function agentDraftState(
export class AgentDraftUnavailable extends Error {}
/**
* An agent's resource together with the draft a run of it would use.
* Land the agent editor's pending save of `path`, or refuse: a read of the stored draft would
* otherwise be older than the edit still held in the browser.
*
* The flush is what makes the answer current. Autosave is debounced by 1.5s (10s ceiling), and
* The flush is what makes the stored row current. Autosave is debounced by 1.5s (10s ceiling), and
* closing the agent editor releases the in-memory cell without cancelling that pending POST — so
* testing or deploying right after closing would otherwise read a row the last edits have not
* reached yet. `flush` replays the parked save and is a no-op when there is none.
*/
export async function fetchAgentWithDraft(
path: string,
workspace: string
): Promise<{ response: Resource; draft: AgentResourceState | undefined }> {
export async function settleAgentDraft(path: string, workspace: string): Promise<void> {
const query = { workspace, itemKind: 'resource' as const, path }
await UserDraftDbSyncer.flush(query)
// `flush` resolves whether or not the save actually landed: `postSave` catches network and
@@ -106,6 +104,14 @@ export async function fetchAgentWithDraft(
`The unsaved changes to ${path} could not be saved because it was edited elsewhere. Open the agent to resolve it.`
)
}
}
/** An agent's resource together with the draft a run of it would use, its pending save landed. */
export async function fetchAgentWithDraft(
path: string,
workspace: string
): Promise<{ response: Resource; draft: AgentResourceState | undefined }> {
await settleAgentDraft(path, workspace)
const response = await ResourceService.getResource({ workspace, path, getDraft: true })
return { response, draft: agentDraftState(response, path, workspace) }
}
@@ -81,7 +81,7 @@ export function sortResourceTypesByMatch<T>(
* that carry a capital inside the word. Anything absent is capitalized, which is right for
* the great majority of types (`stripe` -> `Stripe`).
*/
const RESOURCE_TYPE_WORDS: Record<string, string> = {
export const RESOURCE_TYPE_WORDS: Record<string, string> = {
ai: 'AI',
ai21: 'AI21',
amqp: 'AMQP',
@@ -0,0 +1,67 @@
// The MCP server settles a run's arguments in Rust (`ai_session_tools/runs.rs`), read against
// the same cases by `run_args_match_the_shared_cases`.
import { readFileSync } from 'node:fs'
import { describe, expect, it, vi } from 'vitest'
const stored: string[] = []
vi.mock('$lib/gen', () => ({
VariableService: {
createVariable: vi.fn(async ({ requestBody }: any) => {
stored.push(requestBody.value)
return requestBody.path
})
}
}))
vi.mock('$lib/stores', async () => {
const { writable } = await import('svelte/store')
return { workspaceStore: writable('test-ws'), userStore: writable({ username: 'ada' }) }
})
import {
dropUndeclaredArgs,
enforceDisabledDefaults,
redactFileArgs,
redactSecretArgs
} from './job_args'
import { processSecretArgs } from './secretArgUtils'
const { cases } = JSON.parse(
readFileSync(
new URL(
'../../../../backend/windmill-api/src/ai_session_tools/run_args_cases.json',
import.meta.url
),
'utf8'
)
)
describe('run arguments, against the cases the server shares', () => {
for (const c of cases) {
it(c.name, async () => {
if (c.declared) {
const { args, undeclaredKeys } = dropUndeclaredArgs(c.args, c.schema)
expect(args).toEqual(c.declared.args)
expect(undeclaredKeys).toEqual(c.declared.undeclared)
}
if (c.disabled_defaults) {
const { args, resetKeys } = enforceDisabledDefaults(c.args, c.schema)
expect(args).toEqual(c.disabled_defaults.args)
expect(resetKeys).toEqual(c.disabled_defaults.reset)
}
if (c.secrets) {
stored.length = 0
const out = await processSecretArgs(c.args, c.schema, 'test-ws')
const named = JSON.parse(
JSON.stringify(out).replace(/u\/ada\/secret_arg\/[A-Za-z0-9]+/g, '<minted>')
)
expect(named).toEqual(c.secrets.args)
expect([...stored].sort()).toEqual(c.secrets.stored)
}
if (c.redacted) {
expect(redactFileArgs(redactSecretArgs(c.args, c.schema), c.schema)).toEqual(c.redacted)
}
})
}
})
+1 -1
View File
@@ -10,7 +10,7 @@ are built from it:
`cli/src/guidance/skills.gen.ts`, next to the `AGENTS.wmill.md` template in
`cli/src/guidance/core.ts`.
The workspace MCP server (`backend/windmill-api/src/mcp/session/`) is a third reader, one step
The workspace MCP server (`backend/windmill-api/src/ai_session_tools/`) is a third reader, one step
removed: it serves the global chat's own `get_instructions` texts and system prompt, rendered for
an external client by `npm run gen:mcp-session` in `frontend/`. Rerun that after `generate.py`.