diff --git a/.agents/skills/ai-chat/SKILL.md b/.agents/skills/ai-chat/SKILL.md new file mode 100644 index 0000000000..f670b2ffb7 --- /dev/null +++ b/.agents/skills/ai-chat/SKILL.md @@ -0,0 +1,40 @@ +--- +name: ai-chat +description: Guidance for improving the Windmill AI chat (copilot), especially global mode — tools, prompts, and context-window discipline. Use when editing chat tools, system prompts, or tool-result shapes under frontend/src/lib/components/copilot/chat, or when changing how the chat manages its context window. +--- + +## Always benchmark before and after + +No context or behavior change ships without an `ai_evals` A/B on the affected mode. +Add or adjust cases for exactly what you changed — see the `ai-evals` skill for +authoring and the full run reference. + +Run the affected mode **before** your change and **after**, same model(s), same cases. + +## Measure the window first, and cumulative second + +Optimize **`finalContextTokens`** (window occupancy — what drives overflow and +compaction), then cumulative prompt tokens. + +## Context discipline + +The dominant fixed cost is per-iteration overhead: the system prompt **plus every +tool schema** is re-sent on every loop iteration. So: + +- **Every tool and every parameter is a permanent tax.** Justify each one and measure + it; an extra "locate" round-trip can cost more than the reads it saves. Strip dead + params rather than leaving them in the schema. +- **Tool results return the minimum.** Never echo content the model already has. The + canonical mistake: a write tool that returns the whole edited artifact right after + the model authored it — return `{ success, message }` instead. When you touch a + *shared* write helper (e.g. `finishAppDraftWrite` in `global/core.ts`), re-check + this invariant for **all** the write tools routing through it — the echo has + regressed before via a shared refactor. + +## Prompts and tool descriptions are part of the surface + +The system prompt and tool descriptions steer behavior as much as the tools +themselves, and are benchmarkable the same way. A description that advertises +truncation makes the model self-limit; the path-conventions block changes where +drafts land. Treat prompt/description edits as real changes and A/B them — a +pure-prompt change is a legitimate, measurable improvement. \ No newline at end of file diff --git a/.agents/skills/ai-evals/SKILL.md b/.agents/skills/ai-evals/SKILL.md new file mode 100644 index 0000000000..ad334e2c6b --- /dev/null +++ b/.agents/skills/ai-evals/SKILL.md @@ -0,0 +1,87 @@ +--- +name: ai-evals +description: Author and run black-box benchmark cases for the Windmill AI generation modes (flow/app/script/cli/global) in ai_evals/. Use when adding or changing eval cases, or when running before/after benchmarks for AI chat / copilot changes. +--- + +# AI evals — authoring and running benchmark cases + +`ai_evals/` is a black-box benchmark runner for the Windmill AI generation modes: +`flow`, `app`, `script`, `cli`, `global`. It always tests the **current** production +prompts, tools, and guidance in this checkout. Each attempt runs the real production +path, deterministic validation, then LLM judging. + +The goal is to test current production guidance with realistic user requests — **not** +to pin one exact implementation shape. + +## Running benchmarks + +```bash +cd ai_evals +bun install # first time; frontend modes also need `cd frontend && bun install` +bun run cli -- models # list model aliases +bun run cli -- cases global # list cases for a mode +bun run cli -- run global global-test1-script-create --model sonnet +``` + +Frontend modes (`flow`/`script`/`app`/`global`) route model calls through a Windmill +backend's `/api/w//ai/proxy`, so you need **any** reachable backend: + +```bash +WMILL_AI_EVAL_BACKEND_URL=http://127.0.0.1: WMILL_AI_EVAL_BACKEND_WORKSPACE=integration-tests \ + bun run cli -- run global --models sonnet,gpt-5.5,gemini-3.1-pro-preview +``` + +- **Reuse an existing workspace.** CE builds cap workspaces, so temp-workspace + creation 400s ("reached workspace limit"). Always set + `WMILL_AI_EVAL_BACKEND_WORKSPACE=integration-tests` (or any existing workspace) to + reuse one. The only side effect of a run is upserting an `f/evals/ai/` + resource there. +- Provider keys live in `ai_evals/.env` and are auto-loaded by bun. The judge is a + separate Anthropic call (default `claude-sonnet-4-6`) regardless of the model under + test. + +## Authoring core rules + +1. Write prompts like a real user request. +2. Prefer behavior, inputs, constraints, and outcomes over internal implementation. +3. Keep deterministic validation narrow and hard. +4. Put semantic expectations in `judgeChecklist`. +5. Use `expected` fixtures only when exact structure really matters. + +### Prompt writing + +Prompts should sound like something a user would naturally ask. Do not write prompts +as if the user knows Windmill internals unless the case explicitly tests a power-user +workflow. + +Good: +- "Create a flow that routes support requests based on customer tier." +- "Add a reset button that sets the counter back to 0." +- "Create a flow that reuses the existing greeting script instead of duplicating the logic." + +Bad: +- "Use `branchone` with 3 branches and a default branch." +- "Create a `rawscript` step with this exact topology." +- "This is a benchmark harness." + +### Deterministic validation + +Use deterministic checks only for hard failures: missing required files; unexpected +extra files when the prompt says not to create them; syntax errors; unresolved flow +refs; missing required special modules or suspend config; obvious corruption. + +Do **not** encode one preferred implementation. Bad hard checks: exact step topology +for a creation flow; exact branch structure when the prompt only asked for routing; +exact input shape when multiple reasonable shapes are acceptable. + +### Judge checklist + +Every non-trivial case should have a `judgeChecklist` capturing user-visible behavior +that must be present, important constraints, and key completion criteria — not +low-level implementation details unless truly required. + +Good: "the flow calculates the order total with 8% tax"; "the flow reuses the existing +workspace script instead of rewriting the logic". Bad: "uses `branchone`"; "contains a +`rawscript` node". + +See `ai_evals/README.md` for the full case format, fields, and fixture details. \ No newline at end of file diff --git a/.claude/skills/ai-chat/SKILL.md b/.claude/skills/ai-chat/SKILL.md new file mode 120000 index 0000000000..f8c527f3c9 --- /dev/null +++ b/.claude/skills/ai-chat/SKILL.md @@ -0,0 +1 @@ +../../../.agents/skills/ai-chat/SKILL.md \ No newline at end of file diff --git a/.claude/skills/ai-evals/SKILL.md b/.claude/skills/ai-evals/SKILL.md new file mode 120000 index 0000000000..8b714cb50d --- /dev/null +++ b/.claude/skills/ai-evals/SKILL.md @@ -0,0 +1 @@ +../../../.agents/skills/ai-evals/SKILL.md \ No newline at end of file diff --git a/ai_evals/AGENTS.md b/ai_evals/AGENTS.md index af5427abae..6e63c4037e 100644 --- a/ai_evals/AGENTS.md +++ b/ai_evals/AGENTS.md @@ -1,208 +1,14 @@ -# AI Evals Authoring Guide +# AI Evals -This folder contains black-box benchmark cases for: +Black-box benchmark cases for the Windmill AI generation modes (`flow`, `app`, +`script`, `cli`, `global`). -- `flow` -- `app` -- `script` -- `cli` -- `global` +**Authoring and running cases is documented in the `ai-evals` skill** — load it +before adding/changing a case or running a benchmark. Claude Code reads +`.claude/skills/ai-evals/SKILL.md`; Codex and Pi read +`.agents/skills/ai-evals/SKILL.md` (same canonical file). Invoke with `/ai-evals` in +Claude Code, `$ai-evals` in Codex, or `pi --skill ai-evals`. -The goal is to test the current production prompts and guidance with realistic user requests, not to test one exact implementation shape. +For AI chat / copilot changes that these evals measure, see the `ai-chat` skill. -## Core rules - -1. Write prompts like a real user request. -2. Prefer behavior, inputs, constraints, and outcomes over internal implementation details. -3. Keep deterministic validation narrow and hard. -4. Put semantic expectations in `judgeChecklist`. -5. Use `expected` fixtures only when exact structure really matters. - -## Prompt writing - -Prompts should sound like something a user would naturally ask. - -Good: - -- "Create a flow that routes support requests based on customer tier." -- "Add a reset button that sets the counter back to 0." -- "Create a flow that reuses the existing greeting script instead of duplicating the logic." - -Bad: - -- "Use `branchone` with 3 branches and a default branch." -- "Create a `rawscript` step with this exact topology." -- "This is a benchmark harness." - -Do not write prompts as if the user knows Windmill internals unless the case is explicitly testing a power-user workflow. - -## Flow-specific rules - -This is the main principle you asked for: - -- flow prompts should read like requests from a user who does not know the product internals -- the user should ask for behavior, not for `branchone`, `branchall`, `rawscript`, `preprocessor_module`, `failure_module`, exact graph topology, or other internal constructs - -That means: - -- creation cases should describe the business behavior and expected result -- modification cases may mention existing step names, because the user can see the current flow -- only mention special Windmill constructs when the case is explicitly about those constructs - -Examples: - -- acceptable creation prompt: - "Create a purchase approval flow that pauses for approval and asks the approver for a comment." -- avoid: - "Create a suspend step with one required event and a resume form." - -For flow cases, do not fail a case just because the model chose a different valid topology. - -## App-specific rules - -App prompts should focus on user-visible behavior: - -- what the UI should let the user do -- what should persist -- what backend behavior is needed - -Avoid prompting in terms of React structure, component names, or implementation unless the case is specifically about editing an existing app. - -## CLI-specific rules - -CLI prompts can be more explicit about paths and file names because real CLI users often do specify them. - -Still, avoid benchmark phrasing. The prompt should read like a repo task, not a harness instruction. - -When relevant, ask the assistant to tell the user which `wmill` commands to run next. That is part of the benchmarked behavior. - -## Global-specific rules - -Global prompts should exercise workspace-level drafting behavior: - -- inspecting existing scripts, flows, apps, schedules, triggers, resources, and variables when relevant -- writing AI drafts rather than saving or deploying by default -- producing coherent multi-artifact changes when the request crosses artifact boundaries - -Keep deterministic validation focused on the draft contract: required draft type/path, required content snippets, forbidden draft paths, and forbidden mutating tools such as deploy/delete unless the case explicitly asks for them. - -Datatable cases should set `skipJudge: true` and validate through tool-use -(`requiredToolsUsed` / `forbiddenToolsUsed`) and SQL-argument assertions -(`toolCallArgs` with `stringIncludesAnyOf`, e.g. `['select']`, `['create table']`, -`['update', 'insert into']`). Two reasons the judge is unreliable here: - -- `list_datatables`, `get_datatable_table_schema`, and `exec_datatable_sql` - produce no drafts, and the global judge only sees the drafts artifact — it - scores a no-draft conversational answer as empty (same as the - `askUserQuestion` cases). -- Even a case that *does* produce a draft (a script reading the data table via - `wmill.datatable()` at runtime) is mis-judged: the judge has no datatable SDK - reference and penalizes correct `wmill.datatable()` usage as wrong. Verify the - SDK call deterministically instead — `requiredDrafts.valueIncludes: ['wmill.datatable(']` - plus forbidding `exec_datatable_sql` (keeping chat-time SQL distinct from - runtime SDK use). - -`stringIncludesAnyOf` is existential over calls (at least one matching call), so a -mutation case still passes when the model mixes its UPDATE/INSERT with -verification SELECTs. The in-memory engine (`datatableSqlEngine.ts`) is stateful -within a case — writes persist, so a model that re-queries to verify its -CREATE/UPDATE sees the change and does not loop. But the engine is best-effort -(SELECT returns all rows of the referenced/first table with no WHERE/projection), -so still never assert specific returned row values. Seed data via -`workspace.datatables` in the `initial` fixture (see README). - -## Deterministic validation - -Use deterministic validation only for hard failures such as: - -- missing required files -- unexpected extra files when the prompt says not to create them -- syntax errors -- unresolved flow refs -- missing required special modules or suspend config -- obvious artifact corruption - -Do not use deterministic validation to enforce one preferred implementation for broad creation tasks. - -Examples of bad hard checks: - -- exact step topology for a creation flow -- exact branch structure when the prompt only asked for routing behavior -- exact input shape when multiple reasonable shapes are acceptable - -## Judge checklist - -Every non-trivial case should have a `judgeChecklist`. - -The checklist should capture: - -- the user-visible behavior that must be present -- important constraints -- key completion criteria - -The checklist should not duplicate low-level implementation details unless they are truly required by the task. - -Good checklist items: - -- "the flow calculates the order total with 8% tax" -- "the app persists recipes appropriately for a raw Windmill app" -- "the flow reuses the existing workspace script instead of rewriting the logic" - -Bad checklist items: - -- "uses `branchone`" -- "contains a `rawscript` node" - -## When to use `expected` - -Use `expected` fixtures when the case is structure-sensitive, for example: - -- exact file creation -- exact script content -- modification cases where a specific file must change in a specific way -- cases where preserving an existing structure is part of the requirement - -Do not use a full `expected` artifact as the semantic oracle for broad creation tasks when multiple valid outputs should pass. - -## When to use `initial` - -Use `initial` when the benchmark is about: - -- editing an existing artifact -- reusing existing workspace assets -- preserving existing behavior while adding a change - -If the case is greenfield, prefer no `initial`. - -## Case design ladder - -Prefer suites that get gradually harder: - -1. trivial create case -2. realistic create case -3. reuse-existing-assets case -4. modification case -5. refactor case -6. edge-case or niche product behavior - -The last cases in a suite should cover unusual or product-specific behavior. - -## Anti-patterns - -Avoid these: - -- benchmark framing in prompts -- over-specified internal topology for creation tasks -- judge checklists that just restate implementation details -- deterministic validation that encodes one preferred solution -- fixtures that are so minimal or brittle that they create false negatives - -## Before adding a case - -Ask: - -1. Would a real user plausibly write this prompt? -2. If the model solves it in a different valid way, would the case still pass? -3. Are the hard deterministic checks only catching objectively broken output? -4. Does the `judgeChecklist` describe the real success criteria? -5. If this case fails, will the reason be understandable from the saved artifacts? +The full case format, fields, and fixture details remain in `ai_evals/README.md`.