mirror of
https://github.com/windmill-labs/windmill.git
synced 2026-08-18 16:02:10 +00:00
docs: add ai-chat and ai-evals skills (#9770)
Add two agent skills under .agents/skills (symlinked into .claude/skills): - ai-chat: guidance for improving the Windmill AI chat / copilot, especially global mode — benchmark before/after with ai_evals, optimize finalContextTokens over cumulative, keep tool params and tool-result payloads minimal (no echoing content the model already has), treat prompts/tool-descriptions as benchmarkable surface. - ai-evals: author and run black-box benchmark cases for the AI generation modes, migrated from ai_evals/AGENTS.md and extended with run mechanics (workspace reuse, reading the summary). ai_evals/AGENTS.md becomes a pointer stub to the skill. Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
centdix
GitHub
@@ -0,0 +1,40 @@
|
||||
---
|
||||
name: ai-chat
|
||||
description: Guidance for improving the Windmill AI chat (copilot), especially global mode — tools, prompts, and context-window discipline. Use when editing chat tools, system prompts, or tool-result shapes under frontend/src/lib/components/copilot/chat, or when changing how the chat manages its context window.
|
||||
---
|
||||
|
||||
## Always benchmark before and after
|
||||
|
||||
No context or behavior change ships without an `ai_evals` A/B on the affected mode.
|
||||
Add or adjust cases for exactly what you changed — see the `ai-evals` skill for
|
||||
authoring and the full run reference.
|
||||
|
||||
Run the affected mode **before** your change and **after**, same model(s), same cases.
|
||||
|
||||
## Measure the window first, and cumulative second
|
||||
|
||||
Optimize **`finalContextTokens`** (window occupancy — what drives overflow and
|
||||
compaction), then cumulative prompt tokens.
|
||||
|
||||
## Context discipline
|
||||
|
||||
The dominant fixed cost is per-iteration overhead: the system prompt **plus every
|
||||
tool schema** is re-sent on every loop iteration. So:
|
||||
|
||||
- **Every tool and every parameter is a permanent tax.** Justify each one and measure
|
||||
it; an extra "locate" round-trip can cost more than the reads it saves. Strip dead
|
||||
params rather than leaving them in the schema.
|
||||
- **Tool results return the minimum.** Never echo content the model already has. The
|
||||
canonical mistake: a write tool that returns the whole edited artifact right after
|
||||
the model authored it — return `{ success, message }` instead. When you touch a
|
||||
*shared* write helper (e.g. `finishAppDraftWrite` in `global/core.ts`), re-check
|
||||
this invariant for **all** the write tools routing through it — the echo has
|
||||
regressed before via a shared refactor.
|
||||
|
||||
## Prompts and tool descriptions are part of the surface
|
||||
|
||||
The system prompt and tool descriptions steer behavior as much as the tools
|
||||
themselves, and are benchmarkable the same way. A description that advertises
|
||||
truncation makes the model self-limit; the path-conventions block changes where
|
||||
drafts land. Treat prompt/description edits as real changes and A/B them — a
|
||||
pure-prompt change is a legitimate, measurable improvement.
|
||||
@@ -0,0 +1,87 @@
|
||||
---
|
||||
name: ai-evals
|
||||
description: Author and run black-box benchmark cases for the Windmill AI generation modes (flow/app/script/cli/global) in ai_evals/. Use when adding or changing eval cases, or when running before/after benchmarks for AI chat / copilot changes.
|
||||
---
|
||||
|
||||
# AI evals — authoring and running benchmark cases
|
||||
|
||||
`ai_evals/` is a black-box benchmark runner for the Windmill AI generation modes:
|
||||
`flow`, `app`, `script`, `cli`, `global`. It always tests the **current** production
|
||||
prompts, tools, and guidance in this checkout. Each attempt runs the real production
|
||||
path, deterministic validation, then LLM judging.
|
||||
|
||||
The goal is to test current production guidance with realistic user requests — **not**
|
||||
to pin one exact implementation shape.
|
||||
|
||||
## Running benchmarks
|
||||
|
||||
```bash
|
||||
cd ai_evals
|
||||
bun install # first time; frontend modes also need `cd frontend && bun install`
|
||||
bun run cli -- models # list model aliases
|
||||
bun run cli -- cases global # list cases for a mode
|
||||
bun run cli -- run global global-test1-script-create --model sonnet
|
||||
```
|
||||
|
||||
Frontend modes (`flow`/`script`/`app`/`global`) route model calls through a Windmill
|
||||
backend's `/api/w/<ws>/ai/proxy`, so you need **any** reachable backend:
|
||||
|
||||
```bash
|
||||
WMILL_AI_EVAL_BACKEND_URL=http://127.0.0.1:<port> WMILL_AI_EVAL_BACKEND_WORKSPACE=integration-tests \
|
||||
bun run cli -- run global <caseIds...> --models sonnet,gpt-5.5,gemini-3.1-pro-preview
|
||||
```
|
||||
|
||||
- **Reuse an existing workspace.** CE builds cap workspaces, so temp-workspace
|
||||
creation 400s ("reached workspace limit"). Always set
|
||||
`WMILL_AI_EVAL_BACKEND_WORKSPACE=integration-tests` (or any existing workspace) to
|
||||
reuse one. The only side effect of a run is upserting an `f/evals/ai/<provider>`
|
||||
resource there.
|
||||
- Provider keys live in `ai_evals/.env` and are auto-loaded by bun. The judge is a
|
||||
separate Anthropic call (default `claude-sonnet-4-6`) regardless of the model under
|
||||
test.
|
||||
|
||||
## Authoring core rules
|
||||
|
||||
1. Write prompts like a real user request.
|
||||
2. Prefer behavior, inputs, constraints, and outcomes over internal implementation.
|
||||
3. Keep deterministic validation narrow and hard.
|
||||
4. Put semantic expectations in `judgeChecklist`.
|
||||
5. Use `expected` fixtures only when exact structure really matters.
|
||||
|
||||
### Prompt writing
|
||||
|
||||
Prompts should sound like something a user would naturally ask. Do not write prompts
|
||||
as if the user knows Windmill internals unless the case explicitly tests a power-user
|
||||
workflow.
|
||||
|
||||
Good:
|
||||
- "Create a flow that routes support requests based on customer tier."
|
||||
- "Add a reset button that sets the counter back to 0."
|
||||
- "Create a flow that reuses the existing greeting script instead of duplicating the logic."
|
||||
|
||||
Bad:
|
||||
- "Use `branchone` with 3 branches and a default branch."
|
||||
- "Create a `rawscript` step with this exact topology."
|
||||
- "This is a benchmark harness."
|
||||
|
||||
### Deterministic validation
|
||||
|
||||
Use deterministic checks only for hard failures: missing required files; unexpected
|
||||
extra files when the prompt says not to create them; syntax errors; unresolved flow
|
||||
refs; missing required special modules or suspend config; obvious corruption.
|
||||
|
||||
Do **not** encode one preferred implementation. Bad hard checks: exact step topology
|
||||
for a creation flow; exact branch structure when the prompt only asked for routing;
|
||||
exact input shape when multiple reasonable shapes are acceptable.
|
||||
|
||||
### Judge checklist
|
||||
|
||||
Every non-trivial case should have a `judgeChecklist` capturing user-visible behavior
|
||||
that must be present, important constraints, and key completion criteria — not
|
||||
low-level implementation details unless truly required.
|
||||
|
||||
Good: "the flow calculates the order total with 8% tax"; "the flow reuses the existing
|
||||
workspace script instead of rewriting the logic". Bad: "uses `branchone`"; "contains a
|
||||
`rawscript` node".
|
||||
|
||||
See `ai_evals/README.md` for the full case format, fields, and fixture details.
|
||||
Symlink
+1
@@ -0,0 +1 @@
|
||||
../../../.agents/skills/ai-chat/SKILL.md
|
||||
Symlink
+1
@@ -0,0 +1 @@
|
||||
../../../.agents/skills/ai-evals/SKILL.md
|
||||
+10
-204
@@ -1,208 +1,14 @@
|
||||
# AI Evals Authoring Guide
|
||||
# AI Evals
|
||||
|
||||
This folder contains black-box benchmark cases for:
|
||||
Black-box benchmark cases for the Windmill AI generation modes (`flow`, `app`,
|
||||
`script`, `cli`, `global`).
|
||||
|
||||
- `flow`
|
||||
- `app`
|
||||
- `script`
|
||||
- `cli`
|
||||
- `global`
|
||||
**Authoring and running cases is documented in the `ai-evals` skill** — load it
|
||||
before adding/changing a case or running a benchmark. Claude Code reads
|
||||
`.claude/skills/ai-evals/SKILL.md`; Codex and Pi read
|
||||
`.agents/skills/ai-evals/SKILL.md` (same canonical file). Invoke with `/ai-evals` in
|
||||
Claude Code, `$ai-evals` in Codex, or `pi --skill ai-evals`.
|
||||
|
||||
The goal is to test the current production prompts and guidance with realistic user requests, not to test one exact implementation shape.
|
||||
For AI chat / copilot changes that these evals measure, see the `ai-chat` skill.
|
||||
|
||||
## Core rules
|
||||
|
||||
1. Write prompts like a real user request.
|
||||
2. Prefer behavior, inputs, constraints, and outcomes over internal implementation details.
|
||||
3. Keep deterministic validation narrow and hard.
|
||||
4. Put semantic expectations in `judgeChecklist`.
|
||||
5. Use `expected` fixtures only when exact structure really matters.
|
||||
|
||||
## Prompt writing
|
||||
|
||||
Prompts should sound like something a user would naturally ask.
|
||||
|
||||
Good:
|
||||
|
||||
- "Create a flow that routes support requests based on customer tier."
|
||||
- "Add a reset button that sets the counter back to 0."
|
||||
- "Create a flow that reuses the existing greeting script instead of duplicating the logic."
|
||||
|
||||
Bad:
|
||||
|
||||
- "Use `branchone` with 3 branches and a default branch."
|
||||
- "Create a `rawscript` step with this exact topology."
|
||||
- "This is a benchmark harness."
|
||||
|
||||
Do not write prompts as if the user knows Windmill internals unless the case is explicitly testing a power-user workflow.
|
||||
|
||||
## Flow-specific rules
|
||||
|
||||
This is the main principle you asked for:
|
||||
|
||||
- flow prompts should read like requests from a user who does not know the product internals
|
||||
- the user should ask for behavior, not for `branchone`, `branchall`, `rawscript`, `preprocessor_module`, `failure_module`, exact graph topology, or other internal constructs
|
||||
|
||||
That means:
|
||||
|
||||
- creation cases should describe the business behavior and expected result
|
||||
- modification cases may mention existing step names, because the user can see the current flow
|
||||
- only mention special Windmill constructs when the case is explicitly about those constructs
|
||||
|
||||
Examples:
|
||||
|
||||
- acceptable creation prompt:
|
||||
"Create a purchase approval flow that pauses for approval and asks the approver for a comment."
|
||||
- avoid:
|
||||
"Create a suspend step with one required event and a resume form."
|
||||
|
||||
For flow cases, do not fail a case just because the model chose a different valid topology.
|
||||
|
||||
## App-specific rules
|
||||
|
||||
App prompts should focus on user-visible behavior:
|
||||
|
||||
- what the UI should let the user do
|
||||
- what should persist
|
||||
- what backend behavior is needed
|
||||
|
||||
Avoid prompting in terms of React structure, component names, or implementation unless the case is specifically about editing an existing app.
|
||||
|
||||
## CLI-specific rules
|
||||
|
||||
CLI prompts can be more explicit about paths and file names because real CLI users often do specify them.
|
||||
|
||||
Still, avoid benchmark phrasing. The prompt should read like a repo task, not a harness instruction.
|
||||
|
||||
When relevant, ask the assistant to tell the user which `wmill` commands to run next. That is part of the benchmarked behavior.
|
||||
|
||||
## Global-specific rules
|
||||
|
||||
Global prompts should exercise workspace-level drafting behavior:
|
||||
|
||||
- inspecting existing scripts, flows, apps, schedules, triggers, resources, and variables when relevant
|
||||
- writing AI drafts rather than saving or deploying by default
|
||||
- producing coherent multi-artifact changes when the request crosses artifact boundaries
|
||||
|
||||
Keep deterministic validation focused on the draft contract: required draft type/path, required content snippets, forbidden draft paths, and forbidden mutating tools such as deploy/delete unless the case explicitly asks for them.
|
||||
|
||||
Datatable cases should set `skipJudge: true` and validate through tool-use
|
||||
(`requiredToolsUsed` / `forbiddenToolsUsed`) and SQL-argument assertions
|
||||
(`toolCallArgs` with `stringIncludesAnyOf`, e.g. `['select']`, `['create table']`,
|
||||
`['update', 'insert into']`). Two reasons the judge is unreliable here:
|
||||
|
||||
- `list_datatables`, `get_datatable_table_schema`, and `exec_datatable_sql`
|
||||
produce no drafts, and the global judge only sees the drafts artifact — it
|
||||
scores a no-draft conversational answer as empty (same as the
|
||||
`askUserQuestion` cases).
|
||||
- Even a case that *does* produce a draft (a script reading the data table via
|
||||
`wmill.datatable()` at runtime) is mis-judged: the judge has no datatable SDK
|
||||
reference and penalizes correct `wmill.datatable()` usage as wrong. Verify the
|
||||
SDK call deterministically instead — `requiredDrafts.valueIncludes: ['wmill.datatable(']`
|
||||
plus forbidding `exec_datatable_sql` (keeping chat-time SQL distinct from
|
||||
runtime SDK use).
|
||||
|
||||
`stringIncludesAnyOf` is existential over calls (at least one matching call), so a
|
||||
mutation case still passes when the model mixes its UPDATE/INSERT with
|
||||
verification SELECTs. The in-memory engine (`datatableSqlEngine.ts`) is stateful
|
||||
within a case — writes persist, so a model that re-queries to verify its
|
||||
CREATE/UPDATE sees the change and does not loop. But the engine is best-effort
|
||||
(SELECT returns all rows of the referenced/first table with no WHERE/projection),
|
||||
so still never assert specific returned row values. Seed data via
|
||||
`workspace.datatables` in the `initial` fixture (see README).
|
||||
|
||||
## Deterministic validation
|
||||
|
||||
Use deterministic validation only for hard failures such as:
|
||||
|
||||
- missing required files
|
||||
- unexpected extra files when the prompt says not to create them
|
||||
- syntax errors
|
||||
- unresolved flow refs
|
||||
- missing required special modules or suspend config
|
||||
- obvious artifact corruption
|
||||
|
||||
Do not use deterministic validation to enforce one preferred implementation for broad creation tasks.
|
||||
|
||||
Examples of bad hard checks:
|
||||
|
||||
- exact step topology for a creation flow
|
||||
- exact branch structure when the prompt only asked for routing behavior
|
||||
- exact input shape when multiple reasonable shapes are acceptable
|
||||
|
||||
## Judge checklist
|
||||
|
||||
Every non-trivial case should have a `judgeChecklist`.
|
||||
|
||||
The checklist should capture:
|
||||
|
||||
- the user-visible behavior that must be present
|
||||
- important constraints
|
||||
- key completion criteria
|
||||
|
||||
The checklist should not duplicate low-level implementation details unless they are truly required by the task.
|
||||
|
||||
Good checklist items:
|
||||
|
||||
- "the flow calculates the order total with 8% tax"
|
||||
- "the app persists recipes appropriately for a raw Windmill app"
|
||||
- "the flow reuses the existing workspace script instead of rewriting the logic"
|
||||
|
||||
Bad checklist items:
|
||||
|
||||
- "uses `branchone`"
|
||||
- "contains a `rawscript` node"
|
||||
|
||||
## When to use `expected`
|
||||
|
||||
Use `expected` fixtures when the case is structure-sensitive, for example:
|
||||
|
||||
- exact file creation
|
||||
- exact script content
|
||||
- modification cases where a specific file must change in a specific way
|
||||
- cases where preserving an existing structure is part of the requirement
|
||||
|
||||
Do not use a full `expected` artifact as the semantic oracle for broad creation tasks when multiple valid outputs should pass.
|
||||
|
||||
## When to use `initial`
|
||||
|
||||
Use `initial` when the benchmark is about:
|
||||
|
||||
- editing an existing artifact
|
||||
- reusing existing workspace assets
|
||||
- preserving existing behavior while adding a change
|
||||
|
||||
If the case is greenfield, prefer no `initial`.
|
||||
|
||||
## Case design ladder
|
||||
|
||||
Prefer suites that get gradually harder:
|
||||
|
||||
1. trivial create case
|
||||
2. realistic create case
|
||||
3. reuse-existing-assets case
|
||||
4. modification case
|
||||
5. refactor case
|
||||
6. edge-case or niche product behavior
|
||||
|
||||
The last cases in a suite should cover unusual or product-specific behavior.
|
||||
|
||||
## Anti-patterns
|
||||
|
||||
Avoid these:
|
||||
|
||||
- benchmark framing in prompts
|
||||
- over-specified internal topology for creation tasks
|
||||
- judge checklists that just restate implementation details
|
||||
- deterministic validation that encodes one preferred solution
|
||||
- fixtures that are so minimal or brittle that they create false negatives
|
||||
|
||||
## Before adding a case
|
||||
|
||||
Ask:
|
||||
|
||||
1. Would a real user plausibly write this prompt?
|
||||
2. If the model solves it in a different valid way, would the case still pass?
|
||||
3. Are the hard deterministic checks only catching objectively broken output?
|
||||
4. Does the `judgeChecklist` describe the real success criteria?
|
||||
5. If this case fails, will the reason be understandable from the saved artifacts?
|
||||
The full case format, fields, and fixture details remain in `ai_evals/README.md`.
|
||||
|
||||
Reference in New Issue
Block a user