windmill/ai_evals/AGENTS.md

AI Evals Authoring Guide

This folder contains black-box benchmark cases for:

• flow
• app
• script
• cli
• global

The goal is to test the current production prompts and guidance with realistic user requests, not to test one exact implementation shape.

Core rules

1. Write prompts like a real user request.
2. Prefer behavior, inputs, constraints, and outcomes over internal implementation details.
3. Keep deterministic validation narrow and hard.
4. Put semantic expectations in judgeChecklist.
5. Use expected fixtures only when exact structure really matters.

Prompt writing

Prompts should sound like something a user would naturally ask.

Good:

• "Create a flow that routes support requests based on customer tier."
• "Add a reset button that sets the counter back to 0."
• "Create a flow that reuses the existing greeting script instead of duplicating the logic."

Bad:

• "Use branchone with 3 branches and a default branch."
• "Create a rawscript step with this exact topology."
• "This is a benchmark harness."

Do not write prompts as if the user knows Windmill internals unless the case is explicitly testing a power-user workflow.

Flow-specific rules

This is the main principle you asked for:

• flow prompts should read like requests from a user who does not know the product internals
• the user should ask for behavior, not for branchone, branchall, rawscript, preprocessor_module, failure_module, exact graph topology, or other internal constructs

That means:

• creation cases should describe the business behavior and expected result
• modification cases may mention existing step names, because the user can see the current flow
• only mention special Windmill constructs when the case is explicitly about those constructs

Examples:

• acceptable creation prompt: "Create a purchase approval flow that pauses for approval and asks the approver for a comment."
• avoid: "Create a suspend step with one required event and a resume form."

For flow cases, do not fail a case just because the model chose a different valid topology.

App-specific rules

App prompts should focus on user-visible behavior:

• what the UI should let the user do
• what should persist
• what backend behavior is needed

Avoid prompting in terms of React structure, component names, or implementation unless the case is specifically about editing an existing app.

CLI-specific rules

CLI prompts can be more explicit about paths and file names because real CLI users often do specify them.

Still, avoid benchmark phrasing. The prompt should read like a repo task, not a harness instruction.

When relevant, ask the assistant to tell the user which wmill commands to run next. That is part of the benchmarked behavior.

Global-specific rules

Global prompts should exercise workspace-level drafting behavior:

• inspecting existing scripts, flows, apps, schedules, triggers, resources, and variables when relevant
• writing AI drafts rather than saving or deploying by default
• producing coherent multi-artifact changes when the request crosses artifact boundaries

Keep deterministic validation focused on the draft contract: required draft type/path, required content snippets, forbidden draft paths, and forbidden mutating tools such as deploy/delete unless the case explicitly asks for them.

Datatable cases should set skipJudge: true and validate through tool-use (requiredToolsUsed / forbiddenToolsUsed) and SQL-argument assertions (toolCallArgs with stringIncludesAnyOf, e.g. ['select'], ['create table'], ['update', 'insert into']). Two reasons the judge is unreliable here:

• list_datatables, get_datatable_table_schema, and exec_datatable_sql produce no drafts, and the global judge only sees the drafts artifact — it scores a no-draft conversational answer as empty (same as the askUserQuestion cases).
• Even a case that does produce a draft (a script reading the data table via wmill.datatable() at runtime) is mis-judged: the judge has no datatable SDK reference and penalizes correct wmill.datatable() usage as wrong. Verify the SDK call deterministically instead — requiredDrafts.valueIncludes: ['wmill.datatable('] plus forbidding exec_datatable_sql (keeping chat-time SQL distinct from runtime SDK use).

stringIncludesAnyOf is existential over calls (at least one matching call), so a mutation case still passes when the model mixes its UPDATE/INSERT with verification SELECTs. The in-memory engine (datatableSqlEngine.ts) is stateful within a case — writes persist, so a model that re-queries to verify its CREATE/UPDATE sees the change and does not loop. But the engine is best-effort (SELECT returns all rows of the referenced/first table with no WHERE/projection), so still never assert specific returned row values. Seed data via workspace.datatables in the initial fixture (see README).

Deterministic validation

Use deterministic validation only for hard failures such as:

• missing required files
• unexpected extra files when the prompt says not to create them
• syntax errors
• unresolved flow refs
• missing required special modules or suspend config
• obvious artifact corruption

Do not use deterministic validation to enforce one preferred implementation for broad creation tasks.

Examples of bad hard checks:

• exact step topology for a creation flow
• exact branch structure when the prompt only asked for routing behavior
• exact input shape when multiple reasonable shapes are acceptable

Judge checklist

Every non-trivial case should have a judgeChecklist.

The checklist should capture:

• the user-visible behavior that must be present
• important constraints
• key completion criteria

The checklist should not duplicate low-level implementation details unless they are truly required by the task.

Good checklist items:

• "the flow calculates the order total with 8% tax"
• "the app persists recipes appropriately for a raw Windmill app"
• "the flow reuses the existing workspace script instead of rewriting the logic"

Bad checklist items:

• "uses branchone"
• "contains a rawscript node"

When to use expected

Use expected fixtures when the case is structure-sensitive, for example:

• exact file creation
• exact script content
• modification cases where a specific file must change in a specific way
• cases where preserving an existing structure is part of the requirement

Do not use a full expected artifact as the semantic oracle for broad creation tasks when multiple valid outputs should pass.

When to use initial

Use initial when the benchmark is about:

• editing an existing artifact
• reusing existing workspace assets
• preserving existing behavior while adding a change

If the case is greenfield, prefer no initial.

Case design ladder

Prefer suites that get gradually harder:

1. trivial create case
2. realistic create case
3. reuse-existing-assets case
4. modification case
5. refactor case
6. edge-case or niche product behavior

The last cases in a suite should cover unusual or product-specific behavior.

Anti-patterns

Avoid these:

• benchmark framing in prompts
• over-specified internal topology for creation tasks
• judge checklists that just restate implementation details
• deterministic validation that encodes one preferred solution
• fixtures that are so minimal or brittle that they create false negatives

Before adding a case

Ask:

1. Would a real user plausibly write this prompt?
2. If the model solves it in a different valid way, would the case still pass?
3. Are the hard deterministic checks only catching objectively broken output?
4. Does the judgeChecklist describe the real success criteria?
5. If this case fails, will the reason be understandable from the saved artifacts?