windmill/ai_evals/cli

Benchmark CLI

This directory is the start of the repo-level benchmark CLI described in docs/system-prompt-testing-plan.md.

Current Scope

The current implementation is intentionally small:

• run command
• compare command
• history command
• list-cases and list-variants discovery commands
• cli surface only

This is the benchmark entrypoint for prompt and artifact evaluation.

The CLI benchmark workspace now uses the same shared AI-guidance writer as wmill init, so benchmark runs and local CLI bootstraps go through the same project-guidance generation path.

Usage

Install dependencies once:

cd ai_evals
bun install

List available CLI cases:

cd ai_evals
bun run cli -- list-cases --surface cli

List available CLI variants:

cd ai_evals
bun run cli -- list-variants --surface cli

Snapshot the current guidance bundle into a named CLI variant:

cd ai_evals
bun run cli -- snapshot-variant --surface cli --variant candidate --description "Candidate CLI guidance bundle"

Run one CLI case:

cd ai_evals
bun run cli -- run --surface cli --case bun-hello-script --variant baseline

Keep the temp workspace for inspection:

cd ai_evals
bun run cli -- run --surface cli --case bun-hello-script --variant baseline --keep-workspace

Print machine-readable output:

cd ai_evals
bun run cli -- run --surface cli --case bun-hello-script --variant baseline --json

Run the same case multiple times to measure reliability:

cd ai_evals
bun run cli -- run --surface cli --case bun-hello-script --variant baseline --runs 5 --json

Compare two variant selections on one or more cases:

cd ai_evals
bun run cli -- compare --surface cli --case bun-hello-script --variant baseline --variant baseline --json

Write official benchmark snapshots while comparing distinct variants:

cd ai_evals
bun run cli -- compare --surface cli --case bun-hello-script --variant baseline-frozen --variant candidate --runs 5 --write-history

Inspect the tracked history:

cd ai_evals
bun run cli -- history --view latest
bun run cli -- history --view summary --limit 10

Benchmarking A CLI Skill Change

If you change one of the generated CLI skills and want to know whether the change improved the system, do not compare against the moving baseline variant alone.

baseline points at the repo's current generated skills, so it changes when the repo changes. To make a real before-vs-after comparison, snapshot both sides as named variants.

Before changing the skills, freeze the current bundle:

cd ai_evals
bun run cli -- snapshot-variant --surface cli --variant baseline-frozen --description "Frozen CLI skills before the change"

After changing and regenerating the guidance, snapshot the candidate:

cd ai_evals
bun run cli -- snapshot-variant --surface cli --variant candidate --description "CLI skills after the change"

Then compare them on one or more cases:

cd ai_evals
bun run cli -- compare --surface cli --case bun-hello-script --case bun-hello-flow --variant baseline-frozen --variant candidate --runs 5 --json

Today, the most meaningful improvement signal is still output quality:

• required artifact checks passing more often
• more cases passing
• fewer required failures across repeated runs

The benchmark CLI now reports basic reliability and efficiency-adjacent metrics for repeated runs:

• pass rate
• average duration
• average assistant message count
• average tool-call count
• average skill-invocation count
• aggregated required-check failures

When --write-history is used on compare, the benchmark CLI also writes one official snapshot per compared variant into ai_evals/history/ and rebuilds:

• summary.jsonl
• rollups/latest.json
• rollups/by_surface.json
• rollups/by_variant.json
• rollups/by_model.json

The compare output also includes tool usage and invoked skills as diagnostics.

True efficiency metrics such as latency, token usage, and cost are planned, but the current CLI does not emit them yet. Until that lands, use compare primarily to answer "did this skill bundle produce better artifacts on the same cases?"

Variant manifests can also override the top-level project instructions in addition to the skills bundle:

{
  "id": "candidate",
  "description": "Candidate guidance bundle",
  "skillsSource": {
    "type": "path",
    "path": "./snapshots/candidate-skills"
  },
  "agentsSourcePath": "./snapshots/candidate-AGENTS.md",
  "claudeSourcePath": "./snapshots/candidate-CLAUDE.md"
}

That matches the internal wmill init override env vars:

WMILL_INIT_AI_SKILLS_SOURCE=./ai_evals/variants/cli/snapshots/candidate-skills wmill init --use-default
WMILL_INIT_AI_SKILLS_SOURCE=./ai_evals/variants/cli/snapshots/candidate-skills WMILL_INIT_AI_AGENTS_SOURCE=./my-candidate-AGENTS.md wmill init --use-default

The snapshot-variant command also honors those same env vars. If they are set when you run the snapshot command, it will freeze that overridden bundle instead of the generated default.

Next Steps

Later iterations should add:

• frontend adapters
• variant cleanup and diff helpers
• token and cost metrics in compare output
• richer history views and filtering