From 455fcb0fefa3f66d55f59786b2725be162529329 Mon Sep 17 00:00:00 2001
From: centdix <farhadg110@gmail.com>
Date: Tue, 31 Mar 2026 22:23:56 +0200
Subject: [PATCH] docs: add cli benchmark comparison workflow

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
---
 ai_evals/README.md     |  3 ++
 ai_evals/cli/README.md | 81 +++++++++++++++++++++++++++++++++++++++++-
 2 files changed, 83 insertions(+), 1 deletion(-)

diff --git a/ai_evals/README.md b/ai_evals/README.md
index b03b127387..9678dad82c 100644
--- a/ai_evals/README.md
+++ b/ai_evals/README.md
@@ -63,3 +63,6 @@ bun run cli -- compare --surface cli --case bun-hello-script --variant baseline
 
 At the moment this is still intentionally small, but it is the only benchmark
 entrypoint.
+
+For the concrete workflow to benchmark a CLI skill change with frozen
+before/after variants, see [cli/README.md](/home/farhad/windmill__worktrees/prompt-testing-plan/ai_evals/cli/README.md).
diff --git a/ai_evals/cli/README.md b/ai_evals/cli/README.md
index 58a7f602e5..51bd06b097 100644
--- a/ai_evals/cli/README.md
+++ b/ai_evals/cli/README.md
@@ -8,6 +8,8 @@ This directory is the start of the repo-level benchmark CLI described in
 The current implementation is intentionally small:
 
 - `run` command
+- `compare` command
+- `list-cases` and `list-variants` discovery commands
 - `cli` surface only
 
 This is the benchmark entrypoint for prompt and artifact evaluation.
@@ -63,11 +65,88 @@ cd ai_evals
 bun run cli -- compare --surface cli --case bun-hello-script --variant baseline --variant baseline --json
 ```
 
+## Benchmarking A CLI Skill Change
+
+If you change one of the generated CLI skills and want to know whether the
+change improved the system, do not compare against the moving `baseline`
+variant alone.
+
+`baseline` points at the repo's current generated skills, so it changes when
+the repo changes. To make a real before-vs-after comparison, freeze both sides
+as path-based variants.
+
+Create a snapshot directory:
+
+```bash
+mkdir -p ai_evals/variants/cli/snapshots
+```
+
+Before changing the skill, snapshot the current generated skills:
+
+```bash
+cp -R system_prompts/auto-generated/skills ai_evals/variants/cli/snapshots/baseline-skills
+```
+
+Create a frozen baseline variant manifest in
+`ai_evals/variants/cli/baseline-frozen.json`:
+
+```json
+{
+  "id": "baseline-frozen",
+  "description": "Frozen CLI skills before the change",
+  "skillsSource": {
+    "type": "path",
+    "path": "./snapshots/baseline-skills"
+  }
+}
+```
+
+After changing and regenerating the skills, snapshot the candidate:
+
+```bash
+cp -R system_prompts/auto-generated/skills ai_evals/variants/cli/snapshots/candidate-skills
+```
+
+Create `ai_evals/variants/cli/candidate.json`:
+
+```json
+{
+  "id": "candidate",
+  "description": "CLI skills after the change",
+  "skillsSource": {
+    "type": "path",
+    "path": "./snapshots/candidate-skills"
+  }
+}
+```
+
+Then compare them on one or more cases:
+
+```bash
+cd ai_evals
+bun run cli -- compare --surface cli --case bun-hello-script --case bun-hello-flow --variant baseline-frozen --variant candidate --json
+```
+
+Today, the most meaningful improvement signal is still output quality:
+
+- required artifact checks passing more often
+- more cases passing
+- fewer required failures across repeated runs
+
+The compare output also includes tool usage and invoked skills as diagnostics.
+
+True efficiency metrics such as latency, token usage, and cost are planned, but
+the current CLI does not emit them yet. Until that lands, use `compare`
+primarily to answer "did this skill bundle produce better artifacts on the same
+cases?"
+
 ## Next Steps
 
 Later iterations should add:
 
-- `compare` command
 - `history` command
 - frontend adapters
+- repeated-run reliability mode
+- frozen-variant helper commands
+- latency, token, and cost metrics in compare output
 - shared result/history writing from this entrypoint