diff --git a/ai_evals/README.md b/ai_evals/README.md index 2b3cc4465d..dec5d43a14 100644 --- a/ai_evals/README.md +++ b/ai_evals/README.md @@ -80,6 +80,15 @@ Frontend benchmark ownership also lives here now: - frontend flow/app fixtures are under `ai_evals/fixtures/frontend/` - the frontend benchmark runner is under `ai_evals/adapters/frontend/` - the frontend source tree no longer owns a separate AI chat benchmark suite +- production frontend prompt builders, tool definitions, and `runChatLoop` are still reused +- benchmark-only infrastructure also lives here: + - case loading + - variant loading + - judge scoring + - benchmark result shaping +- file-backed helper adapters also live here: + - production helpers mutate UI/editor state + - benchmark helpers mutate temp-workspace files For the concrete workflow to benchmark a CLI skill change with frozen before/after variants, see [cli/README.md](/home/farhad/windmill__worktrees/prompt-testing-plan/ai_evals/cli/README.md). diff --git a/ai_evals/cli/README.md b/ai_evals/cli/README.md index 6d88fb3f09..649f2d60eb 100644 --- a/ai_evals/cli/README.md +++ b/ai_evals/cli/README.md @@ -211,6 +211,8 @@ That means: - the benchmark entrypoint remains `ai_evals/cli` - frontend flow/app fixtures live under `ai_evals/fixtures/frontend/` - the frontend source tree no longer owns a separate AI chat benchmark suite +- production frontend prompt builders, tool definitions, and `runChatLoop` are still reused +- benchmark-only infrastructure and file-backed helper adapters live under `ai_evals` By default, the repo only ships `baseline` frontend variants. To compare distinct prompt variants or write official history snapshots, add a second @@ -256,3 +258,4 @@ Later iterations should add: - token and cost metrics in compare output - richer history views and filtering - frontend `script` surface +- optional reduction of the thin frontend Vitest bridge diff --git a/docs/system-prompt-testing-plan.md b/docs/system-prompt-testing-plan.md index b3f85f59ca..0c91375c20 100644 --- a/docs/system-prompt-testing-plan.md +++ b/docs/system-prompt-testing-plan.md @@ -156,30 +156,60 @@ The exact binary name can change, but the architecture should not: ## Temporary Bootstrap Code -Existing frontend test files under `frontend/.../__tests__/...` are acceptable -as bootstrap code while the benchmark CLI is being built. +This bootstrap phase is now complete for frontend `flow` and `app`. -They should not be treated as the final user-facing interface for prompt -evaluation. +Frontend AI benchmark ownership has moved into `ai_evals/`, and the frontend +source tree no longer owns a separate AI benchmark suite under +`frontend/.../__tests__/...`. Benchmark authors should only need the repo-level benchmark CLI to run the long-term suite. +The only temporary frontend-specific piece that remains is a thin Vitest/Vite +loader bridge so the benchmark runner can import the production chat modules in +the same module/runtime environment they already expect. + ## Frontend: What Exists Today -The current frontend evals are already **mostly decoupled from the UI**. +The current frontend benchmark path is **decoupled from the UI** and now owned +by `ai_evals`. They currently: - run through the shared headless chat loop - use production prompt builders - use production tool definitions -- use in-memory helper implementations instead of browser interactions +- use benchmark-owned helper adapters that write to temp workspaces on disk +- execute through the frontend module/runtime environment only as a loader bridge -This means the current flow and app evals are already closer to a proper black-box prompt harness than to a frontend UI test. +This means the current flow and app evals are now a proper benchmark adapter, +not a frontend test suite. That is the correct direction. +### Frontend Architecture Notes + +There are three categories of code involved: + +- shared production logic: + - production system prompt builders + - production tool definitions + - production `runChatLoop` +- benchmark-only infrastructure: + - case loading + - variant loading + - judge scoring + - benchmark result shaping + - history/reporting integration +- alternate helper adapters: + - production helpers mutate UI/editor state + - benchmark helpers mutate temp-workspace files + +This is important because the benchmark suite is **not** meant to duplicate the +frontend chat logic. It is meant to reuse the production chat loop and tool +definitions while swapping the execution backend from UI state to filesystem +state. + ## Frontend: What Is Missing ### Coverage gaps @@ -189,34 +219,34 @@ That is the correct direction. ### Reliability gaps -- Existing evals are mostly report-oriented rather than pass/fail oriented. -- They write results and logs, but they do not yet act like a proper reliability gate. -- They do not compute pass rate across repeated runs. +- Frontend flow and app can already run with pass/fail results and repeated runs through the shared benchmark CLI. +- The remaining gap is turning that into stronger routine reliability gating with better deterministic validators and broader routine case coverage. +- Frontend reliability reporting is still less mature than the intended end state for official CI tiers and richer failure triage. ### Prompt-iteration gaps -- Testing a new prompt variant is possible in code, but not ergonomic enough. -- Variants are still mostly defined in TypeScript, not in a prompt-candidate workflow that makes experimentation cheap. -- There is no simple "baseline vs candidate" benchmark command that product or prompt authors can run repeatedly. +- Frontend prompt variants are file-backed now, but the repo only ships baseline manifests by default. +- Creating and curating meaningful frontend candidate variants is still a mostly manual workflow compared with the CLI snapshot flow. +- Frontend prompt comparison exists through the shared `compare` command, but it still needs broader routine use and better variant coverage. ### Artifact-validation gaps -- The current flow and app helpers mock several effects instead of validating the resulting artifact deeply enough. +- The current flow and app helpers are file-backed now, but several effects are still lightweight and should become more realistic over time. - Linting and runnable validation are currently too lightweight in the eval path. - Datatable interactions are mocked rather than validated as output constraints. - The suite does not yet enforce a strong deterministic validator layer before using an LLM judge. ### Corpus gaps -- There is no single shared evaluation corpus for frontend surfaces. -- Cases are not yet organized as a reusable benchmark set with metadata, tags, and expected constraints. -- There is no explicit regression library built from real failures. +- Frontend surfaces already use shared case manifests under `ai_evals/cases/frontend/`. +- The remaining gap is breadth and representativeness, not the absence of a shared corpus. +- Cases still need richer metadata, stronger deterministic constraints, and a larger regression library built from real failures. ### Reporting gaps -- There is no stable result format for comparing runs over time. -- There is no leaderboard view by prompt variant, provider, model, or surface. -- There is no "worst failures first" report for debugging regressions. +- Frontend runs already emit the shared benchmark result shape and can write official history snapshots through the shared benchmark CLI. +- There is still no rich leaderboard or trend-oriented debugging workflow for frontend surfaces specifically. +- There is still no strong "worst failures first" report for debugging regressions. ## Frontend: Perfect Testing Logic @@ -242,6 +272,14 @@ The runner should directly invoke: - the production tool list - the production chat loop +It is acceptable for the benchmark adapter to use the frontend Vitest/Vite +runtime as a thin loader bridge when production chat modules still depend on +that environment, as long as: + +- the benchmark entrypoint remains the shared benchmark CLI +- the benchmark logic and fixtures live under `ai_evals` +- the frontend source tree does not own a separate benchmark suite + This keeps the suite decorrelated from the frontend UI while still testing the real AI logic. ### 2. Test the three frontend AI surfaces separately diff --git a/docs/system-prompt-testing-status.md b/docs/system-prompt-testing-status.md index 90ceffbcc4..b6d0f82ca4 100644 --- a/docs/system-prompt-testing-status.md +++ b/docs/system-prompt-testing-status.md @@ -20,6 +20,7 @@ What is true today: - the CLI surface is the first real benchmark adapter behind it - frontend flow and app are now exposed through that same benchmark CLI - frontend AI benchmark logic now lives under `ai_evals`, not under `frontend/__tests__` +- frontend benchmarks reuse production prompt builders, tool definitions, and `runChatLoop` - CLI cases are now skill-sensitive rather than only artifact-sensitive - the benchmark harness and `wmill init` now share the same AI-guidance writer - variants can be frozen as named snapshots through the benchmark CLI @@ -32,6 +33,7 @@ What is not true yet: - frontend `script` is not yet exposed through the benchmark CLI - frontend still has only baseline prompt variants wired by default +- frontend still uses a thin Vitest bridge for module/runtime loading - CLI still needs broader case coverage and richer efficiency metrics - token and cost metrics are not implemented - no UI studio exists yet @@ -198,6 +200,7 @@ Implemented: - frontend benchmark fixtures were moved under `ai_evals/fixtures/frontend/` - repo-level benchmark CLI support for `frontend-flow` and `frontend-app` - ai_evals-owned frontend benchmark runner and adapter executed through the frontend Vitest environment +- frontend benchmark helpers now write to temp workspaces instead of mutating in-memory UI state - repeated runs for frontend flow/app through the shared benchmark CLI - frontend prompt variants loaded from files under `ai_evals/variants/frontend/` - frontend run/compare payloads aligned to the shared benchmark result shape @@ -211,6 +214,7 @@ Still missing: - more than the baseline prompt variant for frontend flow/app - broader frontend reliability coverage across more cases in routine use - regular use of frontend history writes in routine benchmark workflows +- optional future cleanup to remove or reduce the thin frontend Vitest bridge if production chat modules become directly importable ### Phase 6: Add CI tiers @@ -270,6 +274,8 @@ The most important implemented changes so far are: - Validated frontend `run`, `compare`, and `compare --write-history` through the shared benchmark CLI - Moved frontend AI benchmark fixtures and runner ownership into `ai_evals` - Removed the old frontend AI benchmark test tree +- Kept the production frontend chat loop and tool-definition path as the shared execution core +- Swapped frontend benchmark helpers from in-memory state to temp-workspace files - Shared AI-guidance generation between: - benchmark temp workspaces - `wmill init` @@ -295,6 +301,7 @@ The highest-priority remaining work is: - more filtering 5. Add CI tiers. 6. Build the UI last. +7. Optionally reduce the thin frontend Vitest bridge once production chat modules are easier to load headlessly. ## Recommended Next Step