diff --git a/ai_evals/README.md b/ai_evals/README.md index 854be5c660..1cb983983c 100644 --- a/ai_evals/README.md +++ b/ai_evals/README.md @@ -11,9 +11,6 @@ internal implementation details of the frontend or CLI. The intended end state is a new repo-level benchmark CLI for running the shared eval suite across both the Windmill CLI and the frontend. -Current code under `cli/test-skills/` should be treated as bootstrap/prototype -implementation work, not the final entrypoint. - ## Layout - `cli/`: repo-level benchmark CLI entrypoint and orchestration @@ -57,9 +54,10 @@ bun install Current usage: ```bash -bun ai_evals/cli/index.ts list-cases --surface cli -bun ai_evals/cli/index.ts run --surface cli --case bun-hello-script +cd ai_evals +bun run cli -- list-cases --surface cli +bun run cli -- run --surface cli --case bun-hello-script ``` -At the moment this is intentionally thin and delegates to the bootstrap CLI -adapter code under `cli/test-skills/`. +At the moment this is still intentionally small, but it is the only benchmark +entrypoint. diff --git a/ai_evals/cli/README.md b/ai_evals/cli/README.md index bb568bed41..b65975bca6 100644 --- a/ai_evals/cli/README.md +++ b/ai_evals/cli/README.md @@ -9,10 +9,8 @@ The current implementation is intentionally small: - `run` command - `cli` surface only -- runs the existing CLI artifact-eval adapter from `cli/test-skills/` -This is a migration step away from using `cli/test-skills/` as the primary -entrypoint. +This is the benchmark entrypoint for prompt and artifact evaluation. ## Usage @@ -26,25 +24,29 @@ bun install List available CLI cases: ```bash -bun ai_evals/cli/index.ts list-cases --surface cli +cd ai_evals +bun run cli -- list-cases --surface cli ``` Run one CLI case: ```bash -bun ai_evals/cli/index.ts run --surface cli --case bun-hello-script +cd ai_evals +bun run cli -- run --surface cli --case bun-hello-script ``` Keep the temp workspace for inspection: ```bash -bun ai_evals/cli/index.ts run --surface cli --case bun-hello-script --keep-workspace +cd ai_evals +bun run cli -- run --surface cli --case bun-hello-script --keep-workspace ``` Print machine-readable output: ```bash -bun ai_evals/cli/index.ts run --surface cli --case bun-hello-script --json +cd ai_evals +bun run cli -- run --surface cli --case bun-hello-script --json ``` ## Next Steps diff --git a/ai_evals/cli/index.ts b/ai_evals/cli/index.ts index e2f7965abe..44ef8c8e02 100644 --- a/ai_evals/cli/index.ts +++ b/ai_evals/cli/index.ts @@ -237,10 +237,10 @@ function printHelp() { process.stdout.write( [ "Usage:", - " bun ai_evals/cli/index.ts list-cases --surface cli [--json]", - " bun ai_evals/cli/index.ts run --surface cli --case [--json] [--keep-workspace]", - " bun ai_evals/cli/index.ts compare", - " bun ai_evals/cli/index.ts history", + " cd ai_evals && bun run cli -- list-cases --surface cli [--json]", + " cd ai_evals && bun run cli -- run --surface cli --case [--json] [--keep-workspace]", + " cd ai_evals && bun run cli -- compare", + " cd ai_evals && bun run cli -- history", "", "Current support:", " surfaces: cli" diff --git a/ai_evals/package.json b/ai_evals/package.json index 301b094dc2..dffa49ee2a 100644 --- a/ai_evals/package.json +++ b/ai_evals/package.json @@ -3,6 +3,7 @@ "private": true, "type": "module", "scripts": { + "cli": "bun cli/index.ts", "list-cases": "bun cli/index.ts list-cases --surface cli" }, "dependencies": { diff --git a/cli/test-skills/README.md b/cli/test-skills/README.md deleted file mode 100644 index 6b0221e54d..0000000000 --- a/cli/test-skills/README.md +++ /dev/null @@ -1,110 +0,0 @@ -# Windmill CLI Prompt Tests - -Test suite for verifying how Claude Code behaves with Windmill auto-generated -skills. It currently contains both skill-invocation smoke tests and the first -artifact-evaluation benchmark. - -## Overview - -This framework sends prompts through the Claude Agent SDK using the repo's -generated Windmill skills. - -It currently supports: - -- skill-invocation smoke tests -- CLI artifact evaluation in an isolated temp workspace - -## Prerequisites - -- [Bun](https://bun.sh/) installed -- `ANTHROPIC_API_KEY` environment variable set -- Auto-generated Windmill skills available under `system_prompts/auto-generated/skills/` - -## User Setup - -1. Generate the latest system prompts and CLI skills in the repo: - -```bash -python3 system_prompts/generate.py -``` - -2. Set your API key: -```bash -export ANTHROPIC_API_KEY=your-key-here -``` - -3. Install dependencies and run tests: -```bash -cd cli/test-skills -bun install -bun test -``` - -## Expected Skills - -The tests expect the following auto-generated skills to be present: - -| Skill Name | Purpose | -|------------|---------| -| `write-flow` | Creating Windmill flows/workflows | -| `write-script-python3` | Creating Python scripts | -| `write-script-bun` | Creating TypeScript/Bun scripts | -| `schedules` | Configuring schedules and cron jobs | -| `triggers` | Setting up triggers (webhook, Kafka, etc.) | - -## Test Matrix - -| Prompt | Expected Skill | -|--------|----------------| -| "Create a flow to process user data" | `write-flow` | -| "Build a workflow that fetches and transforms data" | `write-flow` | -| "Write a Python script to fetch API data" | `write-script-python3` | -| "Create a Python function to process CSV files" | `write-script-python3` | -| "Write a TypeScript script using Bun" | `write-script-bun` | -| "Create a Bun script to handle webhooks" | `write-script-bun` | -| "Set up a schedule to run this daily at midnight" | `schedules` | -| "Configure a cron job to run every hour" | `schedules` | -| "Set up a webhook trigger for this flow" | `triggers` | -| "Configure a Kafka trigger" | `triggers` | - -## Running Tests - -Run all tests: -```bash -bun test -``` - -Run only skill invocation tests: -```bash -bun test:skills -``` - -Run the first artifact-evaluation benchmark: -```bash -bun test:artifact -``` - -## Test Utilities - -The `src/test-utils.ts` module provides: - -- `runPromptAndCapture(prompt, cwd?, maxTurns)` - Runs a prompt and captures tool invocations -- `wasToolUsed(result, toolName)` - Checks if a specific tool was used -- `wasSkillInvoked(result, skillName)` - Checks if a specific skill was invoked -- `getToolInputs(result, toolName)` - Gets all inputs for a specific tool -- `getTestSkillsDir()` - Returns the test-skills directory path - -The `src/artifact-eval.ts` module provides: - -- temp-workspace creation with generated skills -- prompt rendering with workspace-root placeholders -- file-based artifact scoring for benchmark cases - -## Notes - -- Tests have extended timeouts (120 seconds) due to API latency -- Tests run against the actual Claude API, so they consume API credits -- Skill-invocation smoke tests still use `test-folder/` -- Artifact evals use isolated temp workspaces under `/tmp` -- The first artifact benchmark uses an explicit absolute target path so file - outputs are scoreable even when Claude executes inside a skill context diff --git a/cli/test-skills/bun.lock b/cli/test-skills/bun.lock deleted file mode 100644 index 9e4443cca8..0000000000 --- a/cli/test-skills/bun.lock +++ /dev/null @@ -1,61 +0,0 @@ -{ - "lockfileVersion": 1, - "configVersion": 1, - "workspaces": { - "": { - "name": "claude-code-skill-tests", - "dependencies": { - "@anthropic-ai/claude-agent-sdk": "^0.2.25", - }, - "devDependencies": { - "@types/bun": "latest", - "typescript": "^5.0.0", - }, - }, - }, - "packages": { - "@anthropic-ai/claude-agent-sdk": ["@anthropic-ai/claude-agent-sdk@0.2.25", "", { "optionalDependencies": { "@img/sharp-darwin-arm64": "^0.33.5", "@img/sharp-darwin-x64": "^0.33.5", "@img/sharp-linux-arm": "^0.33.5", "@img/sharp-linux-arm64": "^0.33.5", "@img/sharp-linux-x64": "^0.33.5", "@img/sharp-linuxmusl-arm64": "^0.33.5", "@img/sharp-linuxmusl-x64": "^0.33.5", "@img/sharp-win32-x64": "^0.33.5" }, "peerDependencies": { "zod": "^4.0.0" } }, "sha512-YIP3I40+XSkC3zE1Z8KRQY02VA7UfofFamF1cFrLe7FbtCnjpslyDl9coGBh2DAi9xj2yQcKZZf751jEWpB+dQ=="], - - "@img/sharp-darwin-arm64": ["@img/sharp-darwin-arm64@0.33.5", "", { "optionalDependencies": { "@img/sharp-libvips-darwin-arm64": "1.0.4" }, "os": "darwin", "cpu": "arm64" }, "sha512-UT4p+iz/2H4twwAoLCqfA9UH5pI6DggwKEGuaPy7nCVQ8ZsiY5PIcrRvD1DzuY3qYL07NtIQcWnBSY/heikIFQ=="], - - "@img/sharp-darwin-x64": ["@img/sharp-darwin-x64@0.33.5", "", { "optionalDependencies": { "@img/sharp-libvips-darwin-x64": "1.0.4" }, "os": "darwin", "cpu": "x64" }, "sha512-fyHac4jIc1ANYGRDxtiqelIbdWkIuQaI84Mv45KvGRRxSAa7o7d1ZKAOBaYbnepLC1WqxfpimdeWfvqqSGwR2Q=="], - - "@img/sharp-libvips-darwin-arm64": ["@img/sharp-libvips-darwin-arm64@1.0.4", "", { "os": "darwin", "cpu": "arm64" }, "sha512-XblONe153h0O2zuFfTAbQYAX2JhYmDHeWikp1LM9Hul9gVPjFY427k6dFEcOL72O01QxQsWi761svJ/ev9xEDg=="], - - "@img/sharp-libvips-darwin-x64": ["@img/sharp-libvips-darwin-x64@1.0.4", "", { "os": "darwin", "cpu": "x64" }, "sha512-xnGR8YuZYfJGmWPvmlunFaWJsb9T/AO2ykoP3Fz/0X5XV2aoYBPkX6xqCQvUTKKiLddarLaxpzNe+b1hjeWHAQ=="], - - "@img/sharp-libvips-linux-arm": ["@img/sharp-libvips-linux-arm@1.0.5", "", { "os": "linux", "cpu": "arm" }, "sha512-gvcC4ACAOPRNATg/ov8/MnbxFDJqf/pDePbBnuBDcjsI8PssmjoKMAz4LtLaVi+OnSb5FK/yIOamqDwGmXW32g=="], - - "@img/sharp-libvips-linux-arm64": ["@img/sharp-libvips-linux-arm64@1.0.4", "", { "os": "linux", "cpu": "arm64" }, "sha512-9B+taZ8DlyyqzZQnoeIvDVR/2F4EbMepXMc/NdVbkzsJbzkUjhXv/70GQJ7tdLA4YJgNP25zukcxpX2/SueNrA=="], - - "@img/sharp-libvips-linux-x64": ["@img/sharp-libvips-linux-x64@1.0.4", "", { "os": "linux", "cpu": "x64" }, "sha512-MmWmQ3iPFZr0Iev+BAgVMb3ZyC4KeFc3jFxnNbEPas60e1cIfevbtuyf9nDGIzOaW9PdnDciJm+wFFaTlj5xYw=="], - - "@img/sharp-libvips-linuxmusl-arm64": ["@img/sharp-libvips-linuxmusl-arm64@1.0.4", "", { "os": "linux", "cpu": "arm64" }, "sha512-9Ti+BbTYDcsbp4wfYib8Ctm1ilkugkA/uscUn6UXK1ldpC1JjiXbLfFZtRlBhjPZ5o1NCLiDbg8fhUPKStHoTA=="], - - "@img/sharp-libvips-linuxmusl-x64": ["@img/sharp-libvips-linuxmusl-x64@1.0.4", "", { "os": "linux", "cpu": "x64" }, "sha512-viYN1KX9m+/hGkJtvYYp+CCLgnJXwiQB39damAO7WMdKWlIhmYTfHjwSbQeUK/20vY154mwezd9HflVFM1wVSw=="], - - "@img/sharp-linux-arm": ["@img/sharp-linux-arm@0.33.5", "", { "optionalDependencies": { "@img/sharp-libvips-linux-arm": "1.0.5" }, "os": "linux", "cpu": "arm" }, "sha512-JTS1eldqZbJxjvKaAkxhZmBqPRGmxgu+qFKSInv8moZ2AmT5Yib3EQ1c6gp493HvrvV8QgdOXdyaIBrhvFhBMQ=="], - - "@img/sharp-linux-arm64": ["@img/sharp-linux-arm64@0.33.5", "", { "optionalDependencies": { "@img/sharp-libvips-linux-arm64": "1.0.4" }, "os": "linux", "cpu": "arm64" }, "sha512-JMVv+AMRyGOHtO1RFBiJy/MBsgz0x4AWrT6QoEVVTyh1E39TrCUpTRI7mx9VksGX4awWASxqCYLCV4wBZHAYxA=="], - - "@img/sharp-linux-x64": ["@img/sharp-linux-x64@0.33.5", "", { "optionalDependencies": { "@img/sharp-libvips-linux-x64": "1.0.4" }, "os": "linux", "cpu": "x64" }, "sha512-opC+Ok5pRNAzuvq1AG0ar+1owsu842/Ab+4qvU879ippJBHvyY5n2mxF1izXqkPYlGuP/M556uh53jRLJmzTWA=="], - - "@img/sharp-linuxmusl-arm64": ["@img/sharp-linuxmusl-arm64@0.33.5", "", { "optionalDependencies": { "@img/sharp-libvips-linuxmusl-arm64": "1.0.4" }, "os": "linux", "cpu": "arm64" }, "sha512-XrHMZwGQGvJg2V/oRSUfSAfjfPxO+4DkiRh6p2AFjLQztWUuY/o8Mq0eMQVIY7HJ1CDQUJlxGGZRw1a5bqmd1g=="], - - "@img/sharp-linuxmusl-x64": ["@img/sharp-linuxmusl-x64@0.33.5", "", { "optionalDependencies": { "@img/sharp-libvips-linuxmusl-x64": "1.0.4" }, "os": "linux", "cpu": "x64" }, "sha512-WT+d/cgqKkkKySYmqoZ8y3pxx7lx9vVejxW/W4DOFMYVSkErR+w7mf2u8m/y4+xHe7yY9DAXQMWQhpnMuFfScw=="], - - "@img/sharp-win32-x64": ["@img/sharp-win32-x64@0.33.5", "", { "os": "win32", "cpu": "x64" }, "sha512-MpY/o8/8kj+EcnxwvrP4aTJSWw/aZ7JIGR4aBeZkZw5B7/Jn+tY9/VNwtcoGmdT7GfggGIU4kygOMSbYnOrAbg=="], - - "@types/bun": ["@types/bun@1.3.8", "", { "dependencies": { "bun-types": "1.3.8" } }, "sha512-3LvWJ2q5GerAXYxO2mffLTqOzEu5qnhEAlh48Vnu8WQfnmSwbgagjGZV6BoHKJztENYEDn6QmVd949W4uESRJA=="], - - "@types/node": ["@types/node@25.1.0", "", { "dependencies": { "undici-types": "~7.16.0" } }, "sha512-t7frlewr6+cbx+9Ohpl0NOTKXZNV9xHRmNOvql47BFJKcEG1CxtxlPEEe+gR9uhVWM4DwhnvTF110mIL4yP9RA=="], - - "bun-types": ["bun-types@1.3.8", "", { "dependencies": { "@types/node": "*" } }, "sha512-fL99nxdOWvV4LqjmC+8Q9kW3M4QTtTR1eePs94v5ctGqU8OeceWrSUaRw3JYb7tU3FkMIAjkueehrHPPPGKi5Q=="], - - "typescript": ["typescript@5.9.3", "", { "bin": { "tsc": "bin/tsc", "tsserver": "bin/tsserver" } }, "sha512-jl1vZzPDinLr9eUt3J/t7V6FgNEw9QjvBPdysz9KfQDD41fQrC2Y4vKQdiaUpFT4bXlb1RHhLpp8wtm6M5TgSw=="], - - "undici-types": ["undici-types@7.16.0", "", {}, "sha512-Zz+aZWSj8LE6zoxD+xrjh4VfkIG8Ya6LvYkZqtUQGJPZjYl53ypCaUwWqo7eI0x66KBGeRo+mlBEkMSeSZ38Nw=="], - - "zod": ["zod@4.3.6", "", {}, "sha512-rftlrkhHZOcjDwkGlnUtZZkvaPHCsDATp4pGpuOOMDaTdDDXF91wuVDJoWoPsKX/3YPQ5fHuF3STjcYyKr+Qhg=="], - } -} diff --git a/cli/test-skills/package.json b/cli/test-skills/package.json deleted file mode 100644 index d95b6c29b8..0000000000 --- a/cli/test-skills/package.json +++ /dev/null @@ -1,17 +0,0 @@ -{ - "name": "claude-code-skill-tests", - "version": "1.0.0", - "type": "module", - "scripts": { - "test": "bun test", - "test:skills": "bun test src/skill-invocation.test.ts", - "test:artifact": "bun test src/artifact-eval.test.ts" - }, - "dependencies": { - "@anthropic-ai/claude-agent-sdk": "^0.2.25" - }, - "devDependencies": { - "@types/bun": "latest", - "typescript": "^5.0.0" - } -} diff --git a/cli/test-skills/src/artifact-eval.test.ts b/cli/test-skills/src/artifact-eval.test.ts deleted file mode 100644 index 90e4749b92..0000000000 --- a/cli/test-skills/src/artifact-eval.test.ts +++ /dev/null @@ -1,57 +0,0 @@ -import { beforeAll, describe, expect, test } from "bun:test"; -import { - cleanupWorkspace, - loadCliArtifactEvalCases, - runCliArtifactEvalCase, - shouldKeepWorkspace -} from "../../../ai_evals/adapters/cli/artifact-eval"; - -const smokeCaseIds = new Set(["bun-hello-script"]); -const evalCases = (await loadCliArtifactEvalCases()).filter((evalCase) => - smokeCaseIds.has(evalCase.id) -); - -describe("Windmill CLI Artifact Evals", () => { - beforeAll(() => { - if (!process.env.ANTHROPIC_API_KEY) { - throw new Error("ANTHROPIC_API_KEY environment variable is required"); - } - }); - - for (const evalCase of evalCases) { - test( - evalCase.id, - async () => { - const result = await runCliArtifactEvalCase(evalCase); - - try { - console.log( - JSON.stringify( - { - caseId: evalCase.id, - workspaceDir: result.workspaceDir, - passed: result.passed, - checks: result.checks, - skillsInvoked: result.run.skillsInvoked, - toolsUsed: result.run.toolsUsed.map((tool) => tool.tool), - files: result.expectedFiles.map((file) => ({ - path: file.path, - exists: file.exists - })) - }, - null, - 2 - ) - ); - - expect(result.passed).toBe(true); - } finally { - if (!shouldKeepWorkspace()) { - await cleanupWorkspace(result.workspaceDir); - } - } - }, - { timeout: 180000 } - ); - } -}); diff --git a/cli/test-skills/src/skill-invocation.test.ts b/cli/test-skills/src/skill-invocation.test.ts deleted file mode 100644 index 9633eb2ef5..0000000000 --- a/cli/test-skills/src/skill-invocation.test.ts +++ /dev/null @@ -1,91 +0,0 @@ -import { describe, test, expect, beforeAll } from "bun:test"; -import { runPromptAndCapture, wasSkillInvoked, wasToolUsed, validateTestFolder } from "./test-utils"; - -describe("Windmill Skill Invocation", () => { - beforeAll(() => { - if (!process.env.ANTHROPIC_API_KEY) { - throw new Error("ANTHROPIC_API_KEY environment variable is required"); - } - validateTestFolder(); - }); - - describe("Flow Creation", () => { - test("'Create a Windmill flow' should invoke write-flow skill", async () => { - const result = await runPromptAndCapture( - "Create a Windmill flow that fetches data from an API and transforms it. Use placeholder URLs.", - undefined, - 3 - ); - - console.log("Tools used:", result.toolsUsed.map(t => t.tool)); - console.log("Skills invoked:", result.skillsInvoked); - - expect(wasToolUsed(result, "Skill")).toBe(true); - expect(wasSkillInvoked(result, "write-flow")).toBe(true); - }, { timeout: 120000 }); - }); - - describe("Python Script Creation", () => { - test("'Write a Windmill Python script' should invoke write-script-python3 skill", async () => { - const result = await runPromptAndCapture( - "Write a Windmill Python script that fetches data from https://api.example.com/users", - undefined, - 3 - ); - - console.log("Tools used:", result.toolsUsed.map(t => t.tool)); - console.log("Skills invoked:", result.skillsInvoked); - - expect(wasToolUsed(result, "Skill")).toBe(true); - expect(wasSkillInvoked(result, "write-script-python3")).toBe(true); - }, { timeout: 120000 }); - }); - - describe("Bun Script Creation", () => { - test("'Write a Windmill Bun/TypeScript script' should invoke write-script-bun skill", async () => { - const result = await runPromptAndCapture( - "Write a Windmill Bun script that processes JSON data", - undefined, - 3 - ); - - console.log("Tools used:", result.toolsUsed.map(t => t.tool)); - console.log("Skills invoked:", result.skillsInvoked); - - expect(wasToolUsed(result, "Skill")).toBe(true); - expect(wasSkillInvoked(result, "write-script-bun")).toBe(true); - }, { timeout: 120000 }); - }); - - describe("Schedule Configuration", () => { - test("'Create a Windmill schedule' should invoke schedules skill", async () => { - const result = await runPromptAndCapture( - "Create a Windmill schedule that runs a script daily at midnight", - undefined, - 3 - ); - - console.log("Tools used:", result.toolsUsed.map(t => t.tool)); - console.log("Skills invoked:", result.skillsInvoked); - - expect(wasToolUsed(result, "Skill")).toBe(true); - expect(wasSkillInvoked(result, "schedules")).toBe(true); - }, { timeout: 120000 }); - }); - - describe("Trigger Configuration", () => { - test("'Set up a Windmill webhook trigger' should invoke triggers skill", async () => { - const result = await runPromptAndCapture( - "Set up a Windmill HTTP trigger for a flow at /api/webhook", - undefined, - 3 - ); - - console.log("Tools used:", result.toolsUsed.map(t => t.tool)); - console.log("Skills invoked:", result.skillsInvoked); - - expect(wasToolUsed(result, "Skill")).toBe(true); - expect(wasSkillInvoked(result, "triggers")).toBe(true); - }, { timeout: 120000 }); - }); -}); diff --git a/cli/test-skills/src/test-utils.ts b/cli/test-skills/src/test-utils.ts deleted file mode 100644 index b998d37832..0000000000 --- a/cli/test-skills/src/test-utils.ts +++ /dev/null @@ -1,87 +0,0 @@ -import { cpSync, existsSync, mkdirSync, rmSync } from "fs"; -import { join } from "path"; -import { - getToolInputs as getSharedToolInputs, - getGeneratedSkillsSource as getSharedGeneratedSkillsSource, - runPromptAndCapture as runSharedPromptAndCapture, - wasSkillInvoked as wasSharedSkillInvoked, - wasToolUsed as wasSharedToolUsed, - type PromptRunResult, - type ToolInvocation -} from "../../../ai_evals/adapters/cli/runtime"; - -export type TestResult = PromptRunResult; - -/** - * Get the test-skills directory path - */ -export function getTestSkillsDir(): string { - return new URL("..", import.meta.url).pathname; -} - -/** - * Get the test-folder directory path (where user places .claude/skills) - */ -export function getTestFolder(): string { - return join(getTestSkillsDir(), "test-folder"); -} - -/** - * Get the generated skills directory from the repo root. - */ -export function getGeneratedSkillsSource(): string { - return getSharedGeneratedSkillsSource(); -} - -/** - * Ensure test-folder exists and mirrors the repo's generated skills. - */ -export function validateTestFolder(): void { - const testFolder = getTestFolder(); - const skillsFolder = join(testFolder, ".claude", "skills"); - const generatedSkillsSource = getGeneratedSkillsSource(); - - if (!existsSync(generatedSkillsSource)) { - throw new Error( - `Generated skills directory not found at: ${generatedSkillsSource}\n` + - `Run system prompt generation first so cli/test-skills can mirror the current repo skill bundle.` - ); - } - - mkdirSync(join(testFolder, ".claude"), { recursive: true }); - rmSync(skillsFolder, { recursive: true, force: true }); - cpSync(generatedSkillsSource, skillsFolder, { recursive: true }); -} - -/** - * Runs a prompt through the Claude Agent SDK and captures tool invocations - * Uses test-folder as cwd where user-provided skills are located - */ -export async function runPromptAndCapture( - prompt: string, - cwd?: string, - maxTurns: number = 3 -): Promise { - return runSharedPromptAndCapture(prompt, cwd ?? getTestFolder(), maxTurns); -} - -/** - * Helper to check if a specific tool was used - */ -export function wasToolUsed(result: TestResult, toolName: string): boolean { - return wasSharedToolUsed(result, toolName); -} - -/** - * Helper to check if a specific skill was invoked - */ -export function wasSkillInvoked(result: TestResult, skillName: string): boolean { - return wasSharedSkillInvoked(result, skillName); -} - -/** - * Helper to get all tool inputs for a specific tool - */ -export function getToolInputs(result: TestResult, toolName: string): Record[] { - return getSharedToolInputs(result, toolName); -} diff --git a/cli/test-skills/tsconfig.json b/cli/test-skills/tsconfig.json deleted file mode 100644 index 45f0069307..0000000000 --- a/cli/test-skills/tsconfig.json +++ /dev/null @@ -1,17 +0,0 @@ -{ - "compilerOptions": { - "target": "ES2022", - "module": "ESNext", - "moduleResolution": "bundler", - "strict": true, - "esModuleInterop": true, - "skipLibCheck": true, - "forceConsistentCasingInFileNames": true, - "outDir": "./dist", - "rootDir": "./src", - "declaration": true, - "types": ["bun-types"] - }, - "include": ["src/**/*"], - "exclude": ["node_modules", "dist"] -} diff --git a/docs/system-prompt-testing-plan.md b/docs/system-prompt-testing-plan.md index a1887cc8e2..b3f85f59ca 100644 --- a/docs/system-prompt-testing-plan.md +++ b/docs/system-prompt-testing-plan.md @@ -156,18 +156,14 @@ The exact binary name can change, but the architecture should not: ## Temporary Bootstrap Code -Existing test files under `frontend/.../__tests__/...` and `cli/test-skills/` -are acceptable as bootstrap code while the benchmark CLI is being built. +Existing frontend test files under `frontend/.../__tests__/...` are acceptable +as bootstrap code while the benchmark CLI is being built. They should not be treated as the final user-facing interface for prompt evaluation. -In particular: - -- `cli/test-skills/` is currently a prototype adapter and proving ground -- it should eventually be migrated behind the repo-level benchmark CLI -- benchmark authors should not have to know about `cli/test-skills/` to run the - long-term suite +Benchmark authors should only need the repo-level benchmark CLI to run the +long-term suite. ## Frontend: What Exists Today @@ -947,9 +943,6 @@ This project is successful when all of the following are true: The current frontend evals should be treated as a useful starting point, not the finished solution. -The current `cli/test-skills` work should also be treated as a useful prototype, -not the final benchmark interface. - They already prove that the repo can test AI behavior without coupling to the browser UI. The main work now is: