mirror of
https://github.com/windmill-labs/windmill.git
synced 2026-08-21 08:02:26 +00:00
refactor: remove legacy cli test harness
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
This commit is contained in:
centdix
+5
-7
@@ -11,9 +11,6 @@ internal implementation details of the frontend or CLI.
|
||||
The intended end state is a new repo-level benchmark CLI for running the shared
|
||||
eval suite across both the Windmill CLI and the frontend.
|
||||
|
||||
Current code under `cli/test-skills/` should be treated as bootstrap/prototype
|
||||
implementation work, not the final entrypoint.
|
||||
|
||||
## Layout
|
||||
|
||||
- `cli/`: repo-level benchmark CLI entrypoint and orchestration
|
||||
@@ -57,9 +54,10 @@ bun install
|
||||
Current usage:
|
||||
|
||||
```bash
|
||||
bun ai_evals/cli/index.ts list-cases --surface cli
|
||||
bun ai_evals/cli/index.ts run --surface cli --case bun-hello-script
|
||||
cd ai_evals
|
||||
bun run cli -- list-cases --surface cli
|
||||
bun run cli -- run --surface cli --case bun-hello-script
|
||||
```
|
||||
|
||||
At the moment this is intentionally thin and delegates to the bootstrap CLI
|
||||
adapter code under `cli/test-skills/`.
|
||||
At the moment this is still intentionally small, but it is the only benchmark
|
||||
entrypoint.
|
||||
|
||||
@@ -9,10 +9,8 @@ The current implementation is intentionally small:
|
||||
|
||||
- `run` command
|
||||
- `cli` surface only
|
||||
- runs the existing CLI artifact-eval adapter from `cli/test-skills/`
|
||||
|
||||
This is a migration step away from using `cli/test-skills/` as the primary
|
||||
entrypoint.
|
||||
This is the benchmark entrypoint for prompt and artifact evaluation.
|
||||
|
||||
## Usage
|
||||
|
||||
@@ -26,25 +24,29 @@ bun install
|
||||
List available CLI cases:
|
||||
|
||||
```bash
|
||||
bun ai_evals/cli/index.ts list-cases --surface cli
|
||||
cd ai_evals
|
||||
bun run cli -- list-cases --surface cli
|
||||
```
|
||||
|
||||
Run one CLI case:
|
||||
|
||||
```bash
|
||||
bun ai_evals/cli/index.ts run --surface cli --case bun-hello-script
|
||||
cd ai_evals
|
||||
bun run cli -- run --surface cli --case bun-hello-script
|
||||
```
|
||||
|
||||
Keep the temp workspace for inspection:
|
||||
|
||||
```bash
|
||||
bun ai_evals/cli/index.ts run --surface cli --case bun-hello-script --keep-workspace
|
||||
cd ai_evals
|
||||
bun run cli -- run --surface cli --case bun-hello-script --keep-workspace
|
||||
```
|
||||
|
||||
Print machine-readable output:
|
||||
|
||||
```bash
|
||||
bun ai_evals/cli/index.ts run --surface cli --case bun-hello-script --json
|
||||
cd ai_evals
|
||||
bun run cli -- run --surface cli --case bun-hello-script --json
|
||||
```
|
||||
|
||||
## Next Steps
|
||||
|
||||
@@ -237,10 +237,10 @@ function printHelp() {
|
||||
process.stdout.write(
|
||||
[
|
||||
"Usage:",
|
||||
" bun ai_evals/cli/index.ts list-cases --surface cli [--json]",
|
||||
" bun ai_evals/cli/index.ts run --surface cli --case <id> [--json] [--keep-workspace]",
|
||||
" bun ai_evals/cli/index.ts compare",
|
||||
" bun ai_evals/cli/index.ts history",
|
||||
" cd ai_evals && bun run cli -- list-cases --surface cli [--json]",
|
||||
" cd ai_evals && bun run cli -- run --surface cli --case <id> [--json] [--keep-workspace]",
|
||||
" cd ai_evals && bun run cli -- compare",
|
||||
" cd ai_evals && bun run cli -- history",
|
||||
"",
|
||||
"Current support:",
|
||||
" surfaces: cli"
|
||||
|
||||
@@ -3,6 +3,7 @@
|
||||
"private": true,
|
||||
"type": "module",
|
||||
"scripts": {
|
||||
"cli": "bun cli/index.ts",
|
||||
"list-cases": "bun cli/index.ts list-cases --surface cli"
|
||||
},
|
||||
"dependencies": {
|
||||
|
||||
@@ -1,110 +0,0 @@
|
||||
# Windmill CLI Prompt Tests
|
||||
|
||||
Test suite for verifying how Claude Code behaves with Windmill auto-generated
|
||||
skills. It currently contains both skill-invocation smoke tests and the first
|
||||
artifact-evaluation benchmark.
|
||||
|
||||
## Overview
|
||||
|
||||
This framework sends prompts through the Claude Agent SDK using the repo's
|
||||
generated Windmill skills.
|
||||
|
||||
It currently supports:
|
||||
|
||||
- skill-invocation smoke tests
|
||||
- CLI artifact evaluation in an isolated temp workspace
|
||||
|
||||
## Prerequisites
|
||||
|
||||
- [Bun](https://bun.sh/) installed
|
||||
- `ANTHROPIC_API_KEY` environment variable set
|
||||
- Auto-generated Windmill skills available under `system_prompts/auto-generated/skills/`
|
||||
|
||||
## User Setup
|
||||
|
||||
1. Generate the latest system prompts and CLI skills in the repo:
|
||||
|
||||
```bash
|
||||
python3 system_prompts/generate.py
|
||||
```
|
||||
|
||||
2. Set your API key:
|
||||
```bash
|
||||
export ANTHROPIC_API_KEY=your-key-here
|
||||
```
|
||||
|
||||
3. Install dependencies and run tests:
|
||||
```bash
|
||||
cd cli/test-skills
|
||||
bun install
|
||||
bun test
|
||||
```
|
||||
|
||||
## Expected Skills
|
||||
|
||||
The tests expect the following auto-generated skills to be present:
|
||||
|
||||
| Skill Name | Purpose |
|
||||
|------------|---------|
|
||||
| `write-flow` | Creating Windmill flows/workflows |
|
||||
| `write-script-python3` | Creating Python scripts |
|
||||
| `write-script-bun` | Creating TypeScript/Bun scripts |
|
||||
| `schedules` | Configuring schedules and cron jobs |
|
||||
| `triggers` | Setting up triggers (webhook, Kafka, etc.) |
|
||||
|
||||
## Test Matrix
|
||||
|
||||
| Prompt | Expected Skill |
|
||||
|--------|----------------|
|
||||
| "Create a flow to process user data" | `write-flow` |
|
||||
| "Build a workflow that fetches and transforms data" | `write-flow` |
|
||||
| "Write a Python script to fetch API data" | `write-script-python3` |
|
||||
| "Create a Python function to process CSV files" | `write-script-python3` |
|
||||
| "Write a TypeScript script using Bun" | `write-script-bun` |
|
||||
| "Create a Bun script to handle webhooks" | `write-script-bun` |
|
||||
| "Set up a schedule to run this daily at midnight" | `schedules` |
|
||||
| "Configure a cron job to run every hour" | `schedules` |
|
||||
| "Set up a webhook trigger for this flow" | `triggers` |
|
||||
| "Configure a Kafka trigger" | `triggers` |
|
||||
|
||||
## Running Tests
|
||||
|
||||
Run all tests:
|
||||
```bash
|
||||
bun test
|
||||
```
|
||||
|
||||
Run only skill invocation tests:
|
||||
```bash
|
||||
bun test:skills
|
||||
```
|
||||
|
||||
Run the first artifact-evaluation benchmark:
|
||||
```bash
|
||||
bun test:artifact
|
||||
```
|
||||
|
||||
## Test Utilities
|
||||
|
||||
The `src/test-utils.ts` module provides:
|
||||
|
||||
- `runPromptAndCapture(prompt, cwd?, maxTurns)` - Runs a prompt and captures tool invocations
|
||||
- `wasToolUsed(result, toolName)` - Checks if a specific tool was used
|
||||
- `wasSkillInvoked(result, skillName)` - Checks if a specific skill was invoked
|
||||
- `getToolInputs(result, toolName)` - Gets all inputs for a specific tool
|
||||
- `getTestSkillsDir()` - Returns the test-skills directory path
|
||||
|
||||
The `src/artifact-eval.ts` module provides:
|
||||
|
||||
- temp-workspace creation with generated skills
|
||||
- prompt rendering with workspace-root placeholders
|
||||
- file-based artifact scoring for benchmark cases
|
||||
|
||||
## Notes
|
||||
|
||||
- Tests have extended timeouts (120 seconds) due to API latency
|
||||
- Tests run against the actual Claude API, so they consume API credits
|
||||
- Skill-invocation smoke tests still use `test-folder/`
|
||||
- Artifact evals use isolated temp workspaces under `/tmp`
|
||||
- The first artifact benchmark uses an explicit absolute target path so file
|
||||
outputs are scoreable even when Claude executes inside a skill context
|
||||
@@ -1,61 +0,0 @@
|
||||
{
|
||||
"lockfileVersion": 1,
|
||||
"configVersion": 1,
|
||||
"workspaces": {
|
||||
"": {
|
||||
"name": "claude-code-skill-tests",
|
||||
"dependencies": {
|
||||
"@anthropic-ai/claude-agent-sdk": "^0.2.25",
|
||||
},
|
||||
"devDependencies": {
|
||||
"@types/bun": "latest",
|
||||
"typescript": "^5.0.0",
|
||||
},
|
||||
},
|
||||
},
|
||||
"packages": {
|
||||
"@anthropic-ai/claude-agent-sdk": ["@anthropic-ai/claude-agent-sdk@0.2.25", "", { "optionalDependencies": { "@img/sharp-darwin-arm64": "^0.33.5", "@img/sharp-darwin-x64": "^0.33.5", "@img/sharp-linux-arm": "^0.33.5", "@img/sharp-linux-arm64": "^0.33.5", "@img/sharp-linux-x64": "^0.33.5", "@img/sharp-linuxmusl-arm64": "^0.33.5", "@img/sharp-linuxmusl-x64": "^0.33.5", "@img/sharp-win32-x64": "^0.33.5" }, "peerDependencies": { "zod": "^4.0.0" } }, "sha512-YIP3I40+XSkC3zE1Z8KRQY02VA7UfofFamF1cFrLe7FbtCnjpslyDl9coGBh2DAi9xj2yQcKZZf751jEWpB+dQ=="],
|
||||
|
||||
"@img/sharp-darwin-arm64": ["@img/sharp-darwin-arm64@0.33.5", "", { "optionalDependencies": { "@img/sharp-libvips-darwin-arm64": "1.0.4" }, "os": "darwin", "cpu": "arm64" }, "sha512-UT4p+iz/2H4twwAoLCqfA9UH5pI6DggwKEGuaPy7nCVQ8ZsiY5PIcrRvD1DzuY3qYL07NtIQcWnBSY/heikIFQ=="],
|
||||
|
||||
"@img/sharp-darwin-x64": ["@img/sharp-darwin-x64@0.33.5", "", { "optionalDependencies": { "@img/sharp-libvips-darwin-x64": "1.0.4" }, "os": "darwin", "cpu": "x64" }, "sha512-fyHac4jIc1ANYGRDxtiqelIbdWkIuQaI84Mv45KvGRRxSAa7o7d1ZKAOBaYbnepLC1WqxfpimdeWfvqqSGwR2Q=="],
|
||||
|
||||
"@img/sharp-libvips-darwin-arm64": ["@img/sharp-libvips-darwin-arm64@1.0.4", "", { "os": "darwin", "cpu": "arm64" }, "sha512-XblONe153h0O2zuFfTAbQYAX2JhYmDHeWikp1LM9Hul9gVPjFY427k6dFEcOL72O01QxQsWi761svJ/ev9xEDg=="],
|
||||
|
||||
"@img/sharp-libvips-darwin-x64": ["@img/sharp-libvips-darwin-x64@1.0.4", "", { "os": "darwin", "cpu": "x64" }, "sha512-xnGR8YuZYfJGmWPvmlunFaWJsb9T/AO2ykoP3Fz/0X5XV2aoYBPkX6xqCQvUTKKiLddarLaxpzNe+b1hjeWHAQ=="],
|
||||
|
||||
"@img/sharp-libvips-linux-arm": ["@img/sharp-libvips-linux-arm@1.0.5", "", { "os": "linux", "cpu": "arm" }, "sha512-gvcC4ACAOPRNATg/ov8/MnbxFDJqf/pDePbBnuBDcjsI8PssmjoKMAz4LtLaVi+OnSb5FK/yIOamqDwGmXW32g=="],
|
||||
|
||||
"@img/sharp-libvips-linux-arm64": ["@img/sharp-libvips-linux-arm64@1.0.4", "", { "os": "linux", "cpu": "arm64" }, "sha512-9B+taZ8DlyyqzZQnoeIvDVR/2F4EbMepXMc/NdVbkzsJbzkUjhXv/70GQJ7tdLA4YJgNP25zukcxpX2/SueNrA=="],
|
||||
|
||||
"@img/sharp-libvips-linux-x64": ["@img/sharp-libvips-linux-x64@1.0.4", "", { "os": "linux", "cpu": "x64" }, "sha512-MmWmQ3iPFZr0Iev+BAgVMb3ZyC4KeFc3jFxnNbEPas60e1cIfevbtuyf9nDGIzOaW9PdnDciJm+wFFaTlj5xYw=="],
|
||||
|
||||
"@img/sharp-libvips-linuxmusl-arm64": ["@img/sharp-libvips-linuxmusl-arm64@1.0.4", "", { "os": "linux", "cpu": "arm64" }, "sha512-9Ti+BbTYDcsbp4wfYib8Ctm1ilkugkA/uscUn6UXK1ldpC1JjiXbLfFZtRlBhjPZ5o1NCLiDbg8fhUPKStHoTA=="],
|
||||
|
||||
"@img/sharp-libvips-linuxmusl-x64": ["@img/sharp-libvips-linuxmusl-x64@1.0.4", "", { "os": "linux", "cpu": "x64" }, "sha512-viYN1KX9m+/hGkJtvYYp+CCLgnJXwiQB39damAO7WMdKWlIhmYTfHjwSbQeUK/20vY154mwezd9HflVFM1wVSw=="],
|
||||
|
||||
"@img/sharp-linux-arm": ["@img/sharp-linux-arm@0.33.5", "", { "optionalDependencies": { "@img/sharp-libvips-linux-arm": "1.0.5" }, "os": "linux", "cpu": "arm" }, "sha512-JTS1eldqZbJxjvKaAkxhZmBqPRGmxgu+qFKSInv8moZ2AmT5Yib3EQ1c6gp493HvrvV8QgdOXdyaIBrhvFhBMQ=="],
|
||||
|
||||
"@img/sharp-linux-arm64": ["@img/sharp-linux-arm64@0.33.5", "", { "optionalDependencies": { "@img/sharp-libvips-linux-arm64": "1.0.4" }, "os": "linux", "cpu": "arm64" }, "sha512-JMVv+AMRyGOHtO1RFBiJy/MBsgz0x4AWrT6QoEVVTyh1E39TrCUpTRI7mx9VksGX4awWASxqCYLCV4wBZHAYxA=="],
|
||||
|
||||
"@img/sharp-linux-x64": ["@img/sharp-linux-x64@0.33.5", "", { "optionalDependencies": { "@img/sharp-libvips-linux-x64": "1.0.4" }, "os": "linux", "cpu": "x64" }, "sha512-opC+Ok5pRNAzuvq1AG0ar+1owsu842/Ab+4qvU879ippJBHvyY5n2mxF1izXqkPYlGuP/M556uh53jRLJmzTWA=="],
|
||||
|
||||
"@img/sharp-linuxmusl-arm64": ["@img/sharp-linuxmusl-arm64@0.33.5", "", { "optionalDependencies": { "@img/sharp-libvips-linuxmusl-arm64": "1.0.4" }, "os": "linux", "cpu": "arm64" }, "sha512-XrHMZwGQGvJg2V/oRSUfSAfjfPxO+4DkiRh6p2AFjLQztWUuY/o8Mq0eMQVIY7HJ1CDQUJlxGGZRw1a5bqmd1g=="],
|
||||
|
||||
"@img/sharp-linuxmusl-x64": ["@img/sharp-linuxmusl-x64@0.33.5", "", { "optionalDependencies": { "@img/sharp-libvips-linuxmusl-x64": "1.0.4" }, "os": "linux", "cpu": "x64" }, "sha512-WT+d/cgqKkkKySYmqoZ8y3pxx7lx9vVejxW/W4DOFMYVSkErR+w7mf2u8m/y4+xHe7yY9DAXQMWQhpnMuFfScw=="],
|
||||
|
||||
"@img/sharp-win32-x64": ["@img/sharp-win32-x64@0.33.5", "", { "os": "win32", "cpu": "x64" }, "sha512-MpY/o8/8kj+EcnxwvrP4aTJSWw/aZ7JIGR4aBeZkZw5B7/Jn+tY9/VNwtcoGmdT7GfggGIU4kygOMSbYnOrAbg=="],
|
||||
|
||||
"@types/bun": ["@types/bun@1.3.8", "", { "dependencies": { "bun-types": "1.3.8" } }, "sha512-3LvWJ2q5GerAXYxO2mffLTqOzEu5qnhEAlh48Vnu8WQfnmSwbgagjGZV6BoHKJztENYEDn6QmVd949W4uESRJA=="],
|
||||
|
||||
"@types/node": ["@types/node@25.1.0", "", { "dependencies": { "undici-types": "~7.16.0" } }, "sha512-t7frlewr6+cbx+9Ohpl0NOTKXZNV9xHRmNOvql47BFJKcEG1CxtxlPEEe+gR9uhVWM4DwhnvTF110mIL4yP9RA=="],
|
||||
|
||||
"bun-types": ["bun-types@1.3.8", "", { "dependencies": { "@types/node": "*" } }, "sha512-fL99nxdOWvV4LqjmC+8Q9kW3M4QTtTR1eePs94v5ctGqU8OeceWrSUaRw3JYb7tU3FkMIAjkueehrHPPPGKi5Q=="],
|
||||
|
||||
"typescript": ["typescript@5.9.3", "", { "bin": { "tsc": "bin/tsc", "tsserver": "bin/tsserver" } }, "sha512-jl1vZzPDinLr9eUt3J/t7V6FgNEw9QjvBPdysz9KfQDD41fQrC2Y4vKQdiaUpFT4bXlb1RHhLpp8wtm6M5TgSw=="],
|
||||
|
||||
"undici-types": ["undici-types@7.16.0", "", {}, "sha512-Zz+aZWSj8LE6zoxD+xrjh4VfkIG8Ya6LvYkZqtUQGJPZjYl53ypCaUwWqo7eI0x66KBGeRo+mlBEkMSeSZ38Nw=="],
|
||||
|
||||
"zod": ["zod@4.3.6", "", {}, "sha512-rftlrkhHZOcjDwkGlnUtZZkvaPHCsDATp4pGpuOOMDaTdDDXF91wuVDJoWoPsKX/3YPQ5fHuF3STjcYyKr+Qhg=="],
|
||||
}
|
||||
}
|
||||
@@ -1,17 +0,0 @@
|
||||
{
|
||||
"name": "claude-code-skill-tests",
|
||||
"version": "1.0.0",
|
||||
"type": "module",
|
||||
"scripts": {
|
||||
"test": "bun test",
|
||||
"test:skills": "bun test src/skill-invocation.test.ts",
|
||||
"test:artifact": "bun test src/artifact-eval.test.ts"
|
||||
},
|
||||
"dependencies": {
|
||||
"@anthropic-ai/claude-agent-sdk": "^0.2.25"
|
||||
},
|
||||
"devDependencies": {
|
||||
"@types/bun": "latest",
|
||||
"typescript": "^5.0.0"
|
||||
}
|
||||
}
|
||||
@@ -1,57 +0,0 @@
|
||||
import { beforeAll, describe, expect, test } from "bun:test";
|
||||
import {
|
||||
cleanupWorkspace,
|
||||
loadCliArtifactEvalCases,
|
||||
runCliArtifactEvalCase,
|
||||
shouldKeepWorkspace
|
||||
} from "../../../ai_evals/adapters/cli/artifact-eval";
|
||||
|
||||
const smokeCaseIds = new Set(["bun-hello-script"]);
|
||||
const evalCases = (await loadCliArtifactEvalCases()).filter((evalCase) =>
|
||||
smokeCaseIds.has(evalCase.id)
|
||||
);
|
||||
|
||||
describe("Windmill CLI Artifact Evals", () => {
|
||||
beforeAll(() => {
|
||||
if (!process.env.ANTHROPIC_API_KEY) {
|
||||
throw new Error("ANTHROPIC_API_KEY environment variable is required");
|
||||
}
|
||||
});
|
||||
|
||||
for (const evalCase of evalCases) {
|
||||
test(
|
||||
evalCase.id,
|
||||
async () => {
|
||||
const result = await runCliArtifactEvalCase(evalCase);
|
||||
|
||||
try {
|
||||
console.log(
|
||||
JSON.stringify(
|
||||
{
|
||||
caseId: evalCase.id,
|
||||
workspaceDir: result.workspaceDir,
|
||||
passed: result.passed,
|
||||
checks: result.checks,
|
||||
skillsInvoked: result.run.skillsInvoked,
|
||||
toolsUsed: result.run.toolsUsed.map((tool) => tool.tool),
|
||||
files: result.expectedFiles.map((file) => ({
|
||||
path: file.path,
|
||||
exists: file.exists
|
||||
}))
|
||||
},
|
||||
null,
|
||||
2
|
||||
)
|
||||
);
|
||||
|
||||
expect(result.passed).toBe(true);
|
||||
} finally {
|
||||
if (!shouldKeepWorkspace()) {
|
||||
await cleanupWorkspace(result.workspaceDir);
|
||||
}
|
||||
}
|
||||
},
|
||||
{ timeout: 180000 }
|
||||
);
|
||||
}
|
||||
});
|
||||
@@ -1,91 +0,0 @@
|
||||
import { describe, test, expect, beforeAll } from "bun:test";
|
||||
import { runPromptAndCapture, wasSkillInvoked, wasToolUsed, validateTestFolder } from "./test-utils";
|
||||
|
||||
describe("Windmill Skill Invocation", () => {
|
||||
beforeAll(() => {
|
||||
if (!process.env.ANTHROPIC_API_KEY) {
|
||||
throw new Error("ANTHROPIC_API_KEY environment variable is required");
|
||||
}
|
||||
validateTestFolder();
|
||||
});
|
||||
|
||||
describe("Flow Creation", () => {
|
||||
test("'Create a Windmill flow' should invoke write-flow skill", async () => {
|
||||
const result = await runPromptAndCapture(
|
||||
"Create a Windmill flow that fetches data from an API and transforms it. Use placeholder URLs.",
|
||||
undefined,
|
||||
3
|
||||
);
|
||||
|
||||
console.log("Tools used:", result.toolsUsed.map(t => t.tool));
|
||||
console.log("Skills invoked:", result.skillsInvoked);
|
||||
|
||||
expect(wasToolUsed(result, "Skill")).toBe(true);
|
||||
expect(wasSkillInvoked(result, "write-flow")).toBe(true);
|
||||
}, { timeout: 120000 });
|
||||
});
|
||||
|
||||
describe("Python Script Creation", () => {
|
||||
test("'Write a Windmill Python script' should invoke write-script-python3 skill", async () => {
|
||||
const result = await runPromptAndCapture(
|
||||
"Write a Windmill Python script that fetches data from https://api.example.com/users",
|
||||
undefined,
|
||||
3
|
||||
);
|
||||
|
||||
console.log("Tools used:", result.toolsUsed.map(t => t.tool));
|
||||
console.log("Skills invoked:", result.skillsInvoked);
|
||||
|
||||
expect(wasToolUsed(result, "Skill")).toBe(true);
|
||||
expect(wasSkillInvoked(result, "write-script-python3")).toBe(true);
|
||||
}, { timeout: 120000 });
|
||||
});
|
||||
|
||||
describe("Bun Script Creation", () => {
|
||||
test("'Write a Windmill Bun/TypeScript script' should invoke write-script-bun skill", async () => {
|
||||
const result = await runPromptAndCapture(
|
||||
"Write a Windmill Bun script that processes JSON data",
|
||||
undefined,
|
||||
3
|
||||
);
|
||||
|
||||
console.log("Tools used:", result.toolsUsed.map(t => t.tool));
|
||||
console.log("Skills invoked:", result.skillsInvoked);
|
||||
|
||||
expect(wasToolUsed(result, "Skill")).toBe(true);
|
||||
expect(wasSkillInvoked(result, "write-script-bun")).toBe(true);
|
||||
}, { timeout: 120000 });
|
||||
});
|
||||
|
||||
describe("Schedule Configuration", () => {
|
||||
test("'Create a Windmill schedule' should invoke schedules skill", async () => {
|
||||
const result = await runPromptAndCapture(
|
||||
"Create a Windmill schedule that runs a script daily at midnight",
|
||||
undefined,
|
||||
3
|
||||
);
|
||||
|
||||
console.log("Tools used:", result.toolsUsed.map(t => t.tool));
|
||||
console.log("Skills invoked:", result.skillsInvoked);
|
||||
|
||||
expect(wasToolUsed(result, "Skill")).toBe(true);
|
||||
expect(wasSkillInvoked(result, "schedules")).toBe(true);
|
||||
}, { timeout: 120000 });
|
||||
});
|
||||
|
||||
describe("Trigger Configuration", () => {
|
||||
test("'Set up a Windmill webhook trigger' should invoke triggers skill", async () => {
|
||||
const result = await runPromptAndCapture(
|
||||
"Set up a Windmill HTTP trigger for a flow at /api/webhook",
|
||||
undefined,
|
||||
3
|
||||
);
|
||||
|
||||
console.log("Tools used:", result.toolsUsed.map(t => t.tool));
|
||||
console.log("Skills invoked:", result.skillsInvoked);
|
||||
|
||||
expect(wasToolUsed(result, "Skill")).toBe(true);
|
||||
expect(wasSkillInvoked(result, "triggers")).toBe(true);
|
||||
}, { timeout: 120000 });
|
||||
});
|
||||
});
|
||||
@@ -1,87 +0,0 @@
|
||||
import { cpSync, existsSync, mkdirSync, rmSync } from "fs";
|
||||
import { join } from "path";
|
||||
import {
|
||||
getToolInputs as getSharedToolInputs,
|
||||
getGeneratedSkillsSource as getSharedGeneratedSkillsSource,
|
||||
runPromptAndCapture as runSharedPromptAndCapture,
|
||||
wasSkillInvoked as wasSharedSkillInvoked,
|
||||
wasToolUsed as wasSharedToolUsed,
|
||||
type PromptRunResult,
|
||||
type ToolInvocation
|
||||
} from "../../../ai_evals/adapters/cli/runtime";
|
||||
|
||||
export type TestResult = PromptRunResult;
|
||||
|
||||
/**
|
||||
* Get the test-skills directory path
|
||||
*/
|
||||
export function getTestSkillsDir(): string {
|
||||
return new URL("..", import.meta.url).pathname;
|
||||
}
|
||||
|
||||
/**
|
||||
* Get the test-folder directory path (where user places .claude/skills)
|
||||
*/
|
||||
export function getTestFolder(): string {
|
||||
return join(getTestSkillsDir(), "test-folder");
|
||||
}
|
||||
|
||||
/**
|
||||
* Get the generated skills directory from the repo root.
|
||||
*/
|
||||
export function getGeneratedSkillsSource(): string {
|
||||
return getSharedGeneratedSkillsSource();
|
||||
}
|
||||
|
||||
/**
|
||||
* Ensure test-folder exists and mirrors the repo's generated skills.
|
||||
*/
|
||||
export function validateTestFolder(): void {
|
||||
const testFolder = getTestFolder();
|
||||
const skillsFolder = join(testFolder, ".claude", "skills");
|
||||
const generatedSkillsSource = getGeneratedSkillsSource();
|
||||
|
||||
if (!existsSync(generatedSkillsSource)) {
|
||||
throw new Error(
|
||||
`Generated skills directory not found at: ${generatedSkillsSource}\n` +
|
||||
`Run system prompt generation first so cli/test-skills can mirror the current repo skill bundle.`
|
||||
);
|
||||
}
|
||||
|
||||
mkdirSync(join(testFolder, ".claude"), { recursive: true });
|
||||
rmSync(skillsFolder, { recursive: true, force: true });
|
||||
cpSync(generatedSkillsSource, skillsFolder, { recursive: true });
|
||||
}
|
||||
|
||||
/**
|
||||
* Runs a prompt through the Claude Agent SDK and captures tool invocations
|
||||
* Uses test-folder as cwd where user-provided skills are located
|
||||
*/
|
||||
export async function runPromptAndCapture(
|
||||
prompt: string,
|
||||
cwd?: string,
|
||||
maxTurns: number = 3
|
||||
): Promise<TestResult> {
|
||||
return runSharedPromptAndCapture(prompt, cwd ?? getTestFolder(), maxTurns);
|
||||
}
|
||||
|
||||
/**
|
||||
* Helper to check if a specific tool was used
|
||||
*/
|
||||
export function wasToolUsed(result: TestResult, toolName: string): boolean {
|
||||
return wasSharedToolUsed(result, toolName);
|
||||
}
|
||||
|
||||
/**
|
||||
* Helper to check if a specific skill was invoked
|
||||
*/
|
||||
export function wasSkillInvoked(result: TestResult, skillName: string): boolean {
|
||||
return wasSharedSkillInvoked(result, skillName);
|
||||
}
|
||||
|
||||
/**
|
||||
* Helper to get all tool inputs for a specific tool
|
||||
*/
|
||||
export function getToolInputs(result: TestResult, toolName: string): Record<string, unknown>[] {
|
||||
return getSharedToolInputs(result, toolName);
|
||||
}
|
||||
@@ -1,17 +0,0 @@
|
||||
{
|
||||
"compilerOptions": {
|
||||
"target": "ES2022",
|
||||
"module": "ESNext",
|
||||
"moduleResolution": "bundler",
|
||||
"strict": true,
|
||||
"esModuleInterop": true,
|
||||
"skipLibCheck": true,
|
||||
"forceConsistentCasingInFileNames": true,
|
||||
"outDir": "./dist",
|
||||
"rootDir": "./src",
|
||||
"declaration": true,
|
||||
"types": ["bun-types"]
|
||||
},
|
||||
"include": ["src/**/*"],
|
||||
"exclude": ["node_modules", "dist"]
|
||||
}
|
||||
@@ -156,18 +156,14 @@ The exact binary name can change, but the architecture should not:
|
||||
|
||||
## Temporary Bootstrap Code
|
||||
|
||||
Existing test files under `frontend/.../__tests__/...` and `cli/test-skills/`
|
||||
are acceptable as bootstrap code while the benchmark CLI is being built.
|
||||
Existing frontend test files under `frontend/.../__tests__/...` are acceptable
|
||||
as bootstrap code while the benchmark CLI is being built.
|
||||
|
||||
They should not be treated as the final user-facing interface for prompt
|
||||
evaluation.
|
||||
|
||||
In particular:
|
||||
|
||||
- `cli/test-skills/` is currently a prototype adapter and proving ground
|
||||
- it should eventually be migrated behind the repo-level benchmark CLI
|
||||
- benchmark authors should not have to know about `cli/test-skills/` to run the
|
||||
long-term suite
|
||||
Benchmark authors should only need the repo-level benchmark CLI to run the
|
||||
long-term suite.
|
||||
|
||||
## Frontend: What Exists Today
|
||||
|
||||
@@ -947,9 +943,6 @@ This project is successful when all of the following are true:
|
||||
|
||||
The current frontend evals should be treated as a useful starting point, not the finished solution.
|
||||
|
||||
The current `cli/test-skills` work should also be treated as a useful prototype,
|
||||
not the final benchmark interface.
|
||||
|
||||
They already prove that the repo can test AI behavior without coupling to the browser UI.
|
||||
|
||||
The main work now is:
|
||||
|
||||
Reference in New Issue
Block a user