chore(main): release 1.693.2 (#8987)

* chore(main): release 1.693.2

* Apply automatic changes

---------

Co-authored-by: rubenfiszel <275584+rubenfiszel@users.noreply.github.com>

.claude/skills/adding-a-trigger/SKILL.md

@@ -0,0 +1,265 @@
+---
+name: adding-a-trigger
+description: Checklist for adding a new TriggerCrud-based trigger type to Windmill (Azure, GCP, Kafka, etc.). Use when wiring a new trigger kind across backend, frontend, CLI, and capture infrastructure.
+---
+
+# Skill: Adding a New Trigger Type
+
+Use this skill when adding a trigger kind that implements `TriggerCrud` (Kafka, GCP, Azure, MQTT, SQS, NATS, Postgres, Email…). For native triggers (Nextcloud, Google Drive — things wired through `windmill-native-triggers`), use the `native-trigger` skill instead.
+
+The goal of this doc is to enumerate every file that needs to change. Missing any one of them leads to silent regressions: sync drops the trigger, capture button does nothing, workspace forks lose it, sidebar counters undercount. Follow the checklist top-to-bottom — each section is independent enough to be validated on its own.
+
+Throughout this doc, substitute `{kind}` for the new trigger kind (`azure`, `kafka`, …), `{Kind}` for PascalCase (`Azure`, `Kafka`), `{KIND}` for SCREAMING (`AZURE`, `KAFKA`).
+
+## Reference implementations
+
+- **GCP** — closest analogue to Azure. Has push + pull, OIDC auth, ARM-like resource paths, capture handler. Grep for `gcp_trigger` / `GcpTrigger`.
+- **Kafka** — simpler (pull-only, streaming). Good for trivial integrations.
+- **Azure** — most recently added (2026). Shared-secret push auth, Event Grid namespaces + basic topics, ARM resource discovery, Namespace-pull data-plane. Grep for `azure_trigger` / `AzureTrigger`.
+
+## 1. Database migration
+
+Create a migration: `cargo sqlx migrate add -r add_{kind}_trigger` from `backend/`. Never write timestamps manually.
+
+The `up.sql` usually defines:
+- An optional enum type (e.g. `AZURE_MODE`) if the trigger has sub-kinds
+- The `{kind}_trigger` table with at minimum these columns (mirrored from kafka/gcp):
+  - primary: `(workspace_id, path)`
+  - `script_path`, `is_flow`, `enabled`, `mode`, `permissioned_as`, `edited_by`, `email`
+  - `edited_at`, `error`, `server_id`, `last_server_ping`
+  - `error_handler_path`, `error_handler_args jsonb`, `retry jsonb`
+  - trigger-specific fields
+- Indexes on foreign keys + any frequently-filtered columns
+- Foreign key to `workspace`
+
+Down migration drops the table and any enum types.
+
+## 2. Backend crate (`windmill-trigger-{kind}`)
+
+Create a new crate under `backend/windmill-trigger-{kind}/` with:
+
+- `Cargo.toml`: features `enterprise`, `private` if EE, standard deps
+- `src/lib.rs`: `pub use mod_ee::*;` behind `#[cfg(all(feature = "enterprise", feature = "private"))]`
+- `src/mod_ee.rs`: core types + helpers
+- `src/handler_ee.rs`: `TriggerCrud` impl + route handlers
+- `src/listener_ee.rs`: (only if streaming/pull-based) `Listener` trait impl
+
+Required in `mod_ee.rs`:
+- `{Kind}Config` struct (persisted shape, `FromRow`)
+- `{Kind}ConfigRequest` struct (what API receives — usually similar to Config but with validation fields)
+- `{Kind}Trigger` unit struct (implements the traits)
+- `impl TriggerJobArgs for {Kind}Trigger` — sets `TRIGGER_KIND`, `Payload`, `v1_payload_fn`
+
+Required in `handler_ee.rs`:
+- `#[async_trait] impl TriggerCrud for {Kind}Trigger` with:
+  - `type Trigger = Trigger<{Kind}Config>`
+  - `type TriggerConfigRequest = {Kind}ConfigRequest`
+  - `const ROUTE_PREFIX: &'static str = "/{kind}_triggers";`
+  - `const TABLE_NAME`, `ADDITIONAL_SELECT_FIELDS`
+  - `get_deployed_object`, `validate_config`, `create_trigger`, `update_trigger`, `delete_trigger`, `test_connection`
+  - `additional_routes` (optional — mount extra endpoints for things like ARM resource listing, topic discovery)
+
+Register the crate in `backend/Cargo.toml` as a workspace member and as a dep of `windmill-api` behind the feature flag.
+
+## 3. Wire into `windmill-api` (feature-gated everywhere)
+
+**`backend/windmill-api/src/triggers/handler.rs`** — mount the trigger crate:
+```rust
+#[cfg(all(feature = "enterprise", feature = "{kind}_trigger", feature = "private"))]
+{
+    use crate::triggers::{kind}::{Kind}Trigger;
+    router = router.nest({Kind}Trigger::ROUTE_PREFIX, complete_trigger_routes({Kind}Trigger));
+}
+```
+
+**`backend/windmill-api/src/triggers/{kind}/mod.rs`** — re-export the crate:
+```rust
+pub use windmill_trigger_{kind}::*;
+```
+
+**`backend/windmill-api/src/lib.rs`** — if the trigger receives inbound pushes, add a webhook route:
+```rust
+.nest("/{kind}/w/{workspace_id}", {
+    #[cfg(all(feature = "enterprise", feature = "{kind}_trigger", feature = "private"))]
+    { triggers::{kind}::handler_oss::{kind}_push_route_handler() }
+    #[cfg(not(...))]
+    { Router::new() }
+})
+```
+
+## 4. `TriggerKind` enum (`backend/windmill-types/src/triggers.rs`)
+
+Already has slots for most triggers but verify your variant exists:
+- Add `{Kind}` to the `TriggerKind` enum
+- Add match arm in `to_key()`
+- Add match arm in `from_str`
+- Add match arm in `JobTriggerKind` (if jobs need kind tagging)
+
+## 5. OpenAPI (`backend/windmill-api/openapi.yaml`)
+
+This file is huge and the single most-forgotten place. Add:
+
+- `/w/{workspace}/{kind}_triggers/create` + `/update/{path}` + `/delete/{path}` + `/get/{path}` + `/list` + `/exists/{path}` + `/setmode/{path}` + `/test` paths (mirror gcp section)
+- Any `additional_routes` your handler exposes (resource discovery, etc.)
+- Schemas: `{Kind}Trigger`, `{Kind}TriggerData`, `{Kind}Mode` (if enum), `{Kind}DeliveryConfig`, helper request/response types
+- Add `{kind}` to `CaptureTriggerKind` enum
+- Add `{kind}_used: boolean` to the `UsedTriggers` response schema
+
+Regenerate frontend client: `npm run generate-backend-client` from `frontend/`.
+
+## 6. `UsedTriggers` + workspace export
+
+**`backend/windmill-api-workspaces/src/workspaces.rs`** — add `{kind}_used: bool` to the `UsedTriggers` struct and add an `EXISTS(SELECT 1 FROM {kind}_trigger …)` to the `get_used_triggers` query.
+
+**`backend/windmill-api/src/workspaces_export.rs`** — add export block mirroring gcp's (export lists all triggers, serializes them to YAML/JSON).
+
+## 6.5 Hardcoded trigger-kind arrays (silent-failure hotspots)
+
+Several files keep **hardcoded arrays** of trigger kind strings. Miss one and ACL checks / user offboarding / trash drop your kind:
+
+- **`backend/windmill-api-groups/src/granular_acls.rs`** — `KINDS: [&str; N]`. **Increment N** (the compile error is cryptic otherwise). Controls which kinds accept granular ACL operations.
+- **`backend/windmill-api-users/src/users.rs`** (`extra_perms_tables`) — which tables get `extra_perms` entries cleaned when a user is deleted.
+- **`backend/windmill-api/src/offboarding.rs`** — three separate arrays (enumeration, fork-copy, and delete paths). **All three** need the new kind.
+- **`backend/windmill-api/src/trash.rs`** — `valid_tables` for the trash / restore API.
+- **`backend/windmill-git-sync/src/lib.rs`** — add a test assertion for `DeployedObject::{Kind}Trigger.get_kind() == "{kind}_trigger"` (the `get_kind` match arm itself lives in the enum impl — already required by the Rust compiler).
+- **`backend/windmill-api-auth/src/scopes.rs`** — add the `{Kind}Triggers` variant to `ScopeDomain` enum + `as_str` match + `from_str` match. Required for the OAuth/token system to recognise `{kind}_triggers:read|write` scopes.
+- **`backend/windmill-api/src/token.rs`** (`build_trigger_scope_domains` → `TRIGGER_DOMAINS`) — add `("{kind}_triggers", "{Kind display name}")` so the CreateToken UI's scope selector surfaces the `read` / `write` checkboxes.
+
+**OpenAPI enums** to extend (do NOT forget — generated client will allow it but server rejects as 400):
+- `CaptureTriggerKind` enum
+- Three `kind` enums under `/w/{workspace}/acls/{get,add,remove}/{kind}/{path}` (yes, same list repeated three times)
+
+After editing any of these, run a full `cargo check` with your feature flag + `gcp_trigger` + other core flags — the `KINDS: [&str; N]` length mismatch only surfaces when the crate compiles.
+
+## 7. Capture infrastructure (`backend/windmill-api/src/capture.rs`)
+
+If the trigger supports push delivery, it also needs a capture endpoint so users can test it:
+
+- `{Kind}TriggerConfig` struct (gated by feature flags)
+- `TriggerConfig::{Kind}` variant
+- `set_{kind}_trigger_config` function (creates the subscription/equivalent pointing at the capture URL — use your `manage_{kind}_subscription` helper with `trigger_mode=false`)
+- Both real + no-op versions behind feature gates
+- `TriggerKind::{Kind} => set_{kind}_trigger_config(...)` arm in `set_config`
+- `{kind}_payload` async handler — validates auth (if any), processes payload, calls `insert_capture_payload`
+- Route: `.route("/{kind}/{runnable_kind}/{*path}", post({kind}_payload))` inside `workspaced_unauthed_service` — and expand the surrounding `#[cfg(any(...))]` to include your feature flag
+
+## 8. CLI (`cli/`) — easy to miss, breaks sync silently
+
+Check all of these:
+
+**`cli/src/types.ts`:**
+- Add `"{kind}"` to `TRIGGER_TYPES` array
+- Add `"{kind}_trigger"` to `getTypeStrFromPath` return union
+- Add match case in `getTypeStrFromPath`'s `typeEnding ===` chain
+- Add `pushTrigger("{kind}", ...)` branch in `pushObj`
+
+**`cli/src/commands/trigger/trigger.ts`:**
+- Import `{Kind}Trigger` type
+- Add `{kind}: {Kind}Trigger` to the `Trigger` type map
+- Add `{kind}: wmill.get{Kind}Trigger`, `update{Kind}Trigger`, `create{Kind}Trigger` to each function map
+- Add `{kind}: { ... }` template to `triggerTemplates`
+- Add `list{Kind}Triggers` call + spread in the `list` aggregation
+- Update `--kind` option descriptions to mention the new kind
+
+**`cli/src/commands/sync/sync.ts`:**
+- Add `path.endsWith(".{kind}_trigger" + ext)` in the file-type filter
+- Add `typ == "{kind}_trigger"` in `getTypeOrder`
+- Add `"{kind}_trigger"` to the delete-suffix regex (~line 3092)
+- Add a `case "{kind}_trigger"` in the delete switch
+
+**`cli/src/guidance/skills.ts`** — **DO NOT EDIT DIRECTLY**. It's auto-generated by `system_prompts/generate.py`. Instead:
+- Edit `system_prompts/utils.py` → append `('{Kind}Trigger', '{kind}_trigger')` to the `SCHEMA_MAPPINGS['triggers']` list (this is the master list — the one in `generate.py` is duplicated and `utils.py` wins)
+- Then run `python3 system_prompts/generate.py` — it regenerates `cli/src/guidance/skills.ts` with the schema extracted from `backend/windmill-api/openapi.yaml`
+- Commit the regenerated file
+
+## 9. Frontend — editor + drawer
+
+Under `frontend/src/lib/components/triggers/{kind}/`:
+
+- `{Kind}TriggerPanel.svelte` — the tile shown in the triggers listing
+- `{Kind}TriggerEditor.svelte` — outer drawer wrapper
+- `{Kind}TriggerEditorInner.svelte` — state + business logic; must expose:
+  - `openEdit(path, isFlow, defaultValues?)` method
+  - `isEditor` prop, `onConfigChange` + `onCaptureConfigChange` callbacks
+  - `get{Kind}Config()` + `get{Kind}CaptureConfig()` helpers
+  - `captureConfig = $derived.by(untrack(() => isEditor) ? get{Kind}CaptureConfig : () => ({}))`
+  - `$effect(() => { const args = [captureConfig, isValid] as const; untrack(() => onCaptureConfigChange?.(...args)) })`
+- `{Kind}TriggerEditorConfigSection.svelte` — form fields; use design-system components (`TextInput`, `Select`, `Toggle`, `ToggleButtonGroup`), never raw `<input>`
+- `{Kind}Capture.svelte` — capture panel; wraps `CaptureSection` with `captureType="{kind}"`
+- `utils.ts` — `requestBody` builders and any trigger-type-specific helpers
+
+## 10. Frontend — global integration
+
+Easy to miss:
+
+- **`frontend/src/lib/components/triggers.ts`** — add `'{kind}'` to the `TriggerKind` union
+- **`frontend/src/lib/components/triggers/CaptureWrapper.svelte`**:
+  - Import `{Kind}Capture`
+  - Add to `isStreamingCapture()` array (streaming = pull-style; push-style is typically `false`)
+  - Add `{:else if captureType === '{kind}'}` branch with the `<{Kind}Capture>` render
+- **`frontend/src/lib/components/sidebar/SidebarContent.svelte`** — import the icon, add the nav entry
+- **`frontend/src/lib/components/sidebar/OperatorMenu.svelte`** — add the operator-mode entry
+- **`frontend/src/routes/(root)/(logged)/+layout.svelte`** — destructure `{kind}_used` from `/get_used_triggers` response, push `'{kind}'` into `usedKinds`
+- **`frontend/src/lib/components/search/GlobalSearchModal.svelte`** — import icon, add "Go to {Kind} ..." entry
+- **`frontend/src/lib/components/offboarding-utils.ts`** — add mappings `{kind}_trigger: '{kind}_triggers'` and `{kind}_trigger: '{kind} trigger'`
+- **`frontend/src/lib/components/icons/{Kind}Icon.svelte`** — single-path SVG, `fill={color ?? 'currentColor'}`, `size` prop default 16 (match existing icons — don't hardcode colors, don't use `width`/`height` props)
+- **`frontend/src/routes/(root)/(logged)/{kind}_triggers/+page.svelte`** — listing page (mirror `gcp_triggers/+page.svelte` for push+pull, `kafka_triggers` for pure streaming)
+- **`frontend/src/lib/components/CompareWorkspaces.svelte`** — workspace fork / compare tool. Needs: service import, editor import, `{kind}Editor` `$state`, `case '{kind}'` in `openTriggerDetails()`, entry in `triggerServices` object (list/delete/normalize), and `<{Kind}TriggerEditor bind:this={{kind}Editor} />` in the template
+
+## 10.5 AI system prompts (`system_prompts/`)
+
+- **`system_prompts/utils.py`** — append `('{Kind}Trigger', '{kind}_trigger')` to `SCHEMA_MAPPINGS['triggers']` (master list used by code generation + CLI skills)
+- **`system_prompts/generate.py`** — also has a duplicated `schema_types` list (~line 903) for the AI `triggers` skill content. Add `('{Kind}Trigger', '{kind}_trigger')` there too
+- **`system_prompts/generate.py`** `schema_names` (~line 1192) — add `'{Kind}Trigger'` (add `'New{Kind}Trigger'` only if the OpenAPI declares one; GCP and Azure don't)
+- Run `python3 system_prompts/generate.py` — this rewrites `cli/src/guidance/skills.ts` and all `auto-generated/` docs. Commit the regenerated files
+
+## 11. Validation
+
+Run all of these before declaring done:
+
+```bash
+# Backend
+cd backend
+cargo check --features enterprise,{kind}_trigger,private   # minimal
+cargo check --features enterprise,azure_trigger,private,gcp_trigger,http_trigger,mqtt_trigger,postgres_trigger,sqs_trigger,kafka,nats,smtp,websocket  # full
+
+# SQLx offline data (never run `cargo sqlx prepare` directly — use the wrapper)
+./update_sqlx.sh
+
+# Frontend
+cd frontend
+npm run generate-backend-client
+npm run check:fast
+```
+
+Smoke test in the UI: create a trigger, save, check it appears in sidebar + search, delete, re-create via CLI `wmill sync`.
+
+## 12. Common pitfalls
+
+- **Forgetting feature gates in `workspaced_unauthed_service()`** — the surrounding `#[cfg(any(...))]` expression must include your feature flag, not just the inner `#[cfg]` on the route
+- **`.route(path, ...).route(path, ...)` with same path and different methods** — older axum replaced; use `.route(path, post(h1).options(h2))` to chain methods on the same `MethodRouter`
+- **`on:event` directives** — legacy Svelte 4, no-op in runes mode. Use callback props (`onSelected`, `onConfigChange`)
+- **`$bindable(default_value)` on optional props** — banned by project CLAUDE.md. Use `$bindable()` + `$derived(prop ?? default)` instead
+- **CORS layer intercepting OPTIONS** — tower-http CorsLayer short-circuits OPTIONS before reaching your handler. For server-to-server webhook endpoints, drop the CORS layer entirely (CORS is browser-only)
+- **DeliveryAttributeMappings / custom headers for auth** — prefer HMAC or sha256-hashed shared secrets over opaque JWTs when the provider doesn't support signed tokens natively. Store only the hash; regenerate secret on every save
+- **ARM / API resource-listing cascades** — if the trigger's resource type is deep (Azure: subscription → RG → namespace → topic), offer dropdowns in the UI populated from the provider's APIs using the user's credential resource
+- **Clearing stale selections on dependency change** — when a dropdown's underlying data reloads (e.g., user changes SP or edition), clear selections that no longer match the new list
+- **Workspace-scoped tag compatibility** — if the trigger has tags, verify forked workspaces handle them (see commit `0773b5bc85` for a historical fix)
+
+## 13. EE file split
+
+If the trigger is enterprise-only, the code lives in `windmill-ee-private__worktrees/.../windmill-trigger-{kind}/src/*_ee.rs` and is symlinked into the OSS tree. The `windmill-ee-private__worktrees/` directory holds the real files; changes propagate via symlinks. See `docs/enterprise.md` for the workflow.
+
+## 14. Final checklist before PR
+
+- [ ] Migration up/down tested (revert + re-apply)
+- [ ] `./update_sqlx.sh` committed the updated `.sqlx/` offline data
+- [ ] `cargo check` passes with your feature flag + with all trigger features
+- [ ] `npm run check:fast` passes
+- [ ] Trigger visible in sidebar with correct icon weight (not oversized/colored — use `currentColor`)
+- [ ] Create, edit, delete flow all work in the UI
+- [ ] Capture button works (if push-capable)
+- [ ] Trigger appears in `/get_used_triggers` → sidebar pulse
+- [ ] `wmill sync pull` + `wmill sync push` both round-trip the trigger
+- [ ] `wmill trigger list` includes it
+- [ ] OpenAPI schemas are complete (no `null` in generated types)

.claude/skills/native-trigger/SKILL.md

@@ -607,7 +607,18 @@ In `frontend/src/lib/components/triggers/TriggersEditor.svelte`:
 
 Add your service to the `nativeTriggerServices` map in `deleteDeployedTrigger()`. Native triggers use `NativeTriggerService.deleteNativeTrigger({ workspace, serviceName, externalId })` instead of the standard `path`-based delete.
 
-### Step 17: Update OpenAPI Spec and Regenerate Types
+### Step 17: Update `getUsedTriggers` for Sidebar Visibility
+
+The sidebar (`frontend/src/lib/components/sidebar/SidebarContent.svelte`) shows native-trigger links only if `$usedTriggerKinds` includes the service — without this, your trigger page will never appear in the nav bar even when triggers exist.
+
+1. **Backend** — add `{service}_used: bool` to the `UsedTriggers` struct and SELECT in `backend/windmill-api-workspaces/src/workspaces.rs::get_used_triggers()`:
+   ```rust
+   EXISTS(SELECT 1 FROM native_trigger WHERE workspace_id = $1 AND service_name = '{service}'::native_trigger_service) AS "{service}_used!"
+   ```
+2. **OpenAPI** — add `{service}_used: boolean` to the response schema for `GET /w/{workspace}/workspaces/used_triggers` (under both `properties` and `required`).
+3. **Layout** — in `frontend/src/routes/(root)/(logged)/+layout.svelte::loadUsedTriggerKinds()`, destructure `{service}_used` and push `'{service}'` to `usedKinds`.
+
+### Step 18: Update OpenAPI Spec and Regenerate Types
 
 Add to `JobTriggerKind` enum in `backend/windmill-api/openapi.yaml`, then:
 

.codex/config.toml

@@ -0,0 +1,4 @@
+#:schema https://developers.openai.com/codex/config-schema.json
+
+[mcp_servers.svelte]
+url = "https://mcp.svelte.dev/mcp"

.github/change-versions-mac.sh

@@ -20,4 +20,10 @@ sed -i '' -e "/^wmill =/s/= .*/= \">=$VERSION\"/" ${root_dirpath}/lsp/Pipfile
 
 sed -i '' -E "s/name = \"windmill\"\nversion = \"[^\"]*\"\\n(.*)/name = \"windmill\"\nversion = \"$VERSION\"\\n\\1/" ${root_dirpath}/backend/Cargo.lock
 
+# windmill-parser-wasm is its own workspace (excluded from the backend workspace
+# because of nightly-only cargo-features), so its version lives in
+# [workspace.package] and its Cargo.lock is not regenerated by the backend step.
+sed -i '' -e "/^version =/s/= .*/= \"$VERSION\"/" ${root_dirpath}/backend/parsers/windmill-parser-wasm/Cargo.toml
+sed -i '' -E "s/(name = \"windmill[^\"]*\"\nversion = )\"[^\"]*\"/\\1\"$VERSION\"/g" ${root_dirpath}/backend/parsers/windmill-parser-wasm/Cargo.lock
+
 cd ${root_dirpath}/frontend && npm i --package-lock-only

.github/change-versions.sh

@@ -21,4 +21,10 @@ sed -i -e "/^wmill =/s/= .*/= \">=$VERSION\"/" ${root_dirpath}/lsp/Pipfile
 
 sed -i -zE "s/name = \"windmill\"\nversion = \"[^\"]*\"\\n(.*)/name = \"windmill\"\nversion = \"$VERSION\"\\n\\1/" ${root_dirpath}/backend/Cargo.lock
 
+# windmill-parser-wasm is its own workspace (excluded from the backend workspace
+# because of nightly-only cargo-features), so its version lives in
+# [workspace.package] and its Cargo.lock is not regenerated by the backend step.
+sed -i -e "/^version =/s/= .*/= \"$VERSION\"/" ${root_dirpath}/backend/parsers/windmill-parser-wasm/Cargo.toml
+sed -i -zE "s/(name = \"windmill[^\"]*\"\nversion = )\"[^\"]*\"/\\1\"$VERSION\"/g" ${root_dirpath}/backend/parsers/windmill-parser-wasm/Cargo.lock
+
 cd ${root_dirpath}/frontend && npm i --package-lock-only --ignore-scripts

.github/workflows/backend-test-windows.yml

@@ -145,6 +145,10 @@ jobs:
           RUST_LOG_STYLE: never
           CARGO_NET_GIT_FETCH_WITH_CLI: true
           CARGO_BUILD_JOBS: 12
+          # Tests' poll-time stack frames (deep nested async fn chains in
+          # debug builds) reach ~1.8MB. 4MB gives ~2x headroom against flaky
+          # overflows under parallel-test contention.
+          RUST_MIN_STACK: 4194304
           VCPKGRS_DYNAMIC: 1
           OPENSSL_DIR: ${{ env.VCPKG_INSTALLATION_ROOT }}\installed\x64-windows-static
           DENO_PATH: ${{ steps.runtime-paths.outputs.DENO_PATH }}

.github/workflows/backend-test.yml

@@ -244,6 +244,11 @@ jobs:
           RUST_LOG_STYLE: never
           CARGO_NET_GIT_FETCH_WITH_CLI: true
           CARGO_BUILD_JOBS: 12
+          # Tests' poll-time stack frames (deep nested async fn chains in
+          # debug builds) reach ~1.8MB, leaving very thin headroom on the
+          # default 2MB thread stack. 4MB gives ~2x buffer against flaky
+          # overflows under parallel-test contention.
+          RUST_MIN_STACK: 4194304
           WMDEBUG_FORCE_V0_WORKSPACE_DEPENDENCIES: 1
           WMDEBUG_FORCE_RUNNABLE_SETTINGS_V0: 1
           WMDEBUG_FORCE_NO_LEGACY_DEBOUNCING_COMPAT: 1

.github/workflows/check-system-prompts.yml

@@ -10,6 +10,7 @@ on:
       - "backend/windmill-api/openapi.yaml"
       - "cli/src/main.ts"
       - "cli/src/commands/**"
+      - "frontend/src/lib/components/copilot/chat/workspaceToolsZod.gen.ts"
   pull_request:
     paths:
       - "system_prompts/**"
@@ -19,6 +20,7 @@ on:
       - "backend/windmill-api/openapi.yaml"
       - "cli/src/main.ts"
       - "cli/src/commands/**"
+      - "frontend/src/lib/components/copilot/chat/workspaceToolsZod.gen.ts"
 
 jobs:
   check-freshness:

AGENTS.md

@@ -0,0 +1,87 @@
+# Windmill
+
+Open-source platform for internal tools, workflows, API integrations, background jobs, and UIs. Rust backend + Svelte 5 frontend.
+
+## Workflow
+
+1. **Understand**: Before coding, explore the codebase (see Code Navigation below). Use `outline` to understand file structure, `body` to read specific symbols, `def`/`callers`/`callees` to trace code, `Grep` to find usages. Read `docs/` for domain context.
+2. **Plan**: For non-trivial changes, use plan mode. For large features, break into reviewable stages
+3. **Execute**: Follow coding patterns from skills (`rust-backend`, `svelte-frontend`)
+4. **Validate**: After every change, run the appropriate checks per `docs/validation.md`
+
+## Documentation
+
+- **Validation**: `docs/validation.md` — what checks to run based on what you changed
+- **Enterprise**: `docs/enterprise.md` — EE file conventions and PR workflow
+- **Backend patterns**: use the `rust-backend` skill when writing Rust code
+- **Frontend patterns**: use the `svelte-frontend` skill when writing Svelte code. Do NOT edit svelte files unless you have read that skill.
+- **Code review**: use `/local-review` to review a PR for bugs and CLAUDE.md compliance
+- **Domain guides**: `.claude/skills/native-trigger/` and `frontend/tutorial-system-guide.mdc`
+- **Brand/UI guidelines**: `frontend/brand-guidelines.md`
+
+## Dev Environment
+
+- **Backend**: `cargo run` from `backend/` (API at http://localhost:8000)
+- **Frontend**: `REMOTE=http://localhost:8000 npm run dev` from `frontend/` (port 3000+)
+- **DB**: `psql postgres://postgres:changeme@localhost:5432/windmill`
+- **Login**: `admin@windmill.dev` / `changeme`
+- **Instance settings**: navigate to `/#superadmin-settings`
+- **Migrations**: use `cargo sqlx migrate add -r <name>` from `backend/` to create new migrations (never generate timestamps manually)
+
+## Banned Patterns
+
+### `$bindable(default_value)` on optional props
+
+Using `$bindable(default_value)` on props that can be `undefined` is **banned**. This pattern causes subtle bugs because the default value masks the `undefined` state.
+
+**Bad:**
+
+```svelte
+let { my_prop = $bindable(default_value) }: { my_prop?: string } = $props()
+```
+
+**Correct alternatives:**
+
+1. **Use `$derived` with nullish coalescing** — handle the potential `undefined` at the usage site:
+
+   ```svelte
+   let { my_prop = $bindable() }: { my_prop?: string } = $props()
+   let effective_value = $derived(my_prop ?? default_value)
+   ```
+
+2. **Create a `useMyPropState()` helper** — encapsulate the undefined-handling logic in a reusable function and call it higher in the component tree, so the child component always receives a defined value.
+
+## Code Navigation
+
+`wm-ts-nav` is an AST-aware code navigator. Use **wm-ts-nav** for structural queries — it skips comments/strings and understands symbol boundaries.
+
+**MUST use `outline` before `Read`** on unfamiliar files — a 500-line file costs ~500 lines of context, while `outline` costs ~20. Then **MUST use `body "X"`** instead of reading a full file to see one function/struct. Use `Read` with offset/limit only when you need surrounding context that `body` doesn't capture.
+- `refs "X" --caller` instead of reading files to find which function contains each reference
+- `callers "X"` / `callees "X"` for call-graph questions
+
+EE files (`*_ee.rs`, `*_ee.ts`, `*_ee.svelte`) are indexed — you can `outline`, `def`, `body`, `refs` etc. on them just like regular files.
+
+```bash
+NAV="sh wm-ts-nav/nav"
+# Use --root backend for Rust, --root frontend/src for TS/Svelte
+$NAV --root backend outline backend/path/to/file.rs      # file structure
+$NAV --root backend def "ServiceName"                     # find definition
+$NAV --root backend body "decrypt_oauth_data"             # extract source code
+$NAV --root backend search "%" --parent ServiceName       # methods on a type
+$NAV --root backend search "Trigger" --kind struct        # find by kind
+$NAV --root backend refs "X" --file handler.rs --caller   # scoped refs with caller
+$NAV --root backend callers "X"                           # who calls X?
+$NAV --root backend callees "X"                           # what does X call?
+```
+
+**Limitations** — syntax-level analysis, no type inference. Use **Grep** instead when completeness matters (finding all usages, exhaustiveness checks):
+- `refs`/`callers`/`callees` can't follow re-exports, glob imports, or different import paths to the same symbol
+- Trait impls, macro-generated symbols (`sqlx::FromRow`), and namespace member access (`ns.X`) are invisible
+- `callees` shows all identifiers in a function body, not just actual calls
+
+## Core Principles
+
+- **MUST `outline` before `Read`** on unfamiliar files — then `body` or `Read` with offset/limit for specifics
+- Search for existing code to reuse before writing new code
+- Follow established patterns in the codebase
+- Keep changes focused — don't refactor beyond what's asked

CHANGELOG.md

@@ -1,5 +1,209 @@
 # Changelog
 
+## [1.693.2](https://github.com/windmill-labs/windmill/compare/v1.693.1...v1.693.2) (2026-04-30)
+
+
+### Bug Fixes
+
+* avoid effect_update_depth_exceeded when clicking flow node on runs page ([#8986](https://github.com/windmill-labs/windmill/issues/8986)) ([3ebfc2b](https://github.com/windmill-labs/windmill/commit/3ebfc2b0af38f7eb17774de08a214d7813e07952))
+* OAuth popup login reliability + auto-login Safari edge cases ([#8971](https://github.com/windmill-labs/windmill/issues/8971)) ([3c3c034](https://github.com/windmill-labs/windmill/commit/3c3c03455d68fde982937787992e20c3f8eeeaaf))
+
+## [1.693.1](https://github.com/windmill-labs/windmill/compare/v1.693.0...v1.693.1) (2026-04-29)
+
+
+### Bug Fixes
+
+* include labels when loading flow with draft for editing ([#8981](https://github.com/windmill-labs/windmill/issues/8981)) ([485d1d1](https://github.com/windmill-labs/windmill/commit/485d1d1e3785b5ed7e5f1a5ee127c0fed15fba3e)), closes [#8963](https://github.com/windmill-labs/windmill/issues/8963)
+
+## [1.693.0](https://github.com/windmill-labs/windmill/compare/v1.692.0...v1.693.0) (2026-04-29)
+
+
+### Features
+
+* add ai chat schedule and trigger tools ([#8961](https://github.com/windmill-labs/windmill/issues/8961)) ([b883f9a](https://github.com/windmill-labs/windmill/commit/b883f9a9d2e38a5981860de268fa6227cdd645de))
+* add delete_after_secs and sensitive_inputs for raw app runnables ([#8975](https://github.com/windmill-labs/windmill/issues/8975)) ([1169d9b](https://github.com/windmill-labs/windmill/commit/1169d9bfd315e43194f9e5a2bc55af843476ef68))
+* add min release age instance settings for bun and uv ([#8956](https://github.com/windmill-labs/windmill/issues/8956)) ([1d279e7](https://github.com/windmill-labs/windmill/commit/1d279e7a1e77fd183bb0c99d2430cfd9dc0a617c))
+* edit scopes on existing API tokens ([#8967](https://github.com/windmill-labs/windmill/issues/8967)) ([e9e72fb](https://github.com/windmill-labs/windmill/commit/e9e72fbbf83363ba1426b3dde3d657de02ba50b3))
+* OTEL span status on failed jobs + Python stderr severity classification ([#8918](https://github.com/windmill-labs/windmill/issues/8918)) ([cec8484](https://github.com/windmill-labs/windmill/commit/cec84849b9aea92d355dd346546969027c603498))
+* support restart from steps inside BranchOne, ForLoop, Subflow ([#8955](https://github.com/windmill-labs/windmill/issues/8955)) ([c956428](https://github.com/windmill-labs/windmill/commit/c95642863e366c106d529a57540a75df9480397c))
+* support S3Object input args in native SQL scripts ([#8954](https://github.com/windmill-labs/windmill/issues/8954)) ([c0eeea9](https://github.com/windmill-labs/windmill/commit/c0eeea9c833f9be3981389a19d0964400fd2bda8))
+* workspace-shared ui/ folder reusable across raw apps ([#8974](https://github.com/windmill-labs/windmill/issues/8974)) ([de0b6b1](https://github.com/windmill-labs/windmill/commit/de0b6b15285612ef00b94e332dcb203aed88f2cc))
+
+
+### Bug Fixes
+
+* **cli:** debounce wmill dev flow round-trip 200ms ([#8977](https://github.com/windmill-labs/windmill/issues/8977)) ([5861dca](https://github.com/windmill-labs/windmill/commit/5861dcad589df99f57e730d8d9cdb3eabc3424e4))
+* prevent React app editor from overwriting files on theme switch ([#8965](https://github.com/windmill-labs/windmill/issues/8965)) ([70b90c4](https://github.com/windmill-labs/windmill/commit/70b90c41dc28d1a850bce69836cde920c1404c3b))
+* show skipped label on flow progress bar ([#8973](https://github.com/windmill-labs/windmill/issues/8973)) ([8627d3c](https://github.com/windmill-labs/windmill/commit/8627d3c5aeabfac12a9f06211f8e7db020e758f9))
+* split flow prompts for frontend chat ([#8968](https://github.com/windmill-labs/windmill/issues/8968)) ([4098793](https://github.com/windmill-labs/windmill/commit/4098793db22249c5b4467c2adb72131407f5d6d3))
+* strip additionalProperties from google schemas ([#8964](https://github.com/windmill-labs/windmill/issues/8964)) ([77d9a53](https://github.com/windmill-labs/windmill/commit/77d9a534235a1ff8cbfcdb037a65735db987e2fe))
+
+
+### Performance Improvements
+
+* optimize datatable app chat schemas ([#8960](https://github.com/windmill-labs/windmill/issues/8960)) ([34b549c](https://github.com/windmill-labs/windmill/commit/34b549cfe2e2e060561eabe369a99cfb4d9c9568))
+
+## [1.692.0](https://github.com/windmill-labs/windmill/compare/v1.691.1...v1.692.0) (2026-04-27)
+
+
+### Features
+
+* add agents skills to cli init ([#8948](https://github.com/windmill-labs/windmill/issues/8948)) ([abbfd50](https://github.com/windmill-labs/windmill/commit/abbfd504ac6dd8fe3aa2bb3acfab7f72f24b81c6))
+* **cli:** wmill dev with per-flow proxy and responsive Dev UI ([#8529](https://github.com/windmill-labs/windmill/issues/8529)) ([eebe24d](https://github.com/windmill-labs/windmill/commit/eebe24d8b0739a61d308bc53a322a01df984511a))
+
+
+### Bug Fixes
+
+* Audit logs filters UI spacing ([#8944](https://github.com/windmill-labs/windmill/issues/8944)) ([15bba79](https://github.com/windmill-labs/windmill/commit/15bba79ef25988f923b9604a58b719f9f244c641))
+* delete instance settings cleared via bulk endpoint ([#8949](https://github.com/windmill-labs/windmill/issues/8949)) ([e8f7589](https://github.com/windmill-labs/windmill/commit/e8f7589d7a9cba5b050a35ff7b92116e679566f6))
+* prevent flow-dep job stalls under row-lock contention ([#8952](https://github.com/windmill-labs/windmill/issues/8952)) ([e636f58](https://github.com/windmill-labs/windmill/commit/e636f589a534b29f5deceed3674c6b7d312cb6d4))
+* **wac:** recognize [@workflow](https://github.com/workflow) main, list WAC in scripts/list, run preprocessor ([#8951](https://github.com/windmill-labs/windmill/issues/8951)) ([581658d](https://github.com/windmill-labs/windmill/commit/581658d881dd35e39a3fb8f4216d2f91d4184b03))
+
+## [1.691.1](https://github.com/windmill-labs/windmill/compare/v1.691.0...v1.691.1) (2026-04-27)
+
+
+### Bug Fixes
+
+* **cli:** preserve case in raw-app runnable filenames ([#8940](https://github.com/windmill-labs/windmill/issues/8940)) ([2f58a31](https://github.com/windmill-labs/windmill/commit/2f58a31d009025c18e8eba087ea7001f02639615))
+* preserve s3 rootPath when reloading git repo viewer ([#8942](https://github.com/windmill-labs/windmill/issues/8942)) ([b8fcb7f](https://github.com/windmill-labs/windmill/commit/b8fcb7f04b6a47cd913710522ead37cc07c56fdd))
+
+## [1.691.0](https://github.com/windmill-labs/windmill/compare/v1.690.0...v1.691.0) (2026-04-24)
+
+
+### Features
+
+* add auto-login SSO provider instance setting ([#8929](https://github.com/windmill-labs/windmill/issues/8929)) ([4cf53a4](https://github.com/windmill-labs/windmill/commit/4cf53a44bb10b65dcdb45bac97186a10cdbb48d6))
+* cli diff/deploy no-op handling + promotion debouncing ([#8936](https://github.com/windmill-labs/windmill/issues/8936)) ([489337d](https://github.com/windmill-labs/windmill/commit/489337d5333e31164d83efad5f0fb433f4093640))
+* **cli:** non-interactive Slack connect/disconnect + sync round-trip fixes ([#8935](https://github.com/windmill-labs/windmill/issues/8935)) ([95d4c6a](https://github.com/windmill-labs/windmill/commit/95d4c6a94dfdaf311cba44b2049202dcd819c835))
+* WM_TESTED_RUNNABLE env var + wildcards in test: annotation ([#8926](https://github.com/windmill-labs/windmill/issues/8926)) ([8a98650](https://github.com/windmill-labs/windmill/commit/8a986500b932753508bf5f380f6458a9e1375449))
+
+
+### Bug Fixes
+
+* **autoscaling:** native worker stuck at max + wrong TimeAgo ([#8930](https://github.com/windmill-labs/windmill/issues/8930)) ([73fab0c](https://github.com/windmill-labs/windmill/commit/73fab0c26441678c5efaf8b38f57ba0bd7293522))
+* **nativets:** forward OTEL-prefixed console logs to tracing events ([#8937](https://github.com/windmill-labs/windmill/issues/8937)) ([e732004](https://github.com/windmill-labs/windmill/commit/e732004180728a2dfa45083225d204d9fde89d06))
+
+## [1.690.0](https://github.com/windmill-labs/windmill/compare/v1.689.0...v1.690.0) (2026-04-23)
+
+
+### Features
+
+* add ai agent conversation output control ([#8915](https://github.com/windmill-labs/windmill/issues/8915)) ([9a60ff2](https://github.com/windmill-labs/windmill/commit/9a60ff2e77f197786f523755c3a9286a178a245c))
+* add Azure Event Grid triggers ([#8888](https://github.com/windmill-labs/windmill/issues/8888)) ([d6c642b](https://github.com/windmill-labs/windmill/commit/d6c642b170b9547fe1d8db190affa35b305c9c8a))
+* add OTEL_HOST_NAME env override for host.name attribute ([#8923](https://github.com/windmill-labs/windmill/issues/8923)) ([f429cb5](https://github.com/windmill-labs/windmill/commit/f429cb5e486aa5d2bd37f84c1fb30b9d350909e4))
+
+
+### Bug Fixes
+
+* **cli:** use wmill.yaml key consistently for workspace-specific items ([#8900](https://github.com/windmill-labs/windmill/issues/8900)) ([1722a7a](https://github.com/windmill-labs/windmill/commit/1722a7a2af5e00beeae204b78e588cd74a3ceb39))
+* correct flow conversation pagination ([#8919](https://github.com/windmill-labs/windmill/issues/8919)) ([7fa924e](https://github.com/windmill-labs/windmill/commit/7fa924e67e212458726a839dbe366798b2709cd6))
+* ensure schema is inferred on script/flow module load ([#8927](https://github.com/windmill-labs/windmill/issues/8927)) ([664d0f8](https://github.com/windmill-labs/windmill/commit/664d0f838d168978d7c27e88d2bb9019531e7ea1))
+* include endpoint descriptions in mcp tools ([#8925](https://github.com/windmill-labs/windmill/issues/8925)) ([07951e8](https://github.com/windmill-labs/windmill/commit/07951e81ae9a1c26e8fe63bcd7a760b80500ca4c))
+* load job metadata on approval page via approval token ([#8924](https://github.com/windmill-labs/windmill/issues/8924)) ([dac29e7](https://github.com/windmill-labs/windmill/commit/dac29e7d23d6e980c2b6fc4dd3a05a0d2e0170b3))
+* slim app ai chat context ([#8922](https://github.com/windmill-labs/windmill/issues/8922)) ([132d8a6](https://github.com/windmill-labs/windmill/commit/132d8a61f9c109b2b447fab1a39565f52864b746))
+
+## [1.689.0](https://github.com/windmill-labs/windmill/compare/v1.688.0...v1.689.0) (2026-04-22)
+
+
+### Features
+
+* add s3 stream progress logs to other DB executors ([#8898](https://github.com/windmill-labs/windmill/issues/8898)) ([2d4fadb](https://github.com/windmill-labs/windmill/commit/2d4fadb590590837412d638192fbd62bdc9331e8))
+* allow hiding catalog picker and raw input on s3 form fields ([#8902](https://github.com/windmill-labs/windmill/issues/8902)) ([05baa4a](https://github.com/windmill-labs/windmill/commit/05baa4ab026a307267d11fb827f8abcc246d1ac0))
+* async dep endpoints and queue-position logs in cli ([#8895](https://github.com/windmill-labs/windmill/issues/8895)) ([aaf3a19](https://github.com/windmill-labs/windmill/commit/aaf3a1974746be451adf463fd1f0e584b2fa995e))
+* auto-strip UTF-8 BOM when reading local files in CLI ([#8911](https://github.com/windmill-labs/windmill/issues/8911)) ([99bc96d](https://github.com/windmill-labs/windmill/commit/99bc96d0b231a2af303b28aa87d5de9141ee5cab))
+
+
+### Bug Fixes
+
+* add aws-config to private feature to restore ce build ([18eed92](https://github.com/windmill-labs/windmill/commit/18eed92dd66d635305a72818e2fcf0ee8b9672cc))
+* add flow conversation token scope ([#8903](https://github.com/windmill-labs/windmill/issues/8903)) ([aea7444](https://github.com/windmill-labs/windmill/commit/aea74445a31d30abb8030763db91e1829db772f0))
+* add proxy eval coverage for gemini schemas ([#8897](https://github.com/windmill-labs/windmill/issues/8897)) ([fddd8e2](https://github.com/windmill-labs/windmill/commit/fddd8e288fc0fd7af3b1df1ddd4476fb57694ed3))
+* apply powershell workspace dependencies to deployed scripts ([#8912](https://github.com/windmill-labs/windmill/issues/8912)) ([dc89673](https://github.com/windmill-labs/windmill/commit/dc896737ac1dcd90ab96314b2bc2f044ff833b8a))
+* detect and clearly label OOM in zombie flow alerts ([#8901](https://github.com/windmill-labs/windmill/issues/8901)) ([680c711](https://github.com/windmill-labs/windmill/commit/680c711f9262683c046a78532b22be5f5a4121a8))
+* omit default_permissioned_as from tarball export when empty ([bbb564c](https://github.com/windmill-labs/windmill/commit/bbb564c1420593014d17f352ba38d3ed38c248e1))
+* persist flow groups from AI chat tool calls ([#8906](https://github.com/windmill-labs/windmill/issues/8906)) ([932d183](https://github.com/windmill-labs/windmill/commit/932d18331196ef3e87c45d8a06ae45ac8013bd7a))
+* push parent resource on fileset child add/delete ([#8910](https://github.com/windmill-labs/windmill/issues/8910)) ([f29badc](https://github.com/windmill-labs/windmill/commit/f29badcf368e7c712f1515fa30a6a0e179a4bdc5))
+* rust nsjail RUSTUP_HOME mount and arch-aware cache keys ([#8890](https://github.com/windmill-labs/windmill/issues/8890)) ([f8c916c](https://github.com/windmill-labs/windmill/commit/f8c916cb6073f5566289c395ec39ec3919f449e7))
+* skip opus 4.7 sampling params ([#8904](https://github.com/windmill-labs/windmill/issues/8904)) ([1e83278](https://github.com/windmill-labs/windmill/commit/1e83278fe2ef5a5c6959a9351e7286d9dbf2453a))
+* support windmill chat answer override ([#8909](https://github.com/windmill-labs/windmill/issues/8909)) ([eeb5d12](https://github.com/windmill-labs/windmill/commit/eeb5d12be3ba2aedf2ebc4843d4395e241ecc8d3))
+* track dollar-quoted strings in SQL block splitter ([#8891](https://github.com/windmill-labs/windmill/issues/8891)) ([53badf1](https://github.com/windmill-labs/windmill/commit/53badf1a8cff576bb4ccbc75f045efb457b6a07d))
+* trigger failure_module when branchone predicate throws ([#8905](https://github.com/windmill-labs/windmill/issues/8905)) ([dffb89e](https://github.com/windmill-labs/windmill/commit/dffb89e00632bd4e7bbe9998bccb1f14357d9c07)), closes [#8889](https://github.com/windmill-labs/windmill/issues/8889)
+
+## [1.688.0](https://github.com/windmill-labs/windmill/compare/v1.687.0...v1.688.0) (2026-04-20)
+
+
+### Features
+
+* add homepage connect drawer ([#8880](https://github.com/windmill-labs/windmill/issues/8880)) ([f35e10c](https://github.com/windmill-labs/windmill/commit/f35e10cc0aed86e5230db5786ad1d6d33bb37b94))
+* improve app evals and localized app edits ([#8863](https://github.com/windmill-labs/windmill/issues/8863)) ([46b2915](https://github.com/windmill-labs/windmill/commit/46b2915a9d6350452d1f43ef108e7925b7f879e0))
+
+
+### Bug Fixes
+
+* batch cancel dropping jobs from other workspaces ([#8887](https://github.com/windmill-labs/windmill/issues/8887)) ([10d1a93](https://github.com/windmill-labs/windmill/commit/10d1a932d50044bedfd3c837c7980d68df49bbe6))
+* log boolean predicate eval errors to root flow logs ([#8885](https://github.com/windmill-labs/windmill/issues/8885)) ([94f27af](https://github.com/windmill-labs/windmill/commit/94f27af838294bc76bac57fcd1784a21675aec1c))
+* populate wmill.d.ts schemas in wmill app dev ([#8882](https://github.com/windmill-labs/windmill/issues/8882)) ([12c08cc](https://github.com/windmill-labs/windmill/commit/12c08cc95c2aedbcad50fe0e6392b30fd8438e49))
+* use POST for oidc token request in authed client ([#8883](https://github.com/windmill-labs/windmill/issues/8883)) ([71c4212](https://github.com/windmill-labs/windmill/commit/71c4212a903870357ff62bf90343a2fa0aeaac79))
+
+
+### Performance Improvements
+
+* speed up mssql s3 ingest and add phase logs to job output ([#8884](https://github.com/windmill-labs/windmill/issues/8884)) ([43a6b57](https://github.com/windmill-labs/windmill/commit/43a6b575817ebe386299c022b9bbb5f3e92ccffe))
+
+## [1.687.0](https://github.com/windmill-labs/windmill/compare/v1.686.0...v1.687.0) (2026-04-17)
+
+
+### Features
+
+* add disable_password_login global setting ([#8873](https://github.com/windmill-labs/windmill/issues/8873)) ([0cfa131](https://github.com/windmill-labs/windmill/commit/0cfa131254a517331124a83ae21fbe6508a8c24f))
+* add GitHub as a native trigger service ([#8856](https://github.com/windmill-labs/windmill/issues/8856)) ([4f998cc](https://github.com/windmill-labs/windmill/commit/4f998cc231119ba0ba5224223222b1d4d5976e22))
+
+
+### Bug Fixes
+
+* default null script/flow schema to empty in bg runnable ([#8872](https://github.com/windmill-labs/windmill/issues/8872)) ([1d2d12a](https://github.com/windmill-labs/windmill/commit/1d2d12a27de5731b6bf3e4dae14025ec87b7b4b8))
+* mint fresh Google channel IDs on update/renew ([#8870](https://github.com/windmill-labs/windmill/issues/8870)) ([ad2e855](https://github.com/windmill-labs/windmill/commit/ad2e855a83c94299c6fa9f572acb7e4688346e2e))
+
+## [1.686.0](https://github.com/windmill-labs/windmill/compare/v1.685.0...v1.686.0) (2026-04-17)
+
+
+### Features
+
+* add empty inline script warnings to flow chat ([#8853](https://github.com/windmill-labs/windmill/issues/8853)) ([51b09ac](https://github.com/windmill-labs/windmill/commit/51b09ace45440acd055f95b16c4ad6451ea5c8d5))
+* generate tsconfig.json during wmill init for IDE type support ([#8855](https://github.com/windmill-labs/windmill/issues/8855)) ([0b6874f](https://github.com/windmill-labs/windmill/commit/0b6874fb0d356e7f71b3914066af310e746b4b97))
+* migrate slack OAuth to v2 ([#8859](https://github.com/windmill-labs/windmill/issues/8859)) ([b1a4c78](https://github.com/windmill-labs/windmill/commit/b1a4c780dcfc06766b79683d37f1ae66e021adf9))
+
+
+### Bug Fixes
+
+* fix otel tracing on nativets ([172a7d1](https://github.com/windmill-labs/windmill/commit/172a7d16dbac420ba1b0174d8e7eee7ba78406cd))
+* include app owner in GitHub App URL for GHE Cloud ([#8846](https://github.com/windmill-labs/windmill/issues/8846)) ([e9ea06f](https://github.com/windmill-labs/windmill/commit/e9ea06f4c2b35dad2177a2fafc727463e2ce9f4e))
+* serve populated jwks at /.well-known/jwks.json for vault ([#8865](https://github.com/windmill-labs/windmill/issues/8865)) ([8514347](https://github.com/windmill-labs/windmill/commit/85143477841498fb8e8c947feb89d58d06d425d3))
+* update on_behalf_of_email in app policy on offboarding ([#8858](https://github.com/windmill-labs/windmill/issues/8858)) ([d99a176](https://github.com/windmill-labs/windmill/commit/d99a176b6ace181d5c99140a33e237bc8646fdc3))
+
+## [1.685.0](https://github.com/windmill-labs/windmill/compare/v1.684.1...v1.685.0) (2026-04-16)
+
+
+### Features
+
+* add compact json patch tool to flow chat ([#8840](https://github.com/windmill-labs/windmill/issues/8840)) ([b39671d](https://github.com/windmill-labs/windmill/commit/b39671d933e789301017d3efce6a448f35b3d407))
+
+
+### Bug Fixes
+
+* classify fileset resource files with script extensions correctly ([#8851](https://github.com/windmill-labs/windmill/issues/8851)) ([b1aeb33](https://github.com/windmill-labs/windmill/commit/b1aeb33adeb63a1ee521e7a1aacc552faab833da))
+* clean ai memory and cache bedrock prompts ([#8847](https://github.com/windmill-labs/windmill/issues/8847)) ([b177827](https://github.com/windmill-labs/windmill/commit/b1778272fc91d53922ee62fe4be304ff72767f0b))
+* encourage subflow reuse in AI chat flow builder prompt ([#8839](https://github.com/windmill-labs/windmill/issues/8839)) ([49844eb](https://github.com/windmill-labs/windmill/commit/49844eb240a24caacd3a86b5eb6b3c228e8c5fbe))
+* improve flow chat and benchmark coverage ([#8825](https://github.com/windmill-labs/windmill/issues/8825)) ([d3cb0c6](https://github.com/windmill-labs/windmill/commit/d3cb0c62204ebfebfa6859f38b3b597d719c573a))
+* include workspacedependencies in default git sync include_type ([#8852](https://github.com/windmill-labs/windmill/issues/8852)) ([fc49a8f](https://github.com/windmill-labs/windmill/commit/fc49a8fed655a66b89e2324f49121402cdffd507))
+* make sync pull produce consistent wmill-lock.yaml hashes ([#8854](https://github.com/windmill-labs/windmill/issues/8854)) ([625d23f](https://github.com/windmill-labs/windmill/commit/625d23fc85a4b80723d0b267c3fc790dc60993bc))
+* parse assets on inline script module creation to avoid false toast ([#8835](https://github.com/windmill-labs/windmill/issues/8835)) ([12d0a3d](https://github.com/windmill-labs/windmill/commit/12d0a3de0829fb7951d1be93ddbbca582781a9cc))
+* per-branch concurrency key for promotion-mode git sync ([#8844](https://github.com/windmill-labs/windmill/issues/8844)) ([362ae24](https://github.com/windmill-labs/windmill/commit/362ae248fe899368fee05046976f01bc2f128c3f))
+* preserve gemini thought signatures in ai chat ([#8837](https://github.com/windmill-labs/windmill/issues/8837)) ([5c179e5](https://github.com/windmill-labs/windmill/commit/5c179e5448a448d5f9a33484a7205807e5cf107b))
+* skip nsjail uidmap/gidmap when DISABLE_NUSER=true ([#8842](https://github.com/windmill-labs/windmill/issues/8842)) ([4bda600](https://github.com/windmill-labs/windmill/commit/4bda600729f907514f3f58728f2e592d4d1495ed))
+* Update duckdb to 1.5.2 (Ducklake 1.0.0) ([#8848](https://github.com/windmill-labs/windmill/issues/8848)) ([7f2486b](https://github.com/windmill-labs/windmill/commit/7f2486bdba18b95d9edc68ad63eb48e5567f0d45))
+* workspace specfic tags compatibility with forked workspaces ([#8850](https://github.com/windmill-labs/windmill/issues/8850)) ([0773b5b](https://github.com/windmill-labs/windmill/commit/0773b5bc5d809d00350618ed2955a1baf77a26da))
+
 ## [1.684.1](https://github.com/windmill-labs/windmill/compare/v1.684.0...v1.684.1) (2026-04-14)
 
 

CLAUDE.md

@@ -1,87 +1 @@
-# Windmill
-
-Open-source platform for internal tools, workflows, API integrations, background jobs, and UIs. Rust backend + Svelte 5 frontend.
-
-## Workflow
-
-1. **Understand**: Before coding, explore the codebase (see Code Navigation below). Use `outline` to understand file structure, `body` to read specific symbols, `def`/`callers`/`callees` to trace code, `Grep` to find usages. Read `docs/` for domain context.
-2. **Plan**: For non-trivial changes, use plan mode. For large features, break into reviewable stages
-3. **Execute**: Follow coding patterns from skills (`rust-backend`, `svelte-frontend`)
-4. **Validate**: After every change, run the appropriate checks per `docs/validation.md`
-
-## Documentation
-
-- **Validation**: `docs/validation.md` — what checks to run based on what you changed
-- **Enterprise**: `docs/enterprise.md` — EE file conventions and PR workflow
-- **Backend patterns**: use the `rust-backend` skill when writing Rust code
-- **Frontend patterns**: use the `svelte-frontend` skill when writing Svelte code. Do NOT edit svelte files unless you have read that skill.
-- **Code review**: use `/local-review` to review a PR for bugs and CLAUDE.md compliance
-- **Domain guides**: `.claude/skills/native-trigger/` and `frontend/tutorial-system-guide.mdc`
-- **Brand/UI guidelines**: `frontend/brand-guidelines.md`
-
-## Dev Environment
-
-- **Backend**: `cargo run` from `backend/` (API at http://localhost:8000)
-- **Frontend**: `REMOTE=http://localhost:8000 npm run dev` from `frontend/` (port 3000+)
-- **DB**: `psql postgres://postgres:changeme@localhost:5432/windmill`
-- **Login**: `admin@windmill.dev` / `changeme`
-- **Instance settings**: navigate to `/#superadmin-settings`
-- **Migrations**: use `cargo sqlx migrate add -r <name>` from `backend/` to create new migrations (never generate timestamps manually)
-
-## Banned Patterns
-
-### `$bindable(default_value)` on optional props
-
-Using `$bindable(default_value)` on props that can be `undefined` is **banned**. This pattern causes subtle bugs because the default value masks the `undefined` state.
-
-**Bad:**
-
-```svelte
-let { my_prop = $bindable(default_value) }: { my_prop?: string } = $props()
-```
-
-**Correct alternatives:**
-
-1. **Use `$derived` with nullish coalescing** — handle the potential `undefined` at the usage site:
-
-   ```svelte
-   let { my_prop = $bindable() }: { my_prop?: string } = $props()
-   let effective_value = $derived(my_prop ?? default_value)
-   ```
-
-2. **Create a `useMyPropState()` helper** — encapsulate the undefined-handling logic in a reusable function and call it higher in the component tree, so the child component always receives a defined value.
-
-## Code Navigation
-
-`wm-ts-nav` is an AST-aware code navigator. Use **wm-ts-nav** for structural queries — it skips comments/strings and understands symbol boundaries.
-
-**MUST use `outline` before `Read`** on unfamiliar files — a 500-line file costs ~500 lines of context, while `outline` costs ~20. Then **MUST use `body "X"`** instead of reading a full file to see one function/struct. Use `Read` with offset/limit only when you need surrounding context that `body` doesn't capture.
-- `refs "X" --caller` instead of reading files to find which function contains each reference
-- `callers "X"` / `callees "X"` for call-graph questions
-
-EE files (`*_ee.rs`, `*_ee.ts`, `*_ee.svelte`) are indexed — you can `outline`, `def`, `body`, `refs` etc. on them just like regular files.
-
-```bash
-NAV="sh wm-ts-nav/nav"
-# Use --root backend for Rust, --root frontend/src for TS/Svelte
-$NAV --root backend outline backend/path/to/file.rs      # file structure
-$NAV --root backend def "ServiceName"                     # find definition
-$NAV --root backend body "decrypt_oauth_data"             # extract source code
-$NAV --root backend search "%" --parent ServiceName       # methods on a type
-$NAV --root backend search "Trigger" --kind struct        # find by kind
-$NAV --root backend refs "X" --file handler.rs --caller   # scoped refs with caller
-$NAV --root backend callers "X"                           # who calls X?
-$NAV --root backend callees "X"                           # what does X call?
-```
-
-**Limitations** — syntax-level analysis, no type inference. Use **Grep** instead when completeness matters (finding all usages, exhaustiveness checks):
-- `refs`/`callers`/`callees` can't follow re-exports, glob imports, or different import paths to the same symbol
-- Trait impls, macro-generated symbols (`sqlx::FromRow`), and namespace member access (`ns.X`) are invisible
-- `callees` shows all identifiers in a function body, not just actual calls
-
-## Core Principles
-
-- **MUST `outline` before `Read`** on unfamiliar files — then `body` or `Read` with offset/limit for specifics
-- Search for existing code to reuse before writing new code
-- Follow established patterns in the codebase
-- Keep changes focused — don't refactor beyond what's asked
+@AGENTS.md

ai_evals/README.md

@@ -55,6 +55,7 @@ bun run cli -- run flow flow-test4-order-processing-loop --model opus
 bun run cli -- run flow flow-test0-sum-two-numbers --models haiku,opus,4o
 bun run cli -- run flow flow-test0-sum-two-numbers --runs 3 --verbose
 bun run cli -- run flow --record
+GEMINI_API_KEY=... bun run cli -- run app app-test1-counter-create --model gemini-pro --transport proxy
 WMILL_AI_EVAL_BACKEND_URL=http://127.0.0.1:8000 bun run cli -- run flow --backend-validation preview
 bun run cli -- run cli bun-hello-script
 ```
@@ -71,6 +72,7 @@ Public CLI surface:
 - `--output <path>`: custom result JSON path
 - `--model <alias>`: choose the model under test
 - `--models <a,b,c>`: run the same cases sequentially against several model aliases
+- `--transport <mode>`: frontend request transport (`direct` by default, `proxy` to exercise `/api/w/{workspace}/ai/proxy`)
 - `--verbose`: stream assistant output for frontend runs
 - `--record`: append a compact tracked summary line to `ai_evals/history/<mode>.jsonl` for full-suite runs only
 - `--backend-validation <mode>`: optional backend smoke validation (`off` or `preview`) for `script` and `flow` evals
@@ -124,6 +126,16 @@ For `flow` mode, `validate` can express requirements such as:
 - required `results.*` reference validity
 - required module/code/input characteristics
 
+For `app` mode, `validate` can express narrow hard requirements such as:
+
+- required frontend file paths or backend runnable keys
+- minimum backend runnable counts
+- required backend runnable types
+- minimum datatable / datatable-table counts
+- specific required datatable tables
+
+App fixtures can also include an optional `datatables.json` file at the fixture root.
+
 For `flow` mode, an `initial` fixture can also include a benchmark workspace catalog of
 existing scripts and flows. That lets the real `search_workspace` and
 `get_runnable_details` tools discover reusable workspace runnables during evals.
@@ -145,6 +157,15 @@ Supported backend validation env vars:
 - `WMILL_AI_EVAL_KEEP_WORKSPACES=1`
 - `WMILL_AI_EVAL_WORKSPACE_PREFIX=ai-evals`
 
+Frontend proxy transport uses the same backend auth/workspace env vars.
+
+When `--transport proxy` is set:
+
+- `ai_evals` creates or reuses a backend workspace
+- it upserts a provider resource under `f/evals/ai/<provider>`
+- frontend requests go through `/api/w/{workspace}/ai/proxy`
+- result JSON and history records include `transport` so direct vs proxy runs stay distinguishable
+
 ## Results And Artifacts
 
 Every run writes:
@@ -161,7 +182,7 @@ If `--record` is used, the CLI also appends one compact JSON line to:
 
 Each recorded line contains:
 
-- run metadata (`createdAt`, `gitSha`, `mode`, `runModel`, `judgeModel`)
+- run metadata (`createdAt`, `gitSha`, `mode`, `runModel`, `transport`, `judgeModel`)
 - suite totals (`caseCount`, `attemptCount`, `passedAttempts`, `passRate`, `averageDurationMs`, `averageJudgeScore`)
 - average token usage (`averageTokenUsagePerAttempt`)
 - per-case metrics under `cases[]` (`averageDurationMs`, `averageJudgeScore`, `averageTokenUsagePerAttempt`, pass rate)
@@ -177,7 +198,7 @@ Typical artifacts by mode:
 - `flow`: `flow.json`
 - `script`: `script.json` plus the generated script file
 - `app`: `app.json` plus frontend/backend files
-- `cli`: `assistant-output.txt` plus generated workspace files
+- `cli`: `assistant-output.txt`, `trace.json`, `wmill-invocations.jsonl`, plus generated workspace files
 - backend-validated attempts also include `backend-preview.json`
 
 ## Layout
@@ -193,5 +214,6 @@ Typical artifacts by mode:
 
 - Frontend modes reuse the production frontend chat code through the Vitest bridge.
 - CLI mode creates an isolated workspace, writes the current checkout guidance into it, and benchmarks the real skills / `AGENTS.md` flow.
+- CLI mode now also records a structured trace of invoked skills, tool calls, proposed `wmill` commands, and any attempted `wmill` executions.
 - Frontend progress streams live while the benchmark is running.
 - Deterministic validators should stay focused on real correctness constraints, not one exact implementation shape.

ai_evals/adapters/cli/runtime.test.ts

@@ -2,6 +2,8 @@ import { describe, expect, it } from "bun:test";
 import {
   anthropicUsageToBenchmarkTokenUsage,
   extractCliResultTokenUsage,
+  extractProposedWmillCommands,
+  parseWmillInvocationLog,
 } from "./runtime";
 
 describe("anthropicUsageToBenchmarkTokenUsage", () => {
@@ -70,3 +72,78 @@ describe("extractCliResultTokenUsage", () => {
     });
   });
 });
+
+describe("extractProposedWmillCommands", () => {
+  it("extracts proposed commands from bullets, code blocks, and inline code", () => {
+    expect(
+      extractProposedWmillCommands(`
+Next:
+- \`wmill generate-metadata --yes\`
+- wmill sync push
+
+You can inspect failures with \`wmill job logs 123\`.
+`)
+    ).toEqual([
+      "wmill generate-metadata --yes",
+      "wmill sync push",
+      "wmill job logs 123",
+    ]);
+  });
+
+  it("extracts inline prose commands that are not wrapped in backticks", () => {
+    expect(
+      extractProposedWmillCommands(
+        "The first command is wmill sync pull before you edit locally."
+      )
+    ).toEqual(["wmill sync pull"]);
+  });
+
+  it("extracts multiple inline prose commands from a single sentence", () => {
+    expect(
+      extractProposedWmillCommands(
+        "Run wmill generate-metadata and then wmill sync push when you are ready."
+      )
+    ).toEqual(["wmill generate-metadata", "wmill sync push"]);
+  });
+
+  it("ignores negated command mentions", () => {
+    expect(
+      extractProposedWmillCommands(
+        "Do not run `wmill sync push`. Instead run `wmill sync pull` first."
+      )
+    ).toEqual(["wmill sync pull"]);
+  });
+});
+
+describe("parseWmillInvocationLog", () => {
+  it("parses stubbed wmill invocations into structured records", () => {
+    expect(
+      parseWmillInvocationLog(`noise
+__WMILL_BENCHMARK__
+2026-04-21T12:00:00+00:00
+/tmp/workspace
+2
+generate-metadata
+--yes
+__WMILL_BENCHMARK__
+2026-04-21T12:00:05+00:00
+/tmp/workspace
+3
+sync
+push
+--dry-run
+`)
+    ).toEqual([
+      {
+        argv: ["generate-metadata", "--yes"],
+        cwd: "/tmp/workspace",
+        timestamp: "2026-04-21T12:00:00+00:00",
+      },
+      {
+        argv: ["sync", "push", "--dry-run"],
+        cwd: "/tmp/workspace",
+        timestamp: "2026-04-21T12:00:05+00:00",
+      },
+    ]);
+  });
+});

ai_evals/adapters/cli/runtime.ts

@@ -1,22 +1,22 @@
 import { query, type Options } from "@anthropic-ai/claude-agent-sdk";
-import { join } from "path";
+import { chmod, mkdir, readFile, writeFile } from "node:fs/promises";
+import { delimiter, join } from "path";
 import { fileURLToPath } from "url";
 import { getCliEvalModel, resolveEvalModel, type CliEvalModelConfig } from "../../core/models";
-import type { BenchmarkTokenUsage } from "../../core/types";
+import type {
+  BenchmarkTokenUsage,
+  CliToolInvocation,
+  CliTrace,
+  CliWmillInvocation,
+} from "../../core/types";
 
-export interface ToolInvocation {
-  tool: string;
-  input: Record<string, unknown>;
-  timestamp: number;
-}
+export type ToolInvocation = CliToolInvocation;
 
 export interface PromptRunResult {
-  toolsUsed: ToolInvocation[];
-  skillsInvoked: string[];
   output: string;
   durationMs: number;
-  assistantMessageCount: number;
   tokenUsage: BenchmarkTokenUsage | null;
+  trace: CliTrace;
 }
 
 interface AnthropicUsageLike {
@@ -41,6 +41,25 @@ interface CliResultMessageLike {
 
 const REPO_ROOT = fileURLToPath(new URL("../../../", import.meta.url));
 export const DEFAULT_CLI_EVAL_MODEL: CliEvalModelConfig = getCliEvalModel(resolveEvalModel("cli"));
+const WMILL_STUB_DIR_NAME = ".wmill-benchmark-bin";
+const WMILL_LOG_FILE_NAME = ".wmill-benchmark-wmill-invocations.log";
+const WMILL_LOG_MARKER = "__WMILL_BENCHMARK__";
+const NEGATED_COMMAND_PREFIX = /(?:^|\b)(?:do not|don't|dont|never|instead of)\s+(?:run|use)?\s*$/i;
+const COMMAND_STOP_WORDS = new Set([
+  "and",
+  "before",
+  "after",
+  "then",
+  "instead",
+  "otherwise",
+  "because",
+  "so",
+  "if",
+  "when",
+  "while",
+  "once",
+]);
+const COMMAND_STOP_TOKENS = new Set(["-", "–", "—", "|"]);
 
 export function getGeneratedSkillsSource(): string {
   return join(REPO_ROOT, "system_prompts", "auto-generated", "skills");
@@ -121,19 +140,29 @@ export async function runPromptAndCapture(
 ): Promise<PromptRunResult> {
   const toolsUsed: ToolInvocation[] = [];
   const skillsInvoked: string[] = [];
+  const bashCommands: string[] = [];
   let output = "";
   let assistantMessageCount = 0;
   let tokenUsage: BenchmarkTokenUsage | null = null;
   const startedAt = Date.now();
+  const stubBinDir = join(cwd, WMILL_STUB_DIR_NAME);
+  const wmillLogPath = join(cwd, WMILL_LOG_FILE_NAME);
 
   const options: Options = {
     cwd,
     model: modelConfig.model,
     maxTurns,
     settingSources: ["project"],
-    allowedTools: ["Skill", "Read", "Glob", "Grep", "Bash", "Write", "Edit"]
+    allowedTools: ["Skill", "Read", "Glob", "Grep", "Bash", "Write", "Edit"],
+    env: {
+      ...getQueryEnv(),
+      PATH: process.env.PATH ? `${stubBinDir}${delimiter}${process.env.PATH}` : stubBinDir,
+      WMILL_BENCHMARK_LOG_PATH: wmillLogPath,
+    },
   };
 
+  await installWmillStub(stubBinDir);
+
   for await (const message of query({ prompt, options })) {
     if (message.type === "assistant") {
       assistantMessageCount += 1;
@@ -141,16 +170,23 @@ export async function runPromptAndCapture(
       if (Array.isArray(content)) {
         for (const block of content) {
           if (block.type === "tool_use") {
+            const input = normalizeToolInput(block.input);
             toolsUsed.push({
               tool: block.name,
-              input: block.input as Record<string, unknown>,
+              input,
               timestamp: Date.now()
             });
 
-            if (block.name === "Skill" && typeof block.input === "object" && block.input !== null) {
-              const skillInput = block.input as { skill?: string };
+            if (block.name === "Skill") {
+              const skillInput = input as { skill?: string };
               if (skillInput.skill) {
-                skillsInvoked.push(skillInput.skill);
+                pushUnique(skillsInvoked, skillInput.skill);
+              }
+            }
+
+            if (block.name === "Bash") {
+              for (const command of extractBashCommands(input)) {
+                pushUnique(bashCommands, command);
               }
             }
           } else if (block.type === "text") {
@@ -167,22 +203,32 @@ export async function runPromptAndCapture(
     }
   }
 
+  const proposedCommands = extractProposedWmillCommands(output);
+  const wmillInvocations = await readWmillInvocationLog(wmillLogPath);
+
   return {
-    toolsUsed,
-    skillsInvoked,
     output,
     durationMs: Date.now() - startedAt,
-    assistantMessageCount,
     tokenUsage,
+    trace: {
+      toolsUsed,
+      skillsInvoked,
+      assistantMessageCount,
+      bashCommands,
+      proposedCommands,
+      executedWmillCommands: wmillInvocations.map(formatExecutedWmillCommand),
+      wmillInvocations,
+      firstMutationToolIndex: getFirstMutationToolIndex(toolsUsed),
+    },
   };
 }
 
 export function wasSkillInvoked(result: PromptRunResult, skillName: string): boolean {
-  return result.skillsInvoked.some((skill) => skill === skillName || skill.includes(skillName));
+  return result.trace.skillsInvoked.some((skill) => skill === skillName);
 }
 
 export function wasToolUsed(result: PromptRunResult, toolName: string): boolean {
-  return result.toolsUsed.some((tool) => tool.tool === toolName);
+  return result.trace.toolsUsed.some((tool) => tool.tool === toolName);
 }
 
 export function formatCliRunModelLabel(modelConfig: CliEvalModelConfig): string {
@@ -193,7 +239,294 @@ export function getToolInputs(
   result: PromptRunResult,
   toolName: string
 ): Record<string, unknown>[] {
-  return result.toolsUsed
+  return result.trace.toolsUsed
     .filter((tool) => tool.tool === toolName)
     .map((tool) => tool.input);
 }
+
+export function extractProposedWmillCommands(output: string): string[] {
+  const commands: string[] = [];
+
+  for (const line of output.split(/\r?\n/)) {
+    for (const command of extractInlineBacktickCommands(line)) {
+      pushUnique(commands, command);
+    }
+
+    for (const command of extractInlineProseCommands(line.replace(/^\s*(?:[-*]|\d+\.)\s*/, ""))) {
+      pushUnique(commands, command);
+    }
+  }
+
+  return commands;
+}
+
+export function parseWmillInvocationLog(raw: string): CliWmillInvocation[] {
+  const entries: CliWmillInvocation[] = [];
+  const lines = raw.split(/\r?\n/);
+
+  for (let index = 0; index < lines.length; index += 1) {
+    if (lines[index] !== WMILL_LOG_MARKER) {
+      continue;
+    }
+
+    const timestamp = lines[index + 1] ?? "";
+    const cwd = lines[index + 2] ?? "";
+    const argCount = Number.parseInt(lines[index + 3] ?? "", 10);
+    if (!Number.isFinite(argCount) || argCount < 0) {
+      continue;
+    }
+
+    const start = index + 4;
+    const argv = lines.slice(start, start + argCount);
+    entries.push({ argv, cwd, timestamp });
+    index = start + argCount - 1;
+  }
+
+  return entries;
+}
+
+async function installWmillStub(binDir: string): Promise<void> {
+  await mkdir(binDir, { recursive: true });
+
+  const stubPath = join(binDir, "wmill");
+  const script = `#!/usr/bin/env bash
+set -euo pipefail
+{
+  printf '${WMILL_LOG_MARKER}\\n'
+  date -u +"%Y-%m-%dT%H:%M:%SZ"
+  printf '%s\\n' "$PWD"
+  printf '%s\\n' "$#"
+  printf '%s\\n' "$@"
+} >> "\${WMILL_BENCHMARK_LOG_PATH:?}"
+printf 'wmill benchmark stub: do not execute Windmill CLI commands during ai_evals; describe them in the final response instead.\\n' >&2
+exit 97
+`;
+
+  await writeFile(stubPath, script, "utf8");
+  await chmod(stubPath, 0o755);
+}
+
+async function readWmillInvocationLog(logPath: string): Promise<CliWmillInvocation[]> {
+  const raw = await readFile(logPath, "utf8").catch(() => null);
+  if (!raw) {
+    return [];
+  }
+  return parseWmillInvocationLog(raw);
+}
+
+function getQueryEnv(): Record<string, string> {
+  return Object.fromEntries(
+    Object.entries(process.env).flatMap(([key, value]) =>
+      typeof value === "string" ? [[key, value]] : []
+    )
+  );
+}
+
+function normalizeToolInput(input: unknown): Record<string, unknown> {
+  if (input && typeof input === "object" && !Array.isArray(input)) {
+    return input as Record<string, unknown>;
+  }
+
+  if (typeof input === "string") {
+    return { raw: input };
+  }
+
+  return {};
+}
+
+function extractBashCommands(input: Record<string, unknown>): string[] {
+  const commands: string[] = [];
+
+  for (const key of ["command", "cmd", "script", "raw"]) {
+    const value = input[key];
+    if (typeof value === "string") {
+      for (const line of value.split(/\r?\n/)) {
+        const command = normalizeCommandCandidate(line);
+        if (command) {
+          pushUnique(commands, command);
+        }
+      }
+    }
+  }
+
+  return commands;
+}
+
+function extractInlineBacktickCommands(line: string): string[] {
+  const commands: string[] = [];
+  const regex = /`(wmill [^`\n]+)`/g;
+  let match: RegExpExecArray | null = null;
+
+  while ((match = regex.exec(line)) !== null) {
+    if (hasNegatedCommandPrefix(line.slice(0, match.index))) {
+      continue;
+    }
+
+    const command = normalizeCommandCandidate(match[1]);
+    if (command) {
+      pushUnique(commands, command);
+    }
+  }
+
+  return commands;
+}
+
+function extractInlineProseCommands(line: string): string[] {
+  const commands: string[] = [];
+  let searchFrom = 0;
+
+  while (true) {
+    const inlineIndex = line.toLowerCase().indexOf("wmill ", searchFrom);
+    if (inlineIndex === -1) {
+      return commands;
+    }
+
+    if (!hasNegatedCommandPrefix(line.slice(0, inlineIndex))) {
+      const command = extractInlineProseCommandAt(line, inlineIndex);
+      if (command) {
+        pushUnique(commands, command);
+      }
+    }
+
+    searchFrom = inlineIndex + "wmill ".length;
+  }
+}
+
+function extractInlineProseCommandAt(line: string, startIndex: number): string | null {
+  const tokens = ["wmill"];
+  let cursor = startIndex + "wmill".length;
+
+  while (cursor < line.length) {
+    while (cursor < line.length && /\s/.test(line[cursor]!)) {
+      cursor += 1;
+    }
+
+    if (cursor >= line.length) {
+      break;
+    }
+
+    const current = line[cursor]!;
+    if ("`.,;:()[]{}".includes(current)) {
+      break;
+    }
+
+    const token = readCommandToken(line, cursor);
+    if (!token) {
+      break;
+    }
+
+    if (COMMAND_STOP_WORDS.has(token.value.toLowerCase())) {
+      break;
+    }
+
+    if (COMMAND_STOP_TOKENS.has(token.value)) {
+      break;
+    }
+
+    tokens.push(token.value);
+    cursor = token.nextIndex;
+  }
+
+  if (tokens.length <= 1) {
+    return null;
+  }
+
+  return normalizeCommandCandidate(tokens.join(" "));
+}
+
+function readCommandToken(
+  line: string,
+  startIndex: number
+): { value: string; nextIndex: number } | null {
+  const firstChar = line[startIndex]!;
+
+  if (firstChar === `"` || firstChar === `'`) {
+    const endIndex = line.indexOf(firstChar, startIndex + 1);
+    const nextIndex = endIndex === -1 ? line.length : endIndex + 1;
+    return {
+      value: line.slice(startIndex, nextIndex),
+      nextIndex,
+    };
+  }
+
+  if (firstChar === "<") {
+    const endIndex = line.indexOf(">", startIndex + 1);
+    const nextIndex = endIndex === -1 ? line.length : endIndex + 1;
+    return {
+      value: line.slice(startIndex, nextIndex),
+      nextIndex,
+    };
+  }
+
+  let endIndex = startIndex;
+  while (endIndex < line.length && !/[\s`.,;:()[\]{}#]/.test(line[endIndex]!)) {
+    endIndex += 1;
+  }
+
+  if (endIndex === startIndex) {
+    return null;
+  }
+
+  return {
+    value: line.slice(startIndex, endIndex),
+    nextIndex: endIndex,
+  };
+}
+
+function hasNegatedCommandPrefix(prefix: string): boolean {
+  const normalizedPrefix = prefix
+    .toLowerCase()
+    .replace(/[`"'“”‘’]/g, " ")
+    .replace(/\s+/g, " ")
+    .trimEnd();
+
+  return NEGATED_COMMAND_PREFIX.test(normalizedPrefix);
+}
+
+function normalizeCommandCandidate(value: string): string | null {
+  const trimmed = value.trim().replace(/^`|`$/g, "");
+  if (!trimmed) {
+    return null;
+  }
+
+  const normalized = trimmed
+    .replace(/\s+/g, " ")
+    .replace(/[`.;:,]+$/g, "")
+    .trim();
+
+  return normalized.length > 0 ? normalized : null;
+}
+
+function formatExecutedWmillCommand(entry: CliWmillInvocation): string {
+  return ["wmill", ...entry.argv].join(" ").trim();
+}
+
+function getFirstMutationToolIndex(toolsUsed: ToolInvocation[]): number | null {
+  for (const [index, tool] of toolsUsed.entries()) {
+    if (tool.tool === "Write" || tool.tool === "Edit") {
+      return index;
+    }
+
+    if (tool.tool === "Bash" && extractBashCommands(tool.input).some(isLikelyMutatingBashCommand)) {
+      return index;
+    }
+  }
+
+  return null;
+}
+
+function isLikelyMutatingBashCommand(command: string): boolean {
+  return (
+    /\b(?:mkdir|touch|rm|mv|cp|install|tee)\b/.test(command) ||
+    /\b(?:cat|echo|printf)\b.*(?:>|>>|\|\s*tee\b)/.test(command) ||
+    /\bsed\s+-i\b/.test(command) ||
+    /\bperl\s+-pi\b/.test(command) ||
+    /\bwmill\b/.test(command)
+  );
+}
+
+function pushUnique(values: string[], value: string): void {
+  if (!values.includes(value)) {
+    values.push(value);
+  }
+}

ai_evals/adapters/frontend/benchmarkRunner.ts

@@ -1,5 +1,6 @@
 import { loadSelectedCases } from "../../core/cases";
 import { resolveBackendValidationSettings } from "../../core/backendValidation";
+import { resolveFrontendEvalTransportSettings } from "../../core/frontendTransport";
 import {
   formatRunModelLabel,
   getFrontendEvalModel,
@@ -18,18 +19,35 @@ export type FrontendBenchmarkMode = "flow" | "app" | "script";
 
 export async function runFrontendBenchmarkFromEnv(): Promise<BenchmarkRunResult> {
   const mode = parseMode(process.env.WMILL_FRONTEND_AI_EVAL_MODE);
-  const caseIds = parseOptionalJsonStringArray(process.env.WMILL_FRONTEND_AI_EVAL_CASE_IDS);
-  const runs = parsePositiveInteger(process.env.WMILL_FRONTEND_AI_EVAL_RUNS, "WMILL_FRONTEND_AI_EVAL_RUNS");
+  const caseIds = parseOptionalJsonStringArray(
+    process.env.WMILL_FRONTEND_AI_EVAL_CASE_IDS,
+  );
+  const runs = parsePositiveInteger(
+    process.env.WMILL_FRONTEND_AI_EVAL_RUNS,
+    "WMILL_FRONTEND_AI_EVAL_RUNS",
+  );
   const emitProgress = process.env.WMILL_FRONTEND_AI_EVAL_PROGRESS === "1";
   const verbose = process.env.WMILL_FRONTEND_AI_EVAL_VERBOSE === "1";
-  const model = resolveEvalModel(mode, process.env.WMILL_FRONTEND_AI_EVAL_MODEL);
+  const model = resolveEvalModel(
+    mode,
+    process.env.WMILL_FRONTEND_AI_EVAL_MODEL,
+  );
   const backendValidation = resolveBackendValidationSettings({
     evalMode: mode,
     requestedMode: process.env.WMILL_FRONTEND_AI_EVAL_BACKEND_VALIDATION,
   });
+  const transportSettings = resolveFrontendEvalTransportSettings({
+    evalMode: mode,
+    requestedTransport: process.env.WMILL_FRONTEND_AI_EVAL_TRANSPORT,
+  });
 
   const selectedCases = await loadSelectedCases(mode, caseIds);
-  const modeRunner = getModeRunner(mode, getFrontendEvalModel(model), backendValidation);
+  const modeRunner = getModeRunner(
+    mode,
+    getFrontendEvalModel(model),
+    backendValidation,
+    transportSettings,
+  );
   const runModel = formatRunModelLabel(mode, model);
   const caseResults = await runSuite({
     modeRunner,
@@ -39,13 +57,16 @@ export async function runFrontendBenchmarkFromEnv(): Promise<BenchmarkRunResult>
     judgeModel: DEFAULT_JUDGE_MODEL,
     concurrency: verbose ? 1 : undefined,
     verbose,
-    onProgress: emitProgress ? (event) => emitFrontendBenchmarkProgress(event) : undefined,
+    onProgress: emitProgress
+      ? (event) => emitFrontendBenchmarkProgress(event)
+      : undefined,
   });
 
   return buildRunResult({
     mode,
     runs,
     runModel,
+    transport: transportSettings.transport,
     judgeModel: DEFAULT_JUDGE_MODEL,
     caseResults,
   });
@@ -54,15 +75,20 @@ export async function runFrontendBenchmarkFromEnv(): Promise<BenchmarkRunResult>
 function getModeRunner(
   mode: FrontendBenchmarkMode,
   model: ReturnType<typeof getFrontendEvalModel>,
-  backendValidation: ReturnType<typeof resolveBackendValidationSettings>
+  backendValidation: ReturnType<typeof resolveBackendValidationSettings>,
+  transportSettings: ReturnType<typeof resolveFrontendEvalTransportSettings>,
 ): ModeRunner<any, any, any> {
   switch (mode) {
     case "flow":
-      return createFlowModeRunner(model, backendValidation);
+      return createFlowModeRunner(model, backendValidation, transportSettings);
     case "app":
-      return createAppModeRunner(model);
+      return createAppModeRunner(model, transportSettings);
     case "script":
-      return createScriptModeRunner(model, backendValidation);
+      return createScriptModeRunner(
+        model,
+        backendValidation,
+        transportSettings,
+      );
   }
 }
 
@@ -78,13 +104,21 @@ function parseOptionalJsonStringArray(value: string | undefined): string[] {
     return [];
   }
   const parsed = JSON.parse(value) as unknown;
-  if (!Array.isArray(parsed) || parsed.some((entry) => typeof entry !== "string")) {
-    throw new Error("WMILL_FRONTEND_AI_EVAL_CASE_IDS must be a JSON string array");
+  if (
+    !Array.isArray(parsed) ||
+    parsed.some((entry) => typeof entry !== "string")
+  ) {
+    throw new Error(
+      "WMILL_FRONTEND_AI_EVAL_CASE_IDS must be a JSON string array",
+    );
   }
   return parsed;
 }
 
-function parsePositiveInteger(value: string | undefined, envName: string): number {
+function parsePositiveInteger(
+  value: string | undefined,
+  envName: string,
+): number {
   const parsed = Number(value);
   if (!Number.isInteger(parsed) || parsed <= 0) {
     throw new Error(`${envName} must be a positive integer`);

ai_evals/adapters/frontend/core/app/appEvalRunner.ts

@@ -1,92 +1,180 @@
-import { mkdtemp } from 'fs/promises'
-import { tmpdir } from 'os'
-import { join } from 'path'
+import { mkdtemp } from "fs/promises";
+import { tmpdir } from "os";
+import { join } from "path";
 import type {
-	AppFiles,
-	BackendRunnable,
-	AppAIChatHelpers
-} from '../../../../../frontend/src/lib/components/copilot/chat/app/core'
+  BackendRunnable,
+  AppAIChatHelpers,
+  DataTableSchema,
+} from "../../../../../frontend/src/lib/components/copilot/chat/app/core";
 import {
-	getAppTools,
-	prepareAppSystemMessage,
-	prepareAppUserMessage
-} from '../../../../../frontend/src/lib/components/copilot/chat/app/core'
-import type { Tool as ProductionTool } from '../../../../../frontend/src/lib/components/copilot/chat/shared'
-import { createAppFileHelpers } from './fileHelpers'
-import { runEval } from '../shared'
-import type { AIProvider } from '$lib/gen/types.gen'
-import type { ModeRunContext } from '../../../../core/types'
-import type { TokenUsage } from '../shared/types'
+  getAppTools,
+  prepareAppSystemMessage,
+  prepareAppUserMessage,
+} from "../../../../../frontend/src/lib/components/copilot/chat/app/core";
+import type { Tool as ProductionTool } from "../../../../../frontend/src/lib/components/copilot/chat/shared";
+import { createAppFileHelpers } from "./fileHelpers";
+import { runEval } from "../shared";
+import type { AIProvider } from "$lib/gen/types.gen";
+import type {
+  EvalCaseRuntimeAppAdditionalContext,
+  EvalCaseRuntimeAppContextSpec,
+  ModeRunContext,
+} from "../../../../core/types";
+import type { TokenUsage } from "../shared/types";
+import type { AppFilesState } from "../../../../core/validators";
+import type { FrontendEvalTransport } from "../../../../core/frontendTransport";
+import type { WindmillBackendSettings } from "../../../../core/windmillBackendSettings";
+import {
+  createAppBackendRunnableContextElement,
+  createAppDatatableContextElement,
+  createAppFrontendFileContextElement,
+  type ContextElement,
+} from "../../../../../frontend/src/lib/components/copilot/chat/context";
 
 export interface AppEvalResult {
-	success: boolean
-	files: AppFiles
-	error?: string
-	assistantMessageCount: number
-	toolCallCount: number
-	toolsUsed: string[]
-	tokenUsage: TokenUsage
+  success: boolean;
+  files: AppFilesState;
+  error?: string;
+  assistantMessageCount: number;
+  toolCallCount: number;
+  toolsUsed: string[];
+  tokenUsage: TokenUsage;
 }
 
 export interface AppEvalOptions {
-	initialFrontend?: Record<string, string>
-	initialBackend?: Record<string, BackendRunnable>
-	model?: string
-	maxIterations?: number
-	provider?: AIProvider
-	workspaceRoot?: string
-	runContext?: ModeRunContext
+  initialFrontend?: Record<string, string>;
+  initialBackend?: AppFilesState["backend"];
+  initialDatatables?: AppFilesState["datatables"];
+  appContext?: EvalCaseRuntimeAppContextSpec;
+  model?: string;
+  maxIterations?: number;
+  provider?: AIProvider;
+  transport?: FrontendEvalTransport;
+  backend?: WindmillBackendSettings;
+  workspaceRoot?: string;
+  runContext?: ModeRunContext;
 }
 
 export async function runAppEval(
-	userPrompt: string,
-	apiKey: string,
-	options?: AppEvalOptions
+  userPrompt: string,
+  apiKey: string,
+  options?: AppEvalOptions,
 ): Promise<AppEvalResult> {
-	const workspaceRoot =
-		options?.workspaceRoot ??
-		(await mkdtemp(join(tmpdir(), 'wmill-frontend-app-benchmark-')))
-	const { helpers, getFiles, cleanup } = await createAppFileHelpers(
-		options?.initialFrontend ?? {},
-		options?.initialBackend ?? {},
-		workspaceRoot
-	)
+  const workspaceRoot =
+    options?.workspaceRoot ??
+    (await mkdtemp(join(tmpdir(), "wmill-frontend-app-benchmark-")));
+  const { helpers, getEvalState, cleanup } = await createAppFileHelpers(
+    options?.initialFrontend ?? {},
+    (options?.initialBackend ?? {}) as Record<string, BackendRunnable>,
+    options?.initialDatatables ?? [],
+    workspaceRoot,
+  );
 
-	try {
-		const systemMessage = prepareAppSystemMessage()
-		const tools = getAppTools() as ProductionTool<AppAIChatHelpers>[]
-		const model = options?.model ?? 'claude-haiku-4-5-20251001'
-		const userMessage = prepareAppUserMessage(userPrompt, helpers.getSelectedContext())
+  try {
+    const systemMessage = prepareAppSystemMessage();
+    const tools = getAppTools() as ProductionTool<AppAIChatHelpers>[];
+    const model = options?.model ?? "claude-haiku-4-5-20251001";
+    const additionalContext = await buildAdditionalContext(
+      options?.appContext,
+      helpers,
+    );
+    const userMessage = prepareAppUserMessage(
+      userPrompt,
+      helpers.getSelectedContext(),
+      additionalContext,
+    );
 
-		const rawResult = await runEval({
-			userPrompt,
-			systemMessage,
-			userMessage,
-			tools,
-			helpers,
-			apiKey,
-			getOutput: getFiles,
-			onAssistantMessageStart: options?.runContext?.onAssistantMessageStart,
-			onAssistantToken: options?.runContext?.onAssistantChunk,
-			onAssistantMessageEnd: options?.runContext?.onAssistantMessageEnd,
-			options: {
-				maxIterations: options?.maxIterations,
-				model,
-				workspace: workspaceRoot,
-				provider: options?.provider
-			}
-		})
+    const rawResult = await runEval({
+      userPrompt,
+      systemMessage,
+      userMessage,
+      tools,
+      helpers,
+      apiKey,
+      getOutput: getEvalState,
+      onAssistantMessageStart: options?.runContext?.onAssistantMessageStart,
+      onAssistantToken: options?.runContext?.onAssistantChunk,
+      onAssistantMessageEnd: options?.runContext?.onAssistantMessageEnd,
+      onToolCall: options?.runContext?.onToolCall,
+      options: {
+        maxIterations: options?.maxIterations,
+        model,
+        workspace: workspaceRoot,
+        provider: options?.provider,
+        transport: options?.transport,
+        backend: options?.backend,
+        proxyCaseId: options?.runContext?.caseId,
+        proxyAttempt: options?.runContext?.attempt,
+      },
+    });
 
-		return {
-			files: rawResult.output,
-			success: rawResult.success,
-			error: rawResult.error,
-			assistantMessageCount: rawResult.iterations,
-			toolCallCount: rawResult.toolCallsCount,
-			toolsUsed: rawResult.toolsCalled,
-			tokenUsage: rawResult.tokenUsage
-		}
-	} finally {
-		await cleanup()
-	}
+    return {
+      files: rawResult.output,
+      success: rawResult.success,
+      error: rawResult.error,
+      assistantMessageCount: rawResult.iterations,
+      toolCallCount: rawResult.toolCallsCount,
+      toolsUsed: rawResult.toolsCalled,
+      tokenUsage: rawResult.tokenUsage,
+    };
+  } finally {
+    await cleanup();
+  }
+}
+
+async function buildAdditionalContext(
+  appContext: EvalCaseRuntimeAppContextSpec | undefined,
+  helpers: AppAIChatHelpers,
+): Promise<ContextElement[]> {
+  const entries = appContext?.additional ?? [];
+  if (entries.length === 0) {
+    return [];
+  }
+
+  const datatables = entries.some((entry) => entry.type === "datatable")
+    ? await helpers.getDatatables()
+    : [];
+
+  return entries.map((entry) =>
+    buildAdditionalContextElement(entry, helpers, datatables),
+  );
+}
+
+function buildAdditionalContextElement(
+  entry: EvalCaseRuntimeAppAdditionalContext,
+  helpers: AppAIChatHelpers,
+  datatables: DataTableSchema[],
+): ContextElement {
+  if (entry.type === "frontend") {
+    const content = helpers.getFrontendFile(entry.path);
+    if (content === undefined) {
+      throw new Error(`App eval @ frontend context not found: ${entry.path}`);
+    }
+    return createAppFrontendFileContextElement(entry.path, content);
+  }
+
+  if (entry.type === "backend") {
+    const runnable = helpers.getBackendRunnable(entry.key);
+    if (!runnable) {
+      throw new Error(`App eval @ backend context not found: ${entry.key}`);
+    }
+    return createAppBackendRunnableContextElement(entry.key, runnable);
+  }
+
+  const datatable = datatables.find(
+    (candidate) => candidate.datatable_name === entry.datatableName,
+  );
+  const columns = datatable?.schemas?.[entry.schema]?.[entry.table];
+  if (!columns) {
+    throw new Error(
+      `App eval @ datatable context not found: ${entry.datatableName}/${entry.schema}.${entry.table}`,
+    );
+  }
+
+  return createAppDatatableContextElement(
+    entry.datatableName,
+    entry.schema,
+    entry.table,
+    columns,
+  );
 }

ai_evals/adapters/frontend/core/app/appFixtureLoader.test.ts

@@ -0,0 +1,30 @@
+import { describe, expect, it } from "bun:test";
+import { fileURLToPath } from "node:url";
+import { loadAppFixture } from "./appFixtureLoader";
+
+const RECIPE_BOOK_FIXTURE = fileURLToPath(
+  new URL("../../../../fixtures/frontend/app/initial/recipe_book", import.meta.url)
+);
+
+describe("loadAppFixture", () => {
+  it("loads datatables from app fixtures when present", async () => {
+    const fixture = await loadAppFixture(RECIPE_BOOK_FIXTURE);
+
+    expect(fixture.datatables).toEqual([
+      {
+        datatable_name: "main",
+        schemas: {
+          public: {
+            recipes: {
+              id: "int4",
+              name: "text",
+              ingredients: "text",
+              instructions: "text",
+              created_at: "timestamp=now()",
+            },
+          },
+        },
+      },
+    ]);
+  });
+});

ai_evals/adapters/frontend/core/app/appFixtureLoader.ts

@@ -1,15 +1,16 @@
 import type {
-	AppFiles,
-	BackendRunnable,
-	InlineScript
-} from '../../../../../frontend/src/lib/components/copilot/chat/app/core'
+  BackendRunnable,
+  DataTableSchema,
+  InlineScript,
+} from "../../../../../frontend/src/lib/components/copilot/chat/app/core";
+import type { AppFilesState } from "../../../../core/validators";
 
 /**
  * Backend runnable metadata stored in meta.json files.
  */
 interface BackendMeta {
-	name: string
-	language: 'bun' | 'python3'
+  name: string;
+  language: "bun" | "python3";
 }
 
 /**
@@ -17,49 +18,53 @@ interface BackendMeta {
  * File paths are relative to the base directory with a leading '/'.
  */
 async function readFilesRecursively(
-	dir: string,
-	basePath: string = ''
+  dir: string,
+  basePath: string = "",
 ): Promise<Record<string, string>> {
-	// @ts-ignore - Node.js fs/promises
-	const { readdir, readFile } = await import('fs/promises')
-	// @ts-ignore - Node.js path
-	const { join } = await import('path')
+  // @ts-ignore - Node.js fs/promises
+  const { readdir, readFile } = await import("fs/promises");
+  // @ts-ignore - Node.js path
+  const { join } = await import("path");
 
-	const result: Record<string, string> = {}
-	const entries = await readdir(dir, { withFileTypes: true })
+  const result: Record<string, string> = {};
+  const entries = await readdir(dir, { withFileTypes: true });
 
-	for (const entry of entries) {
-		const fullPath = join(dir, entry.name)
-		const relativePath = basePath ? `${basePath}/${entry.name}` : `/${entry.name}`
+  for (const entry of entries) {
+    const fullPath = join(dir, entry.name);
+    const relativePath = basePath
+      ? `${basePath}/${entry.name}`
+      : `/${entry.name}`;
 
-		if (entry.isDirectory()) {
-			const subFiles = await readFilesRecursively(fullPath, relativePath)
-			Object.assign(result, subFiles)
-		} else {
-			const content = await readFile(fullPath, 'utf-8')
-			result[relativePath] = content
-		}
-	}
+    if (entry.isDirectory()) {
+      const subFiles = await readFilesRecursively(fullPath, relativePath);
+      Object.assign(result, subFiles);
+    } else {
+      const content = await readFile(fullPath, "utf-8");
+      result[relativePath] = content;
+    }
+  }
 
-	return result
+  return result;
 }
 
 /**
  * Loads frontend files from a directory.
  * All files are read recursively and paths become keys with leading '/'.
  */
-async function loadFrontend(frontendPath: string): Promise<Record<string, string>> {
-	// @ts-ignore - Node.js fs/promises
-	const { access } = await import('fs/promises')
+async function loadFrontend(
+  frontendPath: string,
+): Promise<Record<string, string>> {
+  // @ts-ignore - Node.js fs/promises
+  const { access } = await import("fs/promises");
 
-	try {
-		await access(frontendPath)
-	} catch {
-		// Directory doesn't exist, return empty
-		return {}
-	}
+  try {
+    await access(frontendPath);
+  } catch {
+    // Directory doesn't exist, return empty
+    return {};
+  }
 
-	return readFilesRecursively(frontendPath)
+  return readFilesRecursively(frontendPath);
 }
 
 /**
@@ -68,63 +73,89 @@ async function loadFrontend(frontendPath: string): Promise<Record<string, string
  * - main.ts or main.py: The code content
  * - meta.json: Metadata { name, language }
  */
-async function loadBackend(backendPath: string): Promise<Record<string, BackendRunnable>> {
-	// @ts-ignore - Node.js fs/promises
-	const { readdir, readFile, access } = await import('fs/promises')
-	// @ts-ignore - Node.js path
-	const { join } = await import('path')
+async function loadBackend(
+  backendPath: string,
+): Promise<Record<string, BackendRunnable>> {
+  // @ts-ignore - Node.js fs/promises
+  const { readdir, readFile, access } = await import("fs/promises");
+  // @ts-ignore - Node.js path
+  const { join } = await import("path");
 
-	try {
-		await access(backendPath)
-	} catch {
-		// Directory doesn't exist, return empty
-		return {}
-	}
+  try {
+    await access(backendPath);
+  } catch {
+    // Directory doesn't exist, return empty
+    return {};
+  }
 
-	const result: Record<string, BackendRunnable> = {}
-	const entries = await readdir(backendPath, { withFileTypes: true })
+  const result: Record<string, BackendRunnable> = {};
+  const entries = await readdir(backendPath, { withFileTypes: true });
 
-	for (const entry of entries) {
-		if (!entry.isDirectory()) continue
+  for (const entry of entries) {
+    if (!entry.isDirectory()) continue;
 
-		const runnableKey = entry.name
-		const runnablePath = join(backendPath, entry.name)
+    const runnableKey = entry.name;
+    const runnablePath = join(backendPath, entry.name);
 
-		// Read meta.json
-		const metaPath = join(runnablePath, 'meta.json')
-		let meta: BackendMeta
-		try {
-			const metaContent = await readFile(metaPath, 'utf-8')
-			meta = JSON.parse(metaContent)
-		} catch {
-			console.warn(`Missing or invalid meta.json for runnable '${runnableKey}', skipping`)
-			continue
-		}
+    // Read meta.json
+    const metaPath = join(runnablePath, "meta.json");
+    let meta: BackendMeta;
+    try {
+      const metaContent = await readFile(metaPath, "utf-8");
+      meta = JSON.parse(metaContent);
+    } catch {
+      console.warn(
+        `Missing or invalid meta.json for runnable '${runnableKey}', skipping`,
+      );
+      continue;
+    }
 
-		// Find and read the main file (main.ts or main.py)
-		const runnableFiles = await readdir(runnablePath)
-		const mainFile = runnableFiles.find((f) => f === 'main.ts' || f === 'main.py')
+    // Find and read the main file (main.ts or main.py)
+    const runnableFiles = await readdir(runnablePath);
+    const mainFile = runnableFiles.find(
+      (f) => f === "main.ts" || f === "main.py",
+    );
 
-		if (!mainFile) {
-			console.warn(`No main.ts or main.py found for runnable '${runnableKey}', skipping`)
-			continue
-		}
+    if (!mainFile) {
+      console.warn(
+        `No main.ts or main.py found for runnable '${runnableKey}', skipping`,
+      );
+      continue;
+    }
 
-		const content = await readFile(join(runnablePath, mainFile), 'utf-8')
+    const content = await readFile(join(runnablePath, mainFile), "utf-8");
 
-		const inlineScript: InlineScript = {
-			language: meta.language,
-			content
-		}
+    const inlineScript: InlineScript = {
+      language: meta.language,
+      content,
+    };
 
-		result[runnableKey] = {
-			name: meta.name,
-			type: 'inline',
-			inlineScript
-		}
-	}
+    result[runnableKey] = {
+      name: meta.name,
+      type: "inline",
+      inlineScript,
+    };
+  }
 
-	return result
+  return result;
+}
+
+async function loadDatatables(fixturePath: string): Promise<DataTableSchema[]> {
+  // @ts-ignore - Node.js fs/promises
+  const { readFile } = await import("fs/promises");
+  // @ts-ignore - Node.js path
+  const { join } = await import("path");
+
+  try {
+    const content = await readFile(
+      join(fixturePath, "datatables.json"),
+      "utf-8",
+    );
+    const parsed = JSON.parse(content);
+    return Array.isArray(parsed) ? (parsed as DataTableSchema[]) : [];
+  } catch {
+    return [];
+  }
 }
 
 /**
@@ -146,29 +177,32 @@ async function loadBackend(backendPath: string): Promise<Record<string, BackendR
  * @param fixturePath - Path to the fixture directory
  * @returns AppFiles object with frontend and backend
  */
-export async function loadAppFixture(fixturePath: string): Promise<AppFiles> {
-	// @ts-ignore - Node.js path
-	const { join } = await import('path')
+export async function loadAppFixture(
+  fixturePath: string,
+): Promise<AppFilesState> {
+  // @ts-ignore - Node.js path
+  const { join } = await import("path");
 
-	const frontend = await loadFrontend(join(fixturePath, 'frontend'))
-	const backend = await loadBackend(join(fixturePath, 'backend'))
+  const frontend = await loadFrontend(join(fixturePath, "frontend"));
+  const backend = await loadBackend(join(fixturePath, "backend"));
+  const datatables = await loadDatatables(fixturePath);
 
-	return { frontend, backend }
+  return { frontend, backend, datatables };
 }
 
 /**
  * Loads an app fixture and returns the separate frontend and backend objects.
  * Convenience function for use with runAppEval options.
  */
-export async function loadAppFixtureForEval(
-	fixturePath: string
-): Promise<{
-	initialFrontend: Record<string, string>
-	initialBackend: Record<string, BackendRunnable>
+export async function loadAppFixtureForEval(fixturePath: string): Promise<{
+  initialFrontend: Record<string, string>;
+  initialBackend: AppFilesState["backend"];
+  initialDatatables: DataTableSchema[];
 }> {
-	const { frontend, backend } = await loadAppFixture(fixturePath)
-	return {
-		initialFrontend: frontend,
-		initialBackend: backend
-	}
+  const { frontend, backend, datatables } = await loadAppFixture(fixturePath);
+  return {
+    initialFrontend: frontend,
+    initialBackend: backend,
+    initialDatatables: datatables,
+  };
 }

ai_evals/adapters/frontend/core/app/fileHelpers.test.ts

@@ -0,0 +1,40 @@
+import { describe, expect, it } from "bun:test";
+import { createAppFileHelpers } from "./fileHelpers";
+
+describe("createAppFileHelpers", () => {
+  it("exposes generated wmill typings and returns real lint diagnostics", async () => {
+    const { helpers, cleanup } = await createAppFileHelpers(
+      {
+        "/index.tsx":
+          "import { backend } from 'wmill'\nexport default function App() { void backend.listRecipes(); return <div /> }\n",
+      },
+      {
+        listRecipes: {
+          name: "List recipes",
+          type: "inline",
+          inlineScript: {
+            language: "bun",
+            content: "export async function main() { return [] }\n",
+          },
+        },
+      }
+    );
+
+    try {
+      expect(helpers.listFrontendFiles()).toContain("/wmill.d.ts");
+      expect(helpers.getFrontendFile("/wmill.d.ts")).toContain("listRecipes");
+
+      const lintResult = helpers.setFrontendFile(
+        "/index.tsx",
+        "import { backend } from 'wmill'\nexport default function App() { void backend.deleteRecipe({ id: 1 }); return <div /> }\n"
+      );
+
+      expect(lintResult.errorCount).toBeGreaterThan(0);
+      expect(lintResult.errors.frontend["/index.tsx"]?.join("\n")).toContain(
+        "Property 'deleteRecipe' does not exist"
+      );
+    } finally {
+      await cleanup();
+    }
+  });
+});

ai_evals/adapters/frontend/core/app/fileHelpers.ts

@@ -8,15 +8,7 @@ import type {
 	LintResult,
 	SelectedContext
 } from '../../../../../frontend/src/lib/components/copilot/chat/app/core'
-
-function createEmptyLintResult(): LintResult {
-	return {
-		errorCount: 0,
-		warningCount: 0,
-		errors: { frontend: {}, backend: {} },
-		warnings: { frontend: {}, backend: {} }
-	}
-}
+import { buildAppWmillTypes, collectAppDiagnostics } from '../../../../core/appDiagnostics'
 
 async function writeFrontendFile(
 	workspaceRoot: string | undefined,
@@ -97,12 +89,19 @@ async function persistDatatables(
 export async function createAppFileHelpers(
 	initialFrontend: Record<string, string> = {},
 	initialBackend: Record<string, BackendRunnable> = {},
+	initialDatatables: DataTableSchema[] = [],
 	workspaceRoot?: string
 ): Promise<{
 	helpers: AppAIChatHelpers
 	getFiles: () => AppFiles
+	getEvalState: () => {
+		frontend: Record<string, string>
+		backend: Record<string, BackendRunnable>
+		datatables: DataTableSchema[]
+	}
 	getFrontend: () => Record<string, string>
 	getBackend: () => Record<string, BackendRunnable>
+	getDatatables: () => DataTableSchema[]
 	cleanup: () => Promise<void>
 	workspaceDir: string | null
 }> {
@@ -111,9 +110,24 @@ export async function createAppFileHelpers(
 	let snapshotId = 0
 	const snapshots = new Map<
 		number,
-		{ frontend: Record<string, string>; backend: Record<string, BackendRunnable> }
+		{
+			frontend: Record<string, string>
+			backend: Record<string, BackendRunnable>
+			datatables: DataTableSchema[]
+		}
 	>()
-	const datatables: DataTableSchema[] = []
+	const datatables: DataTableSchema[] = structuredClone(initialDatatables)
+
+	function lint(): LintResult {
+		return collectAppDiagnostics({
+			frontend,
+			backend
+		}).lintResult
+	}
+
+	function getGeneratedWmillTypes(): string {
+		return buildAppWmillTypes(backend)
+	}
 
 	for (const [path, content] of Object.entries(frontend)) {
 		await writeFrontendFile(workspaceRoot, path, content)
@@ -124,15 +138,34 @@ export async function createAppFileHelpers(
 	await persistDatatables(workspaceRoot, datatables)
 
 	const helpers: AppAIChatHelpers = {
-		listFrontendFiles: () => Object.keys(frontend),
-		getFrontendFile: (path: string) => frontend[path],
-		getFrontendFiles: () => ({ ...frontend }),
+		listFrontendFiles: () => [
+			...Object.keys(frontend).filter((path) => path !== '/wmill.d.ts'),
+			'/wmill.d.ts'
+		],
+		getFrontendFile: (path: string) => {
+			if (path === '/wmill.d.ts') {
+				return getGeneratedWmillTypes()
+			}
+			return frontend[path]
+		},
+		getFrontendFiles: () => ({
+			...Object.fromEntries(
+				Object.entries(frontend).filter(([path]) => path !== '/wmill.d.ts')
+			),
+			'/wmill.d.ts': getGeneratedWmillTypes()
+		}),
 		setFrontendFile: (path: string, content: string) => {
+			if (path === '/wmill.d.ts') {
+				return lint()
+			}
 			frontend[path] = content
 			void writeFrontendFile(workspaceRoot, path, content)
-			return createEmptyLintResult()
+			return lint()
 		},
 		deleteFrontendFile: (path: string) => {
+			if (path === '/wmill.d.ts') {
+				return
+			}
 			delete frontend[path]
 			void removeFrontendFile(workspaceRoot, path)
 		},
@@ -146,7 +179,7 @@ export async function createAppFileHelpers(
 		setBackendRunnable: async (key: string, runnable: BackendRunnable) => {
 			backend[key] = runnable
 			await writeBackendRunnable(workspaceRoot, key, runnable)
-			return createEmptyLintResult()
+			return lint()
 		},
 		deleteBackendRunnable: (key: string) => {
 			delete backend[key]
@@ -156,12 +189,13 @@ export async function createAppFileHelpers(
 			frontend: { ...frontend },
 			backend: { ...backend }
 		}),
-		getSelectedContext: (): SelectedContext => ({ type: 'none' }),
+		getSelectedContext: (): SelectedContext => ({}),
 		snapshot: () => {
 			const id = ++snapshotId
 			snapshots.set(id, {
 				frontend: { ...frontend },
-				backend: { ...backend }
+				backend: { ...backend },
+				datatables: structuredClone(datatables)
 			})
 			return id
 		},
@@ -172,9 +206,10 @@ export async function createAppFileHelpers(
 			}
 			frontend = { ...snapshot.frontend }
 			backend = { ...snapshot.backend }
+			datatables.splice(0, datatables.length, ...structuredClone(snapshot.datatables))
 			void syncWorkspace()
 		},
-		lint: () => createEmptyLintResult(),
+		lint,
 		getDatatables: async () => structuredClone(datatables),
 		getAvailableDatatableNames: () => datatables.map((datatable) => datatable.datatable_name),
 		execDatatableSql: async (
@@ -243,8 +278,14 @@ export async function createAppFileHelpers(
 			frontend: { ...frontend },
 			backend: { ...backend }
 		}),
+		getEvalState: () => ({
+			frontend: { ...frontend },
+			backend: { ...backend },
+			datatables: structuredClone(datatables)
+		}),
 		getFrontend: () => ({ ...frontend }),
 		getBackend: () => ({ ...backend }),
+		getDatatables: () => structuredClone(datatables),
 		cleanup: async () => {
 			if (workspaceRoot) {
 				await rm(workspaceRoot, { recursive: true, force: true })

ai_evals/adapters/frontend/core/flow/fileHelpers.ts

@@ -4,10 +4,14 @@ import type { FlowModule, InputTransform } from '../../../../../frontend/src/lib
 import type { ExtendedOpenFlow } from '../../../../../frontend/src/lib/components/flows/types'
 import type { FlowAIChatHelpers } from '../../../../../frontend/src/lib/components/copilot/chat/flow/core'
 import type { ScriptLintResult } from '../../../../../frontend/src/lib/components/copilot/chat/shared'
-import { findModuleById } from '../../../../../frontend/src/lib/components/copilot/chat/shared'
+import type { WorkspaceMutationTarget } from '../../../../../frontend/src/lib/components/copilot/chat/workspaceTools'
 import {
 	createInlineScriptSession
 } from '../../../../../frontend/src/lib/components/copilot/chat/flow/inlineScriptsUtils'
+import {
+	applyFlowJsonUpdate,
+	updateRawScriptModuleContent
+} from '../../../../../frontend/src/lib/components/copilot/chat/flow/helperUtils'
 import {
 	registerBenchmarkWorkspace,
 	registerBenchmarkWorkspaceRunnables,
@@ -32,8 +36,11 @@ export interface FlowWorkspaceFixtures {
 export async function createFlowFileHelpers(
 	initialModules: FlowModule[] = [],
 	initialSchema?: Record<string, any>,
+	initialPreprocessorModule?: FlowModule,
+	initialFailureModule?: FlowModule,
 	workspaceRoot?: string,
-	workspaceFixtures?: FlowWorkspaceFixtures
+	workspaceFixtures?: FlowWorkspaceFixtures,
+	currentFlowPath?: string
 ): Promise<{
 	helpers: FlowAIChatHelpers
 	getFlow: () => ExtendedOpenFlow
@@ -42,7 +49,11 @@ export async function createFlowFileHelpers(
 	workspaceDir: string | null
 }> {
 	let flow: ExtendedOpenFlow = {
-		value: { modules: structuredClone(initialModules) },
+		value: {
+			modules: structuredClone(initialModules),
+			preprocessor_module: structuredClone(initialPreprocessorModule),
+			failure_module: structuredClone(initialFailureModule)
+		},
 		summary: '',
 		schema: initialSchema ?? {
 			$schema: 'https://json-schema.org/draft/2020-12/schema',
@@ -72,42 +83,41 @@ export async function createFlowFileHelpers(
 		}
 	}
 
-	const helpers: FlowAIChatHelpers = {
+	const setFlowJson: FlowAIChatHelpers['setFlowJson'] = async ({
+		modules,
+		schema,
+		preprocessorModule,
+		failureModule
+	}) => {
+		const result = applyFlowJsonUpdate(flow, inlineScriptSession, {
+			modules,
+			schema,
+			preprocessorModule,
+			failureModule
+		})
+		await persistFlow()
+		return result
+	}
+
+	const helpers: FlowAIChatHelpers & {
+		getWorkspaceMutationTarget: () => WorkspaceMutationTarget
+	} = {
 		getFlowAndSelectedId: () => ({ flow, selectedId: '' }),
-		getModules: (id?: string) => {
-			if (!id) return flow.value.modules
-			const module = findModuleById(flow.value.modules, id)
-			return module ? [module] : []
-		},
+		getRootModules: () => flow.value.modules,
 		inlineScriptSession,
+		getWorkspaceMutationTarget: () => ({
+			kind: 'flow',
+			path: currentFlowPath,
+			deployed: Boolean(currentFlowPath)
+		}),
 		setSnapshot: () => {},
 		revertToSnapshot: () => {},
 		setCode: async (id: string, code: string) => {
-			const module = findModuleById(flow.value.modules, id)
-			if (module && module.value.type === 'rawscript') {
-				module.value.content = code
-			}
+			updateRawScriptModuleContent(flow, id, code)
 			inlineScriptSession.set(id, code)
 			await persistFlow()
 		},
-		setFlowJson: async (
-			modules: FlowModule[] | undefined,
-			schema: Record<string, any> | undefined
-		) => {
-			if (modules) {
-				flow.value.modules = inlineScriptSession.restoreInlineScriptReferences(modules)
-				const unresolvedRefs = inlineScriptSession.findUnresolvedInlineScriptRefs(flow.value.modules)
-				if (unresolvedRefs.length > 0) {
-					throw new Error(
-						`Unresolved inline script references: ${unresolvedRefs.join(', ')}`
-					)
-				}
-			}
-			if (schema !== undefined) {
-				flow.schema = schema
-			}
-			await persistFlow()
-		},
+		setFlowJson,
 		getFlowInputsSchema: async () => flow.schema ?? {},
 		updateExprsToSet: (_id: string, _inputTransforms: Record<string, InputTransform>) => {},
 		acceptAllModuleActions: () => {},
@@ -122,7 +132,9 @@ export async function createFlowFileHelpers(
 					JSON.stringify(
 						{
 							requestedArgs: args ?? {},
-							modules: flow.value.modules.map((module) => module.id)
+							modules: flow.value.modules.map((module) => module.id),
+							preprocessor_module: flow.value.preprocessor_module?.id ?? null,
+							failure_module: flow.value.failure_module?.id ?? null
 						},
 						null,
 						2
@@ -136,6 +148,8 @@ export async function createFlowFileHelpers(
 				result: {
 					requestedArgs: args ?? {},
 					modules: flow.value.modules.map((module) => module.id),
+					preprocessor_module: flow.value.preprocessor_module?.id ?? null,
+					failure_module: flow.value.failure_module?.id ?? null,
 					mocked: true
 				},
 				logs: 'Mock benchmark flow test run completed successfully.'

ai_evals/adapters/frontend/core/flow/flowEvalRunner.ts

@@ -1,103 +1,123 @@
-import { mkdtemp } from 'fs/promises'
-import { tmpdir } from 'os'
-import { join } from 'path'
-import type { FlowModule } from '$lib/gen'
-import type { AIProvider } from '$lib/gen/types.gen'
-import type { ExtendedOpenFlow } from '$lib/components/flows/types'
+import { mkdtemp } from "fs/promises";
+import { tmpdir } from "os";
+import { join } from "path";
+import type { FlowModule } from "$lib/gen";
+import type { AIProvider } from "$lib/gen/types.gen";
+import type { ExtendedOpenFlow } from "$lib/components/flows/types";
 import {
-	flowTools,
-	prepareFlowSystemMessage,
-	prepareFlowUserMessage,
-	type FlowAIChatHelpers
-} from '../../../../../frontend/src/lib/components/copilot/chat/flow/core'
-import type { Tool as ProductionTool } from '../../../../../frontend/src/lib/components/copilot/chat/shared'
-import { createFlowFileHelpers, type FlowWorkspaceFixtures } from './fileHelpers'
-import { runEval } from '../shared'
-import type { ModeRunContext } from '../../../../core/types'
-import type { TokenUsage } from '../shared/types'
+  flowTools,
+  prepareFlowSystemMessage,
+  prepareFlowUserMessage,
+  type FlowAIChatHelpers,
+} from "../../../../../frontend/src/lib/components/copilot/chat/flow/core";
+import type { Tool as ProductionTool } from "../../../../../frontend/src/lib/components/copilot/chat/shared";
+import {
+  createFlowFileHelpers,
+  type FlowWorkspaceFixtures,
+} from "./fileHelpers";
+import { runEval } from "../shared";
+import type { ModeRunContext } from "../../../../core/types";
+import type { TokenUsage, ToolCallDetail } from "../shared/types";
+import type { FrontendEvalTransport } from "../../../../core/frontendTransport";
+import type { WindmillBackendSettings } from "../../../../core/windmillBackendSettings";
 
 export interface FlowFixture {
-	value?: {
-		modules?: FlowModule[]
-	}
-	schema?: Record<string, unknown>
+  path?: string;
+  value?: {
+    modules?: FlowModule[];
+    preprocessor_module?: FlowModule;
+    failure_module?: FlowModule;
+  };
+  schema?: Record<string, unknown>;
 }
 
 export interface FlowEvalResult {
-	success: boolean
-	flow: ExtendedOpenFlow
-	error?: string
-	assistantMessageCount: number
-	toolCallCount: number
-	toolsUsed: string[]
-	tokenUsage: TokenUsage
+  success: boolean;
+  flow: ExtendedOpenFlow;
+  error?: string;
+  assistantMessageCount: number;
+  toolCallCount: number;
+  toolsUsed: string[];
+  toolCallDetails: ToolCallDetail[];
+  tokenUsage: TokenUsage;
 }
 
 export interface FlowEvalOptions {
-	initialFlow?: FlowFixture
-	workspaceFixtures?: FlowWorkspaceFixtures
-	model?: string
-	maxIterations?: number
-	provider?: AIProvider
-	workspaceRoot?: string
-	runContext?: ModeRunContext
+  initialFlow?: FlowFixture;
+  workspaceFixtures?: FlowWorkspaceFixtures;
+  model?: string;
+  maxIterations?: number;
+  provider?: AIProvider;
+  transport?: FrontendEvalTransport;
+  backend?: WindmillBackendSettings;
+  workspaceRoot?: string;
+  runContext?: ModeRunContext;
 }
 
 export async function runFlowEval(
-	userPrompt: string,
-	apiKey: string,
-	options?: FlowEvalOptions
+  userPrompt: string,
+  apiKey: string,
+  options?: FlowEvalOptions,
 ): Promise<FlowEvalResult> {
-	const workspaceRoot =
-		options?.workspaceRoot ??
-		(await mkdtemp(join(tmpdir(), 'wmill-frontend-flow-benchmark-')))
-	const { helpers, getFlow, cleanup } = await createFlowFileHelpers(
-		options?.initialFlow?.value?.modules ?? [],
-		options?.initialFlow?.schema,
-		workspaceRoot,
-		options?.workspaceFixtures
-	)
+  const workspaceRoot =
+    options?.workspaceRoot ??
+    (await mkdtemp(join(tmpdir(), "wmill-frontend-flow-benchmark-")));
+  const { helpers, getFlow, cleanup } = await createFlowFileHelpers(
+    options?.initialFlow?.value?.modules ?? [],
+    options?.initialFlow?.schema,
+    options?.initialFlow?.value?.preprocessor_module,
+    options?.initialFlow?.value?.failure_module,
+    workspaceRoot,
+    options?.workspaceFixtures,
+    options?.initialFlow?.path,
+  );
 
-	try {
-		const systemMessage = prepareFlowSystemMessage()
-		const tools = flowTools as ProductionTool<FlowAIChatHelpers>[]
-		const model = options?.model ?? 'claude-haiku-4-5-20251001'
-		const userMessage = prepareFlowUserMessage(
-			userPrompt,
-			helpers.getFlowAndSelectedId(),
-			[],
-			helpers.inlineScriptSession
-		)
+  try {
+    const systemMessage = prepareFlowSystemMessage();
+    const tools = flowTools as ProductionTool<FlowAIChatHelpers>[];
+    const model = options?.model ?? "claude-haiku-4-5-20251001";
+    const userMessage = prepareFlowUserMessage(
+      userPrompt,
+      helpers.getFlowAndSelectedId(),
+      [],
+      helpers.inlineScriptSession,
+    );
 
-		const rawResult = await runEval({
-			userPrompt,
-			systemMessage,
-			userMessage,
-			tools,
-			helpers,
-			apiKey,
-			getOutput: getFlow,
-			onAssistantMessageStart: options?.runContext?.onAssistantMessageStart,
-			onAssistantToken: options?.runContext?.onAssistantChunk,
-			onAssistantMessageEnd: options?.runContext?.onAssistantMessageEnd,
-			options: {
-				maxIterations: options?.maxIterations,
-				model,
-				workspace: workspaceRoot,
-				provider: options?.provider
-			}
-		})
+    const rawResult = await runEval({
+      userPrompt,
+      systemMessage,
+      userMessage,
+      tools,
+      helpers,
+      apiKey,
+      getOutput: getFlow,
+      onAssistantMessageStart: options?.runContext?.onAssistantMessageStart,
+      onAssistantToken: options?.runContext?.onAssistantChunk,
+      onAssistantMessageEnd: options?.runContext?.onAssistantMessageEnd,
+      onToolCall: options?.runContext?.onToolCall,
+      options: {
+        maxIterations: options?.maxIterations,
+        model,
+        workspace: workspaceRoot,
+        provider: options?.provider,
+        transport: options?.transport,
+        backend: options?.backend,
+        proxyCaseId: options?.runContext?.caseId,
+        proxyAttempt: options?.runContext?.attempt,
+      },
+    });
 
-		return {
-			flow: rawResult.output,
-			success: rawResult.success,
-			error: rawResult.error,
-			assistantMessageCount: rawResult.iterations,
-			toolCallCount: rawResult.toolCallsCount,
-			toolsUsed: rawResult.toolsCalled,
-			tokenUsage: rawResult.tokenUsage
-		}
-	} finally {
-		await cleanup()
-	}
+    return {
+      flow: rawResult.output,
+      success: rawResult.success,
+      error: rawResult.error,
+      assistantMessageCount: rawResult.iterations,
+      toolCallCount: rawResult.toolCallsCount,
+      toolsUsed: rawResult.toolsCalled,
+      toolCallDetails: rawResult.toolCallDetails,
+      tokenUsage: rawResult.tokenUsage,
+    };
+  } finally {
+    await cleanup();
+  }
 }

ai_evals/adapters/frontend/core/script/fileHelpers.ts

@@ -1,8 +1,8 @@
 import { mkdir, rm, writeFile } from 'fs/promises'
 import { dirname, join } from 'path'
 import type { ScriptLang } from '../../../../../frontend/src/lib/gen/types.gen'
-import type { ReviewChangesOpts } from '../../../../../frontend/src/lib/components/copilot/chat/monaco-adapter'
 import type { ScriptChatHelpers } from '../../../../../frontend/src/lib/components/copilot/chat/script/core'
+import type { WorkspaceMutationTarget } from '../../../../../frontend/src/lib/components/copilot/chat/workspaceTools'
 import { buildScriptLintResult } from './preview'
 import { registerBenchmarkWorkspace, unregisterBenchmarkWorkspace } from '../../mockBackend'
 
@@ -13,6 +13,10 @@ export interface ScriptEvalState {
 	args: Record<string, any>
 }
 
+function toRunnablePath(filePath: string): string {
+	return filePath.replace(/\.[^/.]+$/, '')
+}
+
 export async function createScriptFileHelpers(
 	initialScript: ScriptEvalState,
 	workspaceRoot?: string
@@ -39,24 +43,39 @@ export async function createScriptFileHelpers(
 		registerBenchmarkWorkspace(workspaceRoot)
 	}
 
-	const helpers: ScriptChatHelpers = {
+	const applyCode: NonNullable<ScriptChatHelpers['applyCode']> = async (
+		code,
+		opts
+	) => {
+		if (opts?.mode === 'revert') {
+			return
+		}
+		script = {
+			...script,
+			code
+		}
+		await persistScript()
+	}
+
+	const getLintErrors: NonNullable<ScriptChatHelpers['getLintErrors']> = () =>
+		buildScriptLintResult(script.code, script.lang)
+
+	const helpers: ScriptChatHelpers & {
+		getWorkspaceMutationTarget: () => WorkspaceMutationTarget
+	} = {
 		getScriptOptions: () => ({
 			code: script.code,
 			lang: script.lang,
 			path: script.path,
 			args: structuredClone(script.args)
 		}),
-		applyCode: async (code: string, opts?: ReviewChangesOpts) => {
-			if (opts?.mode === 'revert') {
-				return
-			}
-			script = {
-				...script,
-				code
-			}
-			await persistScript()
-		},
-		getLintErrors: () => buildScriptLintResult(script.code, script.lang)
+		getWorkspaceMutationTarget: () => ({
+			kind: 'script',
+			path: script.path ? toRunnablePath(script.path) : undefined,
+			deployed: Boolean(script.path)
+		}),
+		applyCode,
+		getLintErrors
 	}
 
 	return {

ai_evals/adapters/frontend/core/script/scriptEvalRunner.ts

@@ -1,109 +1,121 @@
-import { mkdtemp } from 'fs/promises'
-import { tmpdir } from 'os'
-import { join } from 'path'
-import type { AIProvider, AIProviderModel, ScriptLang } from '$lib/gen/types.gen'
-import type { ContextElement } from '../../../../../frontend/src/lib/components/copilot/chat/context'
+import { mkdtemp } from "fs/promises";
+import { tmpdir } from "os";
+import { join } from "path";
+import type { AIProvider, AIProviderModel } from "$lib/gen/types.gen";
+import type { ContextElement } from "../../../../../frontend/src/lib/components/copilot/chat/context";
 import {
-	prepareScriptSystemMessage,
-	prepareScriptTools,
-	prepareScriptUserMessage,
-	type ScriptChatHelpers
-} from '../../../../../frontend/src/lib/components/copilot/chat/script/core'
-import type { Tool as ProductionTool } from '../../../../../frontend/src/lib/components/copilot/chat/shared'
-import { createScriptFileHelpers, type ScriptEvalState } from './fileHelpers'
-import { runEval } from '../shared'
-import type { ModeRunContext } from '../../../../core/types'
-import type { TokenUsage } from '../shared/types'
+  prepareScriptSystemMessage,
+  prepareScriptTools,
+  prepareScriptUserMessage,
+  type ScriptChatHelpers,
+} from "../../../../../frontend/src/lib/components/copilot/chat/script/core";
+import type { Tool as ProductionTool } from "../../../../../frontend/src/lib/components/copilot/chat/shared";
+import { createScriptFileHelpers, type ScriptEvalState } from "./fileHelpers";
+import { runEval } from "../shared";
+import type { ModeRunContext } from "../../../../core/types";
+import type { TokenUsage, ToolCallDetail } from "../shared/types";
+import type { FrontendEvalTransport } from "../../../../core/frontendTransport";
+import type { WindmillBackendSettings } from "../../../../core/windmillBackendSettings";
 
 export interface ScriptEvalResult {
-	success: boolean
-	script: ScriptEvalState
-	error?: string
-	assistantMessageCount: number
-	toolCallCount: number
-	toolsUsed: string[]
-	tokenUsage: TokenUsage
+  success: boolean;
+  script: ScriptEvalState;
+  error?: string;
+  assistantMessageCount: number;
+  toolCallCount: number;
+  toolsUsed: string[];
+  toolCallDetails: ToolCallDetail[];
+  tokenUsage: TokenUsage;
 }
 
 export interface ScriptEvalOptions {
-	initialScript: ScriptEvalState
-	model?: string
-	maxIterations?: number
-	provider?: AIProvider
-	workspaceRoot?: string
-	runContext?: ModeRunContext
+  initialScript: ScriptEvalState;
+  model?: string;
+  maxIterations?: number;
+  provider?: AIProvider;
+  transport?: FrontendEvalTransport;
+  backend?: WindmillBackendSettings;
+  workspaceRoot?: string;
+  runContext?: ModeRunContext;
 }
 
 function resolveModelProvider(
-	model: string,
-	provider?: AIProvider
+  model: string,
+  provider?: AIProvider,
 ): AIProviderModel {
-	if (provider) {
-		return { provider, model }
-	}
-	if (model.startsWith('claude')) {
-		return { provider: 'anthropic', model }
-	}
-	return { provider: 'openai', model }
+  if (provider) {
+    return { provider, model };
+  }
+  if (model.startsWith("claude")) {
+    return { provider: "anthropic", model };
+  }
+  return { provider: "openai", model };
 }
 
 export async function runScriptEval(
-	userPrompt: string,
-	apiKey: string,
-	options: ScriptEvalOptions
+  userPrompt: string,
+  apiKey: string,
+  options: ScriptEvalOptions,
 ): Promise<ScriptEvalResult> {
-	const workspaceRoot =
-		options.workspaceRoot ?? (await mkdtemp(join(tmpdir(), 'wmill-frontend-script-benchmark-')))
-	const { helpers, getScript, cleanup } = await createScriptFileHelpers(
-		options.initialScript,
-		workspaceRoot
-	)
+  const workspaceRoot =
+    options.workspaceRoot ??
+    (await mkdtemp(join(tmpdir(), "wmill-frontend-script-benchmark-")));
+  const { helpers, getScript, cleanup } = await createScriptFileHelpers(
+    options.initialScript,
+    workspaceRoot,
+  );
 
-	try {
-		const model = options.model ?? 'claude-haiku-4-5-20251001'
-		const modelProvider = resolveModelProvider(model, options.provider)
-		const selectedContext: ContextElement[] = []
-		const systemMessage = prepareScriptSystemMessage(
-			modelProvider,
-			options.initialScript.lang,
-			{}
-		)
-		const tools = prepareScriptTools(
-			modelProvider,
-			options.initialScript.lang,
-			selectedContext
-		) as ProductionTool<ScriptChatHelpers>[]
-		const userMessage = prepareScriptUserMessage(userPrompt, selectedContext)
+  try {
+    const model = options.model ?? "claude-haiku-4-5-20251001";
+    const modelProvider = resolveModelProvider(model, options.provider);
+    const selectedContext: ContextElement[] = [];
+    const systemMessage = prepareScriptSystemMessage(
+      modelProvider,
+      options.initialScript.lang,
+      {},
+    );
+    const tools = prepareScriptTools(
+      modelProvider,
+      options.initialScript.lang,
+      selectedContext,
+    ) as ProductionTool<ScriptChatHelpers>[];
+    const userMessage = prepareScriptUserMessage(userPrompt, selectedContext);
 
-		const rawResult = await runEval({
-			userPrompt,
-			systemMessage,
-			userMessage,
-			tools,
-			helpers,
-			apiKey,
-			getOutput: getScript,
-			onAssistantMessageStart: options.runContext?.onAssistantMessageStart,
-			onAssistantToken: options.runContext?.onAssistantChunk,
-			onAssistantMessageEnd: options.runContext?.onAssistantMessageEnd,
-			options: {
-				maxIterations: options.maxIterations,
-				model,
-				workspace: workspaceRoot,
-				provider: modelProvider.provider
-			}
-		})
+    const rawResult = await runEval({
+      userPrompt,
+      systemMessage,
+      userMessage,
+      tools,
+      helpers,
+      apiKey,
+      getOutput: getScript,
+      onAssistantMessageStart: options.runContext?.onAssistantMessageStart,
+      onAssistantToken: options.runContext?.onAssistantChunk,
+      onAssistantMessageEnd: options.runContext?.onAssistantMessageEnd,
+      onToolCall: options.runContext?.onToolCall,
+      options: {
+        maxIterations: options.maxIterations,
+        model,
+        workspace: workspaceRoot,
+        provider: modelProvider.provider,
+        transport: options.transport,
+        backend: options.backend,
+        proxyCaseId: options.runContext?.caseId,
+        proxyAttempt: options.runContext?.attempt,
+      },
+    });
 
-		return {
-			script: rawResult.output,
-			success: rawResult.success,
-			error: rawResult.error,
-			assistantMessageCount: rawResult.iterations,
-			toolCallCount: rawResult.toolCallsCount,
-			toolsUsed: rawResult.toolsCalled,
-			tokenUsage: rawResult.tokenUsage
-		}
-	} finally {
-		await cleanup()
-	}
+    return {
+      script: rawResult.output,
+      success: rawResult.success,
+      error: rawResult.error,
+      assistantMessageCount: rawResult.iterations,
+      toolCallCount: rawResult.toolCallsCount,
+      toolsUsed: rawResult.toolsCalled,
+      toolCallDetails: rawResult.toolCallDetails,
+      tokenUsage: rawResult.tokenUsage,
+    };
+  } finally {
+    await cleanup();
+  }
 }

ai_evals/adapters/frontend/core/shared/baseEvalRunner.ts

@@ -1,45 +1,51 @@
 import type {
-	ChatCompletionMessageParam,
-	ChatCompletionSystemMessageParam
-} from 'openai/resources/chat/completions.mjs'
-import type { AIProviderModel } from '$lib/gen/types.gen'
-import type { TokenUsage, ToolCallDetail, EvalRunnerOptions, RawEvalResult } from './types'
-import { runChatLoop, type ChatClients } from '../../../../../frontend/src/lib/components/copilot/chat/chatLoop'
-import type {
-	Tool as ProductionTool,
-	ToolCallbacks
-} from '../../../../../frontend/src/lib/components/copilot/chat/shared'
+  ChatCompletionMessageParam,
+  ChatCompletionSystemMessageParam,
+} from "openai/resources/chat/completions.mjs";
+import type { AIProvider } from "$lib/gen/types.gen";
+import type { ToolCallDetail, EvalRunnerOptions, RawEvalResult } from "./types";
 import {
-	createEvalClients,
-	type FrontendEvalProvider,
-	resolveEvalModelProvider
-} from './providerConfig'
+  runChatLoop,
+  type ChatClients,
+} from "../../../../../frontend/src/lib/components/copilot/chat/chatLoop";
+import type {
+  Tool as ProductionTool,
+  ToolCallbacks,
+} from "../../../../../frontend/src/lib/components/copilot/chat/shared";
+import {
+  buildProxyResourcePath,
+  createEvalClients,
+  type FrontendEvalProvider,
+  resolveEvalModelProvider,
+} from "./providerConfig";
+import { WindmillBackendClient } from "../../windmillBackend";
 
 /**
  * Parameters for running a base evaluation.
  */
 export interface RunEvalParams<THelpers, TOutput> {
-	/** The user's prompt/instruction */
-	userPrompt: string
-	/** System message for the LLM */
-	systemMessage: ChatCompletionSystemMessageParam
-	/** User message for the LLM */
-	userMessage: ChatCompletionMessageParam
-	/** Tool definitions for the LLM API (unused — derived from tools) */
-	toolDefs?: unknown
-	/** Full tool implementations for execution */
-	tools: ProductionTool<THelpers>[]
-	/** Domain-specific helpers for tool execution */
-	helpers: THelpers
-	/** API key for the provider */
-	apiKey: string
-	/** Function to get the current output state */
-	getOutput: () => TOutput
-	/** Optional configuration */
-	options?: EvalRunnerOptions
-	onAssistantMessageStart?: () => void
-	onAssistantToken?: (token: string) => void
-	onAssistantMessageEnd?: () => void
+  /** The user's prompt/instruction */
+  userPrompt: string;
+  /** System message for the LLM */
+  systemMessage: ChatCompletionSystemMessageParam;
+  /** User message for the LLM */
+  userMessage: ChatCompletionMessageParam;
+  /** Tool definitions for the LLM API (unused — derived from tools) */
+  toolDefs?: unknown;
+  /** Full tool implementations for execution */
+  tools: ProductionTool<THelpers>[];
+  /** Domain-specific helpers for tool execution */
+  helpers: THelpers;
+  /** API key for the provider */
+  apiKey: string;
+  /** Function to get the current output state */
+  getOutput: () => TOutput;
+  /** Optional configuration */
+  options?: EvalRunnerOptions;
+  onAssistantMessageStart?: () => void;
+  onAssistantToken?: (token: string) => void;
+  onAssistantMessageEnd?: () => void;
+  onToolCall?: (input: { toolName: string; argumentsText: string }) => void;
 }
 
 /**
@@ -47,127 +53,206 @@ export interface RunEvalParams<THelpers, TOutput> {
  * Uses streaming via real provider SDKs instead of OpenRouter non-streaming.
  */
 export async function runEval<THelpers, TOutput>(
-	params: RunEvalParams<THelpers, TOutput>
+  params: RunEvalParams<THelpers, TOutput>,
 ): Promise<RawEvalResult<TOutput>> {
-	const {
-		systemMessage,
-		userMessage,
-		tools,
-		helpers,
-		apiKey,
-		getOutput,
-		options,
-		onAssistantMessageStart,
-		onAssistantToken,
-		onAssistantMessageEnd
-	} = params
-	let shouldEmitMessageStart = true
+  const {
+    systemMessage,
+    userMessage,
+    tools,
+    helpers,
+    apiKey,
+    getOutput,
+    options,
+    onAssistantMessageStart,
+    onAssistantToken,
+    onAssistantMessageEnd,
+    onToolCall,
+  } = params;
+  let shouldEmitMessageStart = true;
 
-	const model = options?.model ?? 'gpt-4o'
-	const maxIterations = options?.maxIterations ?? 20
-	const workspace = options?.workspace ?? 'test-workspace'
-	const provider = options?.provider
+  const model = options?.model ?? "gpt-4o";
+  const maxIterations = options?.maxIterations ?? 20;
+  const workspace = options?.workspace ?? "test-workspace";
+  const provider = toFrontendEvalProvider(options?.provider);
 
-	const modelProvider = resolveEvalModelProvider(
-		model,
-		provider as FrontendEvalProvider | undefined
-	) as AIProviderModel
-	const clients = createEvalClients(modelProvider.provider, apiKey) as ChatClients
+  const modelProvider = resolveEvalModelProvider(model, provider);
 
-	const messages: ChatCompletionMessageParam[] = [userMessage]
-	let toolCallsCount = 0
-	const toolsCalled: string[] = []
-	const toolCallDetails: ToolCallDetail[] = []
+  const messages: ChatCompletionMessageParam[] = [userMessage];
+  let toolCallsCount = 0;
+  const toolsCalled: string[] = [];
+  const toolCallDetails: ToolCallDetail[] = [];
 
-	// Wrap tools to intercept fn calls for tracking.
-	// Cast to ProductionTool since the eval Tool has a narrower toolCallbacks type
-	// but the actual callbacks passed at runtime will satisfy both interfaces.
-	const wrappedTools = tools.map((tool) => ({
-		...tool,
-		fn: async (p: any) => {
-			toolCallsCount++
-			toolsCalled.push(tool.def.function.name)
-			try {
-				const args =
-					typeof p.args === 'string' ? JSON.parse(p.args) : p.args
-				toolCallDetails.push({ name: tool.def.function.name, arguments: args })
-			} catch {
-				toolCallDetails.push({
-					name: tool.def.function.name,
-					arguments: p.args
-				})
-			}
-			return tool.fn(p)
-		}
-	}))
+  // Wrap tools to intercept fn calls for tracking.
+  // Cast to ProductionTool since the eval Tool has a narrower toolCallbacks type
+  // but the actual callbacks passed at runtime will satisfy both interfaces.
+  const wrappedTools = tools.map((tool) => ({
+    ...tool,
+    fn: async (p: any) => {
+      toolCallsCount++;
+      toolsCalled.push(tool.def.function.name);
+      let argumentsText = "";
+      try {
+        const args = typeof p.args === "string" ? JSON.parse(p.args) : p.args;
+        toolCallDetails.push({ name: tool.def.function.name, arguments: args });
+        argumentsText = JSON.stringify(args);
+      } catch {
+        toolCallDetails.push({
+          name: tool.def.function.name,
+          arguments: p.args,
+        });
+        argumentsText =
+          typeof p.args === "string" ? p.args : JSON.stringify(p.args);
+      }
+      onToolCall?.({
+        toolName: tool.def.function.name,
+        argumentsText,
+      });
+      return tool.fn(p);
+    },
+  }));
 
-	// No-op callbacks for eval
-	const callbacks: ToolCallbacks & {
-		onNewToken: (token: string) => void
-		onMessageEnd: () => void
-	} = {
-		setToolStatus: () => {},
-		removeToolStatus: () => {},
-		onNewToken: (token: string) => {
-			if (shouldEmitMessageStart) {
-				onAssistantMessageStart?.()
-				shouldEmitMessageStart = false
-			}
-			onAssistantToken?.(token)
-		},
-		onMessageEnd: () => {
-			if (!shouldEmitMessageStart) {
-				onAssistantMessageEnd?.()
-			}
-			shouldEmitMessageStart = true
-		}
-	}
+  // No-op callbacks for eval
+  const callbacks: ToolCallbacks & {
+    onNewToken: (token: string) => void;
+    onMessageEnd: () => void;
+  } = {
+    setToolStatus: () => {},
+    removeToolStatus: () => {},
+    onNewToken: (token: string) => {
+      if (shouldEmitMessageStart) {
+        onAssistantMessageStart?.();
+        shouldEmitMessageStart = false;
+      }
+      onAssistantToken?.(token);
+    },
+    onMessageEnd: () => {
+      if (!shouldEmitMessageStart) {
+        onAssistantMessageEnd?.();
+      }
+      shouldEmitMessageStart = true;
+    },
+  };
 
-	const abortController = new AbortController()
+  const abortController = new AbortController();
 
-	try {
-		const result = await runChatLoop({
-			messages,
-			systemMessage,
-			tools: wrappedTools,
-			helpers,
-			abortController,
-			callbacks,
-			modelProvider,
-			clients,
-			workspace,
-			maxIterations,
-			skipResponsesApi: modelProvider.provider !== 'openai' && modelProvider.provider !== 'azure_openai'
-		})
+  const executeChatLoop = async (clients: ChatClients) => {
+    try {
+      const result = await runChatLoop({
+        messages,
+        systemMessage,
+        tools: wrappedTools,
+        helpers,
+        abortController,
+        callbacks,
+        modelProvider,
+        clients,
+        workspace,
+        maxIterations,
+        skipResponsesApi: modelProvider.provider !== "openai",
+      });
 
-		return {
-			success: true,
-			output: getOutput(),
-			tokenUsage: result.tokenUsage,
-			toolCallsCount,
-			toolsCalled,
-			toolCallDetails,
-			iterations: Math.max(1, result.addedMessages.filter((m) => m.role === 'assistant').length),
-			messages
-		}
-	} catch (err) {
-		let errorMessage: string
-		if (err instanceof Error) {
-			errorMessage = err.stack ?? err.message
-		} else {
-			errorMessage = String(err)
-		}
+      if (result.hitMaxIterations) {
+        return {
+          success: false,
+          output: getOutput(),
+          error: `Reached max turns (${maxIterations})`,
+          tokenUsage: result.tokenUsage,
+          toolCallsCount,
+          toolsCalled,
+          toolCallDetails,
+          iterations: Math.max(
+            1,
+            result.addedMessages.filter((m) => m.role === "assistant").length,
+          ),
+          messages,
+        };
+      }
 
-		return {
-			success: false,
-			output: getOutput(),
-			error: errorMessage,
-			tokenUsage: { prompt: 0, completion: 0, total: 0 },
-			toolCallsCount,
-			toolsCalled,
-			toolCallDetails,
-			iterations: 0,
-			messages
-		}
-	}
+      return {
+        success: true,
+        output: getOutput(),
+        tokenUsage: result.tokenUsage,
+        toolCallsCount,
+        toolsCalled,
+        toolCallDetails,
+        iterations: Math.max(
+          1,
+          result.addedMessages.filter((m) => m.role === "assistant").length,
+        ),
+        messages,
+      };
+    } catch (err) {
+      let errorMessage: string;
+      if (err instanceof Error) {
+        errorMessage = err.stack ?? err.message;
+      } else {
+        errorMessage = String(err);
+      }
+
+      return {
+        success: false,
+        output: getOutput(),
+        error: errorMessage,
+        tokenUsage: { prompt: 0, completion: 0, total: 0 },
+        toolCallsCount,
+        toolsCalled,
+        toolCallDetails,
+        iterations: 0,
+        messages,
+      };
+    }
+  };
+
+  if (options?.transport === "proxy") {
+    const backendSettings = options.backend;
+    if (!backendSettings) {
+      throw new Error("Missing backend settings for proxy transport");
+    }
+
+    const backendClient = new WindmillBackendClient(backendSettings);
+    return await backendClient.withWorkspace(
+      options.proxyCaseId ?? "eval",
+      options.proxyAttempt ?? 1,
+      async (proxyWorkspaceId) => {
+        const resourcePath = buildProxyResourcePath(modelProvider.provider);
+        await backendClient.upsertResource({
+          workspaceId: proxyWorkspaceId,
+          path: resourcePath,
+          resourceType: modelProvider.provider,
+          value: { api_key: apiKey },
+        });
+        const token = await backendClient.getToken();
+        const clients = createEvalClients({
+          provider: modelProvider.provider,
+          apiKey,
+          transport: "proxy",
+          proxy: {
+            baseURL: `${backendSettings.baseUrl}/api/w/${encodeURIComponent(proxyWorkspaceId)}/ai/proxy`,
+            bearerToken: token,
+            resourcePath,
+          },
+        }) as unknown as ChatClients;
+        return await executeChatLoop(clients);
+      },
+    );
+  }
+
+  const clients = createEvalClients({
+    provider: modelProvider.provider,
+    apiKey,
+  }) as unknown as ChatClients;
+  return await executeChatLoop(clients);
+}
+
+function toFrontendEvalProvider(
+  provider?: AIProvider,
+): FrontendEvalProvider | undefined {
+  if (
+    provider === "anthropic" ||
+    provider === "openai" ||
+    provider === "googleai"
+  ) {
+    return provider;
+  }
+  return undefined;
 }

ai_evals/adapters/frontend/core/shared/providerConfig.test.ts

@@ -1,12 +1,17 @@
 import { describe, expect, it } from "bun:test";
 import {
+  buildProxyHeaders,
+  buildProxyResourcePath,
   buildOpenAICompatibleClientOptions,
   resolveEvalModelProvider,
 } from "./providerConfig";
 
 describe("buildOpenAICompatibleClientOptions", () => {
   it("adds Gemini's OpenAI-compatible base URL and client header", () => {
-    const options = buildOpenAICompatibleClientOptions("googleai", "gemini-test-key");
+    const options = buildOpenAICompatibleClientOptions(
+      "googleai",
+      "gemini-test-key",
+    );
 
     expect(options).toMatchObject({
       apiKey: "gemini-test-key",
@@ -18,12 +23,28 @@ describe("buildOpenAICompatibleClientOptions", () => {
   });
 
   it("keeps the default OpenAI-compatible config for OpenAI", () => {
-    expect(buildOpenAICompatibleClientOptions("openai", "openai-test-key")).toEqual({
+    expect(
+      buildOpenAICompatibleClientOptions("openai", "openai-test-key"),
+    ).toEqual({
       apiKey: "openai-test-key",
     });
   });
 });
 
+describe("proxy helpers", () => {
+  it("builds provider-scoped proxy resource paths", () => {
+    expect(buildProxyResourcePath("googleai")).toBe("f/evals/ai/googleai");
+    expect(buildProxyResourcePath("anthropic")).toBe("f/evals/ai/anthropic");
+  });
+
+  it("adds auth and resource headers for workspace proxy requests", () => {
+    expect(buildProxyHeaders("token-123", "f/evals/ai/googleai")).toEqual({
+      Authorization: "Bearer token-123",
+      "X-Resource-Path": "f/evals/ai/googleai",
+    });
+  });
+});
+
 describe("resolveEvalModelProvider", () => {
   it("infers googleai from Gemini model ids", () => {
     expect(resolveEvalModelProvider("gemini-2.5-flash")).toEqual({

ai_evals/adapters/frontend/core/shared/providerConfig.ts

@@ -1,6 +1,7 @@
 import Anthropic from "@anthropic-ai/sdk";
 import OpenAI from "openai";
 import type { FrontendEvalModelConfig } from "../../../../core/models";
+import type { FrontendEvalTransport } from "../../../../core/frontendTransport";
 
 export type FrontendEvalProvider = FrontendEvalModelConfig["provider"];
 
@@ -14,12 +15,34 @@ export interface ResolvedEvalModelProvider {
   model: string;
 }
 
-const GEMINI_OPENAI_BASE_URL = "https://generativelanguage.googleapis.com/v1beta/openai/";
+export interface EvalProxyClientConfig {
+  baseURL: string;
+  bearerToken: string;
+  resourcePath: string;
+}
+
+const GEMINI_OPENAI_BASE_URL =
+  "https://generativelanguage.googleapis.com/v1beta/openai/";
 const GEMINI_GOOG_API_CLIENT = "windmill-ai-evals/1.0";
+const EVAL_PROXY_RESOURCE_PREFIX = "f/evals/ai";
+
+export function buildProxyHeaders(
+  bearerToken: string,
+  resourcePath: string,
+): Record<string, string> {
+  return {
+    Authorization: `Bearer ${bearerToken}`,
+    "X-Resource-Path": resourcePath,
+  };
+}
+
+export function buildProxyResourcePath(provider: FrontendEvalProvider): string {
+  return `${EVAL_PROXY_RESOURCE_PREFIX}/${provider}`;
+}
 
 export function buildOpenAICompatibleClientOptions(
   provider: Exclude<FrontendEvalProvider, "anthropic">,
-  apiKey: string
+  apiKey: string,
 ): ConstructorParameters<typeof OpenAI>[0] {
   if (provider === "googleai") {
     return {
@@ -34,26 +57,71 @@ export function buildOpenAICompatibleClientOptions(
   return { apiKey };
 }
 
-export function createEvalClients(
-  provider: FrontendEvalProvider,
-  apiKey: string
-): EvalClients {
-  if (provider === "anthropic") {
+function buildProxyOpenAIClientOptions(
+  proxy: EvalProxyClientConfig,
+): ConstructorParameters<typeof OpenAI>[0] {
+  return {
+    apiKey: "unused",
+    baseURL: proxy.baseURL,
+    defaultHeaders: buildProxyHeaders(proxy.bearerToken, proxy.resourcePath),
+  };
+}
+
+export function createEvalClients(input: {
+  provider: FrontendEvalProvider;
+  apiKey: string;
+  transport?: FrontendEvalTransport;
+  proxy?: EvalProxyClientConfig;
+}): EvalClients {
+  const transport = input.transport ?? "direct";
+
+  if (input.provider === "anthropic") {
+    if (transport === "proxy") {
+      if (!input.proxy) {
+        throw new Error(
+          "Missing proxy client configuration for proxy transport",
+        );
+      }
+      return {
+        openai: new OpenAI({ apiKey: "unused" }),
+        anthropic: new Anthropic({
+          apiKey: "unused",
+          baseURL: input.proxy.baseURL,
+          defaultHeaders: buildProxyHeaders(
+            input.proxy.bearerToken,
+            input.proxy.resourcePath,
+          ),
+        }),
+      };
+    }
+
     return {
       openai: new OpenAI({ apiKey: "unused" }),
-      anthropic: new Anthropic({ apiKey }),
+      anthropic: new Anthropic({ apiKey: input.apiKey }),
+    };
+  }
+
+  if (transport === "proxy") {
+    if (!input.proxy) {
+      throw new Error("Missing proxy client configuration for proxy transport");
+    }
+    return {
+      openai: new OpenAI(buildProxyOpenAIClientOptions(input.proxy)),
+      anthropic: new Anthropic({ apiKey: "unused" }),
     };
   }
 
   return {
-    openai: new OpenAI(buildOpenAICompatibleClientOptions(provider, apiKey)),
+    openai: new OpenAI(
+      buildOpenAICompatibleClientOptions(input.provider, input.apiKey),
+    ),
     anthropic: new Anthropic({ apiKey: "unused" }),
   };
 }
 
 export function resolveEvalModelProvider(
   model: string,
-  provider?: FrontendEvalProvider
+  provider?: FrontendEvalProvider,
 ): ResolvedEvalModelProvider {
   if (provider) {
     return { provider, model };

ai_evals/adapters/frontend/core/shared/types.ts

@@ -1,32 +1,38 @@
-import type { ChatCompletionMessageParam } from 'openai/resources/chat/completions.mjs'
-import type { AIProvider } from '$lib/gen/types.gen'
+import type { ChatCompletionMessageParam } from "openai/resources/chat/completions.mjs";
+import type { AIProvider } from "$lib/gen/types.gen";
+import type { FrontendEvalTransport } from "../../../../core/frontendTransport";
+import type { WindmillBackendSettings } from "../../../../core/windmillBackendSettings";
 
 export interface TokenUsage {
-	prompt: number
-	completion: number
-	total: number
+  prompt: number;
+  completion: number;
+  total: number;
 }
 
 export interface ToolCallDetail {
-	name: string
-	arguments: Record<string, unknown>
+  name: string;
+  arguments: Record<string, unknown>;
 }
 
 export interface EvalRunnerOptions {
-	maxIterations?: number
-	model?: string
-	workspace?: string
-	provider?: AIProvider
+  maxIterations?: number;
+  model?: string;
+  workspace?: string;
+  provider?: AIProvider;
+  transport?: FrontendEvalTransport;
+  backend?: WindmillBackendSettings;
+  proxyCaseId?: string;
+  proxyAttempt?: number;
 }
 
 export interface RawEvalResult<TOutput> {
-	success: boolean
-	output: TOutput
-	error?: string
-	tokenUsage: TokenUsage
-	toolCallsCount: number
-	toolsCalled: string[]
-	toolCallDetails: ToolCallDetail[]
-	iterations: number
-	messages: ChatCompletionMessageParam[]
+  success: boolean;
+  output: TOutput;
+  error?: string;
+  tokenUsage: TokenUsage;
+  toolCallsCount: number;
+  toolsCalled: string[];
+  toolCallDetails: ToolCallDetail[];
+  iterations: number;
+  messages: ChatCompletionMessageParam[];
 }

ai_evals/adapters/frontend/mockBackend.ts

@@ -227,6 +227,60 @@ export function runBenchmarkFlowByPath(input: {
 	})
 }
 
+export function previewBenchmarkSchedule(input: {
+	requestBody?: Record<string, unknown>
+}): Record<string, unknown> {
+	const schedule = input.requestBody?.schedule
+	if (typeof schedule !== 'string' || schedule.trim().split(/\s+/).length !== 6) {
+		throw new Error(`schedule must use a six-field cron expression, got ${JSON.stringify(schedule)}`)
+	}
+
+	return {
+		next_runs: ['1970-01-02T00:00:00.000Z']
+	}
+}
+
+export function createBenchmarkSchedule(input: {
+	workspace: string
+	requestBody: Record<string, unknown>
+}): Record<string, unknown> {
+	assertBenchmarkWorkspacePath('schedule', input.requestBody.path)
+	assertBenchmarkWorkspacePath('target', input.requestBody.script_path)
+	return {
+		path: input.requestBody.path,
+		target_path: input.requestBody.script_path,
+		is_flow: input.requestBody.is_flow,
+		mocked: true
+	}
+}
+
+export function createBenchmarkHttpTrigger(input: {
+	workspace: string
+	requestBody: Record<string, unknown>
+}): Record<string, unknown> {
+	assertBenchmarkWorkspacePath('trigger', input.requestBody.path)
+	assertBenchmarkWorkspacePath('target', input.requestBody.script_path)
+	if (
+		typeof input.requestBody.route_path === 'string' &&
+		input.requestBody.route_path.startsWith('/')
+	) {
+		throw new Error(`HTTP trigger route_path must not start with /, got "${input.requestBody.route_path}"`)
+	}
+	return {
+		path: input.requestBody.path,
+		target_path: input.requestBody.script_path,
+		route_path: input.requestBody.route_path,
+		is_flow: input.requestBody.is_flow,
+		mocked: true
+	}
+}
+
+function assertBenchmarkWorkspacePath(label: string, value: unknown): void {
+	if (typeof value !== 'string' || (!value.startsWith('f/') && !value.startsWith('u/'))) {
+		throw new Error(`${label} path must start with f/ or u/, got ${JSON.stringify(value)}`)
+	}
+}
+
 function buildBenchmarkScriptHash(path: string): string {
 	return `benchmark:${path}`
 }

ai_evals/adapters/frontend/progress.ts

@@ -58,6 +58,17 @@ export type FrontendBenchmarkProgressEvent =
 			attempt: number
 			runs: number
 	  }
+	| {
+			type: 'tool-call'
+			surface: FrontendBenchmarkProgressSurface
+			caseId: string
+			caseNumber: number
+			totalCases: number
+			attempt: number
+			runs: number
+			toolName: string
+			argumentsText: string
+	  }
 
 export const FRONTEND_BENCHMARK_PROGRESS_PREFIX = 'WMILL_FRONTEND_AI_EVAL_PROGRESS '
 
@@ -109,6 +120,8 @@ export function formatFrontendBenchmarkProgressEvent(
 		case 'assistant-chunk':
 		case 'assistant-message-end':
 			return ''
+		case 'tool-call':
+			return `${formatCasePrefix(event.caseNumber, event.totalCases)} ${event.caseId} attempt ${event.attempt}/${event.runs} tool ${event.toolName} ${truncateSingleLine(event.argumentsText, 200)}`
 	}
 }
 

ai_evals/adapters/frontend/runtime.ts

@@ -1,218 +1,228 @@
-import { spawn } from 'node:child_process'
-import { mkdtemp, readFile, rm } from 'node:fs/promises'
-import { tmpdir } from 'node:os'
-import path from 'node:path'
-import { fileURLToPath } from 'node:url'
+import { spawn } from "node:child_process";
+import { mkdtemp, readFile, rm } from "node:fs/promises";
+import { tmpdir } from "node:os";
+import path from "node:path";
+import { fileURLToPath } from "node:url";
 import {
-	formatFrontendBenchmarkProgressEvent,
-	parseFrontendBenchmarkProgressLine
-} from './progress'
-import type { BenchmarkRunResult } from '../../core/types'
+  formatFrontendBenchmarkProgressEvent,
+  parseFrontendBenchmarkProgressLine,
+} from "./progress";
+import type { BenchmarkRunResult } from "../../core/types";
 
-const REPO_ROOT = fileURLToPath(new URL('../../../', import.meta.url))
-const FRONTEND_DIR = path.join(REPO_ROOT, 'frontend')
-const FRONTEND_BENCHMARK_TEST = '../ai_evals/adapters/frontend/vitestAdapter.test.ts'
-const FRONTEND_BENCHMARK_CONFIG = '../ai_evals/adapters/frontend/vitest.config.ts'
+const REPO_ROOT = fileURLToPath(new URL("../../../", import.meta.url));
+const FRONTEND_DIR = path.join(REPO_ROOT, "frontend");
+const FRONTEND_BENCHMARK_TEST =
+  "../ai_evals/adapters/frontend/vitestAdapter.test.ts";
+const FRONTEND_BENCHMARK_CONFIG =
+  "../ai_evals/adapters/frontend/vitest.config.ts";
 
-export type FrontendMode = 'flow' | 'app' | 'script'
+export type FrontendMode = "flow" | "app" | "script";
 
 export async function runFrontendBenchmarkAdapter(input: {
-	mode: FrontendMode
-	caseIds: string[]
-	runs: number
-	model?: string
-	verbose?: boolean
-	backendValidation?: string
+  mode: FrontendMode;
+  caseIds: string[];
+  runs: number;
+  model?: string;
+  transport?: string;
+  verbose?: boolean;
+  backendValidation?: string;
 }): Promise<BenchmarkRunResult> {
-	const tempDir = await mkdtemp(path.join(tmpdir(), 'wmill-frontend-benchmark-'))
-	const outputPath = path.join(tempDir, 'result.json')
+  const tempDir = await mkdtemp(
+    path.join(tmpdir(), "wmill-frontend-benchmark-"),
+  );
+  const outputPath = path.join(tempDir, "result.json");
+  const env: NodeJS.ProcessEnv = {
+    ...process.env,
+    BROWSERSLIST_IGNORE_OLD_DATA: "1",
+    WMILL_FRONTEND_AI_EVAL_OUTPUT_PATH: outputPath,
+    WMILL_FRONTEND_AI_EVAL_MODE: input.mode,
+    WMILL_FRONTEND_AI_EVAL_CASE_IDS: JSON.stringify(input.caseIds),
+    WMILL_FRONTEND_AI_EVAL_RUNS: String(input.runs),
+    WMILL_FRONTEND_AI_EVAL_MODEL: input.model ?? "",
+    WMILL_FRONTEND_AI_EVAL_PROGRESS: "1",
+    WMILL_FRONTEND_AI_EVAL_VERBOSE: input.verbose ? "1" : "0",
+    WMILL_FRONTEND_AI_EVAL_BACKEND_VALIDATION: input.backendValidation ?? "",
+  };
 
-	try {
-		await runVitestBenchmark(
-			path.join(FRONTEND_DIR, 'node_modules', '.bin', 'vitest'),
-			[
-				'run',
-				FRONTEND_BENCHMARK_TEST,
-				'--project',
-				'server',
-				'--config',
-				FRONTEND_BENCHMARK_CONFIG
-			],
-			{
-				cwd: FRONTEND_DIR,
-				env: {
-					...process.env,
-					BROWSERSLIST_IGNORE_OLD_DATA: '1',
-					WMILL_FRONTEND_AI_EVAL_OUTPUT_PATH: outputPath,
-					WMILL_FRONTEND_AI_EVAL_MODE: input.mode,
-					WMILL_FRONTEND_AI_EVAL_CASE_IDS: JSON.stringify(input.caseIds),
-					WMILL_FRONTEND_AI_EVAL_RUNS: String(input.runs),
-					WMILL_FRONTEND_AI_EVAL_MODEL: input.model ?? "",
-					WMILL_FRONTEND_AI_EVAL_PROGRESS: '1',
-					WMILL_FRONTEND_AI_EVAL_VERBOSE: input.verbose ? '1' : '0',
-					WMILL_FRONTEND_AI_EVAL_BACKEND_VALIDATION: input.backendValidation ?? ''
-				}
-			}
-		)
+  if (input.transport) {
+    env.WMILL_FRONTEND_AI_EVAL_TRANSPORT = input.transport;
+  }
 
-		const raw = await readFile(outputPath, 'utf8')
-		return JSON.parse(raw) as BenchmarkRunResult
-	} catch (error) {
-		throw new Error(`Frontend benchmark adapter failed:\n${toErrorMessage(error)}`)
-	} finally {
-		await rm(tempDir, { recursive: true, force: true })
-	}
+  try {
+    await runVitestBenchmark(
+      path.join(FRONTEND_DIR, "node_modules", ".bin", "vitest"),
+      [
+        "run",
+        FRONTEND_BENCHMARK_TEST,
+        "--project",
+        "server",
+        "--config",
+        FRONTEND_BENCHMARK_CONFIG,
+      ],
+      {
+        cwd: FRONTEND_DIR,
+        env,
+      },
+    );
+
+    const raw = await readFile(outputPath, "utf8");
+    return JSON.parse(raw) as BenchmarkRunResult;
+  } catch (error) {
+    throw new Error(
+      `Frontend benchmark adapter failed:\n${toErrorMessage(error)}`,
+    );
+  } finally {
+    await rm(tempDir, { recursive: true, force: true });
+  }
 }
 
 async function runVitestBenchmark(
-	command: string,
-	args: string[],
-	options: {
-		cwd: string
-		env: NodeJS.ProcessEnv
-	}
+  command: string,
+  args: string[],
+  options: {
+    cwd: string;
+    env: NodeJS.ProcessEnv;
+  },
 ): Promise<void> {
-	const child = spawn(command, args, {
-		cwd: options.cwd,
-		env: options.env,
-		stdio: ['ignore', 'pipe', 'pipe']
-	})
+  const child = spawn(command, args, {
+    cwd: options.cwd,
+    env: options.env,
+    stdio: ["ignore", "pipe", "pipe"],
+  });
 
-	let stdout = ''
-	let stderr = ''
-	let stderrLineBuffer = ''
-	let assistantStreamOpen = false
+  let stdout = "";
+  let stderr = "";
+  let stderrLineBuffer = "";
+  let assistantStreamOpen = false;
 
-	child.stdout?.setEncoding('utf8')
-	child.stdout?.on('data', (chunk: string) => {
-		stdout += chunk
-	})
+  child.stdout?.setEncoding("utf8");
+  child.stdout?.on("data", (chunk: string) => {
+    stdout += chunk;
+  });
 
-	child.stderr?.setEncoding('utf8')
-	child.stderr?.on('data', (chunk: string) => {
-		stderrLineBuffer += chunk
-		const { remainder, passthrough, nextAssistantStreamOpen } = drainProgressLines(
-			stderrLineBuffer,
-			assistantStreamOpen
-		)
-		stderrLineBuffer = remainder
-		stderr += passthrough
-		assistantStreamOpen = nextAssistantStreamOpen
-	})
+  child.stderr?.setEncoding("utf8");
+  child.stderr?.on("data", (chunk: string) => {
+    stderrLineBuffer += chunk;
+    const { remainder, passthrough, nextAssistantStreamOpen } =
+      drainProgressLines(stderrLineBuffer, assistantStreamOpen);
+    stderrLineBuffer = remainder;
+    stderr += passthrough;
+    assistantStreamOpen = nextAssistantStreamOpen;
+  });
 
-	await new Promise<void>((resolve, reject) => {
-		child.once('error', reject)
-		child.once('close', (code) => {
-			if (stderrLineBuffer.length > 0) {
-				const {
-					remainder,
-					passthrough,
-					nextAssistantStreamOpen
-				} = drainProgressLines(`${stderrLineBuffer}\n`, assistantStreamOpen)
-				stderrLineBuffer = remainder
-				stderr += passthrough
-				assistantStreamOpen = nextAssistantStreamOpen
-			}
+  await new Promise<void>((resolve, reject) => {
+    child.on("error", reject);
+    child.on("close", (code) => {
+      if (stderrLineBuffer.length > 0) {
+        const { remainder, passthrough, nextAssistantStreamOpen } =
+          drainProgressLines(`${stderrLineBuffer}\n`, assistantStreamOpen);
+        stderrLineBuffer = remainder;
+        stderr += passthrough;
+        assistantStreamOpen = nextAssistantStreamOpen;
+      }
 
-			if (code === 0) {
-				if (assistantStreamOpen) {
-					process.stderr.write('\n')
-				}
-				resolve()
-				return
-			}
+      if (code === 0) {
+        if (assistantStreamOpen) {
+          process.stderr.write("\n");
+        }
+        resolve();
+        return;
+      }
 
-			const details = [`vitest exited with code ${code}`, stdout, stderr].filter(Boolean).join('\n')
-			reject(new Error(details))
-		})
-	})
+      const details = [`vitest exited with code ${code}`, stdout, stderr]
+        .filter(Boolean)
+        .join("\n");
+      reject(new Error(details));
+    });
+  });
 }
 
-function drainProgressLines(buffer: string): {
-	remainder: string
-	passthrough: string
-	nextAssistantStreamOpen: boolean
-}
 function drainProgressLines(
-	buffer: string,
-	initialAssistantStreamOpen: boolean
+  buffer: string,
+  initialAssistantStreamOpen: boolean,
 ): {
-	remainder: string
-	passthrough: string
-	nextAssistantStreamOpen: boolean
+  remainder: string;
+  passthrough: string;
+  nextAssistantStreamOpen: boolean;
 } {
-	let remainder = buffer
-	let passthrough = ''
-	let assistantStreamOpen = initialAssistantStreamOpen
+  let remainder = buffer;
+  let passthrough = "";
+  let assistantStreamOpen = initialAssistantStreamOpen;
 
-	while (true) {
-		const newlineIndex = remainder.indexOf('\n')
-		if (newlineIndex === -1) {
-			return { remainder, passthrough, nextAssistantStreamOpen: assistantStreamOpen }
-		}
+  while (true) {
+    const newlineIndex = remainder.indexOf("\n");
+    if (newlineIndex === -1) {
+      return {
+        remainder,
+        passthrough,
+        nextAssistantStreamOpen: assistantStreamOpen,
+      };
+    }
 
-		const line = remainder.slice(0, newlineIndex).replace(/\r$/, '')
-		remainder = remainder.slice(newlineIndex + 1)
+    const line = remainder.slice(0, newlineIndex).replace(/\r$/, "");
+    remainder = remainder.slice(newlineIndex + 1);
 
-		const progressEvent = parseFrontendBenchmarkProgressLine(line)
-		if (progressEvent) {
-			if (progressEvent.type === 'assistant-message-start') {
-				if (assistantStreamOpen) {
-					process.stderr.write('\n')
-				}
-				process.stderr.write(
-					`${formatCasePrefix(progressEvent.caseNumber, progressEvent.totalCases)} ${progressEvent.caseId} attempt ${progressEvent.attempt}/${progressEvent.runs} assistant:\n`
-				)
-				assistantStreamOpen = true
-				continue
-			}
+    const progressEvent = parseFrontendBenchmarkProgressLine(line);
+    if (progressEvent) {
+      if (progressEvent.type === "assistant-message-start") {
+        if (assistantStreamOpen) {
+          process.stderr.write("\n");
+        }
+        process.stderr.write(
+          `${formatCasePrefix(progressEvent.caseNumber, progressEvent.totalCases)} ${progressEvent.caseId} attempt ${progressEvent.attempt}/${progressEvent.runs} assistant:\n`,
+        );
+        assistantStreamOpen = true;
+        continue;
+      }
 
-			if (progressEvent.type === 'assistant-chunk') {
-				process.stderr.write(progressEvent.chunk)
-				continue
-			}
+      if (progressEvent.type === "assistant-chunk") {
+        process.stderr.write(progressEvent.chunk);
+        continue;
+      }
 
-			if (progressEvent.type === 'assistant-message-end') {
-				if (assistantStreamOpen) {
-					process.stderr.write('\n')
-				}
-				assistantStreamOpen = false
-				continue
-			}
+      if (progressEvent.type === "assistant-message-end") {
+        if (assistantStreamOpen) {
+          process.stderr.write("\n");
+        }
+        assistantStreamOpen = false;
+        continue;
+      }
 
-			if (assistantStreamOpen) {
-				process.stderr.write('\n')
-				assistantStreamOpen = false
-			}
-			process.stderr.write(`${formatFrontendBenchmarkProgressEvent(progressEvent)}\n`)
-			continue
-		}
+      if (assistantStreamOpen) {
+        process.stderr.write("\n");
+        assistantStreamOpen = false;
+      }
+      process.stderr.write(
+        `${formatFrontendBenchmarkProgressEvent(progressEvent)}\n`,
+      );
+      continue;
+    }
 
-		if (shouldSuppressFrontendStderrLine(line)) {
-			continue
-		}
+    if (shouldSuppressFrontendStderrLine(line)) {
+      continue;
+    }
 
-		passthrough += `${line}\n`
-		process.stderr.write(`${line}\n`)
-	}
+    passthrough += `${line}\n`;
+    process.stderr.write(`${line}\n`);
+  }
 }
 
 function formatCasePrefix(caseNumber: number, totalCases: number): string {
-	return `[${caseNumber}/${totalCases}]`
+  return `[${caseNumber}/${totalCases}]`;
 }
 
 function shouldSuppressFrontendStderrLine(line: string): boolean {
-	return (
-		line.startsWith('[baseline-browser-mapping] ') ||
-		line.startsWith('Browserslist: browsers data (caniuse-lite) is ') ||
-		line.includes('update-browserslist-db@latest') ||
-		line.includes('update-db#readme')
-	)
+  return (
+    line.startsWith("[baseline-browser-mapping] ") ||
+    line.startsWith("Browserslist: browsers data (caniuse-lite) is ") ||
+    line.includes("update-browserslist-db@latest") ||
+    line.includes("update-db#readme")
+  );
 }
 
 function toErrorMessage(error: unknown): string {
-	if (error instanceof Error) {
-		return error.message
-	}
-	return String(error)
+  if (error instanceof Error) {
+    return error.message;
+  }
+  return String(error);
 }

ai_evals/adapters/frontend/vitestAdapter.test.ts

@@ -40,6 +40,9 @@ vi.mock('$lib/gen', async () => {
 		hasBenchmarkWorkspace,
 		listBenchmarkFlows,
 		listBenchmarkScripts,
+		createBenchmarkHttpTrigger,
+		createBenchmarkSchedule,
+		previewBenchmarkSchedule,
 		runBenchmarkFlowByPath,
 		runBenchmarkScriptPreview
 	} = await import('./mockBackend')
@@ -137,6 +140,20 @@ vi.mock('$lib/gen', async () => {
 				}
 				return actual.JobService.getJob(data)
 			}
+		}),
+		ScheduleService: wrapService(actual.ScheduleService, {
+			previewSchedule: async (data: { requestBody?: Record<string, unknown> }) =>
+				previewBenchmarkSchedule(data),
+			createSchedule: async (data: { workspace: string; requestBody: Record<string, unknown> }) =>
+				hasBenchmarkWorkspace(data.workspace)
+					? createBenchmarkSchedule(data)
+					: actual.ScheduleService.createSchedule(data)
+		}),
+		HttpTriggerService: wrapService(actual.HttpTriggerService, {
+			createHttpTrigger: async (data: { workspace: string; requestBody: Record<string, unknown> }) =>
+				hasBenchmarkWorkspace(data.workspace)
+					? createBenchmarkHttpTrigger(data)
+					: actual.HttpTriggerService.createHttpTrigger(data)
 		})
 	}
 })

ai_evals/adapters/frontend/windmillBackend.ts

@@ -0,0 +1,184 @@
+import { randomUUID } from "node:crypto";
+import type { WindmillBackendSettings } from "../../core/windmillBackendSettings";
+
+const tokenCache = new Map<string, Promise<string>>();
+const sharedWorkspaceQueue = new Map<string, Promise<void>>();
+
+export class WindmillBackendClient {
+  constructor(private readonly settings: WindmillBackendSettings) {}
+
+  async withWorkspace<T>(
+    caseId: string,
+    attempt: number,
+    body: (workspaceId: string) => Promise<T>,
+  ): Promise<T> {
+    const workspaceId =
+      this.settings.workspaceOverride ??
+      buildWorkspaceId(this.settings.workspacePrefix, caseId, attempt);
+
+    const run = async () => {
+      await this.ensureWorkspace(workspaceId);
+
+      try {
+        return await body(workspaceId);
+      } finally {
+        if (!this.settings.keepWorkspaces && !this.settings.workspaceOverride) {
+          await this.deleteWorkspace(workspaceId).catch(() => undefined);
+        }
+      }
+    };
+
+    if (this.settings.workspaceOverride) {
+      return await withSharedWorkspaceLock(workspaceId, run);
+    }
+
+    return await run();
+  }
+
+  async request(path: string, init?: RequestInit): Promise<Response> {
+    const token = await this.getToken();
+    return await fetch(`${this.settings.baseUrl}/api${path}`, {
+      ...init,
+      headers: {
+        Authorization: `Bearer ${token}`,
+        ...(init?.headers ?? {}),
+      },
+    });
+  }
+
+  async getToken(): Promise<string> {
+    const cacheKey = `${this.settings.baseUrl}|${this.settings.email}`;
+    let tokenPromise = tokenCache.get(cacheKey);
+    if (!tokenPromise) {
+      tokenPromise = this.login().catch((error) => {
+        if (tokenCache.get(cacheKey) === tokenPromise) {
+          tokenCache.delete(cacheKey);
+        }
+        throw error;
+      });
+      tokenCache.set(cacheKey, tokenPromise);
+    }
+    return await tokenPromise;
+  }
+
+  async upsertResource(input: {
+    workspaceId: string;
+    path: string;
+    resourceType: string;
+    value: Record<string, unknown>;
+  }): Promise<void> {
+    const response = await this.request(
+      `/w/${encodeURIComponent(input.workspaceId)}/resources/create?update_if_exists=true`,
+      {
+        method: "POST",
+        headers: { "Content-Type": "application/json" },
+        body: JSON.stringify({
+          path: input.path,
+          resource_type: input.resourceType,
+          value: input.value,
+        }),
+      },
+    );
+    await expectOk(response, `upsert resource ${input.path}`);
+  }
+
+  private async ensureWorkspace(workspaceId: string): Promise<void> {
+    const existsResponse = await this.request("/workspaces/exists", {
+      method: "POST",
+      headers: { "Content-Type": "application/json" },
+      body: JSON.stringify({ id: workspaceId }),
+    });
+    await expectOk(existsResponse, `check workspace ${workspaceId}`);
+
+    if ((await existsResponse.text()).trim() === "true") {
+      return;
+    }
+
+    const createResponse = await this.request("/workspaces/create", {
+      method: "POST",
+      headers: { "Content-Type": "application/json" },
+      body: JSON.stringify({ id: workspaceId, name: workspaceId }),
+    });
+    try {
+      await expectOk(createResponse, `create workspace ${workspaceId}`);
+    } catch (error) {
+      const message = error instanceof Error ? error.message : String(error);
+      if (message.includes("maximum number of workspaces")) {
+        throw new Error(
+          `${message}. Reuse an existing workspace with WMILL_AI_EVAL_BACKEND_WORKSPACE=<workspace-id>.`,
+        );
+      }
+      throw error;
+    }
+  }
+
+  private async deleteWorkspace(workspaceId: string): Promise<void> {
+    const response = await this.request(
+      `/workspaces/delete/${encodeURIComponent(workspaceId)}`,
+      {
+        method: "DELETE",
+      },
+    );
+    await expectOk(response, `delete workspace ${workspaceId}`);
+  }
+
+  private async login(): Promise<string> {
+    const response = await fetch(`${this.settings.baseUrl}/api/auth/login`, {
+      method: "POST",
+      headers: { "Content-Type": "application/json" },
+      body: JSON.stringify({
+        email: this.settings.email,
+        password: this.settings.password,
+      }),
+    });
+    await expectOk(response, "login to Windmill backend");
+    return (await response.text()).trim();
+  }
+}
+
+async function withSharedWorkspaceLock<T>(
+  workspaceId: string,
+  body: () => Promise<T>,
+): Promise<T> {
+  const previous = sharedWorkspaceQueue.get(workspaceId) ?? Promise.resolve();
+  let releaseCurrent: (() => void) | undefined;
+  const current = new Promise<void>((resolve) => {
+    releaseCurrent = resolve;
+  });
+  const tail = previous.catch(() => undefined).then(() => current);
+  sharedWorkspaceQueue.set(workspaceId, tail);
+
+  await previous.catch(() => undefined);
+
+  try {
+    return await body();
+  } finally {
+    releaseCurrent?.();
+    if (sharedWorkspaceQueue.get(workspaceId) === tail) {
+      sharedWorkspaceQueue.delete(workspaceId);
+    }
+  }
+}
+
+function buildWorkspaceId(
+  prefix: string,
+  caseId: string,
+  attempt: number,
+): string {
+  const caseSlug = caseId
+    .toLowerCase()
+    .replace(/[^a-z0-9-]+/g, "-")
+    .replace(/^-+|-+$/g, "")
+    .slice(0, 30);
+  const suffix = randomUUID().slice(0, 8);
+  return `${prefix}-${caseSlug || "case"}-a${attempt}-${suffix}`;
+}
+
+async function expectOk(response: Response, context: string): Promise<void> {
+  if (response.ok) {
+    return;
+  }
+  throw new Error(
+    `${context} failed: ${response.status} ${response.statusText} - ${await response.text()}`,
+  );
+}

ai_evals/cases/app.yaml

@@ -47,15 +47,30 @@
     - updates the visible file list from the search query
     - keeps the rest of the file manager usable
 
-- id: app-test6-file-manager-inline-rename
+- id: app-test6-file-manager-rename-save-cancel
   prompt: |-
-    Let users rename files and folders directly from the file list without leaving the page.
+    Improve the existing inline rename flow for files and folders.
+    When renaming, show explicit Save and Cancel buttons next to the name input.
+    Pressing Enter should save, pressing Escape should cancel, and Cancel should restore the original name without calling rename.
+    Keep the existing backend rename behavior for successful saves.
   initial: ai_evals/fixtures/frontend/app/initial/file_manager
+  validate:
+    requiredFrontendPaths:
+      - /index.tsx
+      - /components/FileItem.tsx
+    requiredFrontendFileContent:
+      - path: /components/FileItem.tsx
+        includes:
+          - Save
+          - Cancel
+          - Escape
+    forbiddenAppContent:
+      - onBlur={handleRename}
   judgeChecklist:
-    - adds a visible rename action or inline edit mode in the file list
-    - lets users edit an item's name directly from the list
-    - saves the renamed item through the app's existing rename behavior
-    - refreshes the displayed name after a successful rename
+    - keeps the existing visible rename action in the file list
+    - shows explicit Save and Cancel controls while editing a name
+    - pressing Enter saves the new name through the existing rename behavior
+    - pressing Escape or Cancel exits rename mode and restores the original name without saving
 
 - id: app-test7-file-manager-select-all
   prompt: |-
@@ -68,26 +83,244 @@
     - shows a delete-selected action only when there is a selection
     - deleting selected items updates the visible list
 
-- id: app-test8-inventory-tracker-create
+- id: app-test8-inventory-tracker-search-delete
   prompt: |-
-    Create an inventory tracker app for a small store.
-    Users should be able to add items with a name, sku, quantity, and price, search items by name or sku, and delete items.
-    The inventory should persist between sessions.
+    Update this inventory tracker app so users can search items by name or sku and delete existing items.
+    Keep the existing add-item flow and datatable-backed persistence working.
+  initial: ai_evals/fixtures/frontend/app/initial/inventory_tracker
+  validate:
+    requiredFrontendPaths:
+      - /index.tsx
+    requiredBackendRunnableKeys:
+      - listInventory
+      - addInventory
+      - deleteInventory
+    requiredBackendRunnableTypes:
+      - key: listInventory
+        type: inline
+      - key: addInventory
+        type: inline
+      - key: deleteInventory
+        type: inline
+    requiredDatatables:
+      - datatableName: main
+        schema: public
+        table: inventory_items
   judgeChecklist:
-    - includes a form to add inventory items with name, sku, quantity, and price
-    - shows a list or table of saved inventory items
-    - supports searching or filtering by name or sku
-    - lets users delete existing inventory items
-    - persists the inventory data appropriately for a raw Windmill app
+    - keeps the existing add-item form working
+    - adds a search input that filters inventory by name or sku
+    - adds a delete action for existing inventory items
+    - deleting an inventory item updates the visible list
+    - keeps inventory persistence working through the existing datatable-backed app setup
 
-- id: app-test9-recipe-book-create
+- id: app-test9-recipe-book-search-delete
   prompt: |-
-    Create a recipe book app where users can add recipes with a name, ingredients list, and instructions.
-    Include a search bar to filter recipes by name and the ability to delete recipes.
-    Recipes should persist between sessions.
+    Update this recipe book app so users can search recipes by name and delete existing recipes.
+    Keep the existing add-recipe flow and datatable-backed persistence working.
+  initial: ai_evals/fixtures/frontend/app/initial/recipe_book
+  validate:
+    requiredFrontendPaths:
+      - /index.tsx
+    requiredBackendRunnableKeys:
+      - listRecipes
+      - addRecipe
+      - deleteRecipe
+    requiredBackendRunnableTypes:
+      - key: listRecipes
+        type: inline
+      - key: addRecipe
+        type: inline
+      - key: deleteRecipe
+        type: inline
+    requiredDatatables:
+      - datatableName: main
+        schema: public
+        table: recipes
   judgeChecklist:
-    - includes a form to add recipes with name, ingredients, and instructions
-    - shows saved recipes in the app
-    - supports searching recipes by name
-    - lets users delete recipes
-    - persists recipes appropriately for a raw Windmill app
+    - keeps the existing add-recipe form working
+    - adds a search input that filters recipes by name
+    - adds a delete action for existing recipes
+    - deleting a recipe updates the visible list
+    - keeps recipe persistence working through the existing datatable-backed app setup
+
+- id: app-datatable-persistent-notes
+  prompt: |-
+    Build a notes app that persists notes in the existing datatable table.
+    Inspect the existing datatable and table schema before writing code.
+    Use the existing main/public.notes table, and do not create any new tables.
+    Create backend runnables named exactly listNotes, addNote, and deleteNote.
+    The UI should list notes, add a note with title/body, and delete notes.
+    Do not use localStorage, sessionStorage, IndexedDB, or in-memory-only persistence.
+  initial: ai_evals/fixtures/frontend/app/initial/notes_datatable
+  runtime:
+    maxTurns: 10
+  validate:
+    requiredFrontendPaths:
+      - /index.tsx
+    requiredFrontendFileContent:
+      - path: /index.tsx
+        includes:
+          - backend.listNotes
+          - backend.addNote
+          - backend.deleteNote
+    requiredBackendRunnableKeys:
+      - listNotes
+      - addNote
+      - deleteNote
+    requiredBackendRunnableTypes:
+      - key: listNotes
+        type: inline
+      - key: addNote
+        type: inline
+      - key: deleteNote
+        type: inline
+    requiredBackendRunnableContent:
+      - key: listNotes
+        includes:
+          - wmill.datatable
+          - select
+          - notes
+      - key: addNote
+        includes:
+          - wmill.datatable
+          - insert
+          - notes
+      - key: deleteNote
+        includes:
+          - wmill.datatable
+          - delete
+          - notes
+    datatableTableCountExactly: 1
+    requiredDatatables:
+      - datatableName: main
+        schema: public
+        table: notes
+    requiredToolsUsed:
+      - list_datatables
+      - get_datatable_table_schema
+    forbiddenAppContent:
+      - localStorage
+      - sessionStorage
+      - indexedDB
+  judgeChecklist:
+    - creates a notes UI that lists notes from the backend
+    - adds notes through backend datatable persistence
+    - deletes notes through backend datatable persistence
+    - reuses the existing main/public.notes table without creating new tables
+    - does not use browser storage or in-memory-only persistence as the source of truth
+
+- id: app-test10-session-id-no-crypto
+  prompt: |-
+    Update `generateSessionId` so it no longer uses `crypto.randomUUID()`.
+    Make it return a handmade string id built from the current time and random characters.
+    Keep the existing sessionStorage-based chat session behavior unchanged.
+  initial: ai_evals/fixtures/frontend/app/initial/session_id_chat
+  runtime:
+    maxTurns: 4
+  validate:
+    requiredFrontendPaths:
+      - /index.tsx
+    requiredBackendRunnableKeys:
+      - a
+    requiredBackendRunnableTypes:
+      - key: a
+        type: inline
+  judgeChecklist:
+    - generateSessionId no longer calls crypto.randomUUID
+    - generateSessionId returns a handmade string id without using crypto
+    - getSessionId still stores and reuses chat_session_id in sessionStorage
+    - the existing new chat and send message session behavior remains wired up
+
+- id: app-token-baseline-large-app-small-edit
+  prompt: |-
+    Change the main heading from "Analytics Console" to "Operations Console".
+    Keep the existing filtering, summary loading, and backend calls unchanged.
+  initial: ai_evals/fixtures/frontend/app/initial/token_heavy_context
+  runtime:
+    maxTurns: 8
+  validate:
+    requiredFrontendPaths:
+      - /index.tsx
+    requiredBackendRunnableKeys:
+      - loadAnalytics
+      - refreshSummary
+  judgeChecklist:
+    - changes the visible main heading to Operations Console
+    - keeps the existing summary loading behavior wired to loadAnalytics
+    - keeps the existing filter input and metric list behavior intact
+
+- id: app-token-many-datatable-context
+  prompt: |-
+    Add a short note under the dashboard heading that says "Using existing analytics tables".
+    Do not create any new tables.
+  initial: ai_evals/fixtures/frontend/app/initial/token_heavy_datatables
+  runtime:
+    maxTurns: 8
+    appContext:
+      additional:
+        - type: datatable
+          datatableName: main
+          schema: analytics
+          table: event_log_01
+        - type: datatable
+          datatableName: main
+          schema: analytics
+          table: event_log_02
+        - type: datatable
+          datatableName: main
+          schema: analytics
+          table: event_log_03
+        - type: datatable
+          datatableName: main
+          schema: analytics
+          table: event_log_04
+        - type: datatable
+          datatableName: main
+          schema: analytics
+          table: event_log_05
+        - type: datatable
+          datatableName: main
+          schema: analytics
+          table: event_log_06
+        - type: datatable
+          datatableName: main
+          schema: analytics
+          table: event_log_07
+        - type: datatable
+          datatableName: main
+          schema: analytics
+          table: event_log_08
+        - type: datatable
+          datatableName: main
+          schema: operations
+          table: ops_record_13
+        - type: datatable
+          datatableName: main
+          schema: operations
+          table: ops_record_14
+  validate:
+    requiredFrontendPaths:
+      - /index.tsx
+    datatableCountAtLeast: 1
+    datatableTableCountAtLeast: 18
+  judgeChecklist:
+    - adds the note Using existing analytics tables under or near the heading
+    - does not create new datatable tables
+    - keeps the configured datatable references available in the app artifact
+
+- id: app-token-large-datatable-discovery
+  prompt: |-
+    Build a read-only dashboard page that reuses the existing analytics datatable tables.
+    Show a simple summary of which existing tables are available, and do not create any new tables.
+  initial: ai_evals/fixtures/frontend/app/initial/token_heavy_datatables
+  runtime:
+    maxTurns: 8
+  validate:
+    requiredFrontendPaths:
+      - /index.tsx
+    datatableCountAtLeast: 1
+    datatableTableCountAtLeast: 18
+  judgeChecklist:
+    - reuses the existing datatable configuration rather than creating new tables
+    - presents a read-only dashboard or summary of available analytics data
+    - keeps the configured datatable references available in the app artifact

ai_evals/cases/cli.yaml

@@ -3,6 +3,23 @@
     Create a Windmill Bun script at `f/evals/hello.ts`.
     It should take a `name` input and return a greeting object like `{ greeting: "Hello, Alice!" }`.
   expected: ai_evals/fixtures/cli/expected/bun-hello-script
+  cliExpect:
+    requiredSkills:
+      - write-script-bun
+    requiredSkillsBeforeFirstMutation:
+      - write-script-bun
+    forbiddenSkills:
+      - write-script-python3
+      - write-flow
+    orderedAssistantMentions:
+      - wmill generate-metadata
+      - wmill sync push
+    orderedProposedCommands:
+      - wmill generate-metadata
+      - wmill sync push
+    forbiddenExecutedCommands:
+      - ^wmill generate-metadata
+      - ^wmill sync push
   judgeChecklist:
     - creates the requested Bun script at f/evals/hello.ts
     - takes a name input
@@ -14,6 +31,22 @@
     It should take a `name` input and return a greeting object like `{ greeting: "Hello, Alice!" }`.
     Put the step code in `hello.ts`.
   expected: ai_evals/fixtures/cli/expected/bun-hello-flow
+  cliExpect:
+    requiredSkills:
+      - write-flow
+    requiredSkillsBeforeFirstMutation:
+      - write-flow
+    forbiddenSkills:
+      - write-script-python3
+    orderedAssistantMentions:
+      - wmill generate-metadata
+      - wmill sync push
+    orderedProposedCommands:
+      - wmill generate-metadata
+      - wmill sync push
+    forbiddenExecutedCommands:
+      - ^wmill generate-metadata
+      - ^wmill sync push
   judgeChecklist:
     - creates the requested flow folder with flow.yaml and hello.ts
     - wires the name input into the flow step
@@ -24,6 +57,23 @@
     Add a Windmill Python script at `f/evals/add_numbers.py`.
     It should take `a` and `b` as inputs and return `{ "total": a + b }`.
   expected: ai_evals/fixtures/cli/expected/python-add-numbers-script
+  cliExpect:
+    requiredSkills:
+      - write-script-python3
+    requiredSkillsBeforeFirstMutation:
+      - write-script-python3
+    forbiddenSkills:
+      - write-script-bun
+      - write-flow
+    orderedAssistantMentions:
+      - wmill generate-metadata
+      - wmill sync push
+    orderedProposedCommands:
+      - wmill generate-metadata
+      - wmill sync push
+    forbiddenExecutedCommands:
+      - ^wmill generate-metadata
+      - ^wmill sync push
   judgeChecklist:
     - creates the requested Python script at f/evals/add_numbers.py
     - takes `a` and `b` as inputs
@@ -59,8 +109,155 @@
     Create a flow at `f/evals/reuse_greeting__flow` that takes a `name` input and reuses that existing script instead of duplicating the logic inline.
   initial: ai_evals/fixtures/cli/initial/flow-reuse-existing-script
   expected: ai_evals/fixtures/cli/expected/flow-reuse-existing-script
+  cliExpect:
+    requiredSkills:
+      - write-flow
+    requiredSkillsBeforeFirstMutation:
+      - write-flow
+    orderedAssistantMentions:
+      - wmill generate-metadata
+      - wmill sync push
+    orderedProposedCommands:
+      - wmill generate-metadata
+      - wmill sync push
+    forbiddenExecutedCommands:
+      - ^wmill generate-metadata
+      - ^wmill sync push
   judgeChecklist:
     - creates the requested flow at f/evals/reuse_greeting__flow
     - reuses the existing script from f/lib by path
     - does not duplicate the greeting logic in a new inline script
     - wires the name input into the reused script
+
+- id: wac-typescript-order-workflow
+  prompt: |-
+    Create a Windmill Workflow-as-Code TypeScript script at `f/evals/order_workflow.ts`.
+    It should take an `orderId` string, load the order in a durable task, checkpoint a processing timestamp with `step`, and return `{ orderId, processedAt, status }`.
+  cliExpect:
+    requiredSkills:
+      - write-workflow-as-code
+    requiredSkillsBeforeFirstMutation:
+      - write-workflow-as-code
+    forbiddenSkills:
+      - write-flow
+      - write-script-bun
+      - write-script-python3
+  judgeChecklist:
+    - creates the requested TypeScript WAC script at f/evals/order_workflow.ts
+    - uses the Workflow-as-Code SDK from windmill-client
+    - wraps the entrypoint with workflow
+    - uses a durable task for loading the order
+    - uses step to checkpoint the processing timestamp
+    - does not create an OpenFlow flow.yaml or flow folder
+
+- id: wac-python-approval-workflow
+  prompt: |-
+    Create a Windmill Workflow-as-Code Python script at `f/evals/approval_workflow.py`.
+    It should take a `request_id` string, prepare an approval summary in a task, create resume URLs inside a durable step, wait for approval, and return the approval result.
+  cliExpect:
+    requiredSkills:
+      - write-workflow-as-code
+    requiredSkillsBeforeFirstMutation:
+      - write-workflow-as-code
+    forbiddenSkills:
+      - write-flow
+      - write-script-bun
+      - write-script-python3
+  judgeChecklist:
+    - creates the requested Python WAC script at f/evals/approval_workflow.py
+    - imports Workflow-as-Code helpers from wmill
+    - decorates an async entrypoint with @workflow
+    - uses @task for the approval summary work
+    - gets resume URLs inside step before waiting for approval
+    - uses wait_for_approval
+    - does not create an OpenFlow flow.yaml or flow folder
+
+- id: wac-not-openflow-disambiguation
+  prompt: |-
+    Create this as Workflow-as-Code, not an OpenFlow YAML flow: a TypeScript script at `f/evals/fanout_workflow.ts`.
+    It should take an array of customer IDs, process each customer with a WAC task, run the independent customer tasks in parallel, and return the collected results.
+  cliExpect:
+    requiredSkills:
+      - write-workflow-as-code
+    requiredSkillsBeforeFirstMutation:
+      - write-workflow-as-code
+    forbiddenSkills:
+      - write-flow
+      - write-script-bun
+      - write-script-python3
+  judgeChecklist:
+    - creates the requested TypeScript script at f/evals/fanout_workflow.ts
+    - treats the request as Workflow-as-Code rather than an OpenFlow flow
+    - uses workflow for the script entrypoint
+    - uses task for each customer processing unit
+    - runs independent customer tasks in parallel
+    - does not create a flow folder or flow.yaml
+
+- id: cli-job-debug-guidance
+  prompt: |-
+    A Windmill job failed.
+    Tell me exactly which `wmill` commands to run to inspect the job details, logs, and final result for job ID `123`.
+    Do not modify any files.
+  cliExpect:
+    requiredSkills:
+      - cli-commands
+    workspaceUnchanged: true
+    orderedProposedCommands:
+      - wmill job get 123
+      - wmill job logs 123
+      - wmill job result 123
+    forbiddenProposedCommands:
+      - wmill sync push
+    forbiddenExecutedCommands:
+      - ^wmill job get
+      - ^wmill job logs
+      - ^wmill job result
+  judgeChecklist:
+    - does not modify the workspace
+    - recommends commands to inspect the job details
+    - recommends commands to inspect the job logs
+    - recommends commands to inspect the final result
+
+- id: cli-sync-pull-guidance
+  prompt: |-
+    I want to review remote workspace changes before editing locally.
+    Tell me the first `wmill` command I should run.
+    Do not modify any files.
+  cliExpect:
+    requiredSkills:
+      - cli-commands
+    workspaceUnchanged: true
+    requiredProposedCommands:
+      - wmill sync pull
+    forbiddenProposedCommands:
+      - wmill sync push
+    forbiddenExecutedCommands:
+      - ^wmill sync pull
+      - ^wmill sync push
+  judgeChecklist:
+    - does not modify the workspace
+    - recommends using sync pull before making local edits
+    - does not recommend pushing first
+
+- id: cli-script-deploy-guidance
+  prompt: |-
+    I already modified a Windmill script locally and now want the next CLI commands to prepare it and deploy it.
+    Tell me the commands to run, in order.
+    Do not modify any files.
+  cliExpect:
+    requiredSkills:
+      - cli-commands
+    workspaceUnchanged: true
+    orderedAssistantMentions:
+      - wmill generate-metadata
+      - wmill sync push
+    orderedProposedCommands:
+      - wmill generate-metadata
+      - wmill sync push
+    forbiddenExecutedCommands:
+      - ^wmill generate-metadata
+      - ^wmill sync push
+  judgeChecklist:
+    - does not modify the workspace
+    - recommends generate-metadata before sync push
+    - presents the commands in order

ai_evals/cases/flow.yaml

@@ -47,6 +47,38 @@
     - "the main step is named `call_add_numbers`"
     - the parent flow delegates to an existing workspace subflow instead of inlining the addition logic
 
+- id: flow-test13-prefer-existing-workspace-flow
+  prompt: |-
+    Create a parent flow that adds two numbers by reusing an existing flow from the workspace if one fits.
+    A reusable script may also be available, but for this task prefer the existing flow rather than calling a script directly or rewriting the logic inline.
+    The parent flow should take `a` and `b` as inputs and use a single top-level step named `call_add_numbers_flow`.
+  initial: ai_evals/fixtures/frontend/flow/initial/test13_prefer_existing_workspace_flow_initial.json
+  expected: ai_evals/fixtures/frontend/flow/expected/test13_prefer_existing_workspace_flow.json
+  validate:
+    exactTopLevelStepIds:
+      - call_add_numbers_flow
+    topLevelStepTypes:
+      - id: call_add_numbers_flow
+        type: flow
+    moduleRules:
+      - id: call_add_numbers_flow
+        requiredInputTransforms:
+          - type: javascript
+            expr: flow_input.a
+          - type: javascript
+            expr: flow_input.b
+  runtime:
+    backendPreview:
+      args:
+        a: 10
+        b: 5
+  judgeChecklist:
+    - "the parent flow takes `a` and `b` as inputs"
+    - "the main step is named `call_add_numbers_flow`"
+    - the parent flow reuses the existing workspace flow as a subflow
+    - the parent flow does not call the standalone workspace script directly
+    - the parent flow does not inline the addition logic
+
 - id: flow-test3-branchone-routing
   prompt: |-
     Create a flow that routes incoming support requests based on the customer's tier.
@@ -136,7 +168,7 @@
     - search FAQs
     - open a support ticket when needed
 
-    After that, log the interaction and return the assistant's response along with any actions it took.
+    After that, log the interaction and return the assistant's response.
   judgeChecklist:
     - "the input schema includes `customer_id` and `query_text`"
     - the flow loads the customer's profile and order history
@@ -146,24 +178,42 @@
     - the assistant can search FAQs
     - the assistant can open a support ticket
     - the flow logs the interaction
-    - the final output returns the assistant response along with any actions taken or resulting support action details
+    - the final output returns the assistant response
 
 - id: flow-test7-simple-modification
   prompt: |-
     Update this flow so it validates processed data before saving it.
 
     After `process_data`, add a `validate_data` step that checks the data array is not empty.
-    If the array is empty, it should return an error object with the message `No data to save`.
+    If the array is empty, the flow should surface the message `No data to save` and prevent saving.
     If validation passes, let the save continue normally.
-    Update `save_results` so it handles the validation result correctly.
+    Update `save_results` so it uses the validation outcome instead of bypassing it.
   initial: ai_evals/fixtures/frontend/flow/initial/test5_initial.json
-  expected: ai_evals/fixtures/frontend/flow/expected/test5_modify_simple.json
+  runtime:
+    maxTurns: 8
+  validate:
+    topLevelStepIds:
+      - fetch_data
+      - process_data
+      - validate_data
+    topLevelStepOrder:
+      - fetch_data
+      - process_data
+      - validate_data
+    topLevelStepTypes:
+      - id: fetch_data
+        type: rawscript
+      - id: process_data
+        type: rawscript
+      - id: validate_data
+        type: rawscript
   judgeChecklist:
     - the updated flow keeps the original fetch and process steps intact
     - "a `validate_data` step is added after `process_data`"
     - "`validate_data` checks that the processed data array is not empty"
-    - "empty data returns an error object with the message `No data to save`"
-    - "`save_results` handles the validation result correctly"
+    - "when processed data is empty, the flow surfaces the message `No data to save` and does not save results"
+    - "`save_results` uses the validation outcome instead of reading `results.process_data` directly"
+    - "exact field names or wrapper object shape for the validation result are not important"
 
 - id: flow-test8-branching-in-loop
   prompt: |-
@@ -193,7 +243,29 @@
     Update `combine_data` so it merges the enrichment results and sets a `hasFallbacks` flag when any fallback was used.
     Keep `get_item` as the first step and `return_result` as the last step.
   initial: ai_evals/fixtures/frontend/flow/initial/test7_initial.json
-  expected: ai_evals/fixtures/frontend/flow/expected/test7_modify_complex.json
+  validate:
+    topLevelStepIds:
+      - get_item
+      - combine_data
+      - return_result
+    topLevelStepOrder:
+      - get_item
+      - combine_data
+      - return_result
+    topLevelStepTypeCountsAtLeast:
+      - type: branchall
+        count: 1
+    topLevelStepTypes:
+      - id: get_item
+        type: rawscript
+      - id: combine_data
+        type: rawscript
+      - id: return_result
+        type: rawscript
+    moduleRules:
+      - id: enrich_price
+      - id: enrich_inventory
+      - id: enrich_reviews
   judgeChecklist:
     - "the updated flow keeps `get_item` as the first step"
     - "the updated flow keeps `return_result` as the last step"
@@ -206,14 +278,42 @@
   prompt: |-
     Create a flow that keeps incrementing a counter until it reaches a target value.
     The input should include a number field named `target`.
-    Name the looping step `count_until_target`.
-    Once the target is reached, return the final counter value.
-  expected: ai_evals/fixtures/frontend/flow/expected/test10_while_loop_counter.json
+    Use a top-level loop step named `count_until_target`.
+    Inside it, use a single step named `increment_counter` that increments the current counter.
+    The loop should stop once the counter reaches `target`.
+    After the loop, add a top-level step named `return_final_counter` that returns the last counter value.
+  validate:
+    exactTopLevelStepIds:
+      - count_until_target
+      - return_final_counter
+    topLevelStepOrder:
+      - count_until_target
+      - return_final_counter
+    topLevelStepTypes:
+      - id: count_until_target
+        type: whileloopflow
+      - id: return_final_counter
+        type: rawscript
+    moduleRules:
+      - id: count_until_target
+        hasStopAfterIf: true
+        hasStopAfterAllItersIf: false
+        exactImmediateChildStepIds:
+          - increment_counter
+        immediateChildStepTypes:
+          - id: increment_counter
+            type: rawscript
+    moduleFieldRules:
+      - id: count_until_target
+        path: stop_after_if.expr
+        equals: result >= flow_input.target
   judgeChecklist:
     - "the input schema includes a number field named `target`"
-    - "the looping step is named `count_until_target`"
-    - the flow keeps incrementing a counter until the target is reached
-    - the final output returns the final counter value
+    - "the top-level while loop step is named `count_until_target`"
+    - "`count_until_target` contains a single increment step named `increment_counter`"
+    - "`count_until_target` uses module-level `stop_after_if` to stop when the counter reaches `target`"
+    - "`increment_counter` uses `flow_input.iter.value` or an equivalent loop-state expression and falls back to `0` on the first iteration"
+    - "`return_final_counter` returns the final counter value"
 
 - id: flow-test11-preprocessor-and-failure-handler
   prompt: |-
@@ -242,8 +342,16 @@
     Add an approval step named `request_approval` that pauses the flow and asks the approver for a comment.
     One approval should be enough to continue.
     After approval, add a final step named `finalize_purchase` that returns an approved status object.
-  expected: ai_evals/fixtures/frontend/flow/expected/test12_approval_step.json
   validate:
+    topLevelStepIds:
+      - request_approval
+      - finalize_purchase
+    topLevelStepOrder:
+      - request_approval
+      - finalize_purchase
+    topLevelStepTypes:
+      - id: finalize_purchase
+        type: rawscript
     schemaRequiredPaths:
       - requester_email
       - amount
@@ -259,3 +367,102 @@
     - one approval is enough to continue
     - "the flow includes a final step named `finalize_purchase`"
     - "`finalize_purchase` returns an approved status object after approval"
+
+- id: flow-test13-loop-resilience-toggle
+  prompt: |-
+    Update `loop_orders` so it can process orders in parallel.
+    If one order fails, the rest should still continue.
+    Keep the existing order-fetching and summary steps the same.
+  initial: ai_evals/fixtures/frontend/flow/initial/test6_initial.json
+  validate:
+    exactTopLevelStepIds:
+      - get_orders
+      - loop_orders
+      - summarize
+    topLevelStepTypes:
+      - id: loop_orders
+        type: forloopflow
+    moduleFieldRules:
+      - id: loop_orders
+        path: value.parallel
+        equals: true
+      - id: loop_orders
+        path: value.skip_failures
+        equals: true
+  judgeChecklist:
+    - "the flow keeps `get_orders` before `loop_orders` and `summarize` after it"
+    - "`loop_orders` processes orders in parallel"
+    - "a failure in one order does not stop the remaining orders from being processed"
+
+- id: flow-test14-modify-existing-special-modules
+  prompt: |-
+    Update this event-processing flow for a string payload.
+
+    Before `process_event` runs, trim the payload and reject empty strings.
+    If anything fails, return a compact error object with the error message and the failing step id.
+    Keep `process_event` as the main step.
+  initial: ai_evals/fixtures/frontend/flow/initial/test11_initial.json
+  expected: ai_evals/fixtures/frontend/flow/expected/test11_preprocessor_failure.json
+  validate:
+    requireSpecialModules:
+      - preprocessor_module
+      - failure_module
+  judgeChecklist:
+    - the updated flow trims the payload before the main processing runs
+    - the updated flow rejects empty payload strings
+    - "the existing `process_event` step remains the main step"
+    - failures return a compact error object with the error message and failing step id
+
+- id: flow-test15-create-current-flow-schedule
+  prompt: |-
+    Update this flow by adding a final step named `return_schedule_status`.
+    It should return an object with `scheduled: true` and the order summary from `results.summarize_orders`.
+    Also create an enabled daily schedule named `order_processing_daily` for the current flow.
+    It should run every day at 07:30 UTC with empty args.
+    Do not ask me for the flow path.
+  initial: ai_evals/fixtures/frontend/flow/initial/scheduled_order_flow.json
+  validate:
+    topLevelStepIds:
+      - return_schedule_status
+  toolExpect:
+    requiredToolsUsed:
+      - create_schedule
+    toolCallArgs:
+      - tool: create_schedule
+        field: path
+        stringStartsWithAnyOf:
+          - f/
+          - u/
+        stringMustNotStartWithAnyOf:
+          - schedules/
+  skipJudge: true
+  judgeChecklist:
+    - "the flow includes a final top-level step named `return_schedule_status`"
+    - "`return_schedule_status` returns `scheduled: true` and the order summary"
+
+- id: flow-test16-create-current-flow-http-trigger
+  prompt: |-
+    Update this flow by adding a final step named `webhook_response`.
+    It should return an object with `ok: true` and the order summary from `results.summarize_orders`.
+    Also create a public POST HTTP endpoint named `order_processing_webhook` for the current flow.
+    Use route path `ai-evals/order-processing` and no authentication.
+    Do not ask me for the flow path.
+  initial: ai_evals/fixtures/frontend/flow/initial/scheduled_order_flow.json
+  validate:
+    topLevelStepIds:
+      - webhook_response
+  toolExpect:
+    requiredToolsUsed:
+      - create_trigger
+    toolCallArgs:
+      - tool: create_trigger
+        field: path
+        stringStartsWithAnyOf:
+          - f/
+          - u/
+        stringMustNotStartWithAnyOf:
+          - schedules/
+  skipJudge: true
+  judgeChecklist:
+    - "the flow includes a final top-level step named `webhook_response`"
+    - "`webhook_response` returns `ok: true` and the order summary"

ai_evals/cases/script.yaml

@@ -9,3 +9,51 @@
     - uses the existing `name` input
     - returns a plain greeting string
     - does not wrap the result in an object or array
+
+- id: script-test2-create-current-script-schedule
+  prompt: |-
+    Update the current Bun script so it takes the existing `name` input and returns a plain greeting string like `Hello, Alice!`.
+    Also create an enabled daily schedule named `greet_user_daily` for the current script.
+    It should run every day at 09:00 UTC and pass `{ "name": "Alice" }` as args.
+    Do not ask me for the script path.
+  initial: ai_evals/fixtures/frontend/script/initial/test1_empty_bun.json
+  expected: ai_evals/fixtures/frontend/script/expected/test1_greet_user.json
+  toolExpect:
+    requiredToolsUsed:
+      - create_schedule
+    toolCallArgs:
+      - tool: create_schedule
+        field: path
+        stringStartsWithAnyOf:
+          - f/
+          - u/
+        stringMustNotStartWithAnyOf:
+          - schedules/
+  skipJudge: true
+  judgeChecklist:
+    - uses the existing `name` input
+    - returns a plain greeting string
+
+- id: script-test3-create-current-script-http-trigger
+  prompt: |-
+    Update the current Bun script so it takes the existing `name` input and returns a plain greeting string like `Hello, Alice!`.
+    Also create a public POST HTTP endpoint named `greet_user_webhook` for the current script.
+    Use route path `ai-evals/greet-user` and no authentication.
+    Do not ask me for the script path.
+  initial: ai_evals/fixtures/frontend/script/initial/test1_empty_bun.json
+  expected: ai_evals/fixtures/frontend/script/expected/test1_greet_user.json
+  toolExpect:
+    requiredToolsUsed:
+      - create_trigger
+    toolCallArgs:
+      - tool: create_trigger
+        field: path
+        stringStartsWithAnyOf:
+          - f/
+          - u/
+        stringMustNotStartWithAnyOf:
+          - schedules/
+  skipJudge: true
+  judgeChecklist:
+    - uses the existing `name` input
+    - returns a plain greeting string

ai_evals/cli/index.ts

@@ -27,11 +27,18 @@ import { EVAL_MODES, type EvalMode } from "../core/types";
 import { DEFAULT_JUDGE_MODEL } from "../core/judge";
 import { createCliModeRunner } from "../modes/cli";
 import { runFrontendBenchmarkAdapter } from "../adapters/frontend/runtime";
+import {
+  FRONTEND_EVAL_TRANSPORTS,
+  type FrontendEvalTransport,
+  parseFrontendEvalTransport,
+} from "../core/frontendTransport";
 
 async function main() {
   const program = new Command()
     .name("bun run cli --")
-    .description("Run AI eval cases against the current production prompts and guidance")
+    .description(
+      "Run AI eval cases against the current production prompts and guidance",
+    )
     .showHelpAfterError()
     .showSuggestionAfterError()
     .addHelpText(
@@ -53,7 +60,7 @@ async function main() {
         "",
         "Models:",
         getEvalModelHelpText(),
-      ].join("\n")
+      ].join("\n"),
     );
 
   program
@@ -76,15 +83,33 @@ async function main() {
     .description("Run one benchmark mode")
     .argument("<mode>", "cli, flow, script, or app", parseMode)
     .argument("[caseIds...]", "specific case ids to run")
-    .option("--runs <n>", "number of attempts per case", parsePositiveInteger, 1)
+    .option(
+      "--runs <n>",
+      "number of attempts per case",
+      parsePositiveInteger,
+      1,
+    )
     .option("--output <path>", "write the result JSON to this path")
-    .option("--model <name>", `model alias (${EVAL_MODELS.map((entry) => entry.id).join(", ")})`)
-    .option("--models <names>", "comma-separated model aliases to run sequentially")
+    .option(
+      "--model <name>",
+      `model alias (${EVAL_MODELS.map((entry) => entry.id).join(", ")})`,
+    )
+    .option(
+      "--models <names>",
+      "comma-separated model aliases to run sequentially",
+    )
+    .option(
+      "--transport <mode>",
+      `frontend transport (${FRONTEND_EVAL_TRANSPORTS.join(", ")})`,
+    )
     .option("--verbose", "stream assistant output during frontend runs")
-    .option("--record", "append a compact summary line to ai_evals/history/<mode>.jsonl")
+    .option(
+      "--record",
+      "append a compact summary line to ai_evals/history/<mode>.jsonl",
+    )
     .option(
       "--backend-validation <mode>",
-      `backend smoke validation (${BACKEND_VALIDATION_MODES.join(", ")})`
+      `backend smoke validation (${BACKEND_VALIDATION_MODES.join(", ")})`,
     )
     .action(
       async (
@@ -95,10 +120,11 @@ async function main() {
           output?: string;
           model?: string;
           models?: string;
+          transport?: string;
           verbose?: boolean;
           record?: boolean;
           backendValidation?: string;
-        }
+        },
       ) => {
         await handleRun({
           mode,
@@ -107,11 +133,14 @@ async function main() {
           outputPath: options.output,
           model: options.model,
           models: options.models,
+          transport: options.transport
+            ? parseFrontendEvalTransport(options.transport)
+            : undefined,
           verbose: options.verbose ?? false,
           record: options.record ?? false,
           backendValidation: options.backendValidation,
         });
-      }
+      },
     );
 
   await program.parseAsync(process.argv);
@@ -137,7 +166,10 @@ function handleModels() {
       ...(model.frontend ? ["flow", "script", "app"] : []),
       ...(model.cli ? ["cli"] : []),
     ];
-    const aliases = [model.id, ...model.aliases.filter((alias) => alias !== model.id)];
+    const aliases = [
+      model.id,
+      ...model.aliases.filter((alias) => alias !== model.id),
+    ];
     process.stdout.write(`- ${model.id}: ${model.label}\n`);
     process.stdout.write(`  aliases: ${aliases.join(", ")}\n`);
     process.stdout.write(`  modes: ${supports.join(", ")}\n`);
@@ -152,48 +184,72 @@ async function handleRun(input: {
   outputPath?: string;
   model?: string;
   models?: string;
+  transport?: FrontendEvalTransport;
   verbose: boolean;
   record: boolean;
   backendValidation?: string;
 }) {
   if (input.record && input.caseIds.length > 0) {
-    throw new Error("--record only supports full-suite runs; omit case ids to record history");
+    throw new Error(
+      "--record only supports full-suite runs; omit case ids to record history",
+    );
   }
   if (input.model && input.models) {
     throw new Error("Use either --model or --models, not both");
   }
+  if (input.mode === "cli" && input.transport === "proxy") {
+    throw new Error(
+      "--transport proxy is only supported for flow, script, and app modes",
+    );
+  }
 
   const selectedCases = await loadSelectedCases(input.mode, input.caseIds);
   const models = resolveRequestedModels(input.mode, input.model, input.models);
   const backendValidation = parseBackendValidationMode(
-    input.backendValidation ?? process.env.WMILL_AI_EVAL_BACKEND_VALIDATION
+    input.backendValidation ?? process.env.WMILL_AI_EVAL_BACKEND_VALIDATION,
   );
   if (input.outputPath && models.length > 1) {
     throw new Error("--output only supports a single model run");
   }
-  if (backendValidation !== "off" && input.mode !== "flow" && input.mode !== "script") {
-    throw new Error("--backend-validation currently supports only flow and script modes");
+  if (
+    backendValidation !== "off" &&
+    input.mode !== "flow" &&
+    input.mode !== "script"
+  ) {
+    throw new Error(
+      "--backend-validation currently supports only flow and script modes",
+    );
   }
 
-  const summaries: Array<{ label: string; passRate: number; averageDurationMs: number }> = [];
+  const summaries: Array<{
+    label: string;
+    passRate: number;
+    averageDurationMs: number;
+  }> = [];
 
   for (const [index, model] of models.entries()) {
     const runModel = formatRunModelLabel(input.mode, model);
     if (models.length > 1) {
       process.stdout.write(
-        `${index > 0 ? "\n" : ""}=== ${input.mode} ${model.id} (${runModel}) ===\n`
+        `${index > 0 ? "\n" : ""}=== ${input.mode} ${model.id} (${runModel}) ===\n`,
       );
     }
     process.stderr.write(`Starting ${input.mode} benchmark...\n`);
 
     const result =
       input.mode === "cli"
-        ? await runCliBenchmark(selectedCases, input.runs, getCliEvalModel(model), runModel)
+        ? await runCliBenchmark(
+            selectedCases,
+            input.runs,
+            getCliEvalModel(model),
+            runModel,
+          )
         : await runFrontendBenchmarkAdapter({
             mode: input.mode,
             caseIds: input.caseIds,
             runs: input.runs,
             model: model.id,
+            transport: input.transport,
             verbose: input.verbose,
             backendValidation,
           });
@@ -225,7 +281,7 @@ async function handleRun(input: {
     process.stdout.write("\nModel summary\n");
     for (const summary of summaries) {
       process.stdout.write(
-        `- ${summary.label}: ${formatPercent(summary.passRate)} | ${Math.round(summary.averageDurationMs)}ms\n`
+        `- ${summary.label}: ${formatPercent(summary.passRate)} | ${Math.round(summary.averageDurationMs)}ms\n`,
       );
     }
   }
@@ -235,7 +291,7 @@ async function runCliBenchmark(
   cases: Awaited<ReturnType<typeof loadSelectedCases>>,
   runs: number,
   model: ReturnType<typeof getCliEvalModel>,
-  runModel: string
+  runModel: string,
 ) {
   const caseResults = await runSuite({
     modeRunner: createCliModeRunner(model),
@@ -258,7 +314,9 @@ function parseMode(value: string): EvalMode {
   if (EVAL_MODES.includes(value as EvalMode)) {
     return value as EvalMode;
   }
-  throw new InvalidArgumentError(`mode must be one of: ${EVAL_MODES.join(", ")}`);
+  throw new InvalidArgumentError(
+    `mode must be one of: ${EVAL_MODES.join(", ")}`,
+  );
 }
 
 function parseOptionalMode(value: string | undefined): EvalMode | undefined {
@@ -276,7 +334,7 @@ function parsePositiveInteger(value: string): number {
 function resolveRequestedModels(
   mode: EvalMode,
   singleModel?: string,
-  multipleModels?: string
+  multipleModels?: string,
 ): EvalModelSpec[] {
   if (!multipleModels) {
     return [resolveEvalModel(mode, singleModel)];

ai_evals/core/appArtifacts.test.ts

@@ -0,0 +1,20 @@
+import { describe, expect, it } from "bun:test";
+import { buildAppArtifacts } from "./appArtifacts";
+
+describe("buildAppArtifacts", () => {
+  it("emits lint diagnostics as an artifact alongside app files", () => {
+    const artifacts = buildAppArtifacts({
+      frontend: {
+        "/index.tsx":
+          "import { backend } from 'wmill'\nexport default function App() { void backend.deleteRecipe({ id: 1 }); return <div /> }\n",
+      },
+      backend: {},
+      datatables: [],
+    });
+
+    const lintArtifact = artifacts.find((artifact) => artifact.path === "lint.json");
+    expect(lintArtifact).toBeDefined();
+    expect(lintArtifact?.content).toContain('"errorCount": 1');
+    expect(lintArtifact?.content).toContain("deleteRecipe");
+  });
+});

ai_evals/core/appArtifacts.ts

@@ -0,0 +1,52 @@
+import { collectAppDiagnostics } from "./appDiagnostics";
+import type { BenchmarkArtifactFile } from "./types";
+import type { AppFilesState } from "./validators";
+
+export function buildAppArtifacts(actual: AppFilesState): BenchmarkArtifactFile[] {
+  const diagnostics = collectAppDiagnostics({
+    frontend: actual.frontend,
+    backend: actual.backend,
+  });
+  const artifacts: BenchmarkArtifactFile[] = [
+    {
+      path: "app.json",
+      content: JSON.stringify(actual, null, 2) + "\n",
+    },
+    {
+      path: "lint.json",
+      content: JSON.stringify(diagnostics, null, 2) + "\n",
+    },
+  ];
+
+  for (const [filePath, content] of Object.entries(actual.frontend)) {
+    artifacts.push({
+      path: `frontend${filePath.startsWith("/") ? filePath : `/${filePath}`}`,
+      content,
+    });
+  }
+
+  for (const [key, runnable] of Object.entries(actual.backend)) {
+    artifacts.push({
+      path: `backend/${key}/meta.json`,
+      content: JSON.stringify(runnable, null, 2) + "\n",
+    });
+
+    const inlineContent = runnable.inlineScript?.content;
+    if (inlineContent) {
+      const extension = runnable.inlineScript?.language === "python3" ? "py" : "ts";
+      artifacts.push({
+        path: `backend/${key}/main.${extension}`,
+        content: inlineContent,
+      });
+    }
+  }
+
+  if (actual.datatables.length > 0) {
+    artifacts.push({
+      path: "datatables.json",
+      content: JSON.stringify(actual.datatables, null, 2) + "\n",
+    });
+  }
+
+  return artifacts;
+}

ai_evals/core/appDiagnostics.test.ts

@@ -0,0 +1,96 @@
+import { describe, expect, it } from "bun:test";
+import { fileURLToPath } from "node:url";
+import { loadAppFixture } from "../adapters/frontend/core/app/appFixtureLoader";
+import { buildAppWmillTypes, collectAppDiagnostics } from "./appDiagnostics";
+
+const FILE_MANAGER_FIXTURE = fileURLToPath(
+  new URL("../fixtures/frontend/app/initial/file_manager", import.meta.url)
+);
+
+describe("collectAppDiagnostics", () => {
+  it("accepts seeded multi-file apps without static analysis errors", async () => {
+    const fixture = await loadAppFixture(FILE_MANAGER_FIXTURE);
+    const diagnostics = collectAppDiagnostics({
+      frontend: fixture.frontend,
+      backend: fixture.backend,
+    });
+
+    expect(diagnostics.lintResult.errorCount).toBe(0);
+  });
+
+  it("reports missing backend references through the generated wmill typings", () => {
+    const diagnostics = collectAppDiagnostics({
+      frontend: {
+        "/index.tsx":
+          "import { backend } from 'wmill'\nexport default function App() { void backend.deleteRecipe({ id: 1 }); return <div /> }\n",
+      },
+      backend: {
+        listRecipes: {
+          name: "List recipes",
+          type: "inline",
+          inlineScript: {
+            language: "bun",
+            content: "export async function main() { return [] }\n",
+          },
+        },
+      },
+    });
+
+    expect(diagnostics.lintResult.errorCount).toBeGreaterThan(0);
+    expect(diagnostics.lintResult.errors.frontend["/index.tsx"]?.join("\n")).toContain(
+      "Property 'deleteRecipe' does not exist"
+    );
+  });
+
+  it("reports backend argument shape mismatches when the inline main signature is portable", () => {
+    const diagnostics = collectAppDiagnostics({
+      frontend: {
+        "/index.tsx":
+          "import { backend } from 'wmill'\nexport default function App() { void backend.addRecipe({ name: 'Soup' }); return <div /> }\n",
+      },
+      backend: {
+        addRecipe: {
+          name: "Add recipe",
+          type: "inline",
+          inlineScript: {
+            language: "bun",
+            content:
+              "export async function main({ name, ingredients }: { name: string; ingredients: string }) { return { name, ingredients } }\n",
+          },
+        },
+      },
+    });
+
+    expect(diagnostics.lintResult.errorCount).toBeGreaterThan(0);
+    expect(diagnostics.lintResult.errors.frontend["/index.tsx"]?.join("\n")).toContain(
+      "Property 'ingredients' is missing"
+    );
+  });
+});
+
+describe("buildAppWmillTypes", () => {
+  it("generates callable signatures for zero-arg and typed runnables", () => {
+    const wmillTypes = buildAppWmillTypes({
+      listRecipes: {
+        name: "List recipes",
+        type: "inline",
+        inlineScript: {
+          language: "bun",
+          content: "export async function main() { return [] }\n",
+        },
+      },
+      addRecipe: {
+        name: "Add recipe",
+        type: "inline",
+        inlineScript: {
+          language: "bun",
+          content:
+            "export async function main({ name }: { name: string }) { return { name } }\n",
+        },
+      },
+    });
+
+    expect(wmillTypes).toContain('"listRecipes": () => Promise<any>;');
+    expect(wmillTypes).toContain('"addRecipe": (args: { name: string }) => Promise<any>;');
+  });
+});

ai_evals/core/appDiagnostics.ts

@@ -0,0 +1,758 @@
+import path from "node:path";
+import ts from "typescript";
+import type { LintResult } from "../../frontend/src/lib/components/copilot/chat/app/core";
+
+const FRONTEND_ROOT = "/__ai_evals__/frontend";
+const BACKEND_ROOT = "/__ai_evals__/backend";
+const FRONTEND_REACT_SHIM_PATH = `${FRONTEND_ROOT}/__react_shim__.d.ts`;
+const FRONTEND_WMILL_TYPES_PATH = `${FRONTEND_ROOT}/wmill.d.ts`;
+const BACKEND_WINDMILL_CLIENT_SHIM_PATH = `${BACKEND_ROOT}/__windmill_client__.d.ts`;
+const TS_LIKE_LANGUAGES = new Set([
+  "bun",
+  "deno",
+  "nativets",
+  "bunnative",
+  "ts",
+  "typescript",
+]);
+const JS_LIKE_LANGUAGES = new Set(["javascript", "js", "nodejs"]);
+const SAFE_TYPE_REFERENCE_NAMES = new Set([
+  "Array",
+  "Date",
+  "Exclude",
+  "Extract",
+  "NonNullable",
+  "Omit",
+  "Partial",
+  "Pick",
+  "Promise",
+  "Readonly",
+  "ReadonlyArray",
+  "Record",
+  "Required",
+  "ReturnType",
+  "Uppercase",
+  "Lowercase",
+  "Capitalize",
+  "Uncapitalize",
+]);
+
+const FRONTEND_REACT_SHIM = `declare namespace React {
+  type SetStateAction<S> = S | ((prevState: S) => S);
+  type Dispatch<A> = (value: A) => void;
+  type FC<P = {}> = (props: P) => any;
+  type ReactNode = any;
+  interface FormEvent<T = EventTarget> {
+    preventDefault(): void;
+    target: T;
+    currentTarget: T;
+  }
+  interface ChangeEvent<T = EventTarget> {
+    target: T;
+    currentTarget: T;
+  }
+}
+
+declare namespace JSX {
+  interface IntrinsicAttributes {
+    key?: any;
+  }
+  interface IntrinsicElements {
+    [elementName: string]: any;
+  }
+}
+
+declare module "react" {
+  export type SetStateAction<S> = React.SetStateAction<S>;
+  export type Dispatch<A> = React.Dispatch<A>;
+  export type FC<P = {}> = React.FC<P>;
+  export type ReactNode = React.ReactNode;
+  export type FormEvent<T = EventTarget> = React.FormEvent<T>;
+  export type ChangeEvent<T = EventTarget> = React.ChangeEvent<T>;
+  export function useState<S>(initialState: S | (() => S)): [S, Dispatch<SetStateAction<S>>];
+  export function useEffect(effect: () => void | (() => void), deps?: readonly unknown[]): void;
+  const React: any;
+  export default React;
+}
+`;
+
+const BACKEND_WINDMILL_CLIENT_SHIM = `declare const console: {
+  log: (...args: any[]) => void;
+  error: (...args: any[]) => void;
+  warn: (...args: any[]) => void;
+};
+
+declare module "windmill-client" {
+  interface SqlQueryResult {
+    fetch(): Promise<any>;
+    fetchOne(): Promise<any>;
+  }
+
+  interface SqlTemplateFunction {
+    (strings: TemplateStringsArray, ...values: any[]): SqlQueryResult;
+  }
+
+  interface WindmillClient {
+    datatable(name?: string): SqlTemplateFunction;
+    ducklake(name?: string): SqlTemplateFunction;
+    [key: string]: any;
+  }
+
+  const wmill: WindmillClient;
+  export = wmill;
+}
+`;
+
+export interface AppDiagnosticRunnable {
+  name?: string;
+  type?: string;
+  path?: string;
+  inlineScript?: {
+    language?: string;
+    content?: string;
+  };
+}
+
+export interface AppStaticDiagnostic {
+  source: "frontend" | "backend";
+  target: string;
+  message: string;
+  line?: number;
+  column?: number;
+  code?: number;
+}
+
+export interface AppDiagnosticsResult {
+  lintResult: LintResult;
+  diagnostics: AppStaticDiagnostic[];
+}
+
+export function buildAppWmillTypes(
+  backend: Record<string, AppDiagnosticRunnable> = {},
+): string {
+  return `// THIS FILE IS READ-ONLY
+// AND GENERATED AUTOMATICALLY FROM YOUR RUNNABLES
+
+export declare const backend: {
+${Object.entries(backend)
+  .map(
+    ([key, runnable]) =>
+      `  ${JSON.stringify(key)}: ${getRunnableSignature(runnable, false)};`,
+  )
+  .join("\n")}
+};
+
+export declare const backendAsync: {
+${Object.entries(backend)
+  .map(
+    ([key, runnable]) =>
+      `  ${JSON.stringify(key)}: ${getRunnableSignature(runnable, true)};`,
+  )
+  .join("\n")}
+};
+
+export type Job = {
+  type: "QueuedJob" | "CompletedJob";
+  id: string;
+  created_at: number;
+  started_at: number | undefined;
+  duration_ms: number;
+  success: boolean;
+  args: any;
+  result: any;
+};
+
+export declare function waitJob(id: string): Promise<Job>;
+export declare function getJob(id: string): Promise<Job>;
+
+export type StreamUpdate = {
+  new_result_stream?: string;
+  stream_offset?: number;
+};
+
+export declare function streamJob(id: string, onUpdate?: (data: StreamUpdate) => void): Promise<any>;
+`;
+}
+
+export function collectAppDiagnostics(input: {
+  frontend: Record<string, string>;
+  backend: Record<string, AppDiagnosticRunnable>;
+}): AppDiagnosticsResult {
+  const frontendDiagnostics = collectFrontendDiagnostics(
+    input.frontend,
+    input.backend,
+  );
+  const backendDiagnostics = collectBackendDiagnostics(input.backend);
+  const diagnostics = dedupeDiagnostics([
+    ...frontendDiagnostics,
+    ...backendDiagnostics,
+  ]).sort(compareDiagnostics);
+
+  return {
+    diagnostics,
+    lintResult: {
+      errors: {
+        frontend: groupMessages(
+          diagnostics.filter((diagnostic) => diagnostic.source === "frontend"),
+        ),
+        backend: groupMessages(
+          diagnostics.filter((diagnostic) => diagnostic.source === "backend"),
+        ),
+      },
+      warnings: {
+        frontend: {},
+        backend: {},
+      },
+      errorCount: diagnostics.length,
+      warningCount: 0,
+    },
+  };
+}
+
+function collectFrontendDiagnostics(
+  frontend: Record<string, string>,
+  backend: Record<string, AppDiagnosticRunnable>,
+): AppStaticDiagnostic[] {
+  const frontendFiles = Object.entries(frontend)
+    .filter(([filePath]) => isFrontendCodeFile(filePath))
+    .map(
+      ([filePath, content]) =>
+        [toFrontendVirtualPath(filePath), content] as const,
+    );
+
+  const virtualFiles = new Map<string, string>([
+    [FRONTEND_REACT_SHIM_PATH, FRONTEND_REACT_SHIM],
+    [
+      FRONTEND_WMILL_TYPES_PATH,
+      wrapModuleDeclaration("wmill", buildAppWmillTypes(backend)),
+    ],
+    ...frontendFiles,
+  ]);
+  const host = createVirtualCompilerHost(
+    virtualFiles,
+    getFrontendCompilerOptions(),
+  );
+  const rootNames = [...virtualFiles.keys()];
+  const program = ts.createProgram({
+    rootNames,
+    options: getFrontendCompilerOptions(),
+    host,
+  });
+
+  return ts.getPreEmitDiagnostics(program).flatMap((diagnostic) =>
+    mapTypeScriptDiagnostic({
+      diagnostic,
+      source: "frontend",
+      toTarget(fileName) {
+        const normalized = normalizeFileName(fileName);
+        if (normalized === FRONTEND_WMILL_TYPES_PATH) {
+          return "/wmill.d.ts";
+        }
+        if (!normalized.startsWith(`${FRONTEND_ROOT}/`)) {
+          return null;
+        }
+        if (normalized === FRONTEND_REACT_SHIM_PATH) {
+          return null;
+        }
+        return normalized.slice(FRONTEND_ROOT.length);
+      },
+    }),
+  );
+}
+
+function collectBackendDiagnostics(
+  backend: Record<string, AppDiagnosticRunnable>,
+): AppStaticDiagnostic[] {
+  const backendFiles = Object.entries(backend)
+    .filter(([, runnable]) => isTypeCheckableBackendRunnable(runnable))
+    .map(
+      ([key, runnable]) =>
+        [
+          `${BACKEND_ROOT}/${key}/main.${getBackendFileExtension(runnable.inlineScript?.language)}`,
+          runnable.inlineScript?.content ?? "",
+        ] as const,
+    );
+
+  if (backendFiles.length === 0) {
+    return [];
+  }
+
+  const virtualFiles = new Map<string, string>([
+    [BACKEND_WINDMILL_CLIENT_SHIM_PATH, BACKEND_WINDMILL_CLIENT_SHIM],
+    ...backendFiles,
+  ]);
+  const host = createVirtualCompilerHost(
+    virtualFiles,
+    getBackendCompilerOptions(),
+  );
+  const rootNames = [...virtualFiles.keys()];
+  const program = ts.createProgram({
+    rootNames,
+    options: getBackendCompilerOptions(),
+    host,
+  });
+
+  return ts.getPreEmitDiagnostics(program).flatMap((diagnostic) =>
+    mapTypeScriptDiagnostic({
+      diagnostic,
+      source: "backend",
+      toTarget(fileName) {
+        const normalized = normalizeFileName(fileName);
+        if (normalized === BACKEND_WINDMILL_CLIENT_SHIM_PATH) {
+          return null;
+        }
+        if (!normalized.startsWith(`${BACKEND_ROOT}/`)) {
+          return null;
+        }
+        const relativePath = normalized.slice(BACKEND_ROOT.length + 1);
+        const runnableKey = relativePath.split("/")[0];
+        return runnableKey || null;
+      },
+    }),
+  );
+}
+
+function getFrontendCompilerOptions(): ts.CompilerOptions {
+  return {
+    allowJs: true,
+    checkJs: true,
+    esModuleInterop: true,
+    allowSyntheticDefaultImports: true,
+    jsx: ts.JsxEmit.Preserve,
+    module: ts.ModuleKind.ESNext,
+    moduleResolution: ts.ModuleResolutionKind.Node10,
+    noEmit: true,
+    noImplicitAny: false,
+    skipLibCheck: true,
+    strict: false,
+    target: ts.ScriptTarget.ES2022,
+    lib: ["lib.es2022.d.ts", "lib.dom.d.ts"],
+  };
+}
+
+function getBackendCompilerOptions(): ts.CompilerOptions {
+  return {
+    allowJs: true,
+    checkJs: true,
+    esModuleInterop: true,
+    allowSyntheticDefaultImports: true,
+    module: ts.ModuleKind.ESNext,
+    moduleResolution: ts.ModuleResolutionKind.Node10,
+    noEmit: true,
+    noImplicitAny: false,
+    skipLibCheck: true,
+    strict: false,
+    target: ts.ScriptTarget.ES2022,
+    lib: ["lib.es2022.d.ts"],
+  };
+}
+
+function createVirtualCompilerHost(
+  files: Map<string, string>,
+  options: ts.CompilerOptions,
+): ts.CompilerHost {
+  const originalHost = ts.createCompilerHost(options, true);
+  const originalGetSourceFile = originalHost.getSourceFile.bind(originalHost);
+  const originalReadFile = originalHost.readFile.bind(originalHost);
+  const originalFileExists = originalHost.fileExists.bind(originalHost);
+  const originalDirectoryExists =
+    originalHost.directoryExists?.bind(originalHost);
+  const originalGetDirectories =
+    originalHost.getDirectories?.bind(originalHost);
+
+  return {
+    ...originalHost,
+    getCurrentDirectory: () => "/",
+    getSourceFile(
+      fileName,
+      languageVersion,
+      onError,
+      shouldCreateNewSourceFile,
+    ) {
+      const normalized = normalizeFileName(fileName);
+      const content = files.get(normalized);
+      if (content !== undefined) {
+        return ts.createSourceFile(fileName, content, languageVersion, true);
+      }
+      return originalGetSourceFile(
+        fileName,
+        languageVersion,
+        onError,
+        shouldCreateNewSourceFile,
+      );
+    },
+    readFile(fileName) {
+      const normalized = normalizeFileName(fileName);
+      return files.get(normalized) ?? originalReadFile(fileName);
+    },
+    fileExists(fileName) {
+      const normalized = normalizeFileName(fileName);
+      return files.has(normalized) || originalFileExists(fileName);
+    },
+    directoryExists(dirName) {
+      const normalized = normalizeFileName(dirName);
+      return (
+        hasVirtualDirectory(files, normalized) ||
+        originalDirectoryExists?.(dirName) ||
+        false
+      );
+    },
+    getDirectories(dirName) {
+      const normalized = normalizeFileName(dirName);
+      const virtualDirectories = listVirtualDirectories(files, normalized);
+      const diskDirectories = originalGetDirectories?.(dirName) ?? [];
+      return [...new Set([...diskDirectories, ...virtualDirectories])];
+    },
+    realpath(fileName) {
+      return normalizeFileName(fileName);
+    },
+    writeFile() {},
+  };
+}
+
+function mapTypeScriptDiagnostic(input: {
+  diagnostic: ts.Diagnostic;
+  source: "frontend" | "backend";
+  toTarget: (fileName: string) => string | null;
+}): AppStaticDiagnostic[] {
+  const { diagnostic, source, toTarget } = input;
+  if (!diagnostic.file) {
+    return [];
+  }
+
+  const target = toTarget(diagnostic.file.fileName);
+  if (!target) {
+    return [];
+  }
+
+  const position =
+    diagnostic.start === undefined
+      ? undefined
+      : diagnostic.file.getLineAndCharacterOfPosition(diagnostic.start);
+
+  return [
+    {
+      source,
+      target,
+      message: ts
+        .flattenDiagnosticMessageText(diagnostic.messageText, "\n")
+        .trim(),
+      line: position ? position.line + 1 : undefined,
+      column: position ? position.character + 1 : undefined,
+      code: diagnostic.code,
+    },
+  ];
+}
+
+function groupMessages(
+  diagnostics: AppStaticDiagnostic[],
+): Record<string, string[]> {
+  const grouped: Record<string, string[]> = {};
+
+  for (const diagnostic of diagnostics) {
+    grouped[diagnostic.target] ??= [];
+    grouped[diagnostic.target].push(formatLintMessage(diagnostic));
+  }
+
+  return grouped;
+}
+
+function formatLintMessage(diagnostic: AppStaticDiagnostic): string {
+  if (diagnostic.line !== undefined) {
+    return `Line ${diagnostic.line}: ${diagnostic.message}`;
+  }
+  return diagnostic.message;
+}
+
+function dedupeDiagnostics(
+  diagnostics: AppStaticDiagnostic[],
+): AppStaticDiagnostic[] {
+  const uniqueDiagnostics = new Map<string, AppStaticDiagnostic>();
+
+  for (const diagnostic of diagnostics) {
+    const key = [
+      diagnostic.source,
+      diagnostic.target,
+      diagnostic.code ?? "",
+      diagnostic.line ?? "",
+      diagnostic.column ?? "",
+      diagnostic.message,
+    ].join("::");
+    if (!uniqueDiagnostics.has(key)) {
+      uniqueDiagnostics.set(key, diagnostic);
+    }
+  }
+
+  return [...uniqueDiagnostics.values()];
+}
+
+function compareDiagnostics(
+  a: AppStaticDiagnostic,
+  b: AppStaticDiagnostic,
+): number {
+  if (a.source !== b.source) {
+    return a.source.localeCompare(b.source);
+  }
+  if (a.target !== b.target) {
+    return a.target.localeCompare(b.target);
+  }
+  if ((a.line ?? 0) !== (b.line ?? 0)) {
+    return (a.line ?? 0) - (b.line ?? 0);
+  }
+  if ((a.column ?? 0) !== (b.column ?? 0)) {
+    return (a.column ?? 0) - (b.column ?? 0);
+  }
+  return a.message.localeCompare(b.message);
+}
+
+function toFrontendVirtualPath(filePath: string): string {
+  const normalizedPath = normalizeAppFilePath(filePath);
+  return `${FRONTEND_ROOT}${normalizedPath}`;
+}
+
+function normalizeAppFilePath(filePath: string): string {
+  const normalizedPath = normalizeFileName(filePath);
+  return normalizedPath.startsWith("/") ? normalizedPath : `/${normalizedPath}`;
+}
+
+function normalizeFileName(fileName: string): string {
+  return path.posix.normalize(fileName.replace(/\\/g, "/"));
+}
+
+function hasVirtualDirectory(
+  files: Map<string, string>,
+  dirName: string,
+): boolean {
+  const normalizedDirectory = dirName.endsWith("/") ? dirName : `${dirName}/`;
+  for (const fileName of files.keys()) {
+    if (fileName === dirName || fileName.startsWith(normalizedDirectory)) {
+      return true;
+    }
+  }
+  return false;
+}
+
+function listVirtualDirectories(
+  files: Map<string, string>,
+  dirName: string,
+): string[] {
+  const normalizedDirectory = dirName.endsWith("/") ? dirName : `${dirName}/`;
+  const directories = new Set<string>();
+
+  for (const fileName of files.keys()) {
+    if (!fileName.startsWith(normalizedDirectory)) {
+      continue;
+    }
+    const relativePath = fileName.slice(normalizedDirectory.length);
+    const [segment] = relativePath.split("/");
+    if (segment && relativePath.includes("/")) {
+      directories.add(path.posix.join(dirName, segment));
+    }
+  }
+
+  return [...directories];
+}
+
+function wrapModuleDeclaration(moduleName: string, content: string): string {
+  const indentedContent = content
+    .trim()
+    .split("\n")
+    .map((line) => `  ${line}`)
+    .join("\n");
+
+  return `declare module "${moduleName}" {\n${indentedContent}\n}\n`;
+}
+
+function getRunnableSignature(
+  runnable: AppDiagnosticRunnable | undefined,
+  asyncMode: boolean,
+): string {
+  const returnType = asyncMode ? "Promise<string>" : "Promise<any>";
+  const parameter = getRunnableParameterSignature(runnable);
+  return `${parameter} => ${returnType}`;
+}
+
+function getRunnableParameterSignature(
+  runnable: AppDiagnosticRunnable | undefined,
+): string {
+  const parameterInfo = getRunnableParameterInfo(runnable);
+  if (!parameterInfo) {
+    return "()";
+  }
+
+  const parameterType = parameterInfo.typeText ?? "any";
+  if (parameterInfo.optional) {
+    return `(args?: ${parameterType})`;
+  }
+  return `(args: ${parameterType})`;
+}
+
+function getRunnableParameterInfo(
+  runnable: AppDiagnosticRunnable | undefined,
+): { typeText?: string; optional: boolean } | null {
+  if (
+    !runnable?.inlineScript?.content ||
+    !isTypeCheckableBackendRunnable(runnable)
+  ) {
+    return { typeText: "any", optional: true };
+  }
+
+  const sourceFile = ts.createSourceFile(
+    "main.ts",
+    runnable.inlineScript.content,
+    ts.ScriptTarget.Latest,
+    true,
+    getScriptKindForLanguage(runnable.inlineScript.language),
+  );
+  const mainDeclaration = findExportedMainDeclaration(sourceFile);
+
+  if (!mainDeclaration || mainDeclaration.parameters.length === 0) {
+    return null;
+  }
+
+  const [parameter] = mainDeclaration.parameters;
+  const optional =
+    Boolean(parameter.questionToken) || Boolean(parameter.initializer);
+
+  if (!parameter.type || !isPortableTypeNode(parameter.type)) {
+    return { typeText: "any", optional: true };
+  }
+
+  return {
+    typeText: parameter.type.getText(sourceFile).trim(),
+    optional,
+  };
+}
+
+function findExportedMainDeclaration(
+  sourceFile: ts.SourceFile,
+): ts.SignatureDeclarationBase | null {
+  for (const statement of sourceFile.statements) {
+    if (
+      ts.isFunctionDeclaration(statement) &&
+      statement.name?.text === "main" &&
+      hasExportModifier(statement)
+    ) {
+      return statement;
+    }
+
+    if (!ts.isVariableStatement(statement) || !hasExportModifier(statement)) {
+      continue;
+    }
+
+    for (const declaration of statement.declarationList.declarations) {
+      if (
+        !ts.isIdentifier(declaration.name) ||
+        declaration.name.text !== "main"
+      ) {
+        continue;
+      }
+      const initializer = declaration.initializer;
+      if (
+        initializer &&
+        (ts.isArrowFunction(initializer) ||
+          ts.isFunctionExpression(initializer))
+      ) {
+        return initializer;
+      }
+    }
+  }
+
+  return null;
+}
+
+function hasExportModifier(node: ts.Node): boolean {
+  const modifiers = ts.canHaveModifiers(node)
+    ? ts.getModifiers(node)
+    : undefined;
+  return Boolean(
+    modifiers?.some(
+      (modifier) => modifier.kind === ts.SyntaxKind.ExportKeyword,
+    ),
+  );
+}
+
+function isPortableTypeNode(node: ts.TypeNode): boolean {
+  if (
+    isKeywordTypeNode(node) ||
+    ts.isArrayTypeNode(node) ||
+    ts.isTupleTypeNode(node) ||
+    ts.isLiteralTypeNode(node) ||
+    ts.isTypeLiteralNode(node)
+  ) {
+    return true;
+  }
+
+  if (ts.isParenthesizedTypeNode(node) || ts.isTypeOperatorNode(node)) {
+    return isPortableTypeNode(node.type);
+  }
+
+  if (ts.isUnionTypeNode(node) || ts.isIntersectionTypeNode(node)) {
+    return node.types.every((typeNode) => isPortableTypeNode(typeNode));
+  }
+
+  if (ts.isTypeReferenceNode(node)) {
+    if (
+      !ts.isIdentifier(node.typeName) ||
+      !SAFE_TYPE_REFERENCE_NAMES.has(node.typeName.text)
+    ) {
+      return false;
+    }
+    return (node.typeArguments ?? []).every((typeArgument) =>
+      isPortableTypeNode(typeArgument),
+    );
+  }
+
+  return false;
+}
+
+function isKeywordTypeNode(node: ts.TypeNode): boolean {
+  switch (node.kind) {
+    case ts.SyntaxKind.AnyKeyword:
+    case ts.SyntaxKind.BigIntKeyword:
+    case ts.SyntaxKind.BooleanKeyword:
+    case ts.SyntaxKind.NeverKeyword:
+    case ts.SyntaxKind.NumberKeyword:
+    case ts.SyntaxKind.ObjectKeyword:
+    case ts.SyntaxKind.StringKeyword:
+    case ts.SyntaxKind.SymbolKeyword:
+    case ts.SyntaxKind.UndefinedKeyword:
+    case ts.SyntaxKind.UnknownKeyword:
+    case ts.SyntaxKind.VoidKeyword:
+      return true;
+    default:
+      return false;
+  }
+}
+
+function isFrontendCodeFile(filePath: string): boolean {
+  const extension = path.posix.extname(filePath).toLowerCase();
+  return (
+    extension === ".js" ||
+    extension === ".jsx" ||
+    extension === ".ts" ||
+    extension === ".tsx"
+  );
+}
+
+function isTypeCheckableBackendRunnable(
+  runnable: AppDiagnosticRunnable | undefined,
+): boolean {
+  if (!runnable || runnable.type !== "inline") {
+    return false;
+  }
+  const language = runnable.inlineScript?.language?.toLowerCase() ?? "";
+  return TS_LIKE_LANGUAGES.has(language) || JS_LIKE_LANGUAGES.has(language);
+}
+
+function getBackendFileExtension(language: string | undefined): string {
+  const normalizedLanguage = language?.toLowerCase() ?? "";
+  return JS_LIKE_LANGUAGES.has(normalizedLanguage) ? "js" : "ts";
+}
+
+function getScriptKindForLanguage(language: string | undefined): ts.ScriptKind {
+  const normalizedLanguage = language?.toLowerCase() ?? "";
+  return JS_LIKE_LANGUAGES.has(normalizedLanguage)
+    ? ts.ScriptKind.JS
+    : ts.ScriptKind.TS;
+}

ai_evals/core/backendValidation.ts

@@ -1,4 +1,8 @@
 import type { EvalMode } from "./types";
+import {
+  parsePositiveInteger,
+  resolveWindmillBackendSettings,
+} from "./windmillBackendSettings";
 
 export const BACKEND_VALIDATION_MODES = ["off", "preview"] as const;
 
@@ -16,10 +20,17 @@ export interface BackendValidationSettings {
   maxWaitMs: number;
 }
 
-export function parseBackendValidationMode(value?: string | null): BackendValidationMode {
+export function parseBackendValidationMode(
+  value?: string | null,
+): BackendValidationMode {
   const normalized = value?.trim().toLowerCase();
 
-  if (!normalized || normalized === "off" || normalized === "false" || normalized === "0") {
+  if (
+    !normalized ||
+    normalized === "off" ||
+    normalized === "false" ||
+    normalized === "0"
+  ) {
     return "off";
   }
 
@@ -28,7 +39,7 @@ export function parseBackendValidationMode(value?: string | null): BackendValida
   }
 
   throw new Error(
-    `Unsupported backend validation mode: ${value}. Use one of: ${BACKEND_VALIDATION_MODES.join(", ")}`
+    `Unsupported backend validation mode: ${value}. Use one of: ${BACKEND_VALIDATION_MODES.join(", ")}`,
   );
 }
 
@@ -37,68 +48,29 @@ export function resolveBackendValidationSettings(input: {
   requestedMode?: string | null;
 }): BackendValidationSettings {
   const mode = parseBackendValidationMode(
-    input.requestedMode ?? process.env.WMILL_AI_EVAL_BACKEND_VALIDATION
+    input.requestedMode ?? process.env.WMILL_AI_EVAL_BACKEND_VALIDATION,
   );
 
-  if (mode !== "off" && input.evalMode !== "flow" && input.evalMode !== "script") {
+  if (
+    mode !== "off" &&
+    input.evalMode !== "flow" &&
+    input.evalMode !== "script"
+  ) {
     throw new Error(
-      `Backend validation mode "${mode}" is only supported for flow and script evals`
+      `Backend validation mode "${mode}" is only supported for flow and script evals`,
     );
   }
 
   return {
     mode,
-    baseUrl: normalizeBaseUrl(
-      process.env.WMILL_AI_EVAL_BACKEND_URL ??
-        process.env.WINDMILL_URL ??
-        process.env.WINDMILL_BASE_URL ??
-        process.env.REMOTE ??
-        "http://127.0.0.1:8000"
-    ),
-    email: process.env.WMILL_AI_EVAL_BACKEND_EMAIL ?? "admin@windmill.dev",
-    password: process.env.WMILL_AI_EVAL_BACKEND_PASSWORD ?? "changeme",
-    keepWorkspaces: isTruthy(process.env.WMILL_AI_EVAL_KEEP_WORKSPACES),
-    workspaceOverride: sanitizeOptionalWorkspaceId(process.env.WMILL_AI_EVAL_BACKEND_WORKSPACE),
-    workspacePrefix: sanitizeWorkspacePrefix(
-      process.env.WMILL_AI_EVAL_WORKSPACE_PREFIX ?? "ai-evals"
-    ),
+    ...resolveWindmillBackendSettings(),
     pollIntervalMs: parsePositiveInteger(
       process.env.WMILL_AI_EVAL_BACKEND_POLL_INTERVAL_MS,
-      2000
+      2000,
+    ),
+    maxWaitMs: parsePositiveInteger(
+      process.env.WMILL_AI_EVAL_BACKEND_MAX_WAIT_MS,
+      120000,
     ),
-    maxWaitMs: parsePositiveInteger(process.env.WMILL_AI_EVAL_BACKEND_MAX_WAIT_MS, 120000),
   };
 }
-
-function normalizeBaseUrl(value: string): string {
-  return value.replace(/\/+$/, "");
-}
-
-function sanitizeWorkspacePrefix(value: string): string {
-  const sanitized = value
-    .trim()
-    .toLowerCase()
-    .replace(/[^a-z0-9-]+/g, "-")
-    .replace(/^-+|-+$/g, "");
-  return sanitized.length > 0 ? sanitized : "ai-evals";
-}
-
-function sanitizeOptionalWorkspaceId(value: string | undefined): string | undefined {
-  const trimmed = value?.trim();
-  return trimmed ? trimmed : undefined;
-}
-
-function isTruthy(value: string | undefined): boolean {
-  if (!value) {
-    return false;
-  }
-  return ["1", "true", "yes", "on"].includes(value.trim().toLowerCase());
-}
-
-function parsePositiveInteger(value: string | undefined, fallback: number): number {
-  if (!value) {
-    return fallback;
-  }
-  const parsed = Number(value);
-  return Number.isInteger(parsed) && parsed > 0 ? parsed : fallback;
-}

ai_evals/core/cases.test.ts

@@ -15,4 +15,191 @@ describe("loadCases", () => {
       },
     });
   });
+
+  it("loads the workspace-flow preference benchmark case", async () => {
+    const flowCases = await loadCases("flow");
+    const caseEntry = flowCases.find(
+      (entry) => entry.id === "flow-test13-prefer-existing-workspace-flow"
+    );
+
+    expect(caseEntry).toBeDefined();
+    expect(caseEntry?.runtime).toEqual({
+      backendPreview: {
+        args: {
+          a: 10,
+          b: 5,
+        },
+      },
+    });
+    expect(caseEntry?.initialPath).toContain(
+      "ai_evals/fixtures/frontend/flow/initial/test13_prefer_existing_workspace_flow_initial.json"
+    );
+    expect(caseEntry?.expectedPath).toContain(
+      "ai_evals/fixtures/frontend/flow/expected/test13_prefer_existing_workspace_flow.json"
+    );
+  });
+
+  it("loads app validation config for datatable-backed persistence cases", async () => {
+    const appCases = await loadCases("app");
+    const caseEntry = appCases.find(
+      (entry) => entry.id === "app-test8-inventory-tracker-search-delete"
+    );
+
+    expect(caseEntry?.initialPath).toContain("ai_evals/fixtures/frontend/app/initial/inventory_tracker");
+    expect(caseEntry?.validate).toEqual({
+      requiredFrontendPaths: ["/index.tsx"],
+      requiredBackendRunnableKeys: ["listInventory", "addInventory", "deleteInventory"],
+      requiredBackendRunnableTypes: [
+        { key: "listInventory", type: "inline" },
+        { key: "addInventory", type: "inline" },
+        { key: "deleteInventory", type: "inline" },
+      ],
+      requiredDatatables: [
+        {
+          datatableName: "main",
+          schema: "public",
+          table: "inventory_items",
+        },
+      ],
+    });
+  });
+
+  it("loads the seeded recipe-book app modification case", async () => {
+    const appCases = await loadCases("app");
+    const caseEntry = appCases.find((entry) => entry.id === "app-test9-recipe-book-search-delete");
+
+    expect(caseEntry?.initialPath).toContain("ai_evals/fixtures/frontend/app/initial/recipe_book");
+    expect(caseEntry?.validate).toEqual({
+      requiredFrontendPaths: ["/index.tsx"],
+      requiredBackendRunnableKeys: ["listRecipes", "addRecipe", "deleteRecipe"],
+      requiredBackendRunnableTypes: [
+        { key: "listRecipes", type: "inline" },
+        { key: "addRecipe", type: "inline" },
+        { key: "deleteRecipe", type: "inline" },
+      ],
+      requiredDatatables: [
+        {
+          datatableName: "main",
+          schema: "public",
+          table: "recipes",
+        },
+      ],
+    });
+  });
+
+  it("loads the file-manager rename save/cancel case", async () => {
+    const appCases = await loadCases("app");
+    const caseEntry = appCases.find(
+      (entry) => entry.id === "app-test6-file-manager-rename-save-cancel"
+    );
+
+    expect(caseEntry?.initialPath).toContain("ai_evals/fixtures/frontend/app/initial/file_manager");
+    expect(caseEntry?.validate).toMatchObject({
+      requiredFrontendPaths: ["/index.tsx", "/components/FileItem.tsx"],
+      requiredFrontendFileContent: [
+        {
+          path: "/components/FileItem.tsx",
+          includes: ["Save", "Cancel", "Escape"],
+        },
+      ],
+      forbiddenAppContent: ["onBlur={handleRename}"],
+    });
+  });
+
+  it("loads the datatable-backed notes creation case", async () => {
+    const appCases = await loadCases("app");
+    const caseEntry = appCases.find((entry) => entry.id === "app-datatable-persistent-notes");
+
+    expect(caseEntry?.initialPath).toContain("ai_evals/fixtures/frontend/app/initial/notes_datatable");
+    expect(caseEntry?.runtime).toEqual({
+      maxTurns: 10,
+    });
+    expect(caseEntry?.validate).toMatchObject({
+      requiredFrontendPaths: ["/index.tsx"],
+      requiredBackendRunnableKeys: ["listNotes", "addNote", "deleteNote"],
+      datatableTableCountExactly: 1,
+      requiredDatatables: [
+        {
+          datatableName: "main",
+          schema: "public",
+          table: "notes",
+        },
+      ],
+      requiredToolsUsed: ["list_datatables", "get_datatable_table_schema"],
+      forbiddenAppContent: ["localStorage", "sessionStorage", "indexedDB"],
+    });
+  });
+
+  it("loads the session id micro-edit app case", async () => {
+    const appCases = await loadCases("app");
+    const caseEntry = appCases.find((entry) => entry.id === "app-test10-session-id-no-crypto");
+
+    expect(caseEntry?.initialPath).toContain("ai_evals/fixtures/frontend/app/initial/session_id_chat");
+    expect(caseEntry?.runtime).toEqual({
+      maxTurns: 4,
+    });
+    expect(caseEntry?.validate).toEqual({
+      requiredFrontendPaths: ["/index.tsx"],
+      requiredBackendRunnableKeys: ["a"],
+      requiredBackendRunnableTypes: [{ key: "a", type: "inline" }],
+    });
+  });
+
+  it("loads app token usage cases with additional runtime context", async () => {
+    const appCases = await loadCases("app");
+    const datatableContextCase = appCases.find(
+      (entry) => entry.id === "app-token-many-datatable-context"
+    );
+
+    expect(
+      appCases.find((entry) => entry.id === "app-token-selected-large-frontend-context")
+    ).toBeUndefined();
+    expect(
+      appCases.find((entry) => entry.id === "app-token-selected-large-backend-context")
+    ).toBeUndefined();
+    expect(datatableContextCase?.initialPath).toContain(
+      "ai_evals/fixtures/frontend/app/initial/token_heavy_datatables"
+    );
+    expect(datatableContextCase?.runtime?.appContext?.additional).toHaveLength(10);
+    expect(datatableContextCase?.runtime?.appContext?.additional?.[0]).toEqual({
+      type: "datatable",
+      datatableName: "main",
+      schema: "analytics",
+      table: "event_log_01",
+    });
+  });
+
+  it("loads CLI behavior expectations for deploy-guidance cases", async () => {
+    const cliCases = await loadCases("cli");
+    const caseEntry = cliCases.find((entry) => entry.id === "bun-hello-script");
+
+    expect(caseEntry?.cliExpect).toEqual({
+      requiredSkills: ["write-script-bun"],
+      requiredSkillsBeforeFirstMutation: ["write-script-bun"],
+      forbiddenSkills: ["write-script-python3", "write-flow"],
+      orderedAssistantMentions: ["wmill generate-metadata", "wmill sync push"],
+      orderedProposedCommands: ["wmill generate-metadata", "wmill sync push"],
+      forbiddenExecutedCommands: ["^wmill generate-metadata", "^wmill sync push"],
+    });
+  });
+
+  it("loads tool expectations for workspace mutation cases", async () => {
+    const scriptCases = await loadCases("script");
+    const caseEntry = scriptCases.find(
+      (entry) => entry.id === "script-test2-create-current-script-schedule"
+    );
+
+    expect(caseEntry?.toolExpect).toEqual({
+      requiredToolsUsed: ["create_schedule"],
+      toolCallArgs: [
+        {
+          tool: "create_schedule",
+          field: "path",
+          stringStartsWithAnyOf: ["f/", "u/"],
+          stringMustNotStartWithAnyOf: ["schedules/"],
+        },
+      ],
+    });
+    expect(caseEntry?.skipJudge).toBe(true);
+  });
 });

ai_evals/core/cases.ts

@@ -2,7 +2,13 @@ import { readFile } from "node:fs/promises";
 import path from "node:path";
 import { fileURLToPath } from "node:url";
 import { parse } from "yaml";
-import type { EvalCase, EvalCaseRuntimeSpec, EvalMode, FlowValidationSpec } from "./types";
+import type {
+  CliValidationSpec,
+  EvalCase,
+  EvalCaseRuntimeSpec,
+  EvalMode,
+  EvalValidationSpec,
+} from "./types";
 
 const REPO_ROOT = fileURLToPath(new URL("../../", import.meta.url));
 const CASES_DIR = path.join(REPO_ROOT, "ai_evals", "cases");
@@ -12,11 +18,13 @@ interface RawEvalCase {
   prompt: string;
   initial?: string;
   expected?: string;
-  validate?: FlowValidationSpec;
+  validate?: EvalValidationSpec;
+  toolExpect?: EvalCase["toolExpect"];
+  cliExpect?: CliValidationSpec;
   judgeChecklist?: string[];
+  skipJudge?: boolean;
   runtime?: EvalCaseRuntimeSpec;
 }
-
 export function getRepoRoot(): string {
   return REPO_ROOT;
 }
@@ -34,13 +42,16 @@ export async function loadCases(mode: EvalMode): Promise<EvalCase[]> {
     throw new Error(`Expected ${filePath} to contain a YAML list of cases`);
   }
 
-  return parsed.map((entry) => ({
+  return (parsed as RawEvalCase[]).map((entry) => ({
     id: entry.id,
     prompt: entry.prompt,
     initialPath: resolveFixturePath(entry.initial),
     expectedPath: resolveFixturePath(entry.expected),
     validate: entry.validate,
+    toolExpect: entry.toolExpect,
+    cliExpect: entry.cliExpect,
     judgeChecklist: entry.judgeChecklist,
+    skipJudge: entry.skipJudge,
     runtime: entry.runtime,
   }));
 }

ai_evals/core/frontendTransport.test.ts

@@ -0,0 +1,64 @@
+import { afterEach, describe, expect, it } from "bun:test";
+import {
+  parseFrontendEvalTransport,
+  resolveFrontendEvalTransportSettings,
+} from "./frontendTransport";
+
+const ORIGINAL_ENV = {
+  WMILL_AI_EVAL_BACKEND_URL: process.env.WMILL_AI_EVAL_BACKEND_URL,
+};
+
+afterEach(() => {
+  if (ORIGINAL_ENV.WMILL_AI_EVAL_BACKEND_URL === undefined) {
+    delete process.env.WMILL_AI_EVAL_BACKEND_URL;
+  } else {
+    process.env.WMILL_AI_EVAL_BACKEND_URL =
+      ORIGINAL_ENV.WMILL_AI_EVAL_BACKEND_URL;
+  }
+});
+
+describe("parseFrontendEvalTransport", () => {
+  it("defaults to direct when unset", () => {
+    expect(parseFrontendEvalTransport(undefined)).toBe("direct");
+  });
+
+  it("accepts proxy explicitly", () => {
+    expect(parseFrontendEvalTransport("proxy")).toBe("proxy");
+  });
+
+  it("rejects unsupported values", () => {
+    expect(() => parseFrontendEvalTransport("worker")).toThrow(
+      "Unsupported frontend eval transport: worker",
+    );
+  });
+});
+
+describe("resolveFrontendEvalTransportSettings", () => {
+  it("includes backend settings for proxy transport", () => {
+    process.env.WMILL_AI_EVAL_BACKEND_URL = "http://127.0.0.1:8000/";
+
+    expect(
+      resolveFrontendEvalTransportSettings({
+        evalMode: "app",
+        requestedTransport: "proxy",
+      }),
+    ).toMatchObject({
+      transport: "proxy",
+      backend: {
+        baseUrl: "http://127.0.0.1:8000",
+      },
+    });
+  });
+
+  it("keeps direct transport for cli runs", () => {
+    expect(
+      resolveFrontendEvalTransportSettings({
+        evalMode: "cli",
+        requestedTransport: "direct",
+      }),
+    ).toEqual({
+      transport: "direct",
+      backend: undefined,
+    });
+  });
+});

ai_evals/core/frontendTransport.ts

@@ -0,0 +1,49 @@
+import type { EvalMode } from "./types";
+import type { WindmillBackendSettings } from "./windmillBackendSettings";
+import { resolveWindmillBackendSettings } from "./windmillBackendSettings";
+
+export const FRONTEND_EVAL_TRANSPORTS = ["direct", "proxy"] as const;
+
+export type FrontendEvalTransport = (typeof FRONTEND_EVAL_TRANSPORTS)[number];
+
+export interface FrontendEvalTransportSettings {
+  transport: FrontendEvalTransport;
+  backend?: WindmillBackendSettings;
+}
+
+export function parseFrontendEvalTransport(
+  value?: string | null,
+): FrontendEvalTransport {
+  const normalized = value?.trim().toLowerCase();
+
+  if (!normalized || normalized === "direct") {
+    return "direct";
+  }
+
+  if (normalized === "proxy") {
+    return "proxy";
+  }
+
+  throw new Error(
+    `Unsupported frontend eval transport: ${value}. Use one of: ${FRONTEND_EVAL_TRANSPORTS.join(", ")}`,
+  );
+}
+
+export function resolveFrontendEvalTransportSettings(input: {
+  evalMode: EvalMode;
+  requestedTransport?: string | null;
+}): FrontendEvalTransportSettings {
+  const transport = parseFrontendEvalTransport(input.requestedTransport);
+
+  if (transport === "proxy" && input.evalMode === "cli") {
+    throw new Error(
+      'Frontend eval transport "proxy" is only supported for flow, script, and app evals',
+    );
+  }
+
+  return {
+    transport,
+    backend:
+      transport === "proxy" ? resolveWindmillBackendSettings() : undefined,
+  };
+}

ai_evals/core/judge.ts

@@ -34,7 +34,16 @@ export async function judgeOutput(input: {
     "If a checklist is provided, treat it as the explicit acceptance criteria for this case.",
     "Be strict about missing requested functionality.",
     "When the prompt wording is ambiguous, prefer the checklist over inferred structural requirements.",
+    "Do not invent additional Windmill-specific constraints that are not explicit in the prompt, checklist, or expected state.",
+    "Do not lower the score just because the output uses a different but valid Windmill idiom, naming choice, or equivalent field shape.",
     "Do not require exact ids, exact topology, or exact field names unless the prompt, checklist, or expected state clearly requires them.",
+    ...(input.mode === "app"
+      ? [
+          "For raw app outputs, datatable-backed persistence is a valid Windmill pattern when the app artifact configures datatables.",
+          "Do not mark `wmill.datatable()` usage as fabricated or invalid by itself.",
+          "Judge app persistence against the artifact that was actually produced, including any configured datatables.",
+        ]
+      : []),
     `Always respond by calling the ${JUDGE_TOOL_NAME} tool exactly once.`,
   ].join("\n\n");
 

ai_evals/core/results.ts

@@ -12,26 +12,34 @@ import type {
 
 export async function writeRunResult(
   result: BenchmarkRunResult,
-  outputPath?: string
+  outputPath?: string,
 ): Promise<string> {
   const targetPath = resolveRunOutputPath(result.mode, outputPath);
   await mkdir(path.dirname(targetPath), { recursive: true });
-  await writeFile(targetPath, JSON.stringify(toSerializableRunResult(result), null, 2) + "\n", "utf8");
+  await writeFile(
+    targetPath,
+    JSON.stringify(toSerializableRunResult(result), null, 2) + "\n",
+    "utf8",
+  );
   return targetPath;
 }
 
 export async function appendHistoryRecord(
   result: BenchmarkRunResult,
-  historyPath = resolveHistoryPath(result.mode)
+  historyPath = resolveHistoryPath(result.mode),
 ): Promise<string> {
   await mkdir(path.dirname(historyPath), { recursive: true });
-  await appendFile(historyPath, JSON.stringify(toHistoryRecord(result)) + "\n", "utf8");
+  await appendFile(
+    historyPath,
+    JSON.stringify(toHistoryRecord(result)) + "\n",
+    "utf8",
+  );
   return historyPath;
 }
 
 export async function writeRunArtifacts(
   result: BenchmarkRunResult,
-  outputPath?: string
+  outputPath?: string,
 ): Promise<string | null> {
   const targetPath = resolveRunOutputPath(result.mode, outputPath);
   const artifactRoot = defaultArtifactsRoot(targetPath);
@@ -47,7 +55,11 @@ export async function writeRunArtifacts(
         continue;
       }
 
-      const attemptDir = path.join(artifactRoot, caseResult.id, `attempt-${attempt.attempt}`);
+      const attemptDir = path.join(
+        artifactRoot,
+        caseResult.id,
+        `attempt-${attempt.attempt}`,
+      );
       await writeArtifactFiles(attemptDir, artifactFiles);
       attempt.artifactsPath = attemptDir;
       wroteArtifacts = true;
@@ -62,17 +74,24 @@ export function buildRunResult(input: {
   mode: EvalMode;
   runs: number;
   runModel: string | null;
+  transport?: BenchmarkRunResult["transport"];
   judgeModel: string | null;
   caseResults: BenchmarkCaseResult[];
 }): BenchmarkRunResult {
-  const attemptCount = input.caseResults.reduce((sum, entry) => sum + entry.attempts.length, 0);
+  const attemptCount = input.caseResults.reduce(
+    (sum, entry) => sum + entry.attempts.length,
+    0,
+  );
   const passedAttempts = input.caseResults.reduce(
-    (sum, entry) => sum + entry.attempts.filter((attempt) => attempt.passed).length,
-    0
+    (sum, entry) =>
+      sum + entry.attempts.filter((attempt) => attempt.passed).length,
+    0,
   );
   const durationTotal = input.caseResults.reduce(
-    (sum, entry) => sum + entry.attempts.reduce((inner, attempt) => inner + attempt.durationMs, 0),
-    0
+    (sum, entry) =>
+      sum +
+      entry.attempts.reduce((inner, attempt) => inner + attempt.durationMs, 0),
+    0,
   );
   const tokenUsageTotal = input.caseResults.reduce<BenchmarkTokenUsage | null>(
     (sum, entry) => {
@@ -87,7 +106,7 @@ export function buildRunResult(input: {
       }
       return sum;
     },
-    null
+    null,
   );
 
   return {
@@ -97,6 +116,7 @@ export function buildRunResult(input: {
     gitSha: getGitSha(),
     runs: input.runs,
     runModel: input.runModel,
+    transport: input.transport ?? null,
     judgeModel: input.judgeModel,
     caseCount: input.caseResults.length,
     attemptCount,
@@ -122,6 +142,9 @@ export function formatRunSummary(result: BenchmarkRunResult): string {
     `Pass rate: ${formatPercent(result.passRate)} (${result.passedAttempts}/${result.attemptCount})`,
     `Average duration: ${Math.round(result.averageDurationMs)}ms`,
   ];
+  if (result.transport) {
+    lines.splice(1, 0, `Transport: ${result.transport}`);
+  }
 
   const failures = collectFailures(result);
   if (failures.length > 0) {
@@ -142,9 +165,11 @@ function collectFailures(result: BenchmarkRunResult): string[] {
       if (attempt.passed) {
         continue;
       }
-      const failedChecks = attempt.checks.filter((check) => !check.passed).map((check) => check.name);
+      const failedChecks = attempt.checks
+        .filter((check) => !check.passed)
+        .map((check) => check.name);
       failures.push(
-        `${caseResult.id} attempt ${attempt.attempt}: ${failedChecks.join(", ") || attempt.error || "failed"}`
+        `${caseResult.id} attempt ${attempt.attempt}: ${failedChecks.join(", ") || attempt.error || "failed"}`,
       );
     }
   }
@@ -156,8 +181,13 @@ function defaultFileName(mode: EvalMode): string {
   return `${new Date().toISOString().replaceAll(":", "-")}__${mode}.json`;
 }
 
-export function resolveRunOutputPath(mode: EvalMode, outputPath?: string): string {
-  return outputPath ?? path.join(getAiEvalsRoot(), "results", defaultFileName(mode));
+export function resolveRunOutputPath(
+  mode: EvalMode,
+  outputPath?: string,
+): string {
+  return (
+    outputPath ?? path.join(getAiEvalsRoot(), "results", defaultFileName(mode))
+  );
 }
 
 export function resolveHistoryPath(mode: EvalMode): string {
@@ -172,7 +202,7 @@ function defaultArtifactsRoot(resultPath: string): string {
 
 async function writeArtifactFiles(
   rootDir: string,
-  files: BenchmarkArtifactFile[]
+  files: BenchmarkArtifactFile[],
 ): Promise<void> {
   for (const file of files) {
     const relativePath = normalizeArtifactPath(file.path);
@@ -185,18 +215,25 @@ async function writeArtifactFiles(
 function normalizeArtifactPath(filePath: string): string {
   const normalized = filePath.replaceAll("\\", "/").replace(/^\/+/, "");
   const parts = normalized.split("/").filter(Boolean);
-  if (parts.length === 0 || parts.some((part) => part === "." || part === "..")) {
+  if (
+    parts.length === 0 ||
+    parts.some((part) => part === "." || part === "..")
+  ) {
     throw new Error(`Invalid artifact path: ${filePath}`);
   }
   return parts.join("/");
 }
 
-function toSerializableRunResult(result: BenchmarkRunResult): BenchmarkRunResult {
+function toSerializableRunResult(
+  result: BenchmarkRunResult,
+): BenchmarkRunResult {
   return {
     ...result,
     cases: result.cases.map((caseResult) => ({
       ...caseResult,
-      attempts: caseResult.attempts.map(({ artifactFiles, ...attempt }) => attempt),
+      attempts: caseResult.attempts.map(
+        ({ artifactFiles, ...attempt }) => attempt,
+      ),
     })),
   };
 }
@@ -204,8 +241,8 @@ function toSerializableRunResult(result: BenchmarkRunResult): BenchmarkRunResult
 function toHistoryRecord(result: BenchmarkRunResult) {
   const judgeScores = result.cases.flatMap((caseResult) =>
     caseResult.attempts.flatMap((attempt) =>
-      typeof attempt.judgeScore === "number" ? [attempt.judgeScore] : []
-    )
+      typeof attempt.judgeScore === "number" ? [attempt.judgeScore] : [],
+    ),
   );
 
   return {
@@ -214,6 +251,7 @@ function toHistoryRecord(result: BenchmarkRunResult) {
     mode: result.mode,
     runs: result.runs,
     runModel: result.runModel,
+    transport: result.transport,
     judgeModel: result.judgeModel,
     caseCount: result.caseCount,
     attemptCount: result.attemptCount,
@@ -223,49 +261,57 @@ function toHistoryRecord(result: BenchmarkRunResult) {
     averageJudgeScore:
       judgeScores.length === 0
         ? null
-        : judgeScores.reduce((sum, score) => sum + score, 0) / judgeScores.length,
+        : judgeScores.reduce((sum, score) => sum + score, 0) /
+          judgeScores.length,
     averageTokenUsagePerAttempt: result.averageTokenUsagePerAttempt ?? null,
     failedCaseIds: Array.from(
       new Set(
         result.cases
-          .filter((caseResult) => caseResult.attempts.some((attempt) => !attempt.passed))
-          .map((caseResult) => caseResult.id)
-      )
+          .filter((caseResult) =>
+            caseResult.attempts.some((attempt) => !attempt.passed),
+          )
+          .map((caseResult) => caseResult.id),
+      ),
     ),
     cases: result.cases.map((caseResult) => {
       const attemptCount = caseResult.attempts.length;
-      const passedAttempts = caseResult.attempts.filter((attempt) => attempt.passed).length;
+      const passedAttempts = caseResult.attempts.filter(
+        (attempt) => attempt.passed,
+      ).length;
       const totalDurationMs = caseResult.attempts.reduce(
         (sum, attempt) => sum + attempt.durationMs,
-        0
+        0,
       );
       const judgeScores = caseResult.attempts.flatMap((attempt) =>
-        typeof attempt.judgeScore === "number" ? [attempt.judgeScore] : []
+        typeof attempt.judgeScore === "number" ? [attempt.judgeScore] : [],
       );
-      const totalTokenUsage = caseResult.attempts.reduce<BenchmarkTokenUsage | null>(
-        (sum, attempt) => {
-          if (!attempt.tokenUsage) {
+      const totalTokenUsage =
+        caseResult.attempts.reduce<BenchmarkTokenUsage | null>(
+          (sum, attempt) => {
+            if (!attempt.tokenUsage) {
+              return sum;
+            }
+            sum ??= { prompt: 0, completion: 0, total: 0 };
+            sum.prompt += attempt.tokenUsage.prompt;
+            sum.completion += attempt.tokenUsage.completion;
+            sum.total += attempt.tokenUsage.total;
             return sum;
-          }
-          sum ??= { prompt: 0, completion: 0, total: 0 };
-          sum.prompt += attempt.tokenUsage.prompt;
-          sum.completion += attempt.tokenUsage.completion;
-          sum.total += attempt.tokenUsage.total;
-          return sum;
-        },
-        null
-      );
+          },
+          null,
+        );
 
       return {
         id: caseResult.id,
         attemptCount,
         passedAttempts,
         passRate: attemptCount === 0 ? 0 : passedAttempts / attemptCount,
-        averageDurationMs: attemptCount === 0 ? 0 : totalDurationMs / attemptCount,
+        averageDurationMs:
+          attemptCount === 0 ? 0 : totalDurationMs / attemptCount,
         averageJudgeScore:
           judgeScores.length === 0
             ? null
-            : judgeScores.reduce((sum, score) => sum + score, 0) / judgeScores.length,
+            : judgeScores.reduce((sum, score) => sum + score, 0) /
+              judgeScores.length,
         averageTokenUsagePerAttempt:
           attemptCount === 0 || !totalTokenUsage
             ? null

ai_evals/core/runSuite.ts

@@ -7,6 +7,7 @@ import type {
   FrontendBenchmarkProgressEvent,
   ModeRunner,
 } from "./types";
+import { validateToolExpectations } from "./validators";
 
 export async function runSuite<TInitial, TExpected, TActual>(input: {
   modeRunner: ModeRunner<TInitial, TExpected, TActual>;
@@ -100,6 +101,7 @@ async function runCaseAttempts<TInitial, TExpected, TActual>(input: {
       const initial = await input.modeRunner.loadInitial(input.evalCase.initialPath);
       const expected = await input.modeRunner.loadExpected(input.evalCase.expectedPath);
       const run = await input.modeRunner.run(input.evalCase.prompt, initial, {
+        evalCase: input.evalCase,
         caseId: input.evalCase.id,
         caseNumber: input.caseIndex + 1,
         totalCases: input.totalCases,
@@ -143,6 +145,20 @@ async function runCaseAttempts<TInitial, TExpected, TActual>(input: {
                 runs: input.runs,
               })
           : undefined,
+        onToolCall: input.verbose && surface
+          ? ({ toolName, argumentsText }) =>
+              input.onProgress?.({
+                type: "tool-call",
+                surface,
+                caseId: input.evalCase.id,
+                caseNumber: input.caseIndex + 1,
+                totalCases: input.totalCases,
+                attempt,
+                runs: input.runs,
+                toolName,
+                argumentsText,
+              })
+          : undefined,
       });
       const checks: BenchmarkCheck[] = [
         buildCheck("run succeeded", run.success, run.error),
@@ -154,6 +170,10 @@ async function runCaseAttempts<TInitial, TExpected, TActual>(input: {
           actual: run.actual,
           run,
         }),
+        ...validateToolExpectations({
+          run,
+          toolExpect: input.evalCase.toolExpect,
+        }),
       ];
       const artifactFiles = input.modeRunner.buildArtifacts?.(run.actual) ?? [];
 
@@ -167,6 +187,7 @@ async function runCaseAttempts<TInitial, TExpected, TActual>(input: {
             actual: run.actual,
             run,
             context: {
+              evalCase: input.evalCase,
               caseId: input.evalCase.id,
               caseNumber: input.caseIndex + 1,
               totalCases: input.totalCases,
@@ -197,7 +218,7 @@ async function runCaseAttempts<TInitial, TExpected, TActual>(input: {
       let judgeScore: number | null = null;
       let judgeSummary: string | null = null;
 
-      if (run.success) {
+      if (run.success && !input.evalCase.skipJudge) {
         const judge = await judgeOutput({
           mode: input.modeRunner.mode,
           prompt: input.evalCase.prompt,
@@ -227,6 +248,7 @@ async function runCaseAttempts<TInitial, TExpected, TActual>(input: {
         assistantMessageCount: run.assistantMessageCount,
         toolCallCount: run.toolCallCount,
         toolsUsed: uniqueStrings(run.toolsUsed),
+        toolCallDetails: run.toolCallDetails,
         skillsInvoked: uniqueStrings(run.skillsInvoked),
         checks,
         judgeScore,

ai_evals/core/types.ts

@@ -1,14 +1,37 @@
 export const EVAL_MODES = ["cli", "flow", "script", "app"] as const;
 
 export type EvalMode = (typeof EVAL_MODES)[number];
+export type FrontendEvalTransport = "direct" | "proxy";
 
 export interface EvalCaseRuntimeBackendPreview {
   args?: Record<string, unknown>;
   timeoutSeconds?: number;
 }
 
+export type EvalCaseRuntimeAppAdditionalContext =
+  | {
+      type: "frontend";
+      path: string;
+    }
+  | {
+      type: "backend";
+      key: string;
+    }
+  | {
+      type: "datatable";
+      datatableName: string;
+      schema: string;
+      table: string;
+    };
+
+export interface EvalCaseRuntimeAppContextSpec {
+  additional?: EvalCaseRuntimeAppAdditionalContext[];
+}
+
 export interface EvalCaseRuntimeSpec {
+  maxTurns?: number;
   backendPreview?: EvalCaseRuntimeBackendPreview;
+  appContext?: EvalCaseRuntimeAppContextSpec;
 }
 
 export interface FlowValidationSpec {
@@ -16,6 +39,39 @@ export interface FlowValidationSpec {
   schemaAnyOf?: Array<{
     requiredPaths: string[];
   }>;
+  exactTopLevelStepIds?: string[];
+  topLevelStepIds?: string[];
+  topLevelStepOrder?: string[];
+  topLevelStepTypeCountsAtLeast?: Array<{
+    type: string;
+    count: number;
+  }>;
+  topLevelStepTypes?: Array<{
+    id: string;
+    type: string;
+  }>;
+  moduleRules?: Array<{
+    id: string;
+    hasStopAfterIf?: boolean;
+    hasStopAfterAllItersIf?: boolean;
+    immediateChildStepIds?: string[];
+    exactImmediateChildStepIds?: string[];
+    immediateChildStepTypes?: Array<{
+      id: string;
+      type: string;
+    }>;
+    requiredInputTransforms?: Array<{
+      type?: string;
+      expr?: string;
+      exprAnyOf?: string[];
+      value?: string | number | boolean | null;
+    }>;
+  }>;
+  moduleFieldRules?: Array<{
+    id: string;
+    path: string;
+    equals: string | number | boolean | null;
+  }>;
   resolveResultsRefs?: boolean;
   requireSpecialModules?: Array<"preprocessor_module" | "failure_module">;
   requireSuspendSteps?: Array<{
@@ -25,13 +81,77 @@ export interface FlowValidationSpec {
   }>;
 }
 
+export interface AppValidationSpec {
+  requiredFrontendPaths?: string[];
+  requiredFrontendFileContent?: Array<{
+    path: string;
+    includes: string[];
+  }>;
+  requiredBackendRunnableKeys?: string[];
+  requiredBackendRunnableTypes?: Array<{
+    key: string;
+    type: string;
+  }>;
+  requiredBackendRunnableContent?: Array<{
+    key: string;
+    includes: string[];
+  }>;
+  backendRunnableCountAtLeast?: number;
+  datatableCountAtLeast?: number;
+  datatableTableCountAtLeast?: number;
+  datatableTableCountExactly?: number;
+  requiredDatatables?: Array<{
+    schema: string;
+    table: string;
+    datatableName?: string;
+  }>;
+  requiredToolsUsed?: string[];
+  forbiddenAppContent?: string[];
+}
+
+export interface CliValidationSpec {
+  requiredSkills?: string[];
+  forbiddenSkills?: string[];
+  requiredSkillsBeforeFirstMutation?: string[];
+  requiredAssistantMentions?: string[];
+  forbiddenAssistantMentions?: string[];
+  orderedAssistantMentions?: string[];
+  requiredProposedCommands?: string[];
+  forbiddenProposedCommands?: string[];
+  orderedProposedCommands?: string[];
+  forbiddenExecutedCommands?: string[];
+  workspaceUnchanged?: boolean;
+}
+
+export interface ToolCallDetail {
+  name: string;
+  arguments: unknown;
+}
+
+export interface ToolCallArgumentRule {
+  tool: string;
+  field: string;
+  stringStartsWithAnyOf?: string[];
+  stringMustNotStartWithAnyOf?: string[];
+}
+
+export interface ToolValidationSpec {
+  requiredToolsUsed?: string[];
+  toolCallArgs?: ToolCallArgumentRule[];
+}
+
+export type EvalValidationSpec = FlowValidationSpec | AppValidationSpec;
+
 export interface EvalCase {
   id: string;
   prompt: string;
   initialPath?: string;
   expectedPath?: string;
-  validate?: FlowValidationSpec;
+  validate?: EvalValidationSpec;
+  toolExpect?: ToolValidationSpec;
+  cliExpect?: CliValidationSpec;
   judgeChecklist?: string[];
+  skipJudge?: boolean;
   runtime?: EvalCaseRuntimeSpec;
 }
 
@@ -64,6 +184,29 @@ export interface BenchmarkTokenUsage {
   total: number;
 }
 
+export interface CliToolInvocation {
+  tool: string;
+  input: Record<string, unknown>;
+  timestamp: number;
+}
+
+export interface CliWmillInvocation {
+  argv: string[];
+  cwd: string;
+  timestamp: string;
+}
+
+export interface CliTrace {
+  toolsUsed: CliToolInvocation[];
+  skillsInvoked: string[];
+  assistantMessageCount: number;
+  bashCommands: string[];
+  proposedCommands: string[];
+  executedWmillCommands: string[];
+  wmillInvocations: CliWmillInvocation[];
+  firstMutationToolIndex: number | null;
+}
+
 export interface ModeRunOutput<TActual> {
   success: boolean;
   actual: TActual;
@@ -71,11 +214,13 @@ export interface ModeRunOutput<TActual> {
   assistantMessageCount: number;
   toolCallCount: number;
   toolsUsed: string[];
+  toolCallDetails?: ToolCallDetail[];
   skillsInvoked: string[];
   tokenUsage?: BenchmarkTokenUsage | null;
 }
 
 export interface ModeRunContext {
+  evalCase?: EvalCase;
   caseId: string;
   caseNumber: number;
   totalCases: number;
@@ -85,6 +230,7 @@ export interface ModeRunContext {
   onAssistantMessageStart?: () => void;
   onAssistantChunk?: (chunk: string) => void;
   onAssistantMessageEnd?: () => void;
+  onToolCall?: (input: { toolName: string; argumentsText: string }) => void;
 }
 
 export interface ModeRunner<TInitial, TExpected, TActual> {
@@ -96,7 +242,7 @@ export interface ModeRunner<TInitial, TExpected, TActual> {
   run(
     prompt: string,
     initial: TInitial | undefined,
-    context: ModeRunContext
+    context: ModeRunContext,
   ): Promise<ModeRunOutput<TActual>>;
   validate(input: {
     evalCase: EvalCase;
@@ -125,6 +271,7 @@ export interface BenchmarkAttemptResult {
   assistantMessageCount: number;
   toolCallCount: number;
   toolsUsed: string[];
+  toolCallDetails?: ToolCallDetail[];
   skillsInvoked: string[];
   checks: BenchmarkCheck[];
   judgeScore: number | null;
@@ -150,6 +297,7 @@ export interface BenchmarkRunResult {
   gitSha: string | null;
   runs: number;
   runModel: string | null;
+  transport: FrontendEvalTransport | null;
   judgeModel: string | null;
   caseCount: number;
   attemptCount: number;
@@ -219,4 +367,15 @@ export type FrontendBenchmarkProgressEvent =
       totalCases: number;
       attempt: number;
       runs: number;
+    }
+  | {
+      type: "tool-call";
+      surface: Exclude<EvalMode, "cli">;
+      caseId: string;
+      caseNumber: number;
+      totalCases: number;
+      attempt: number;
+      runs: number;
+      toolName: string;
+      argumentsText: string;
     };

ai_evals/core/validators.test.ts

@@ -1,5 +1,10 @@
 import { describe, expect, it } from "bun:test";
-import { validateScriptState } from "./validators";
+import {
+  validateAppState,
+  validateCliWorkspace,
+  validateScriptState,
+  validateToolExpectations,
+} from "./validators";
 
 describe("validateScriptState", () => {
   it("accepts semantically equivalent script implementations", () => {
@@ -34,3 +39,512 @@ describe("validateScriptState", () => {
     });
   });
 });
+
+describe("validateToolExpectations", () => {
+  it("accepts Windmill-prefixed schedule paths", () => {
+    const checks = validateToolExpectations({
+      run: {
+        success: true,
+        actual: {},
+        assistantMessageCount: 1,
+        toolCallCount: 1,
+        toolsUsed: ["create_schedule"],
+        toolCallDetails: [
+          {
+            name: "create_schedule",
+            arguments: {
+              path: "f/evals/greet_user_daily",
+            },
+          },
+        ],
+        skillsInvoked: [],
+      },
+      toolExpect: {
+        requiredToolsUsed: ["create_schedule"],
+        toolCallArgs: [
+          {
+            tool: "create_schedule",
+            field: "path",
+            stringStartsWithAnyOf: ["f/", "u/"],
+            stringMustNotStartWithAnyOf: ["schedules/"],
+          },
+        ],
+      },
+    });
+
+    expect(checks.every((check) => check.passed)).toBe(true);
+  });
+
+  it("rejects schedule-prefixed tool paths", () => {
+    const checks = validateToolExpectations({
+      run: {
+        success: true,
+        actual: {},
+        assistantMessageCount: 1,
+        toolCallCount: 1,
+        toolsUsed: ["create_schedule"],
+        toolCallDetails: [
+          {
+            name: "create_schedule",
+            arguments: {
+              path: "schedules/greet_user_daily",
+            },
+          },
+        ],
+        skillsInvoked: [],
+      },
+      toolExpect: {
+        requiredToolsUsed: ["create_schedule"],
+        toolCallArgs: [
+          {
+            tool: "create_schedule",
+            field: "path",
+            stringStartsWithAnyOf: ["f/", "u/"],
+            stringMustNotStartWithAnyOf: ["schedules/"],
+          },
+        ],
+      },
+    });
+
+    expect(checks).toContainEqual({
+      name: "create_schedule.path uses an accepted prefix",
+      passed: false,
+      details: 'accepted prefixes: f/, u/; values: "schedules/greet_user_daily"',
+    });
+    expect(checks).toContainEqual({
+      name: "create_schedule.path avoids rejected prefixes",
+      passed: false,
+      details: 'rejected prefixes: schedules/; values: "schedules/greet_user_daily"',
+    });
+  });
+});
+
+describe("validateAppState", () => {
+  it("accepts app persistence requirements when a datatable table is registered", () => {
+    const checks = validateAppState({
+      actual: {
+        frontend: {
+          "/index.tsx": "import { backend } from 'wmill'\nexport default function App() { return <div /> }\n",
+        },
+        backend: {
+          listRecipes: {
+            name: "List recipes",
+            type: "inline",
+            inlineScript: {
+              language: "bun",
+              content:
+                "import * as wmill from 'windmill-client'\nexport async function main() { const sql = wmill.datatable(); return await sql`select * from recipes`.fetch() }\n",
+            },
+          },
+        },
+        datatables: [
+          {
+            datatable_name: "main",
+            schemas: {
+              public: {
+                recipes: {},
+              },
+            },
+          },
+        ],
+      },
+      validate: {
+        datatableTableCountAtLeast: 1,
+      },
+    });
+
+    expect(checks.every((check) => check.passed)).toBe(true);
+  });
+
+  it("fails app persistence requirements when no datatable table exists", () => {
+    const checks = validateAppState({
+      actual: {
+        frontend: {
+          "/index.tsx": "export default function App() { return <div /> }\n",
+        },
+        backend: {},
+        datatables: [],
+      },
+      validate: {
+        datatableTableCountAtLeast: 1,
+      },
+    });
+
+    expect(checks).toContainEqual({
+      name: "app includes at least 1 datatable table",
+      passed: false,
+      details: "expected at least 1, got 0",
+    });
+  });
+
+  it("requires a specific datatable table when requested", () => {
+    const checks = validateAppState({
+      actual: {
+        frontend: {
+          "/index.tsx": "export default function App() { return <div /> }\n",
+        },
+        backend: {},
+        datatables: [
+          {
+            datatable_name: "main",
+            schemas: {
+              public: {
+                recipes: {},
+              },
+            },
+          },
+        ],
+      },
+      validate: {
+        requiredDatatables: [
+          {
+            datatableName: "main",
+            schema: "public",
+            table: "recipes",
+          },
+        ],
+      },
+    });
+
+    expect(checks.every((check) => check.passed)).toBe(true);
+  });
+
+  it("can require an exact datatable table count", () => {
+    const checks = validateAppState({
+      actual: {
+        frontend: {
+          "/index.tsx": "export default function App() { return <div /> }\n",
+        },
+        backend: {},
+        datatables: [
+          {
+            datatable_name: "main",
+            schemas: {
+              public: {
+                notes: {},
+                extra_notes: {},
+              },
+            },
+          },
+        ],
+      },
+      validate: {
+        datatableTableCountExactly: 1,
+      },
+    });
+
+    expect(checks).toContainEqual({
+      name: "app includes exactly 1 datatable table",
+      passed: false,
+      details: "expected exactly 1, got 2",
+    });
+  });
+
+  it("validates app datatable code, tool usage, and forbidden storage", () => {
+    const checks = validateAppState({
+      actual: {
+        frontend: {
+          "/index.tsx":
+            "import { backend } from './wmill'\nexport default function App() { void backend.listNotes(); return <div /> }\n",
+        },
+        backend: {
+          listNotes: {
+            name: "List notes",
+            type: "inline",
+            inlineScript: {
+              language: "bun",
+              content:
+                "import * as wmill from 'windmill-client'\nexport async function main() { const sql = wmill.datatable(); return await sql`SELECT * FROM notes`.fetch() }\n",
+            },
+          },
+        },
+        datatables: [
+          {
+            datatable_name: "main",
+            schemas: {
+              public: {
+                notes: {},
+              },
+            },
+          },
+        ],
+      },
+      toolsUsed: ["list_datatables", "get_datatable_table_schema"],
+      validate: {
+        requiredFrontendFileContent: [
+          {
+            path: "/index.tsx",
+            includes: ["backend.listNotes"],
+          },
+        ],
+        requiredBackendRunnableContent: [
+          {
+            key: "listNotes",
+            includes: ["wmill.datatable", "select", "notes"],
+          },
+        ],
+        requiredToolsUsed: ["list_datatables", "get_datatable_table_schema"],
+        forbiddenAppContent: ["localStorage", "sessionStorage"],
+      },
+    });
+
+    expect(checks.every((check) => check.passed)).toBe(true);
+  });
+
+  it("fails app datatable code validation when required code or tools are missing", () => {
+    const checks = validateAppState({
+      actual: {
+        frontend: {
+          "/index.tsx": "export default function App() { localStorage.setItem('x', 'y'); return <div /> }\n",
+        },
+        backend: {
+          listNotes: {
+            name: "List notes",
+            type: "inline",
+            inlineScript: {
+              language: "bun",
+              content: "export async function main() { return [] }\n",
+            },
+          },
+        },
+        datatables: [],
+      },
+      toolsUsed: ["list_files"],
+      validate: {
+        requiredBackendRunnableContent: [
+          {
+            key: "listNotes",
+            includes: ["wmill.datatable", "notes"],
+          },
+        ],
+        requiredToolsUsed: ["list_datatables"],
+        forbiddenAppContent: ["localStorage"],
+      },
+    });
+
+    expect(checks).toContainEqual({
+      name: "listNotes backend runnable includes required content",
+      passed: false,
+      details: "missing snippets: wmill.datatable, notes",
+    });
+    expect(checks).toContainEqual({
+      name: "tool list_datatables was used",
+      passed: false,
+      details: "tools used: list_files",
+    });
+    expect(checks).toContainEqual({
+      name: "app does not include forbidden content 'localStorage'",
+      passed: false,
+      details: "forbidden snippet: localStorage",
+    });
+  });
+
+  it("fails validation when frontend references a missing backend runnable", () => {
+    const checks = validateAppState({
+      actual: {
+        frontend: {
+          "/index.tsx":
+            "import { backend } from 'wmill'\nexport default function App() { void backend.deleteRecipe({ id: 1 }); return <div /> }\n",
+        },
+        backend: {
+          listRecipes: {
+            name: "List recipes",
+            type: "inline",
+            inlineScript: {
+              language: "bun",
+              content: "export async function main() { return [] }\n",
+            },
+          },
+        },
+        datatables: [],
+      },
+    });
+
+    expect(checks).toContainEqual({
+      name: "frontend backend references resolve",
+      passed: false,
+      details: expect.stringContaining("deleteRecipe"),
+    });
+  });
+});
+
+describe("validateCliWorkspace", () => {
+  it("accepts required CLI skills and proposed commands without execution", () => {
+    const checks = validateCliWorkspace({
+      actualFiles: {
+        "f/evals/hello.ts": "export async function main(name: string) { return { greeting: `Hello, ${name}!` } }\n",
+      },
+      expectedFiles: {
+        "f/evals/hello.ts": "export async function main(name: string)\nreturn { greeting: `Hello, ${name}!` }",
+      },
+      assistantOutput:
+        "Created the script. Next run `wmill generate-metadata --yes` and then `wmill sync push`.",
+      trace: {
+        toolsUsed: [
+          { tool: "Skill", input: { skill: "write-script-bun" }, timestamp: 1 },
+          { tool: "Write", input: { file_path: "f/evals/hello.ts" }, timestamp: 2 },
+        ],
+        skillsInvoked: ["write-script-bun"],
+        assistantMessageCount: 1,
+        bashCommands: [],
+        proposedCommands: ["wmill generate-metadata --yes", "wmill sync push"],
+        executedWmillCommands: [],
+        wmillInvocations: [],
+        firstMutationToolIndex: 1,
+      },
+      cliExpect: {
+        requiredSkills: ["write-script-bun"],
+        requiredSkillsBeforeFirstMutation: ["write-script-bun"],
+        orderedAssistantMentions: ["wmill generate-metadata", "wmill sync push"],
+        orderedProposedCommands: ["wmill generate-metadata", "wmill sync push"],
+        forbiddenExecutedCommands: ["^wmill generate-metadata", "^wmill sync push"],
+      },
+    });
+
+    expect(checks.every((check) => check.passed)).toBe(true);
+  });
+
+  it("fails when a forbidden wmill command is executed", () => {
+    const checks = validateCliWorkspace({
+      actualFiles: {},
+      assistantOutput: "Run `wmill sync push` when ready.",
+      trace: {
+        toolsUsed: [{ tool: "Bash", input: { command: "wmill sync push" }, timestamp: 1 }],
+        skillsInvoked: [],
+        assistantMessageCount: 1,
+        bashCommands: ["wmill sync push"],
+        proposedCommands: ["wmill sync push"],
+        executedWmillCommands: ["wmill sync push"],
+        wmillInvocations: [
+          {
+            argv: ["sync", "push"],
+            cwd: "/tmp/workspace",
+            timestamp: "2026-04-21T12:00:00+00:00",
+          },
+        ],
+        firstMutationToolIndex: 0,
+      },
+      cliExpect: {
+        forbiddenExecutedCommands: ["^wmill sync push"],
+      },
+    });
+
+    expect(checks).toContainEqual({
+      name: "does not execute ^wmill sync push",
+      passed: false,
+      details: "executed=wmill sync push",
+    });
+  });
+
+  it("supports read-only guidance cases that must keep the workspace unchanged", () => {
+    const checks = validateCliWorkspace({
+      actualFiles: {},
+      assistantOutput:
+        "Use `wmill job get 123`, then `wmill job logs 123`, then `wmill job result 123`.",
+      trace: {
+        toolsUsed: [{ tool: "Skill", input: { skill: "cli-commands" }, timestamp: 1 }],
+        skillsInvoked: ["cli-commands"],
+        assistantMessageCount: 1,
+        bashCommands: [],
+        proposedCommands: ["wmill job get 123", "wmill job logs 123", "wmill job result 123"],
+        executedWmillCommands: [],
+        wmillInvocations: [],
+        firstMutationToolIndex: null,
+      },
+      cliExpect: {
+        requiredSkills: ["cli-commands"],
+        workspaceUnchanged: true,
+        orderedProposedCommands: [
+          "wmill job get 123",
+          "wmill job logs 123",
+          "wmill job result 123",
+        ],
+        forbiddenProposedCommands: ["wmill sync push"],
+      },
+    });
+
+    expect(checks.every((check) => check.passed)).toBe(true);
+  });
+
+  it("matches skills by exact name instead of substring", () => {
+    const checks = validateCliWorkspace({
+      actualFiles: {},
+      assistantOutput: "No workspace changes needed.",
+      trace: {
+        toolsUsed: [{ tool: "Skill", input: { skill: "write-flow-helper" }, timestamp: 1 }],
+        skillsInvoked: ["write-flow-helper"],
+        assistantMessageCount: 1,
+        bashCommands: [],
+        proposedCommands: [],
+        executedWmillCommands: [],
+        wmillInvocations: [],
+        firstMutationToolIndex: null,
+      },
+      cliExpect: {
+        requiredSkills: ["write-flow"],
+        forbiddenSkills: ["write-flow"],
+      },
+    });
+
+    expect(checks).toContainEqual({
+      name: "invokes skill write-flow",
+      passed: false,
+      details: "skills=write-flow-helper",
+    });
+    expect(checks).toContainEqual({
+      name: "does not invoke skill write-flow",
+      passed: true,
+    });
+  });
+
+  it("accepts ordered proposed commands when they appear in one concatenated entry", () => {
+    const checks = validateCliWorkspace({
+      actualFiles: {},
+      assistantOutput: "Run wmill generate-metadata and then wmill sync push.",
+      trace: {
+        toolsUsed: [{ tool: "Skill", input: { skill: "cli-commands" }, timestamp: 1 }],
+        skillsInvoked: ["cli-commands"],
+        assistantMessageCount: 1,
+        bashCommands: [],
+        proposedCommands: ["wmill generate-metadata and then wmill sync push"],
+        executedWmillCommands: [],
+        wmillInvocations: [],
+        firstMutationToolIndex: null,
+      },
+      cliExpect: {
+        orderedProposedCommands: ["wmill generate-metadata", "wmill sync push"],
+      },
+    });
+
+    expect(checks).toContainEqual({
+      name: "assistant proposes expected commands in order",
+      passed: true,
+    });
+  });
+
+  it("fails skill-before-mutation checks cleanly when no mutation happened", () => {
+    const checks = validateCliWorkspace({
+      actualFiles: {},
+      assistantOutput: "Run `wmill sync pull` first.",
+      trace: {
+        toolsUsed: [{ tool: "Skill", input: { skill: "cli-commands" }, timestamp: 1 }],
+        skillsInvoked: ["cli-commands"],
+        assistantMessageCount: 1,
+        bashCommands: [],
+        proposedCommands: ["wmill sync pull"],
+        executedWmillCommands: [],
+        wmillInvocations: [],
+        firstMutationToolIndex: null,
+      },
+      cliExpect: {
+        requiredSkillsBeforeFirstMutation: ["cli-commands"],
+      },
+    });
+
+    expect(checks).toContainEqual({
+      name: "invokes skill cli-commands before first mutation",
+      passed: false,
+      details: "firstSkillIndex=0; firstMutationIndex=none",
+    });
+  });
+});

ai_evals/core/windmillBackendSettings.ts

@@ -0,0 +1,67 @@
+export interface WindmillBackendSettings {
+  baseUrl: string;
+  email: string;
+  password: string;
+  keepWorkspaces: boolean;
+  workspaceOverride?: string;
+  workspacePrefix: string;
+}
+
+export function resolveWindmillBackendSettings(): WindmillBackendSettings {
+  return {
+    baseUrl: normalizeBaseUrl(
+      process.env.WMILL_AI_EVAL_BACKEND_URL ??
+        process.env.WINDMILL_URL ??
+        process.env.WINDMILL_BASE_URL ??
+        process.env.REMOTE ??
+        "http://127.0.0.1:8000",
+    ),
+    email: process.env.WMILL_AI_EVAL_BACKEND_EMAIL ?? "admin@windmill.dev",
+    password: process.env.WMILL_AI_EVAL_BACKEND_PASSWORD ?? "changeme",
+    keepWorkspaces: isTruthy(process.env.WMILL_AI_EVAL_KEEP_WORKSPACES),
+    workspaceOverride: sanitizeOptionalWorkspaceId(
+      process.env.WMILL_AI_EVAL_BACKEND_WORKSPACE,
+    ),
+    workspacePrefix: sanitizeWorkspacePrefix(
+      process.env.WMILL_AI_EVAL_WORKSPACE_PREFIX ?? "ai-evals",
+    ),
+  };
+}
+
+export function parsePositiveInteger(
+  value: string | undefined,
+  fallback: number,
+): number {
+  if (!value) {
+    return fallback;
+  }
+  const parsed = Number(value);
+  return Number.isInteger(parsed) && parsed > 0 ? parsed : fallback;
+}
+
+function normalizeBaseUrl(value: string): string {
+  return value.replace(/\/+$/, "");
+}
+
+function sanitizeWorkspacePrefix(value: string): string {
+  const sanitized = value
+    .trim()
+    .toLowerCase()
+    .replace(/[^a-z0-9-]+/g, "-")
+    .replace(/^-+|-+$/g, "");
+  return sanitized.length > 0 ? sanitized : "ai-evals";
+}
+
+function sanitizeOptionalWorkspaceId(
+  value: string | undefined,
+): string | undefined {
+  const trimmed = value?.trim();
+  return trimmed ? trimmed : undefined;
+}
+
+function isTruthy(value: string | undefined): boolean {
+  if (!value) {
+    return false;
+  }
+  return ["1", "true", "yes", "on"].includes(value.trim().toLowerCase());
+}

ai_evals/fixtures/frontend/app/initial/inventory_tracker/backend/addInventory/main.ts

@@ -0,0 +1,29 @@
+import * as wmill from 'windmill-client'
+
+interface InventoryItem {
+	id: number
+	name: string
+	sku: string
+	quantity: number
+	price: number
+	created_at: string
+}
+
+export async function main({
+	name,
+	sku,
+	quantity,
+	price
+}: {
+	name: string
+	sku: string
+	quantity: number
+	price: number
+}): Promise<InventoryItem> {
+	const sql = wmill.datatable()
+	return await sql`
+		INSERT INTO public.inventory_items (name, sku, quantity, price)
+		VALUES (${name}, ${sku}, ${quantity}, ${price})
+		RETURNING id, name, sku, quantity, price, created_at
+	`.fetchOne()
+}

ai_evals/fixtures/frontend/app/initial/inventory_tracker/backend/addInventory/meta.json

@@ -0,0 +1,4 @@
+{
+  "name": "Add inventory",
+  "language": "bun"
+}

ai_evals/fixtures/frontend/app/initial/inventory_tracker/backend/listInventory/main.ts

@@ -0,0 +1,19 @@
+import * as wmill from 'windmill-client'
+
+interface InventoryItem {
+	id: number
+	name: string
+	sku: string
+	quantity: number
+	price: number
+	created_at: string
+}
+
+export async function main(): Promise<InventoryItem[]> {
+	const sql = wmill.datatable()
+	return await sql`
+		SELECT id, name, sku, quantity, price, created_at
+		FROM public.inventory_items
+		ORDER BY created_at DESC
+	`.fetch()
+}

ai_evals/fixtures/frontend/app/initial/inventory_tracker/backend/listInventory/meta.json

@@ -0,0 +1,4 @@
+{
+  "name": "List inventory",
+  "language": "bun"
+}

ai_evals/fixtures/frontend/app/initial/inventory_tracker/datatables.json

@@ -0,0 +1,17 @@
+[
+  {
+    "datatable_name": "main",
+    "schemas": {
+      "public": {
+        "inventory_items": {
+          "id": "int4",
+          "name": "text",
+          "sku": "text",
+          "quantity": "int4",
+          "price": "float8",
+          "created_at": "timestamp=now()"
+        }
+      }
+    }
+  }
+]

ai_evals/fixtures/frontend/app/initial/inventory_tracker/frontend/index.tsx

@@ -0,0 +1,185 @@
+import React, { useEffect, useState } from 'react'
+import { backend } from 'wmill'
+
+interface InventoryItem {
+	id: number
+	name: string
+	sku: string
+	quantity: number
+	price: number
+	created_at: string
+}
+
+const emptyForm = {
+	name: '',
+	sku: '',
+	quantity: '1',
+	price: '0.00'
+}
+
+const InventoryTrackerApp = () => {
+	const [items, setItems] = useState<InventoryItem[]>([])
+	const [form, setForm] = useState(emptyForm)
+	const [loading, setLoading] = useState(true)
+	const [error, setError] = useState<string | null>(null)
+
+	useEffect(() => {
+		void loadItems()
+	}, [])
+
+	const loadItems = async () => {
+		try {
+			setLoading(true)
+			setError(null)
+			const data = await backend.listInventory()
+			setItems(data)
+		} catch (error) {
+			console.error(error)
+			setError('Failed to load inventory')
+		} finally {
+			setLoading(false)
+		}
+	}
+
+	const handleSubmit = async (event: React.FormEvent<HTMLFormElement>) => {
+		event.preventDefault()
+		const name = form.name.trim()
+		const sku = form.sku.trim()
+		const quantity = Number(form.quantity)
+		const price = Number(form.price)
+
+		if (!name || !sku || Number.isNaN(quantity) || Number.isNaN(price) || quantity < 0 || price < 0) {
+			setError('Enter a valid name, sku, quantity, and price')
+			return
+		}
+
+		try {
+			setError(null)
+			const item = await backend.addInventory({
+				name,
+				sku,
+				quantity,
+				price
+			})
+			setItems((previous) => [item, ...previous])
+			setForm(emptyForm)
+		} catch (error) {
+			console.error(error)
+			setError('Failed to save inventory item')
+		}
+	}
+
+	return (
+		<div className="min-h-screen bg-slate-100 p-6">
+			<div className="mx-auto max-w-6xl rounded-3xl bg-white shadow-lg shadow-slate-300/40">
+				<div className="border-b border-slate-200 px-8 py-6">
+					<h1 className="text-3xl font-semibold text-slate-900">Inventory tracker</h1>
+					<p className="mt-2 text-sm text-slate-600">
+						Add products and keep them stored in the existing datatable-backed app.
+					</p>
+				</div>
+
+				<div className="grid gap-8 px-8 py-8 lg:grid-cols-[340px_1fr]">
+					<form className="space-y-4 rounded-2xl border border-slate-200 bg-slate-50 p-5" onSubmit={handleSubmit}>
+						<h2 className="text-lg font-medium text-slate-900">Add item</h2>
+						<label className="block space-y-2">
+							<span className="text-sm font-medium text-slate-700">Name</span>
+							<input
+								className="w-full rounded-xl border border-slate-300 bg-white px-3 py-2 text-sm"
+								value={form.name}
+								onChange={(event) => setForm({ ...form, name: event.target.value })}
+							/>
+						</label>
+						<label className="block space-y-2">
+							<span className="text-sm font-medium text-slate-700">SKU</span>
+							<input
+								className="w-full rounded-xl border border-slate-300 bg-white px-3 py-2 text-sm"
+								value={form.sku}
+								onChange={(event) => setForm({ ...form, sku: event.target.value })}
+							/>
+						</label>
+						<div className="grid gap-4 sm:grid-cols-2">
+							<label className="block space-y-2">
+								<span className="text-sm font-medium text-slate-700">Quantity</span>
+								<input
+									type="number"
+									min="0"
+									className="w-full rounded-xl border border-slate-300 bg-white px-3 py-2 text-sm"
+									value={form.quantity}
+									onChange={(event) => setForm({ ...form, quantity: event.target.value })}
+								/>
+							</label>
+							<label className="block space-y-2">
+								<span className="text-sm font-medium text-slate-700">Price</span>
+								<input
+									type="number"
+									min="0"
+									step="0.01"
+									className="w-full rounded-xl border border-slate-300 bg-white px-3 py-2 text-sm"
+									value={form.price}
+									onChange={(event) => setForm({ ...form, price: event.target.value })}
+								/>
+							</label>
+						</div>
+						<button
+							type="submit"
+							className="rounded-full bg-slate-900 px-4 py-2 text-sm font-medium text-white"
+						>
+							Save item
+						</button>
+					</form>
+
+					<div className="space-y-4">
+						{error ? (
+							<div className="rounded-2xl border border-red-200 bg-red-50 px-4 py-3 text-sm text-red-700">
+								{error}
+							</div>
+						) : null}
+
+						{loading ? (
+							<div className="rounded-2xl border border-slate-200 bg-slate-50 px-4 py-10 text-center text-sm text-slate-500">
+								Loading inventory...
+							</div>
+						) : items.length === 0 ? (
+							<div className="rounded-2xl border border-dashed border-slate-300 px-4 py-10 text-center text-sm text-slate-500">
+								No items saved yet.
+							</div>
+						) : (
+							<div className="overflow-hidden rounded-2xl border border-slate-200">
+								<table className="min-w-full divide-y divide-slate-200 text-sm">
+									<thead className="bg-slate-50 text-left text-slate-600">
+										<tr>
+											<th className="px-4 py-3 font-medium">Item</th>
+											<th className="px-4 py-3 font-medium">SKU</th>
+											<th className="px-4 py-3 font-medium">Qty</th>
+											<th className="px-4 py-3 font-medium">Price</th>
+										</tr>
+									</thead>
+									<tbody className="divide-y divide-slate-200 bg-white">
+										{items.map((item) => (
+											<tr key={item.id}>
+												<td className="px-4 py-3 text-slate-900">
+													<div className="font-medium">{item.name}</div>
+													<div className="text-xs text-slate-500">
+														Added {new Date(item.created_at).toLocaleDateString()}
+													</div>
+												</td>
+												<td className="px-4 py-3 text-slate-600">{item.sku}</td>
+												<td className="px-4 py-3 text-slate-600">{item.quantity}</td>
+												<td className="px-4 py-3 text-slate-600">
+													${item.price.toFixed(2)}
+												</td>
+											</tr>
+										))}
+									</tbody>
+								</table>
+							</div>
+						)}
+					</div>
+				</div>
+			</div>
+		</div>
+	)
+}
+
+export default InventoryTrackerApp

ai_evals/fixtures/frontend/app/initial/notes_datatable/datatables.json

@@ -0,0 +1,15 @@
+[
+  {
+    "datatable_name": "main",
+    "schemas": {
+      "public": {
+        "notes": {
+          "id": "int4",
+          "title": "text",
+          "body": "text?",
+          "created_at": "timestamp=now()"
+        }
+      }
+    }
+  }
+]

ai_evals/fixtures/frontend/app/initial/recipe_book/backend/addRecipe/main.ts

@@ -0,0 +1,26 @@
+import * as wmill from 'windmill-client'
+
+interface Recipe {
+	id: number
+	name: string
+	ingredients: string
+	instructions: string
+	created_at: string
+}
+
+export async function main({
+	name,
+	ingredients,
+	instructions
+}: {
+	name: string
+	ingredients: string
+	instructions: string
+}): Promise<Recipe> {
+	const sql = wmill.datatable()
+	return await sql`
+		INSERT INTO public.recipes (name, ingredients, instructions)
+		VALUES (${name}, ${ingredients}, ${instructions})
+		RETURNING id, name, ingredients, instructions, created_at
+	`.fetchOne()
+}

ai_evals/fixtures/frontend/app/initial/recipe_book/backend/addRecipe/meta.json

@@ -0,0 +1,4 @@
+{
+  "name": "Add recipe",
+  "language": "bun"
+}

ai_evals/fixtures/frontend/app/initial/recipe_book/backend/listRecipes/main.ts

@@ -0,0 +1,18 @@
+import * as wmill from 'windmill-client'
+
+interface Recipe {
+	id: number
+	name: string
+	ingredients: string
+	instructions: string
+	created_at: string
+}
+
+export async function main(): Promise<Recipe[]> {
+	const sql = wmill.datatable()
+	return await sql`
+		SELECT id, name, ingredients, instructions, created_at
+		FROM public.recipes
+		ORDER BY created_at DESC
+	`.fetch()
+}

ai_evals/fixtures/frontend/app/initial/recipe_book/backend/listRecipes/meta.json

@@ -0,0 +1,4 @@
+{
+  "name": "List recipes",
+  "language": "bun"
+}

ai_evals/fixtures/frontend/app/initial/recipe_book/datatables.json

@@ -0,0 +1,16 @@
+[
+  {
+    "datatable_name": "main",
+    "schemas": {
+      "public": {
+        "recipes": {
+          "id": "int4",
+          "name": "text",
+          "ingredients": "text",
+          "instructions": "text",
+          "created_at": "timestamp=now()"
+        }
+      }
+    }
+  }
+]

ai_evals/fixtures/frontend/app/initial/recipe_book/frontend/index.tsx

@@ -0,0 +1,168 @@
+import React, { useEffect, useState } from 'react'
+import { backend } from 'wmill'
+
+interface Recipe {
+	id: number
+	name: string
+	ingredients: string
+	instructions: string
+	created_at: string
+}
+
+const emptyForm = {
+	name: '',
+	ingredients: '',
+	instructions: ''
+}
+
+const RecipeBookApp = () => {
+	const [recipes, setRecipes] = useState<Recipe[]>([])
+	const [form, setForm] = useState(emptyForm)
+	const [loading, setLoading] = useState(true)
+	const [error, setError] = useState<string | null>(null)
+
+	useEffect(() => {
+		void loadRecipes()
+	}, [])
+
+	const loadRecipes = async () => {
+		try {
+			setLoading(true)
+			setError(null)
+			const data = await backend.listRecipes()
+			setRecipes(data)
+		} catch (error) {
+			console.error(error)
+			setError('Failed to load recipes')
+		} finally {
+			setLoading(false)
+		}
+	}
+
+	const handleSubmit = async (event: React.FormEvent<HTMLFormElement>) => {
+		event.preventDefault()
+		const name = form.name.trim()
+		const ingredients = form.ingredients.trim()
+		const instructions = form.instructions.trim()
+
+		if (!name || !ingredients || !instructions) {
+			setError('Please fill in every field')
+			return
+		}
+
+		try {
+			setError(null)
+			const recipe = await backend.addRecipe({
+				name,
+				ingredients,
+				instructions
+			})
+			setRecipes((previous) => [recipe, ...previous])
+			setForm(emptyForm)
+		} catch (error) {
+			console.error(error)
+			setError('Failed to save recipe')
+		}
+	}
+
+	return (
+		<div className="min-h-screen bg-stone-100 p-6">
+			<div className="mx-auto max-w-5xl rounded-3xl bg-white shadow-lg shadow-stone-300/40">
+				<div className="border-b border-stone-200 px-8 py-6">
+					<h1 className="text-3xl font-semibold text-stone-900">Recipe book</h1>
+					<p className="mt-2 text-sm text-stone-600">
+						Add recipes and keep them stored in the existing datatable-backed app.
+					</p>
+				</div>
+
+				<div className="grid gap-8 px-8 py-8 lg:grid-cols-[320px_1fr]">
+					<form className="space-y-4 rounded-2xl border border-stone-200 bg-stone-50 p-5" onSubmit={handleSubmit}>
+						<h2 className="text-lg font-medium text-stone-900">Add recipe</h2>
+						<label className="block space-y-2">
+							<span className="text-sm font-medium text-stone-700">Name</span>
+							<input
+								className="w-full rounded-xl border border-stone-300 bg-white px-3 py-2 text-sm"
+								value={form.name}
+								onChange={(event) => setForm({ ...form, name: event.target.value })}
+							/>
+						</label>
+						<label className="block space-y-2">
+							<span className="text-sm font-medium text-stone-700">Ingredients</span>
+							<textarea
+								className="min-h-28 w-full rounded-xl border border-stone-300 bg-white px-3 py-2 text-sm"
+								value={form.ingredients}
+								onChange={(event) => setForm({ ...form, ingredients: event.target.value })}
+							/>
+						</label>
+						<label className="block space-y-2">
+							<span className="text-sm font-medium text-stone-700">Instructions</span>
+							<textarea
+								className="min-h-32 w-full rounded-xl border border-stone-300 bg-white px-3 py-2 text-sm"
+								value={form.instructions}
+								onChange={(event) => setForm({ ...form, instructions: event.target.value })}
+							/>
+						</label>
+						<button
+							type="submit"
+							className="rounded-full bg-stone-900 px-4 py-2 text-sm font-medium text-white"
+						>
+							Save recipe
+						</button>
+					</form>
+
+					<div className="space-y-4">
+						{error ? (
+							<div className="rounded-2xl border border-red-200 bg-red-50 px-4 py-3 text-sm text-red-700">
+								{error}
+							</div>
+						) : null}
+
+						{loading ? (
+							<div className="rounded-2xl border border-stone-200 bg-stone-50 px-4 py-10 text-center text-sm text-stone-500">
+								Loading recipes...
+							</div>
+						) : recipes.length === 0 ? (
+							<div className="rounded-2xl border border-dashed border-stone-300 px-4 py-10 text-center text-sm text-stone-500">
+								No recipes saved yet.
+							</div>
+						) : (
+							<ul className="space-y-4">
+								{recipes.map((recipe) => (
+									<li key={recipe.id} className="rounded-2xl border border-stone-200 p-5">
+										<div className="flex items-start justify-between gap-4">
+											<div>
+												<h3 className="text-lg font-medium text-stone-900">{recipe.name}</h3>
+												<p className="mt-1 text-xs uppercase tracking-[0.18em] text-stone-500">
+													Stored recipe
+												</p>
+											</div>
+											<span className="rounded-full bg-stone-100 px-3 py-1 text-xs text-stone-600">
+												{new Date(recipe.created_at).toLocaleDateString()}
+											</span>
+										</div>
+										<div className="mt-4 grid gap-4 md:grid-cols-2">
+											<div>
+												<h4 className="text-sm font-medium text-stone-700">Ingredients</h4>
+												<p className="mt-2 whitespace-pre-wrap text-sm text-stone-600">
+													{recipe.ingredients}
+												</p>
+											</div>
+											<div>
+												<h4 className="text-sm font-medium text-stone-700">Instructions</h4>
+												<p className="mt-2 whitespace-pre-wrap text-sm text-stone-600">
+													{recipe.instructions}
+												</p>
+											</div>
+										</div>
+									</li>
+								))}
+							</ul>
+						)}
+					</div>
+				</div>
+			</div>
+		</div>
+	)
+}
+
+export default RecipeBookApp

ai_evals/fixtures/frontend/app/initial/session_id_chat/backend/a/main.ts

@@ -0,0 +1,15 @@
+export async function main({
+	session_id,
+	message,
+	system_prompt
+}: {
+	session_id: string
+	message: string
+	system_prompt: string
+}) {
+	return {
+		content: `Session ${session_id}: ${message}`,
+		tool_calls: [],
+		system_prompt_length: system_prompt.length
+	}
+}

ai_evals/fixtures/frontend/app/initial/session_id_chat/backend/a/meta.json

@@ -0,0 +1,4 @@
+{
+	"name": "Chat Assistant",
+	"language": "bun"
+}

ai_evals/fixtures/frontend/app/initial/session_id_chat/frontend/index.tsx

@@ -0,0 +1,98 @@
+import React, { useState } from 'react'
+import { backend } from 'wmill'
+
+type Message = {
+	role: 'user' | 'assistant'
+	content: string
+}
+
+const SYSTEM_PROMPT =
+	'You are Boris, the internal AI assistant for Acme Corporation. Be helpful, friendly, and concise.'
+
+function generateSessionId(): string {
+	return crypto.randomUUID()
+}
+
+function getSessionId(): string {
+	let sessionId = sessionStorage.getItem('chat_session_id')
+	if (!sessionId) {
+		sessionId = generateSessionId()
+		sessionStorage.setItem('chat_session_id', sessionId)
+	}
+	return sessionId
+}
+
+const App = () => {
+	const [messages, setMessages] = useState<Message[]>([])
+	const [input, setInput] = useState('')
+	const [loading, setLoading] = useState(false)
+	const [sessionId, setSessionId] = useState<string>(getSessionId())
+
+	async function sendMessage(e: React.FormEvent) {
+		e.preventDefault()
+		if (!input.trim() || loading) return
+
+		const userMessage = input.trim()
+		setInput('')
+		setLoading(true)
+		setMessages((prev) => [...prev, { role: 'user', content: userMessage }])
+
+		try {
+			const response = await backend.a({
+				session_id: sessionId,
+				message: userMessage,
+				system_prompt: SYSTEM_PROMPT
+			})
+			const content = typeof response === 'string' ? response : response.content
+			setMessages((prev) => [...prev, { role: 'assistant', content }])
+		} finally {
+			setLoading(false)
+		}
+	}
+
+	function startNewChat() {
+		const newSessionId = generateSessionId()
+		sessionStorage.setItem('chat_session_id', newSessionId)
+		setSessionId(newSessionId)
+		setMessages([])
+	}
+
+	return (
+		<div className="chat-app">
+			<header>
+				<h1>Boris</h1>
+				<p>Session: {sessionId}</p>
+				<button onClick={startNewChat}>New Chat</button>
+			</header>
+
+			<main>
+				{messages.length === 0 ? (
+					<p>Ask Boris anything about the company.</p>
+				) : (
+					<ul>
+						{messages.map((message, index) => (
+							<li key={index}>
+								<strong>{message.role}:</strong> {message.content}
+							</li>
+						))}
+					</ul>
+				)}
+			</main>
+
+			<form onSubmit={sendMessage}>
+				<input
+					type="text"
+					value={input}
+					onChange={(e) => setInput(e.target.value)}
+					placeholder="Type your message..."
+					disabled={loading}
+				/>
+				<button type="submit" disabled={loading || !input.trim()}>
+					Send
+				</button>
+			</form>
+		</div>
+	)
+}
+
+export default App

ai_evals/fixtures/frontend/app/initial/token_heavy_context/backend/loadAnalytics/main.ts

@@ -0,0 +1,207 @@
+import * as wmill from 'windmill-client'
+
+type AnalyticsRow = {
+	metric: string
+	value: number
+	segment: string
+	notes: string
+}
+
+const fallbackRows: AnalyticsRow[] = [
+	{ metric: 'metric_1', value: 3, segment: 'segment_1', notes: 'Verbose fallback analytics row 1 with context about historical cohorts and operating constraints.' },
+	{ metric: 'metric_2', value: 6, segment: 'segment_2', notes: 'Verbose fallback analytics row 2 with context about historical cohorts and operating constraints.' },
+	{ metric: 'metric_3', value: 9, segment: 'segment_3', notes: 'Verbose fallback analytics row 3 with context about historical cohorts and operating constraints.' },
+	{ metric: 'metric_4', value: 12, segment: 'segment_4', notes: 'Verbose fallback analytics row 4 with context about historical cohorts and operating constraints.' },
+	{ metric: 'metric_5', value: 15, segment: 'segment_5', notes: 'Verbose fallback analytics row 5 with context about historical cohorts and operating constraints.' },
+	{ metric: 'metric_6', value: 18, segment: 'segment_6', notes: 'Verbose fallback analytics row 6 with context about historical cohorts and operating constraints.' },
+	{ metric: 'metric_7', value: 21, segment: 'segment_0', notes: 'Verbose fallback analytics row 7 with context about historical cohorts and operating constraints.' },
+	{ metric: 'metric_8', value: 24, segment: 'segment_1', notes: 'Verbose fallback analytics row 8 with context about historical cohorts and operating constraints.' },
+	{ metric: 'metric_9', value: 27, segment: 'segment_2', notes: 'Verbose fallback analytics row 9 with context about historical cohorts and operating constraints.' },
+	{ metric: 'metric_10', value: 30, segment: 'segment_3', notes: 'Verbose fallback analytics row 10 with context about historical cohorts and operating constraints.' },
+	{ metric: 'metric_11', value: 33, segment: 'segment_4', notes: 'Verbose fallback analytics row 11 with context about historical cohorts and operating constraints.' },
+	{ metric: 'metric_12', value: 36, segment: 'segment_5', notes: 'Verbose fallback analytics row 12 with context about historical cohorts and operating constraints.' },
+	{ metric: 'metric_13', value: 39, segment: 'segment_6', notes: 'Verbose fallback analytics row 13 with context about historical cohorts and operating constraints.' },
+	{ metric: 'metric_14', value: 42, segment: 'segment_0', notes: 'Verbose fallback analytics row 14 with context about historical cohorts and operating constraints.' },
+	{ metric: 'metric_15', value: 45, segment: 'segment_1', notes: 'Verbose fallback analytics row 15 with context about historical cohorts and operating constraints.' },
+	{ metric: 'metric_16', value: 48, segment: 'segment_2', notes: 'Verbose fallback analytics row 16 with context about historical cohorts and operating constraints.' },
+	{ metric: 'metric_17', value: 51, segment: 'segment_3', notes: 'Verbose fallback analytics row 17 with context about historical cohorts and operating constraints.' },
+	{ metric: 'metric_18', value: 54, segment: 'segment_4', notes: 'Verbose fallback analytics row 18 with context about historical cohorts and operating constraints.' },
+	{ metric: 'metric_19', value: 57, segment: 'segment_5', notes: 'Verbose fallback analytics row 19 with context about historical cohorts and operating constraints.' },
+	{ metric: 'metric_20', value: 60, segment: 'segment_6', notes: 'Verbose fallback analytics row 20 with context about historical cohorts and operating constraints.' },
+	{ metric: 'metric_21', value: 63, segment: 'segment_0', notes: 'Verbose fallback analytics row 21 with context about historical cohorts and operating constraints.' },
+	{ metric: 'metric_22', value: 66, segment: 'segment_1', notes: 'Verbose fallback analytics row 22 with context about historical cohorts and operating constraints.' },
+	{ metric: 'metric_23', value: 69, segment: 'segment_2', notes: 'Verbose fallback analytics row 23 with context about historical cohorts and operating constraints.' },
+	{ metric: 'metric_24', value: 72, segment: 'segment_3', notes: 'Verbose fallback analytics row 24 with context about historical cohorts and operating constraints.' },
+	{ metric: 'metric_25', value: 75, segment: 'segment_4', notes: 'Verbose fallback analytics row 25 with context about historical cohorts and operating constraints.' },
+	{ metric: 'metric_26', value: 78, segment: 'segment_5', notes: 'Verbose fallback analytics row 26 with context about historical cohorts and operating constraints.' },
+	{ metric: 'metric_27', value: 81, segment: 'segment_6', notes: 'Verbose fallback analytics row 27 with context about historical cohorts and operating constraints.' },
+	{ metric: 'metric_28', value: 84, segment: 'segment_0', notes: 'Verbose fallback analytics row 28 with context about historical cohorts and operating constraints.' },
+	{ metric: 'metric_29', value: 87, segment: 'segment_1', notes: 'Verbose fallback analytics row 29 with context about historical cohorts and operating constraints.' },
+	{ metric: 'metric_30', value: 90, segment: 'segment_2', notes: 'Verbose fallback analytics row 30 with context about historical cohorts and operating constraints.' },
+	{ metric: 'metric_31', value: 93, segment: 'segment_3', notes: 'Verbose fallback analytics row 31 with context about historical cohorts and operating constraints.' },
+	{ metric: 'metric_32', value: 96, segment: 'segment_4', notes: 'Verbose fallback analytics row 32 with context about historical cohorts and operating constraints.' },
+	{ metric: 'metric_33', value: 99, segment: 'segment_5', notes: 'Verbose fallback analytics row 33 with context about historical cohorts and operating constraints.' },
+	{ metric: 'metric_34', value: 102, segment: 'segment_6', notes: 'Verbose fallback analytics row 34 with context about historical cohorts and operating constraints.' },
+	{ metric: 'metric_35', value: 105, segment: 'segment_0', notes: 'Verbose fallback analytics row 35 with context about historical cohorts and operating constraints.' },
+	{ metric: 'metric_36', value: 108, segment: 'segment_1', notes: 'Verbose fallback analytics row 36 with context about historical cohorts and operating constraints.' },
+	{ metric: 'metric_37', value: 111, segment: 'segment_2', notes: 'Verbose fallback analytics row 37 with context about historical cohorts and operating constraints.' },
+	{ metric: 'metric_38', value: 114, segment: 'segment_3', notes: 'Verbose fallback analytics row 38 with context about historical cohorts and operating constraints.' },
+	{ metric: 'metric_39', value: 117, segment: 'segment_4', notes: 'Verbose fallback analytics row 39 with context about historical cohorts and operating constraints.' },
+	{ metric: 'metric_40', value: 120, segment: 'segment_5', notes: 'Verbose fallback analytics row 40 with context about historical cohorts and operating constraints.' },
+	{ metric: 'metric_41', value: 123, segment: 'segment_6', notes: 'Verbose fallback analytics row 41 with context about historical cohorts and operating constraints.' },
+	{ metric: 'metric_42', value: 126, segment: 'segment_0', notes: 'Verbose fallback analytics row 42 with context about historical cohorts and operating constraints.' },
+	{ metric: 'metric_43', value: 129, segment: 'segment_1', notes: 'Verbose fallback analytics row 43 with context about historical cohorts and operating constraints.' },
+	{ metric: 'metric_44', value: 132, segment: 'segment_2', notes: 'Verbose fallback analytics row 44 with context about historical cohorts and operating constraints.' },
+	{ metric: 'metric_45', value: 135, segment: 'segment_3', notes: 'Verbose fallback analytics row 45 with context about historical cohorts and operating constraints.' },
+	{ metric: 'metric_46', value: 138, segment: 'segment_4', notes: 'Verbose fallback analytics row 46 with context about historical cohorts and operating constraints.' },
+	{ metric: 'metric_47', value: 141, segment: 'segment_5', notes: 'Verbose fallback analytics row 47 with context about historical cohorts and operating constraints.' },
+	{ metric: 'metric_48', value: 144, segment: 'segment_6', notes: 'Verbose fallback analytics row 48 with context about historical cohorts and operating constraints.' },
+	{ metric: 'metric_49', value: 147, segment: 'segment_0', notes: 'Verbose fallback analytics row 49 with context about historical cohorts and operating constraints.' },
+	{ metric: 'metric_50', value: 150, segment: 'segment_1', notes: 'Verbose fallback analytics row 50 with context about historical cohorts and operating constraints.' },
+	{ metric: 'metric_51', value: 153, segment: 'segment_2', notes: 'Verbose fallback analytics row 51 with context about historical cohorts and operating constraints.' },
+	{ metric: 'metric_52', value: 156, segment: 'segment_3', notes: 'Verbose fallback analytics row 52 with context about historical cohorts and operating constraints.' },
+	{ metric: 'metric_53', value: 159, segment: 'segment_4', notes: 'Verbose fallback analytics row 53 with context about historical cohorts and operating constraints.' },
+	{ metric: 'metric_54', value: 162, segment: 'segment_5', notes: 'Verbose fallback analytics row 54 with context about historical cohorts and operating constraints.' },
+	{ metric: 'metric_55', value: 165, segment: 'segment_6', notes: 'Verbose fallback analytics row 55 with context about historical cohorts and operating constraints.' },
+	{ metric: 'metric_56', value: 168, segment: 'segment_0', notes: 'Verbose fallback analytics row 56 with context about historical cohorts and operating constraints.' },
+	{ metric: 'metric_57', value: 171, segment: 'segment_1', notes: 'Verbose fallback analytics row 57 with context about historical cohorts and operating constraints.' },
+	{ metric: 'metric_58', value: 174, segment: 'segment_2', notes: 'Verbose fallback analytics row 58 with context about historical cohorts and operating constraints.' },
+	{ metric: 'metric_59', value: 177, segment: 'segment_3', notes: 'Verbose fallback analytics row 59 with context about historical cohorts and operating constraints.' },
+	{ metric: 'metric_60', value: 180, segment: 'segment_4', notes: 'Verbose fallback analytics row 60 with context about historical cohorts and operating constraints.' },
+	{ metric: 'metric_61', value: 183, segment: 'segment_5', notes: 'Verbose fallback analytics row 61 with context about historical cohorts and operating constraints.' },
+	{ metric: 'metric_62', value: 186, segment: 'segment_6', notes: 'Verbose fallback analytics row 62 with context about historical cohorts and operating constraints.' },
+	{ metric: 'metric_63', value: 189, segment: 'segment_0', notes: 'Verbose fallback analytics row 63 with context about historical cohorts and operating constraints.' },
+	{ metric: 'metric_64', value: 192, segment: 'segment_1', notes: 'Verbose fallback analytics row 64 with context about historical cohorts and operating constraints.' },
+	{ metric: 'metric_65', value: 195, segment: 'segment_2', notes: 'Verbose fallback analytics row 65 with context about historical cohorts and operating constraints.' },
+	{ metric: 'metric_66', value: 198, segment: 'segment_3', notes: 'Verbose fallback analytics row 66 with context about historical cohorts and operating constraints.' },
+	{ metric: 'metric_67', value: 201, segment: 'segment_4', notes: 'Verbose fallback analytics row 67 with context about historical cohorts and operating constraints.' },
+	{ metric: 'metric_68', value: 204, segment: 'segment_5', notes: 'Verbose fallback analytics row 68 with context about historical cohorts and operating constraints.' },
+	{ metric: 'metric_69', value: 207, segment: 'segment_6', notes: 'Verbose fallback analytics row 69 with context about historical cohorts and operating constraints.' },
+	{ metric: 'metric_70', value: 210, segment: 'segment_0', notes: 'Verbose fallback analytics row 70 with context about historical cohorts and operating constraints.' },
+	{ metric: 'metric_71', value: 213, segment: 'segment_1', notes: 'Verbose fallback analytics row 71 with context about historical cohorts and operating constraints.' },
+	{ metric: 'metric_72', value: 216, segment: 'segment_2', notes: 'Verbose fallback analytics row 72 with context about historical cohorts and operating constraints.' },
+	{ metric: 'metric_73', value: 219, segment: 'segment_3', notes: 'Verbose fallback analytics row 73 with context about historical cohorts and operating constraints.' },
+	{ metric: 'metric_74', value: 222, segment: 'segment_4', notes: 'Verbose fallback analytics row 74 with context about historical cohorts and operating constraints.' },
+	{ metric: 'metric_75', value: 225, segment: 'segment_5', notes: 'Verbose fallback analytics row 75 with context about historical cohorts and operating constraints.' },
+	{ metric: 'metric_76', value: 228, segment: 'segment_6', notes: 'Verbose fallback analytics row 76 with context about historical cohorts and operating constraints.' },
+	{ metric: 'metric_77', value: 231, segment: 'segment_0', notes: 'Verbose fallback analytics row 77 with context about historical cohorts and operating constraints.' },
+	{ metric: 'metric_78', value: 234, segment: 'segment_1', notes: 'Verbose fallback analytics row 78 with context about historical cohorts and operating constraints.' },
+	{ metric: 'metric_79', value: 237, segment: 'segment_2', notes: 'Verbose fallback analytics row 79 with context about historical cohorts and operating constraints.' },
+	{ metric: 'metric_80', value: 240, segment: 'segment_3', notes: 'Verbose fallback analytics row 80 with context about historical cohorts and operating constraints.' },
+	{ metric: 'metric_81', value: 243, segment: 'segment_4', notes: 'Verbose fallback analytics row 81 with context about historical cohorts and operating constraints.' },
+	{ metric: 'metric_82', value: 246, segment: 'segment_5', notes: 'Verbose fallback analytics row 82 with context about historical cohorts and operating constraints.' },
+	{ metric: 'metric_83', value: 249, segment: 'segment_6', notes: 'Verbose fallback analytics row 83 with context about historical cohorts and operating constraints.' },
+	{ metric: 'metric_84', value: 252, segment: 'segment_0', notes: 'Verbose fallback analytics row 84 with context about historical cohorts and operating constraints.' },
+	{ metric: 'metric_85', value: 255, segment: 'segment_1', notes: 'Verbose fallback analytics row 85 with context about historical cohorts and operating constraints.' },
+	{ metric: 'metric_86', value: 258, segment: 'segment_2', notes: 'Verbose fallback analytics row 86 with context about historical cohorts and operating constraints.' },
+	{ metric: 'metric_87', value: 261, segment: 'segment_3', notes: 'Verbose fallback analytics row 87 with context about historical cohorts and operating constraints.' },
+	{ metric: 'metric_88', value: 264, segment: 'segment_4', notes: 'Verbose fallback analytics row 88 with context about historical cohorts and operating constraints.' },
+	{ metric: 'metric_89', value: 267, segment: 'segment_5', notes: 'Verbose fallback analytics row 89 with context about historical cohorts and operating constraints.' },
+	{ metric: 'metric_90', value: 270, segment: 'segment_6', notes: 'Verbose fallback analytics row 90 with context about historical cohorts and operating constraints.' },
+	{ metric: 'metric_91', value: 273, segment: 'segment_0', notes: 'Verbose fallback analytics row 91 with context about historical cohorts and operating constraints.' },
+	{ metric: 'metric_92', value: 276, segment: 'segment_1', notes: 'Verbose fallback analytics row 92 with context about historical cohorts and operating constraints.' },
+	{ metric: 'metric_93', value: 279, segment: 'segment_2', notes: 'Verbose fallback analytics row 93 with context about historical cohorts and operating constraints.' },
+	{ metric: 'metric_94', value: 282, segment: 'segment_3', notes: 'Verbose fallback analytics row 94 with context about historical cohorts and operating constraints.' },
+	{ metric: 'metric_95', value: 285, segment: 'segment_4', notes: 'Verbose fallback analytics row 95 with context about historical cohorts and operating constraints.' },
+	{ metric: 'metric_96', value: 288, segment: 'segment_5', notes: 'Verbose fallback analytics row 96 with context about historical cohorts and operating constraints.' },
+	{ metric: 'metric_97', value: 291, segment: 'segment_6', notes: 'Verbose fallback analytics row 97 with context about historical cohorts and operating constraints.' },
+	{ metric: 'metric_98', value: 294, segment: 'segment_0', notes: 'Verbose fallback analytics row 98 with context about historical cohorts and operating constraints.' },
+	{ metric: 'metric_99', value: 297, segment: 'segment_1', notes: 'Verbose fallback analytics row 99 with context about historical cohorts and operating constraints.' },
+	{ metric: 'metric_100', value: 300, segment: 'segment_2', notes: 'Verbose fallback analytics row 100 with context about historical cohorts and operating constraints.' },
+	{ metric: 'metric_101', value: 303, segment: 'segment_3', notes: 'Verbose fallback analytics row 101 with context about historical cohorts and operating constraints.' },
+	{ metric: 'metric_102', value: 306, segment: 'segment_4', notes: 'Verbose fallback analytics row 102 with context about historical cohorts and operating constraints.' },
+	{ metric: 'metric_103', value: 309, segment: 'segment_5', notes: 'Verbose fallback analytics row 103 with context about historical cohorts and operating constraints.' },
+	{ metric: 'metric_104', value: 312, segment: 'segment_6', notes: 'Verbose fallback analytics row 104 with context about historical cohorts and operating constraints.' },
+	{ metric: 'metric_105', value: 315, segment: 'segment_0', notes: 'Verbose fallback analytics row 105 with context about historical cohorts and operating constraints.' },
+	{ metric: 'metric_106', value: 318, segment: 'segment_1', notes: 'Verbose fallback analytics row 106 with context about historical cohorts and operating constraints.' },
+	{ metric: 'metric_107', value: 321, segment: 'segment_2', notes: 'Verbose fallback analytics row 107 with context about historical cohorts and operating constraints.' },
+	{ metric: 'metric_108', value: 324, segment: 'segment_3', notes: 'Verbose fallback analytics row 108 with context about historical cohorts and operating constraints.' },
+	{ metric: 'metric_109', value: 327, segment: 'segment_4', notes: 'Verbose fallback analytics row 109 with context about historical cohorts and operating constraints.' },
+	{ metric: 'metric_110', value: 330, segment: 'segment_5', notes: 'Verbose fallback analytics row 110 with context about historical cohorts and operating constraints.' },
+	{ metric: 'metric_111', value: 333, segment: 'segment_6', notes: 'Verbose fallback analytics row 111 with context about historical cohorts and operating constraints.' },
+	{ metric: 'metric_112', value: 336, segment: 'segment_0', notes: 'Verbose fallback analytics row 112 with context about historical cohorts and operating constraints.' },
+	{ metric: 'metric_113', value: 339, segment: 'segment_1', notes: 'Verbose fallback analytics row 113 with context about historical cohorts and operating constraints.' },
+	{ metric: 'metric_114', value: 342, segment: 'segment_2', notes: 'Verbose fallback analytics row 114 with context about historical cohorts and operating constraints.' },
+	{ metric: 'metric_115', value: 345, segment: 'segment_3', notes: 'Verbose fallback analytics row 115 with context about historical cohorts and operating constraints.' },
+	{ metric: 'metric_116', value: 348, segment: 'segment_4', notes: 'Verbose fallback analytics row 116 with context about historical cohorts and operating constraints.' },
+	{ metric: 'metric_117', value: 351, segment: 'segment_5', notes: 'Verbose fallback analytics row 117 with context about historical cohorts and operating constraints.' },
+	{ metric: 'metric_118', value: 354, segment: 'segment_6', notes: 'Verbose fallback analytics row 118 with context about historical cohorts and operating constraints.' },
+	{ metric: 'metric_119', value: 357, segment: 'segment_0', notes: 'Verbose fallback analytics row 119 with context about historical cohorts and operating constraints.' },
+	{ metric: 'metric_120', value: 360, segment: 'segment_1', notes: 'Verbose fallback analytics row 120 with context about historical cohorts and operating constraints.' },
+	{ metric: 'metric_121', value: 363, segment: 'segment_2', notes: 'Verbose fallback analytics row 121 with context about historical cohorts and operating constraints.' },
+	{ metric: 'metric_122', value: 366, segment: 'segment_3', notes: 'Verbose fallback analytics row 122 with context about historical cohorts and operating constraints.' },
+	{ metric: 'metric_123', value: 369, segment: 'segment_4', notes: 'Verbose fallback analytics row 123 with context about historical cohorts and operating constraints.' },
+	{ metric: 'metric_124', value: 372, segment: 'segment_5', notes: 'Verbose fallback analytics row 124 with context about historical cohorts and operating constraints.' },
+	{ metric: 'metric_125', value: 375, segment: 'segment_6', notes: 'Verbose fallback analytics row 125 with context about historical cohorts and operating constraints.' },
+	{ metric: 'metric_126', value: 378, segment: 'segment_0', notes: 'Verbose fallback analytics row 126 with context about historical cohorts and operating constraints.' },
+	{ metric: 'metric_127', value: 381, segment: 'segment_1', notes: 'Verbose fallback analytics row 127 with context about historical cohorts and operating constraints.' },
+	{ metric: 'metric_128', value: 384, segment: 'segment_2', notes: 'Verbose fallback analytics row 128 with context about historical cohorts and operating constraints.' },
+	{ metric: 'metric_129', value: 387, segment: 'segment_3', notes: 'Verbose fallback analytics row 129 with context about historical cohorts and operating constraints.' },
+	{ metric: 'metric_130', value: 390, segment: 'segment_4', notes: 'Verbose fallback analytics row 130 with context about historical cohorts and operating constraints.' },
+	{ metric: 'metric_131', value: 393, segment: 'segment_5', notes: 'Verbose fallback analytics row 131 with context about historical cohorts and operating constraints.' },
+	{ metric: 'metric_132', value: 396, segment: 'segment_6', notes: 'Verbose fallback analytics row 132 with context about historical cohorts and operating constraints.' },
+	{ metric: 'metric_133', value: 399, segment: 'segment_0', notes: 'Verbose fallback analytics row 133 with context about historical cohorts and operating constraints.' },
+	{ metric: 'metric_134', value: 402, segment: 'segment_1', notes: 'Verbose fallback analytics row 134 with context about historical cohorts and operating constraints.' },
+	{ metric: 'metric_135', value: 405, segment: 'segment_2', notes: 'Verbose fallback analytics row 135 with context about historical cohorts and operating constraints.' },
+	{ metric: 'metric_136', value: 408, segment: 'segment_3', notes: 'Verbose fallback analytics row 136 with context about historical cohorts and operating constraints.' },
+	{ metric: 'metric_137', value: 411, segment: 'segment_4', notes: 'Verbose fallback analytics row 137 with context about historical cohorts and operating constraints.' },
+	{ metric: 'metric_138', value: 414, segment: 'segment_5', notes: 'Verbose fallback analytics row 138 with context about historical cohorts and operating constraints.' },
+	{ metric: 'metric_139', value: 417, segment: 'segment_6', notes: 'Verbose fallback analytics row 139 with context about historical cohorts and operating constraints.' },
+	{ metric: 'metric_140', value: 420, segment: 'segment_0', notes: 'Verbose fallback analytics row 140 with context about historical cohorts and operating constraints.' },
+	{ metric: 'metric_141', value: 423, segment: 'segment_1', notes: 'Verbose fallback analytics row 141 with context about historical cohorts and operating constraints.' },
+	{ metric: 'metric_142', value: 426, segment: 'segment_2', notes: 'Verbose fallback analytics row 142 with context about historical cohorts and operating constraints.' },
+	{ metric: 'metric_143', value: 429, segment: 'segment_3', notes: 'Verbose fallback analytics row 143 with context about historical cohorts and operating constraints.' },
+	{ metric: 'metric_144', value: 432, segment: 'segment_4', notes: 'Verbose fallback analytics row 144 with context about historical cohorts and operating constraints.' },
+	{ metric: 'metric_145', value: 435, segment: 'segment_5', notes: 'Verbose fallback analytics row 145 with context about historical cohorts and operating constraints.' },
+	{ metric: 'metric_146', value: 438, segment: 'segment_6', notes: 'Verbose fallback analytics row 146 with context about historical cohorts and operating constraints.' },
+	{ metric: 'metric_147', value: 441, segment: 'segment_0', notes: 'Verbose fallback analytics row 147 with context about historical cohorts and operating constraints.' },
+	{ metric: 'metric_148', value: 444, segment: 'segment_1', notes: 'Verbose fallback analytics row 148 with context about historical cohorts and operating constraints.' },
+	{ metric: 'metric_149', value: 447, segment: 'segment_2', notes: 'Verbose fallback analytics row 149 with context about historical cohorts and operating constraints.' },
+	{ metric: 'metric_150', value: 450, segment: 'segment_3', notes: 'Verbose fallback analytics row 150 with context about historical cohorts and operating constraints.' },
+	{ metric: 'metric_151', value: 453, segment: 'segment_4', notes: 'Verbose fallback analytics row 151 with context about historical cohorts and operating constraints.' },
+	{ metric: 'metric_152', value: 456, segment: 'segment_5', notes: 'Verbose fallback analytics row 152 with context about historical cohorts and operating constraints.' },
+	{ metric: 'metric_153', value: 459, segment: 'segment_6', notes: 'Verbose fallback analytics row 153 with context about historical cohorts and operating constraints.' },
+	{ metric: 'metric_154', value: 462, segment: 'segment_0', notes: 'Verbose fallback analytics row 154 with context about historical cohorts and operating constraints.' },
+	{ metric: 'metric_155', value: 465, segment: 'segment_1', notes: 'Verbose fallback analytics row 155 with context about historical cohorts and operating constraints.' },
+	{ metric: 'metric_156', value: 468, segment: 'segment_2', notes: 'Verbose fallback analytics row 156 with context about historical cohorts and operating constraints.' },
+	{ metric: 'metric_157', value: 471, segment: 'segment_3', notes: 'Verbose fallback analytics row 157 with context about historical cohorts and operating constraints.' },
+	{ metric: 'metric_158', value: 474, segment: 'segment_4', notes: 'Verbose fallback analytics row 158 with context about historical cohorts and operating constraints.' },
+	{ metric: 'metric_159', value: 477, segment: 'segment_5', notes: 'Verbose fallback analytics row 159 with context about historical cohorts and operating constraints.' },
+	{ metric: 'metric_160', value: 480, segment: 'segment_6', notes: 'Verbose fallback analytics row 160 with context about historical cohorts and operating constraints.' },
+	{ metric: 'metric_161', value: 483, segment: 'segment_0', notes: 'Verbose fallback analytics row 161 with context about historical cohorts and operating constraints.' },
+	{ metric: 'metric_162', value: 486, segment: 'segment_1', notes: 'Verbose fallback analytics row 162 with context about historical cohorts and operating constraints.' },
+	{ metric: 'metric_163', value: 489, segment: 'segment_2', notes: 'Verbose fallback analytics row 163 with context about historical cohorts and operating constraints.' },
+	{ metric: 'metric_164', value: 492, segment: 'segment_3', notes: 'Verbose fallback analytics row 164 with context about historical cohorts and operating constraints.' },
+	{ metric: 'metric_165', value: 495, segment: 'segment_4', notes: 'Verbose fallback analytics row 165 with context about historical cohorts and operating constraints.' },
+	{ metric: 'metric_166', value: 498, segment: 'segment_5', notes: 'Verbose fallback analytics row 166 with context about historical cohorts and operating constraints.' },
+	{ metric: 'metric_167', value: 501, segment: 'segment_6', notes: 'Verbose fallback analytics row 167 with context about historical cohorts and operating constraints.' },
+	{ metric: 'metric_168', value: 504, segment: 'segment_0', notes: 'Verbose fallback analytics row 168 with context about historical cohorts and operating constraints.' },
+	{ metric: 'metric_169', value: 507, segment: 'segment_1', notes: 'Verbose fallback analytics row 169 with context about historical cohorts and operating constraints.' },
+	{ metric: 'metric_170', value: 510, segment: 'segment_2', notes: 'Verbose fallback analytics row 170 with context about historical cohorts and operating constraints.' },
+	{ metric: 'metric_171', value: 513, segment: 'segment_3', notes: 'Verbose fallback analytics row 171 with context about historical cohorts and operating constraints.' },
+	{ metric: 'metric_172', value: 516, segment: 'segment_4', notes: 'Verbose fallback analytics row 172 with context about historical cohorts and operating constraints.' },
+	{ metric: 'metric_173', value: 519, segment: 'segment_5', notes: 'Verbose fallback analytics row 173 with context about historical cohorts and operating constraints.' },
+	{ metric: 'metric_174', value: 522, segment: 'segment_6', notes: 'Verbose fallback analytics row 174 with context about historical cohorts and operating constraints.' },
+	{ metric: 'metric_175', value: 525, segment: 'segment_0', notes: 'Verbose fallback analytics row 175 with context about historical cohorts and operating constraints.' },
+	{ metric: 'metric_176', value: 528, segment: 'segment_1', notes: 'Verbose fallback analytics row 176 with context about historical cohorts and operating constraints.' },
+	{ metric: 'metric_177', value: 531, segment: 'segment_2', notes: 'Verbose fallback analytics row 177 with context about historical cohorts and operating constraints.' },
+	{ metric: 'metric_178', value: 534, segment: 'segment_3', notes: 'Verbose fallback analytics row 178 with context about historical cohorts and operating constraints.' },
+	{ metric: 'metric_179', value: 537, segment: 'segment_4', notes: 'Verbose fallback analytics row 179 with context about historical cohorts and operating constraints.' },
+]
+
+export async function main(range: string = '30d') {
+	const sql = wmill.datatable()
+	let rows: AnalyticsRow[] = []
+	try {
+		rows = await sql`SELECT metric, value, segment, notes FROM analytics_rollups WHERE range = ${range} LIMIT 50`.fetch()
+	} catch {
+		rows = []
+	}
+	const sourceRows = Array.isArray(rows) && rows.length > 0 ? rows : fallbackRows
+	const total = sourceRows.reduce((acc, row) => acc + Number(row.value ?? 0), 0)
+	return {
+		range,
+		summary: `Loaded ${sourceRows.length} analytics rows with total ${total}`,
+		rows: sourceRows.slice(0, 10)
+	}
+}

ai_evals/fixtures/frontend/app/initial/token_heavy_context/backend/loadAnalytics/meta.json

@@ -0,0 +1,4 @@
+{
+  "name": "Load analytics data",
+  "language": "bun"
+}

ai_evals/fixtures/frontend/app/initial/token_heavy_context/backend/refreshSummary/main.ts

@@ -0,0 +1,3 @@
+export async function main() {
+	return { refreshed: true, message: 'Summary refreshed' }
+}

ai_evals/fixtures/frontend/app/initial/token_heavy_context/backend/refreshSummary/meta.json

@@ -0,0 +1,4 @@
+{
+  "name": "Refresh summary",
+  "language": "bun"
+}

ai_evals/fixtures/frontend/app/initial/token_heavy_context/frontend/components/ReferencePanel.tsx

@@ -0,0 +1,185 @@
+import React from 'react'
+
+export const copyBlock1 = 'Long reference copy block 1: users compare conversion funnels, weekly retention, activation cohorts, support escalations, and revenue quality indicators across multiple teams.'
+export const copyBlock2 = 'Long reference copy block 2: users compare conversion funnels, weekly retention, activation cohorts, support escalations, and revenue quality indicators across multiple teams.'
+export const copyBlock3 = 'Long reference copy block 3: users compare conversion funnels, weekly retention, activation cohorts, support escalations, and revenue quality indicators across multiple teams.'
+export const copyBlock4 = 'Long reference copy block 4: users compare conversion funnels, weekly retention, activation cohorts, support escalations, and revenue quality indicators across multiple teams.'
+export const copyBlock5 = 'Long reference copy block 5: users compare conversion funnels, weekly retention, activation cohorts, support escalations, and revenue quality indicators across multiple teams.'
+export const copyBlock6 = 'Long reference copy block 6: users compare conversion funnels, weekly retention, activation cohorts, support escalations, and revenue quality indicators across multiple teams.'
+export const copyBlock7 = 'Long reference copy block 7: users compare conversion funnels, weekly retention, activation cohorts, support escalations, and revenue quality indicators across multiple teams.'
+export const copyBlock8 = 'Long reference copy block 8: users compare conversion funnels, weekly retention, activation cohorts, support escalations, and revenue quality indicators across multiple teams.'
+export const copyBlock9 = 'Long reference copy block 9: users compare conversion funnels, weekly retention, activation cohorts, support escalations, and revenue quality indicators across multiple teams.'
+export const copyBlock10 = 'Long reference copy block 10: users compare conversion funnels, weekly retention, activation cohorts, support escalations, and revenue quality indicators across multiple teams.'
+export const copyBlock11 = 'Long reference copy block 11: users compare conversion funnels, weekly retention, activation cohorts, support escalations, and revenue quality indicators across multiple teams.'
+export const copyBlock12 = 'Long reference copy block 12: users compare conversion funnels, weekly retention, activation cohorts, support escalations, and revenue quality indicators across multiple teams.'
+export const copyBlock13 = 'Long reference copy block 13: users compare conversion funnels, weekly retention, activation cohorts, support escalations, and revenue quality indicators across multiple teams.'
+export const copyBlock14 = 'Long reference copy block 14: users compare conversion funnels, weekly retention, activation cohorts, support escalations, and revenue quality indicators across multiple teams.'
+export const copyBlock15 = 'Long reference copy block 15: users compare conversion funnels, weekly retention, activation cohorts, support escalations, and revenue quality indicators across multiple teams.'
+export const copyBlock16 = 'Long reference copy block 16: users compare conversion funnels, weekly retention, activation cohorts, support escalations, and revenue quality indicators across multiple teams.'
+export const copyBlock17 = 'Long reference copy block 17: users compare conversion funnels, weekly retention, activation cohorts, support escalations, and revenue quality indicators across multiple teams.'
+export const copyBlock18 = 'Long reference copy block 18: users compare conversion funnels, weekly retention, activation cohorts, support escalations, and revenue quality indicators across multiple teams.'
+export const copyBlock19 = 'Long reference copy block 19: users compare conversion funnels, weekly retention, activation cohorts, support escalations, and revenue quality indicators across multiple teams.'
+export const copyBlock20 = 'Long reference copy block 20: users compare conversion funnels, weekly retention, activation cohorts, support escalations, and revenue quality indicators across multiple teams.'
+export const copyBlock21 = 'Long reference copy block 21: users compare conversion funnels, weekly retention, activation cohorts, support escalations, and revenue quality indicators across multiple teams.'
+export const copyBlock22 = 'Long reference copy block 22: users compare conversion funnels, weekly retention, activation cohorts, support escalations, and revenue quality indicators across multiple teams.'
+export const copyBlock23 = 'Long reference copy block 23: users compare conversion funnels, weekly retention, activation cohorts, support escalations, and revenue quality indicators across multiple teams.'
+export const copyBlock24 = 'Long reference copy block 24: users compare conversion funnels, weekly retention, activation cohorts, support escalations, and revenue quality indicators across multiple teams.'
+export const copyBlock25 = 'Long reference copy block 25: users compare conversion funnels, weekly retention, activation cohorts, support escalations, and revenue quality indicators across multiple teams.'
+export const copyBlock26 = 'Long reference copy block 26: users compare conversion funnels, weekly retention, activation cohorts, support escalations, and revenue quality indicators across multiple teams.'
+export const copyBlock27 = 'Long reference copy block 27: users compare conversion funnels, weekly retention, activation cohorts, support escalations, and revenue quality indicators across multiple teams.'
+export const copyBlock28 = 'Long reference copy block 28: users compare conversion funnels, weekly retention, activation cohorts, support escalations, and revenue quality indicators across multiple teams.'
+export const copyBlock29 = 'Long reference copy block 29: users compare conversion funnels, weekly retention, activation cohorts, support escalations, and revenue quality indicators across multiple teams.'
+export const copyBlock30 = 'Long reference copy block 30: users compare conversion funnels, weekly retention, activation cohorts, support escalations, and revenue quality indicators across multiple teams.'
+export const copyBlock31 = 'Long reference copy block 31: users compare conversion funnels, weekly retention, activation cohorts, support escalations, and revenue quality indicators across multiple teams.'
+export const copyBlock32 = 'Long reference copy block 32: users compare conversion funnels, weekly retention, activation cohorts, support escalations, and revenue quality indicators across multiple teams.'
+export const copyBlock33 = 'Long reference copy block 33: users compare conversion funnels, weekly retention, activation cohorts, support escalations, and revenue quality indicators across multiple teams.'
+export const copyBlock34 = 'Long reference copy block 34: users compare conversion funnels, weekly retention, activation cohorts, support escalations, and revenue quality indicators across multiple teams.'
+export const copyBlock35 = 'Long reference copy block 35: users compare conversion funnels, weekly retention, activation cohorts, support escalations, and revenue quality indicators across multiple teams.'
+export const copyBlock36 = 'Long reference copy block 36: users compare conversion funnels, weekly retention, activation cohorts, support escalations, and revenue quality indicators across multiple teams.'
+export const copyBlock37 = 'Long reference copy block 37: users compare conversion funnels, weekly retention, activation cohorts, support escalations, and revenue quality indicators across multiple teams.'
+export const copyBlock38 = 'Long reference copy block 38: users compare conversion funnels, weekly retention, activation cohorts, support escalations, and revenue quality indicators across multiple teams.'
+export const copyBlock39 = 'Long reference copy block 39: users compare conversion funnels, weekly retention, activation cohorts, support escalations, and revenue quality indicators across multiple teams.'
+export const copyBlock40 = 'Long reference copy block 40: users compare conversion funnels, weekly retention, activation cohorts, support escalations, and revenue quality indicators across multiple teams.'
+export const copyBlock41 = 'Long reference copy block 41: users compare conversion funnels, weekly retention, activation cohorts, support escalations, and revenue quality indicators across multiple teams.'
+export const copyBlock42 = 'Long reference copy block 42: users compare conversion funnels, weekly retention, activation cohorts, support escalations, and revenue quality indicators across multiple teams.'
+export const copyBlock43 = 'Long reference copy block 43: users compare conversion funnels, weekly retention, activation cohorts, support escalations, and revenue quality indicators across multiple teams.'
+export const copyBlock44 = 'Long reference copy block 44: users compare conversion funnels, weekly retention, activation cohorts, support escalations, and revenue quality indicators across multiple teams.'
+export const copyBlock45 = 'Long reference copy block 45: users compare conversion funnels, weekly retention, activation cohorts, support escalations, and revenue quality indicators across multiple teams.'
+export const copyBlock46 = 'Long reference copy block 46: users compare conversion funnels, weekly retention, activation cohorts, support escalations, and revenue quality indicators across multiple teams.'
+export const copyBlock47 = 'Long reference copy block 47: users compare conversion funnels, weekly retention, activation cohorts, support escalations, and revenue quality indicators across multiple teams.'
+export const copyBlock48 = 'Long reference copy block 48: users compare conversion funnels, weekly retention, activation cohorts, support escalations, and revenue quality indicators across multiple teams.'
+export const copyBlock49 = 'Long reference copy block 49: users compare conversion funnels, weekly retention, activation cohorts, support escalations, and revenue quality indicators across multiple teams.'
+export const copyBlock50 = 'Long reference copy block 50: users compare conversion funnels, weekly retention, activation cohorts, support escalations, and revenue quality indicators across multiple teams.'
+export const copyBlock51 = 'Long reference copy block 51: users compare conversion funnels, weekly retention, activation cohorts, support escalations, and revenue quality indicators across multiple teams.'
+export const copyBlock52 = 'Long reference copy block 52: users compare conversion funnels, weekly retention, activation cohorts, support escalations, and revenue quality indicators across multiple teams.'
+export const copyBlock53 = 'Long reference copy block 53: users compare conversion funnels, weekly retention, activation cohorts, support escalations, and revenue quality indicators across multiple teams.'
+export const copyBlock54 = 'Long reference copy block 54: users compare conversion funnels, weekly retention, activation cohorts, support escalations, and revenue quality indicators across multiple teams.'
+export const copyBlock55 = 'Long reference copy block 55: users compare conversion funnels, weekly retention, activation cohorts, support escalations, and revenue quality indicators across multiple teams.'
+export const copyBlock56 = 'Long reference copy block 56: users compare conversion funnels, weekly retention, activation cohorts, support escalations, and revenue quality indicators across multiple teams.'
+export const copyBlock57 = 'Long reference copy block 57: users compare conversion funnels, weekly retention, activation cohorts, support escalations, and revenue quality indicators across multiple teams.'
+export const copyBlock58 = 'Long reference copy block 58: users compare conversion funnels, weekly retention, activation cohorts, support escalations, and revenue quality indicators across multiple teams.'
+export const copyBlock59 = 'Long reference copy block 59: users compare conversion funnels, weekly retention, activation cohorts, support escalations, and revenue quality indicators across multiple teams.'
+export const copyBlock60 = 'Long reference copy block 60: users compare conversion funnels, weekly retention, activation cohorts, support escalations, and revenue quality indicators across multiple teams.'
+export const copyBlock61 = 'Long reference copy block 61: users compare conversion funnels, weekly retention, activation cohorts, support escalations, and revenue quality indicators across multiple teams.'
+export const copyBlock62 = 'Long reference copy block 62: users compare conversion funnels, weekly retention, activation cohorts, support escalations, and revenue quality indicators across multiple teams.'
+export const copyBlock63 = 'Long reference copy block 63: users compare conversion funnels, weekly retention, activation cohorts, support escalations, and revenue quality indicators across multiple teams.'
+export const copyBlock64 = 'Long reference copy block 64: users compare conversion funnels, weekly retention, activation cohorts, support escalations, and revenue quality indicators across multiple teams.'
+export const copyBlock65 = 'Long reference copy block 65: users compare conversion funnels, weekly retention, activation cohorts, support escalations, and revenue quality indicators across multiple teams.'
+export const copyBlock66 = 'Long reference copy block 66: users compare conversion funnels, weekly retention, activation cohorts, support escalations, and revenue quality indicators across multiple teams.'
+export const copyBlock67 = 'Long reference copy block 67: users compare conversion funnels, weekly retention, activation cohorts, support escalations, and revenue quality indicators across multiple teams.'
+export const copyBlock68 = 'Long reference copy block 68: users compare conversion funnels, weekly retention, activation cohorts, support escalations, and revenue quality indicators across multiple teams.'
+export const copyBlock69 = 'Long reference copy block 69: users compare conversion funnels, weekly retention, activation cohorts, support escalations, and revenue quality indicators across multiple teams.'
+export const copyBlock70 = 'Long reference copy block 70: users compare conversion funnels, weekly retention, activation cohorts, support escalations, and revenue quality indicators across multiple teams.'
+export const copyBlock71 = 'Long reference copy block 71: users compare conversion funnels, weekly retention, activation cohorts, support escalations, and revenue quality indicators across multiple teams.'
+export const copyBlock72 = 'Long reference copy block 72: users compare conversion funnels, weekly retention, activation cohorts, support escalations, and revenue quality indicators across multiple teams.'
+export const copyBlock73 = 'Long reference copy block 73: users compare conversion funnels, weekly retention, activation cohorts, support escalations, and revenue quality indicators across multiple teams.'
+export const copyBlock74 = 'Long reference copy block 74: users compare conversion funnels, weekly retention, activation cohorts, support escalations, and revenue quality indicators across multiple teams.'
+export const copyBlock75 = 'Long reference copy block 75: users compare conversion funnels, weekly retention, activation cohorts, support escalations, and revenue quality indicators across multiple teams.'
+export const copyBlock76 = 'Long reference copy block 76: users compare conversion funnels, weekly retention, activation cohorts, support escalations, and revenue quality indicators across multiple teams.'
+export const copyBlock77 = 'Long reference copy block 77: users compare conversion funnels, weekly retention, activation cohorts, support escalations, and revenue quality indicators across multiple teams.'
+export const copyBlock78 = 'Long reference copy block 78: users compare conversion funnels, weekly retention, activation cohorts, support escalations, and revenue quality indicators across multiple teams.'
+export const copyBlock79 = 'Long reference copy block 79: users compare conversion funnels, weekly retention, activation cohorts, support escalations, and revenue quality indicators across multiple teams.'
+export const copyBlock80 = 'Long reference copy block 80: users compare conversion funnels, weekly retention, activation cohorts, support escalations, and revenue quality indicators across multiple teams.'
+export const copyBlock81 = 'Long reference copy block 81: users compare conversion funnels, weekly retention, activation cohorts, support escalations, and revenue quality indicators across multiple teams.'
+export const copyBlock82 = 'Long reference copy block 82: users compare conversion funnels, weekly retention, activation cohorts, support escalations, and revenue quality indicators across multiple teams.'
+export const copyBlock83 = 'Long reference copy block 83: users compare conversion funnels, weekly retention, activation cohorts, support escalations, and revenue quality indicators across multiple teams.'
+export const copyBlock84 = 'Long reference copy block 84: users compare conversion funnels, weekly retention, activation cohorts, support escalations, and revenue quality indicators across multiple teams.'
+export const copyBlock85 = 'Long reference copy block 85: users compare conversion funnels, weekly retention, activation cohorts, support escalations, and revenue quality indicators across multiple teams.'
+export const copyBlock86 = 'Long reference copy block 86: users compare conversion funnels, weekly retention, activation cohorts, support escalations, and revenue quality indicators across multiple teams.'
+export const copyBlock87 = 'Long reference copy block 87: users compare conversion funnels, weekly retention, activation cohorts, support escalations, and revenue quality indicators across multiple teams.'
+export const copyBlock88 = 'Long reference copy block 88: users compare conversion funnels, weekly retention, activation cohorts, support escalations, and revenue quality indicators across multiple teams.'
+export const copyBlock89 = 'Long reference copy block 89: users compare conversion funnels, weekly retention, activation cohorts, support escalations, and revenue quality indicators across multiple teams.'
+export const copyBlock90 = 'Long reference copy block 90: users compare conversion funnels, weekly retention, activation cohorts, support escalations, and revenue quality indicators across multiple teams.'
+export const copyBlock91 = 'Long reference copy block 91: users compare conversion funnels, weekly retention, activation cohorts, support escalations, and revenue quality indicators across multiple teams.'
+export const copyBlock92 = 'Long reference copy block 92: users compare conversion funnels, weekly retention, activation cohorts, support escalations, and revenue quality indicators across multiple teams.'
+export const copyBlock93 = 'Long reference copy block 93: users compare conversion funnels, weekly retention, activation cohorts, support escalations, and revenue quality indicators across multiple teams.'
+export const copyBlock94 = 'Long reference copy block 94: users compare conversion funnels, weekly retention, activation cohorts, support escalations, and revenue quality indicators across multiple teams.'
+export const copyBlock95 = 'Long reference copy block 95: users compare conversion funnels, weekly retention, activation cohorts, support escalations, and revenue quality indicators across multiple teams.'
+export const copyBlock96 = 'Long reference copy block 96: users compare conversion funnels, weekly retention, activation cohorts, support escalations, and revenue quality indicators across multiple teams.'
+export const copyBlock97 = 'Long reference copy block 97: users compare conversion funnels, weekly retention, activation cohorts, support escalations, and revenue quality indicators across multiple teams.'
+export const copyBlock98 = 'Long reference copy block 98: users compare conversion funnels, weekly retention, activation cohorts, support escalations, and revenue quality indicators across multiple teams.'
+export const copyBlock99 = 'Long reference copy block 99: users compare conversion funnels, weekly retention, activation cohorts, support escalations, and revenue quality indicators across multiple teams.'
+export const copyBlock100 = 'Long reference copy block 100: users compare conversion funnels, weekly retention, activation cohorts, support escalations, and revenue quality indicators across multiple teams.'
+export const copyBlock101 = 'Long reference copy block 101: users compare conversion funnels, weekly retention, activation cohorts, support escalations, and revenue quality indicators across multiple teams.'
+export const copyBlock102 = 'Long reference copy block 102: users compare conversion funnels, weekly retention, activation cohorts, support escalations, and revenue quality indicators across multiple teams.'
+export const copyBlock103 = 'Long reference copy block 103: users compare conversion funnels, weekly retention, activation cohorts, support escalations, and revenue quality indicators across multiple teams.'
+export const copyBlock104 = 'Long reference copy block 104: users compare conversion funnels, weekly retention, activation cohorts, support escalations, and revenue quality indicators across multiple teams.'
+export const copyBlock105 = 'Long reference copy block 105: users compare conversion funnels, weekly retention, activation cohorts, support escalations, and revenue quality indicators across multiple teams.'
+export const copyBlock106 = 'Long reference copy block 106: users compare conversion funnels, weekly retention, activation cohorts, support escalations, and revenue quality indicators across multiple teams.'
+export const copyBlock107 = 'Long reference copy block 107: users compare conversion funnels, weekly retention, activation cohorts, support escalations, and revenue quality indicators across multiple teams.'
+export const copyBlock108 = 'Long reference copy block 108: users compare conversion funnels, weekly retention, activation cohorts, support escalations, and revenue quality indicators across multiple teams.'
+export const copyBlock109 = 'Long reference copy block 109: users compare conversion funnels, weekly retention, activation cohorts, support escalations, and revenue quality indicators across multiple teams.'
+export const copyBlock110 = 'Long reference copy block 110: users compare conversion funnels, weekly retention, activation cohorts, support escalations, and revenue quality indicators across multiple teams.'
+export const copyBlock111 = 'Long reference copy block 111: users compare conversion funnels, weekly retention, activation cohorts, support escalations, and revenue quality indicators across multiple teams.'
+export const copyBlock112 = 'Long reference copy block 112: users compare conversion funnels, weekly retention, activation cohorts, support escalations, and revenue quality indicators across multiple teams.'
+export const copyBlock113 = 'Long reference copy block 113: users compare conversion funnels, weekly retention, activation cohorts, support escalations, and revenue quality indicators across multiple teams.'
+export const copyBlock114 = 'Long reference copy block 114: users compare conversion funnels, weekly retention, activation cohorts, support escalations, and revenue quality indicators across multiple teams.'
+export const copyBlock115 = 'Long reference copy block 115: users compare conversion funnels, weekly retention, activation cohorts, support escalations, and revenue quality indicators across multiple teams.'
+export const copyBlock116 = 'Long reference copy block 116: users compare conversion funnels, weekly retention, activation cohorts, support escalations, and revenue quality indicators across multiple teams.'
+export const copyBlock117 = 'Long reference copy block 117: users compare conversion funnels, weekly retention, activation cohorts, support escalations, and revenue quality indicators across multiple teams.'
+export const copyBlock118 = 'Long reference copy block 118: users compare conversion funnels, weekly retention, activation cohorts, support escalations, and revenue quality indicators across multiple teams.'
+export const copyBlock119 = 'Long reference copy block 119: users compare conversion funnels, weekly retention, activation cohorts, support escalations, and revenue quality indicators across multiple teams.'
+export const copyBlock120 = 'Long reference copy block 120: users compare conversion funnels, weekly retention, activation cohorts, support escalations, and revenue quality indicators across multiple teams.'
+export const copyBlock121 = 'Long reference copy block 121: users compare conversion funnels, weekly retention, activation cohorts, support escalations, and revenue quality indicators across multiple teams.'
+export const copyBlock122 = 'Long reference copy block 122: users compare conversion funnels, weekly retention, activation cohorts, support escalations, and revenue quality indicators across multiple teams.'
+export const copyBlock123 = 'Long reference copy block 123: users compare conversion funnels, weekly retention, activation cohorts, support escalations, and revenue quality indicators across multiple teams.'
+export const copyBlock124 = 'Long reference copy block 124: users compare conversion funnels, weekly retention, activation cohorts, support escalations, and revenue quality indicators across multiple teams.'
+export const copyBlock125 = 'Long reference copy block 125: users compare conversion funnels, weekly retention, activation cohorts, support escalations, and revenue quality indicators across multiple teams.'
+export const copyBlock126 = 'Long reference copy block 126: users compare conversion funnels, weekly retention, activation cohorts, support escalations, and revenue quality indicators across multiple teams.'
+export const copyBlock127 = 'Long reference copy block 127: users compare conversion funnels, weekly retention, activation cohorts, support escalations, and revenue quality indicators across multiple teams.'
+export const copyBlock128 = 'Long reference copy block 128: users compare conversion funnels, weekly retention, activation cohorts, support escalations, and revenue quality indicators across multiple teams.'
+export const copyBlock129 = 'Long reference copy block 129: users compare conversion funnels, weekly retention, activation cohorts, support escalations, and revenue quality indicators across multiple teams.'
+export const copyBlock130 = 'Long reference copy block 130: users compare conversion funnels, weekly retention, activation cohorts, support escalations, and revenue quality indicators across multiple teams.'
+export const copyBlock131 = 'Long reference copy block 131: users compare conversion funnels, weekly retention, activation cohorts, support escalations, and revenue quality indicators across multiple teams.'
+export const copyBlock132 = 'Long reference copy block 132: users compare conversion funnels, weekly retention, activation cohorts, support escalations, and revenue quality indicators across multiple teams.'
+export const copyBlock133 = 'Long reference copy block 133: users compare conversion funnels, weekly retention, activation cohorts, support escalations, and revenue quality indicators across multiple teams.'
+export const copyBlock134 = 'Long reference copy block 134: users compare conversion funnels, weekly retention, activation cohorts, support escalations, and revenue quality indicators across multiple teams.'
+export const copyBlock135 = 'Long reference copy block 135: users compare conversion funnels, weekly retention, activation cohorts, support escalations, and revenue quality indicators across multiple teams.'
+export const copyBlock136 = 'Long reference copy block 136: users compare conversion funnels, weekly retention, activation cohorts, support escalations, and revenue quality indicators across multiple teams.'
+export const copyBlock137 = 'Long reference copy block 137: users compare conversion funnels, weekly retention, activation cohorts, support escalations, and revenue quality indicators across multiple teams.'
+export const copyBlock138 = 'Long reference copy block 138: users compare conversion funnels, weekly retention, activation cohorts, support escalations, and revenue quality indicators across multiple teams.'
+export const copyBlock139 = 'Long reference copy block 139: users compare conversion funnels, weekly retention, activation cohorts, support escalations, and revenue quality indicators across multiple teams.'
+export const copyBlock140 = 'Long reference copy block 140: users compare conversion funnels, weekly retention, activation cohorts, support escalations, and revenue quality indicators across multiple teams.'
+export const copyBlock141 = 'Long reference copy block 141: users compare conversion funnels, weekly retention, activation cohorts, support escalations, and revenue quality indicators across multiple teams.'
+export const copyBlock142 = 'Long reference copy block 142: users compare conversion funnels, weekly retention, activation cohorts, support escalations, and revenue quality indicators across multiple teams.'
+export const copyBlock143 = 'Long reference copy block 143: users compare conversion funnels, weekly retention, activation cohorts, support escalations, and revenue quality indicators across multiple teams.'
+export const copyBlock144 = 'Long reference copy block 144: users compare conversion funnels, weekly retention, activation cohorts, support escalations, and revenue quality indicators across multiple teams.'
+export const copyBlock145 = 'Long reference copy block 145: users compare conversion funnels, weekly retention, activation cohorts, support escalations, and revenue quality indicators across multiple teams.'
+export const copyBlock146 = 'Long reference copy block 146: users compare conversion funnels, weekly retention, activation cohorts, support escalations, and revenue quality indicators across multiple teams.'
+export const copyBlock147 = 'Long reference copy block 147: users compare conversion funnels, weekly retention, activation cohorts, support escalations, and revenue quality indicators across multiple teams.'
+export const copyBlock148 = 'Long reference copy block 148: users compare conversion funnels, weekly retention, activation cohorts, support escalations, and revenue quality indicators across multiple teams.'
+export const copyBlock149 = 'Long reference copy block 149: users compare conversion funnels, weekly retention, activation cohorts, support escalations, and revenue quality indicators across multiple teams.'
+export const copyBlock150 = 'Long reference copy block 150: users compare conversion funnels, weekly retention, activation cohorts, support escalations, and revenue quality indicators across multiple teams.'
+export const copyBlock151 = 'Long reference copy block 151: users compare conversion funnels, weekly retention, activation cohorts, support escalations, and revenue quality indicators across multiple teams.'
+export const copyBlock152 = 'Long reference copy block 152: users compare conversion funnels, weekly retention, activation cohorts, support escalations, and revenue quality indicators across multiple teams.'
+export const copyBlock153 = 'Long reference copy block 153: users compare conversion funnels, weekly retention, activation cohorts, support escalations, and revenue quality indicators across multiple teams.'
+export const copyBlock154 = 'Long reference copy block 154: users compare conversion funnels, weekly retention, activation cohorts, support escalations, and revenue quality indicators across multiple teams.'
+export const copyBlock155 = 'Long reference copy block 155: users compare conversion funnels, weekly retention, activation cohorts, support escalations, and revenue quality indicators across multiple teams.'
+export const copyBlock156 = 'Long reference copy block 156: users compare conversion funnels, weekly retention, activation cohorts, support escalations, and revenue quality indicators across multiple teams.'
+export const copyBlock157 = 'Long reference copy block 157: users compare conversion funnels, weekly retention, activation cohorts, support escalations, and revenue quality indicators across multiple teams.'
+export const copyBlock158 = 'Long reference copy block 158: users compare conversion funnels, weekly retention, activation cohorts, support escalations, and revenue quality indicators across multiple teams.'
+export const copyBlock159 = 'Long reference copy block 159: users compare conversion funnels, weekly retention, activation cohorts, support escalations, and revenue quality indicators across multiple teams.'
+export const copyBlock160 = 'Long reference copy block 160: users compare conversion funnels, weekly retention, activation cohorts, support escalations, and revenue quality indicators across multiple teams.'
+export const copyBlock161 = 'Long reference copy block 161: users compare conversion funnels, weekly retention, activation cohorts, support escalations, and revenue quality indicators across multiple teams.'
+export const copyBlock162 = 'Long reference copy block 162: users compare conversion funnels, weekly retention, activation cohorts, support escalations, and revenue quality indicators across multiple teams.'
+export const copyBlock163 = 'Long reference copy block 163: users compare conversion funnels, weekly retention, activation cohorts, support escalations, and revenue quality indicators across multiple teams.'
+export const copyBlock164 = 'Long reference copy block 164: users compare conversion funnels, weekly retention, activation cohorts, support escalations, and revenue quality indicators across multiple teams.'
+export const copyBlock165 = 'Long reference copy block 165: users compare conversion funnels, weekly retention, activation cohorts, support escalations, and revenue quality indicators across multiple teams.'
+export const copyBlock166 = 'Long reference copy block 166: users compare conversion funnels, weekly retention, activation cohorts, support escalations, and revenue quality indicators across multiple teams.'
+export const copyBlock167 = 'Long reference copy block 167: users compare conversion funnels, weekly retention, activation cohorts, support escalations, and revenue quality indicators across multiple teams.'
+export const copyBlock168 = 'Long reference copy block 168: users compare conversion funnels, weekly retention, activation cohorts, support escalations, and revenue quality indicators across multiple teams.'
+export const copyBlock169 = 'Long reference copy block 169: users compare conversion funnels, weekly retention, activation cohorts, support escalations, and revenue quality indicators across multiple teams.'
+export const copyBlock170 = 'Long reference copy block 170: users compare conversion funnels, weekly retention, activation cohorts, support escalations, and revenue quality indicators across multiple teams.'
+export const copyBlock171 = 'Long reference copy block 171: users compare conversion funnels, weekly retention, activation cohorts, support escalations, and revenue quality indicators across multiple teams.'
+export const copyBlock172 = 'Long reference copy block 172: users compare conversion funnels, weekly retention, activation cohorts, support escalations, and revenue quality indicators across multiple teams.'
+export const copyBlock173 = 'Long reference copy block 173: users compare conversion funnels, weekly retention, activation cohorts, support escalations, and revenue quality indicators across multiple teams.'
+export const copyBlock174 = 'Long reference copy block 174: users compare conversion funnels, weekly retention, activation cohorts, support escalations, and revenue quality indicators across multiple teams.'
+export const copyBlock175 = 'Long reference copy block 175: users compare conversion funnels, weekly retention, activation cohorts, support escalations, and revenue quality indicators across multiple teams.'
+export const copyBlock176 = 'Long reference copy block 176: users compare conversion funnels, weekly retention, activation cohorts, support escalations, and revenue quality indicators across multiple teams.'
+export const copyBlock177 = 'Long reference copy block 177: users compare conversion funnels, weekly retention, activation cohorts, support escalations, and revenue quality indicators across multiple teams.'
+export const copyBlock178 = 'Long reference copy block 178: users compare conversion funnels, weekly retention, activation cohorts, support escalations, and revenue quality indicators across multiple teams.'
+export const copyBlock179 = 'Long reference copy block 179: users compare conversion funnels, weekly retention, activation cohorts, support escalations, and revenue quality indicators across multiple teams.'
+
+export function ReferencePanel() {
+	return <div>Reference panel</div>
+}

ai_evals/fixtures/frontend/app/initial/token_heavy_context/frontend/data/labels.ts

@@ -0,0 +1,221 @@
+export const labels = [
+	'Dashboard label 1 with a verbose explanation for filtering and display',
+	'Dashboard label 2 with a verbose explanation for filtering and display',
+	'Dashboard label 3 with a verbose explanation for filtering and display',
+	'Dashboard label 4 with a verbose explanation for filtering and display',
+	'Dashboard label 5 with a verbose explanation for filtering and display',
+	'Dashboard label 6 with a verbose explanation for filtering and display',
+	'Dashboard label 7 with a verbose explanation for filtering and display',
+	'Dashboard label 8 with a verbose explanation for filtering and display',
+	'Dashboard label 9 with a verbose explanation for filtering and display',
+	'Dashboard label 10 with a verbose explanation for filtering and display',
+	'Dashboard label 11 with a verbose explanation for filtering and display',
+	'Dashboard label 12 with a verbose explanation for filtering and display',
+	'Dashboard label 13 with a verbose explanation for filtering and display',
+	'Dashboard label 14 with a verbose explanation for filtering and display',
+	'Dashboard label 15 with a verbose explanation for filtering and display',
+	'Dashboard label 16 with a verbose explanation for filtering and display',
+	'Dashboard label 17 with a verbose explanation for filtering and display',
+	'Dashboard label 18 with a verbose explanation for filtering and display',
+	'Dashboard label 19 with a verbose explanation for filtering and display',
+	'Dashboard label 20 with a verbose explanation for filtering and display',
+	'Dashboard label 21 with a verbose explanation for filtering and display',
+	'Dashboard label 22 with a verbose explanation for filtering and display',
+	'Dashboard label 23 with a verbose explanation for filtering and display',
+	'Dashboard label 24 with a verbose explanation for filtering and display',
+	'Dashboard label 25 with a verbose explanation for filtering and display',
+	'Dashboard label 26 with a verbose explanation for filtering and display',
+	'Dashboard label 27 with a verbose explanation for filtering and display',
+	'Dashboard label 28 with a verbose explanation for filtering and display',
+	'Dashboard label 29 with a verbose explanation for filtering and display',
+	'Dashboard label 30 with a verbose explanation for filtering and display',
+	'Dashboard label 31 with a verbose explanation for filtering and display',
+	'Dashboard label 32 with a verbose explanation for filtering and display',
+	'Dashboard label 33 with a verbose explanation for filtering and display',
+	'Dashboard label 34 with a verbose explanation for filtering and display',
+	'Dashboard label 35 with a verbose explanation for filtering and display',
+	'Dashboard label 36 with a verbose explanation for filtering and display',
+	'Dashboard label 37 with a verbose explanation for filtering and display',
+	'Dashboard label 38 with a verbose explanation for filtering and display',
+	'Dashboard label 39 with a verbose explanation for filtering and display',
+	'Dashboard label 40 with a verbose explanation for filtering and display',
+	'Dashboard label 41 with a verbose explanation for filtering and display',
+	'Dashboard label 42 with a verbose explanation for filtering and display',
+	'Dashboard label 43 with a verbose explanation for filtering and display',
+	'Dashboard label 44 with a verbose explanation for filtering and display',
+	'Dashboard label 45 with a verbose explanation for filtering and display',
+	'Dashboard label 46 with a verbose explanation for filtering and display',
+	'Dashboard label 47 with a verbose explanation for filtering and display',
+	'Dashboard label 48 with a verbose explanation for filtering and display',
+	'Dashboard label 49 with a verbose explanation for filtering and display',
+	'Dashboard label 50 with a verbose explanation for filtering and display',
+	'Dashboard label 51 with a verbose explanation for filtering and display',
+	'Dashboard label 52 with a verbose explanation for filtering and display',
+	'Dashboard label 53 with a verbose explanation for filtering and display',
+	'Dashboard label 54 with a verbose explanation for filtering and display',
+	'Dashboard label 55 with a verbose explanation for filtering and display',
+	'Dashboard label 56 with a verbose explanation for filtering and display',
+	'Dashboard label 57 with a verbose explanation for filtering and display',
+	'Dashboard label 58 with a verbose explanation for filtering and display',
+	'Dashboard label 59 with a verbose explanation for filtering and display',
+	'Dashboard label 60 with a verbose explanation for filtering and display',
+	'Dashboard label 61 with a verbose explanation for filtering and display',
+	'Dashboard label 62 with a verbose explanation for filtering and display',
+	'Dashboard label 63 with a verbose explanation for filtering and display',
+	'Dashboard label 64 with a verbose explanation for filtering and display',
+	'Dashboard label 65 with a verbose explanation for filtering and display',
+	'Dashboard label 66 with a verbose explanation for filtering and display',
+	'Dashboard label 67 with a verbose explanation for filtering and display',
+	'Dashboard label 68 with a verbose explanation for filtering and display',
+	'Dashboard label 69 with a verbose explanation for filtering and display',
+	'Dashboard label 70 with a verbose explanation for filtering and display',
+	'Dashboard label 71 with a verbose explanation for filtering and display',
+	'Dashboard label 72 with a verbose explanation for filtering and display',
+	'Dashboard label 73 with a verbose explanation for filtering and display',
+	'Dashboard label 74 with a verbose explanation for filtering and display',
+	'Dashboard label 75 with a verbose explanation for filtering and display',
+	'Dashboard label 76 with a verbose explanation for filtering and display',
+	'Dashboard label 77 with a verbose explanation for filtering and display',
+	'Dashboard label 78 with a verbose explanation for filtering and display',
+	'Dashboard label 79 with a verbose explanation for filtering and display',
+	'Dashboard label 80 with a verbose explanation for filtering and display',
+	'Dashboard label 81 with a verbose explanation for filtering and display',
+	'Dashboard label 82 with a verbose explanation for filtering and display',
+	'Dashboard label 83 with a verbose explanation for filtering and display',
+	'Dashboard label 84 with a verbose explanation for filtering and display',
+	'Dashboard label 85 with a verbose explanation for filtering and display',
+	'Dashboard label 86 with a verbose explanation for filtering and display',
+	'Dashboard label 87 with a verbose explanation for filtering and display',
+	'Dashboard label 88 with a verbose explanation for filtering and display',
+	'Dashboard label 89 with a verbose explanation for filtering and display',
+	'Dashboard label 90 with a verbose explanation for filtering and display',
+	'Dashboard label 91 with a verbose explanation for filtering and display',
+	'Dashboard label 92 with a verbose explanation for filtering and display',
+	'Dashboard label 93 with a verbose explanation for filtering and display',
+	'Dashboard label 94 with a verbose explanation for filtering and display',
+	'Dashboard label 95 with a verbose explanation for filtering and display',
+	'Dashboard label 96 with a verbose explanation for filtering and display',
+	'Dashboard label 97 with a verbose explanation for filtering and display',
+	'Dashboard label 98 with a verbose explanation for filtering and display',
+	'Dashboard label 99 with a verbose explanation for filtering and display',
+	'Dashboard label 100 with a verbose explanation for filtering and display',
+	'Dashboard label 101 with a verbose explanation for filtering and display',
+	'Dashboard label 102 with a verbose explanation for filtering and display',
+	'Dashboard label 103 with a verbose explanation for filtering and display',
+	'Dashboard label 104 with a verbose explanation for filtering and display',
+	'Dashboard label 105 with a verbose explanation for filtering and display',
+	'Dashboard label 106 with a verbose explanation for filtering and display',
+	'Dashboard label 107 with a verbose explanation for filtering and display',
+	'Dashboard label 108 with a verbose explanation for filtering and display',
+	'Dashboard label 109 with a verbose explanation for filtering and display',
+	'Dashboard label 110 with a verbose explanation for filtering and display',
+	'Dashboard label 111 with a verbose explanation for filtering and display',
+	'Dashboard label 112 with a verbose explanation for filtering and display',
+	'Dashboard label 113 with a verbose explanation for filtering and display',
+	'Dashboard label 114 with a verbose explanation for filtering and display',
+	'Dashboard label 115 with a verbose explanation for filtering and display',
+	'Dashboard label 116 with a verbose explanation for filtering and display',
+	'Dashboard label 117 with a verbose explanation for filtering and display',
+	'Dashboard label 118 with a verbose explanation for filtering and display',
+	'Dashboard label 119 with a verbose explanation for filtering and display',
+	'Dashboard label 120 with a verbose explanation for filtering and display',
+	'Dashboard label 121 with a verbose explanation for filtering and display',
+	'Dashboard label 122 with a verbose explanation for filtering and display',
+	'Dashboard label 123 with a verbose explanation for filtering and display',
+	'Dashboard label 124 with a verbose explanation for filtering and display',
+	'Dashboard label 125 with a verbose explanation for filtering and display',
+	'Dashboard label 126 with a verbose explanation for filtering and display',
+	'Dashboard label 127 with a verbose explanation for filtering and display',
+	'Dashboard label 128 with a verbose explanation for filtering and display',
+	'Dashboard label 129 with a verbose explanation for filtering and display',
+	'Dashboard label 130 with a verbose explanation for filtering and display',
+	'Dashboard label 131 with a verbose explanation for filtering and display',
+	'Dashboard label 132 with a verbose explanation for filtering and display',
+	'Dashboard label 133 with a verbose explanation for filtering and display',
+	'Dashboard label 134 with a verbose explanation for filtering and display',
+	'Dashboard label 135 with a verbose explanation for filtering and display',
+	'Dashboard label 136 with a verbose explanation for filtering and display',
+	'Dashboard label 137 with a verbose explanation for filtering and display',
+	'Dashboard label 138 with a verbose explanation for filtering and display',
+	'Dashboard label 139 with a verbose explanation for filtering and display',
+	'Dashboard label 140 with a verbose explanation for filtering and display',
+	'Dashboard label 141 with a verbose explanation for filtering and display',
+	'Dashboard label 142 with a verbose explanation for filtering and display',
+	'Dashboard label 143 with a verbose explanation for filtering and display',
+	'Dashboard label 144 with a verbose explanation for filtering and display',
+	'Dashboard label 145 with a verbose explanation for filtering and display',
+	'Dashboard label 146 with a verbose explanation for filtering and display',
+	'Dashboard label 147 with a verbose explanation for filtering and display',
+	'Dashboard label 148 with a verbose explanation for filtering and display',
+	'Dashboard label 149 with a verbose explanation for filtering and display',
+	'Dashboard label 150 with a verbose explanation for filtering and display',
+	'Dashboard label 151 with a verbose explanation for filtering and display',
+	'Dashboard label 152 with a verbose explanation for filtering and display',
+	'Dashboard label 153 with a verbose explanation for filtering and display',
+	'Dashboard label 154 with a verbose explanation for filtering and display',
+	'Dashboard label 155 with a verbose explanation for filtering and display',
+	'Dashboard label 156 with a verbose explanation for filtering and display',
+	'Dashboard label 157 with a verbose explanation for filtering and display',
+	'Dashboard label 158 with a verbose explanation for filtering and display',
+	'Dashboard label 159 with a verbose explanation for filtering and display',
+	'Dashboard label 160 with a verbose explanation for filtering and display',
+	'Dashboard label 161 with a verbose explanation for filtering and display',
+	'Dashboard label 162 with a verbose explanation for filtering and display',
+	'Dashboard label 163 with a verbose explanation for filtering and display',
+	'Dashboard label 164 with a verbose explanation for filtering and display',
+	'Dashboard label 165 with a verbose explanation for filtering and display',
+	'Dashboard label 166 with a verbose explanation for filtering and display',
+	'Dashboard label 167 with a verbose explanation for filtering and display',
+	'Dashboard label 168 with a verbose explanation for filtering and display',
+	'Dashboard label 169 with a verbose explanation for filtering and display',
+	'Dashboard label 170 with a verbose explanation for filtering and display',
+	'Dashboard label 171 with a verbose explanation for filtering and display',
+	'Dashboard label 172 with a verbose explanation for filtering and display',
+	'Dashboard label 173 with a verbose explanation for filtering and display',
+	'Dashboard label 174 with a verbose explanation for filtering and display',
+	'Dashboard label 175 with a verbose explanation for filtering and display',
+	'Dashboard label 176 with a verbose explanation for filtering and display',
+	'Dashboard label 177 with a verbose explanation for filtering and display',
+	'Dashboard label 178 with a verbose explanation for filtering and display',
+	'Dashboard label 179 with a verbose explanation for filtering and display',
+	'Dashboard label 180 with a verbose explanation for filtering and display',
+	'Dashboard label 181 with a verbose explanation for filtering and display',
+	'Dashboard label 182 with a verbose explanation for filtering and display',
+	'Dashboard label 183 with a verbose explanation for filtering and display',
+	'Dashboard label 184 with a verbose explanation for filtering and display',
+	'Dashboard label 185 with a verbose explanation for filtering and display',
+	'Dashboard label 186 with a verbose explanation for filtering and display',
+	'Dashboard label 187 with a verbose explanation for filtering and display',
+	'Dashboard label 188 with a verbose explanation for filtering and display',
+	'Dashboard label 189 with a verbose explanation for filtering and display',
+	'Dashboard label 190 with a verbose explanation for filtering and display',
+	'Dashboard label 191 with a verbose explanation for filtering and display',
+	'Dashboard label 192 with a verbose explanation for filtering and display',
+	'Dashboard label 193 with a verbose explanation for filtering and display',
+	'Dashboard label 194 with a verbose explanation for filtering and display',
+	'Dashboard label 195 with a verbose explanation for filtering and display',
+	'Dashboard label 196 with a verbose explanation for filtering and display',
+	'Dashboard label 197 with a verbose explanation for filtering and display',
+	'Dashboard label 198 with a verbose explanation for filtering and display',
+	'Dashboard label 199 with a verbose explanation for filtering and display',
+	'Dashboard label 200 with a verbose explanation for filtering and display',
+	'Dashboard label 201 with a verbose explanation for filtering and display',
+	'Dashboard label 202 with a verbose explanation for filtering and display',
+	'Dashboard label 203 with a verbose explanation for filtering and display',
+	'Dashboard label 204 with a verbose explanation for filtering and display',
+	'Dashboard label 205 with a verbose explanation for filtering and display',
+	'Dashboard label 206 with a verbose explanation for filtering and display',
+	'Dashboard label 207 with a verbose explanation for filtering and display',
+	'Dashboard label 208 with a verbose explanation for filtering and display',
+	'Dashboard label 209 with a verbose explanation for filtering and display',
+	'Dashboard label 210 with a verbose explanation for filtering and display',
+	'Dashboard label 211 with a verbose explanation for filtering and display',
+	'Dashboard label 212 with a verbose explanation for filtering and display',
+	'Dashboard label 213 with a verbose explanation for filtering and display',
+	'Dashboard label 214 with a verbose explanation for filtering and display',
+	'Dashboard label 215 with a verbose explanation for filtering and display',
+	'Dashboard label 216 with a verbose explanation for filtering and display',
+	'Dashboard label 217 with a verbose explanation for filtering and display',
+	'Dashboard label 218 with a verbose explanation for filtering and display',
+	'Dashboard label 219 with a verbose explanation for filtering and display',
+]

ai_evals/fixtures/frontend/app/initial/token_heavy_context/frontend/index.tsx

@@ -0,0 +1,194 @@
+import React, { useEffect, useMemo, useState } from 'react'
+import { backend } from 'wmill'
+
+type Metric = {
+	id: number
+	label: string
+	description: string
+}
+
+const referenceMetrics: Metric[] = [
+	{ id: 1, label: 'Reference metric 1', description: 'This is intentionally verbose dashboard reference text number 1 used to simulate a large selected app file with lots of inline business copy and configuration.' },
+	{ id: 2, label: 'Reference metric 2', description: 'This is intentionally verbose dashboard reference text number 2 used to simulate a large selected app file with lots of inline business copy and configuration.' },
+	{ id: 3, label: 'Reference metric 3', description: 'This is intentionally verbose dashboard reference text number 3 used to simulate a large selected app file with lots of inline business copy and configuration.' },
+	{ id: 4, label: 'Reference metric 4', description: 'This is intentionally verbose dashboard reference text number 4 used to simulate a large selected app file with lots of inline business copy and configuration.' },
+	{ id: 5, label: 'Reference metric 5', description: 'This is intentionally verbose dashboard reference text number 5 used to simulate a large selected app file with lots of inline business copy and configuration.' },
+	{ id: 6, label: 'Reference metric 6', description: 'This is intentionally verbose dashboard reference text number 6 used to simulate a large selected app file with lots of inline business copy and configuration.' },
+	{ id: 7, label: 'Reference metric 7', description: 'This is intentionally verbose dashboard reference text number 7 used to simulate a large selected app file with lots of inline business copy and configuration.' },
+	{ id: 8, label: 'Reference metric 8', description: 'This is intentionally verbose dashboard reference text number 8 used to simulate a large selected app file with lots of inline business copy and configuration.' },
+	{ id: 9, label: 'Reference metric 9', description: 'This is intentionally verbose dashboard reference text number 9 used to simulate a large selected app file with lots of inline business copy and configuration.' },
+	{ id: 10, label: 'Reference metric 10', description: 'This is intentionally verbose dashboard reference text number 10 used to simulate a large selected app file with lots of inline business copy and configuration.' },
+	{ id: 11, label: 'Reference metric 11', description: 'This is intentionally verbose dashboard reference text number 11 used to simulate a large selected app file with lots of inline business copy and configuration.' },
+	{ id: 12, label: 'Reference metric 12', description: 'This is intentionally verbose dashboard reference text number 12 used to simulate a large selected app file with lots of inline business copy and configuration.' },
+	{ id: 13, label: 'Reference metric 13', description: 'This is intentionally verbose dashboard reference text number 13 used to simulate a large selected app file with lots of inline business copy and configuration.' },
+	{ id: 14, label: 'Reference metric 14', description: 'This is intentionally verbose dashboard reference text number 14 used to simulate a large selected app file with lots of inline business copy and configuration.' },
+	{ id: 15, label: 'Reference metric 15', description: 'This is intentionally verbose dashboard reference text number 15 used to simulate a large selected app file with lots of inline business copy and configuration.' },
+	{ id: 16, label: 'Reference metric 16', description: 'This is intentionally verbose dashboard reference text number 16 used to simulate a large selected app file with lots of inline business copy and configuration.' },
+	{ id: 17, label: 'Reference metric 17', description: 'This is intentionally verbose dashboard reference text number 17 used to simulate a large selected app file with lots of inline business copy and configuration.' },
+	{ id: 18, label: 'Reference metric 18', description: 'This is intentionally verbose dashboard reference text number 18 used to simulate a large selected app file with lots of inline business copy and configuration.' },
+	{ id: 19, label: 'Reference metric 19', description: 'This is intentionally verbose dashboard reference text number 19 used to simulate a large selected app file with lots of inline business copy and configuration.' },
+	{ id: 20, label: 'Reference metric 20', description: 'This is intentionally verbose dashboard reference text number 20 used to simulate a large selected app file with lots of inline business copy and configuration.' },
+	{ id: 21, label: 'Reference metric 21', description: 'This is intentionally verbose dashboard reference text number 21 used to simulate a large selected app file with lots of inline business copy and configuration.' },
+	{ id: 22, label: 'Reference metric 22', description: 'This is intentionally verbose dashboard reference text number 22 used to simulate a large selected app file with lots of inline business copy and configuration.' },
+	{ id: 23, label: 'Reference metric 23', description: 'This is intentionally verbose dashboard reference text number 23 used to simulate a large selected app file with lots of inline business copy and configuration.' },
+	{ id: 24, label: 'Reference metric 24', description: 'This is intentionally verbose dashboard reference text number 24 used to simulate a large selected app file with lots of inline business copy and configuration.' },
+	{ id: 25, label: 'Reference metric 25', description: 'This is intentionally verbose dashboard reference text number 25 used to simulate a large selected app file with lots of inline business copy and configuration.' },
+	{ id: 26, label: 'Reference metric 26', description: 'This is intentionally verbose dashboard reference text number 26 used to simulate a large selected app file with lots of inline business copy and configuration.' },
+	{ id: 27, label: 'Reference metric 27', description: 'This is intentionally verbose dashboard reference text number 27 used to simulate a large selected app file with lots of inline business copy and configuration.' },
+	{ id: 28, label: 'Reference metric 28', description: 'This is intentionally verbose dashboard reference text number 28 used to simulate a large selected app file with lots of inline business copy and configuration.' },
+	{ id: 29, label: 'Reference metric 29', description: 'This is intentionally verbose dashboard reference text number 29 used to simulate a large selected app file with lots of inline business copy and configuration.' },
+	{ id: 30, label: 'Reference metric 30', description: 'This is intentionally verbose dashboard reference text number 30 used to simulate a large selected app file with lots of inline business copy and configuration.' },
+	{ id: 31, label: 'Reference metric 31', description: 'This is intentionally verbose dashboard reference text number 31 used to simulate a large selected app file with lots of inline business copy and configuration.' },
+	{ id: 32, label: 'Reference metric 32', description: 'This is intentionally verbose dashboard reference text number 32 used to simulate a large selected app file with lots of inline business copy and configuration.' },
+	{ id: 33, label: 'Reference metric 33', description: 'This is intentionally verbose dashboard reference text number 33 used to simulate a large selected app file with lots of inline business copy and configuration.' },
+	{ id: 34, label: 'Reference metric 34', description: 'This is intentionally verbose dashboard reference text number 34 used to simulate a large selected app file with lots of inline business copy and configuration.' },
+	{ id: 35, label: 'Reference metric 35', description: 'This is intentionally verbose dashboard reference text number 35 used to simulate a large selected app file with lots of inline business copy and configuration.' },
+	{ id: 36, label: 'Reference metric 36', description: 'This is intentionally verbose dashboard reference text number 36 used to simulate a large selected app file with lots of inline business copy and configuration.' },
+	{ id: 37, label: 'Reference metric 37', description: 'This is intentionally verbose dashboard reference text number 37 used to simulate a large selected app file with lots of inline business copy and configuration.' },
+	{ id: 38, label: 'Reference metric 38', description: 'This is intentionally verbose dashboard reference text number 38 used to simulate a large selected app file with lots of inline business copy and configuration.' },
+	{ id: 39, label: 'Reference metric 39', description: 'This is intentionally verbose dashboard reference text number 39 used to simulate a large selected app file with lots of inline business copy and configuration.' },
+	{ id: 40, label: 'Reference metric 40', description: 'This is intentionally verbose dashboard reference text number 40 used to simulate a large selected app file with lots of inline business copy and configuration.' },
+	{ id: 41, label: 'Reference metric 41', description: 'This is intentionally verbose dashboard reference text number 41 used to simulate a large selected app file with lots of inline business copy and configuration.' },
+	{ id: 42, label: 'Reference metric 42', description: 'This is intentionally verbose dashboard reference text number 42 used to simulate a large selected app file with lots of inline business copy and configuration.' },
+	{ id: 43, label: 'Reference metric 43', description: 'This is intentionally verbose dashboard reference text number 43 used to simulate a large selected app file with lots of inline business copy and configuration.' },
+	{ id: 44, label: 'Reference metric 44', description: 'This is intentionally verbose dashboard reference text number 44 used to simulate a large selected app file with lots of inline business copy and configuration.' },
+	{ id: 45, label: 'Reference metric 45', description: 'This is intentionally verbose dashboard reference text number 45 used to simulate a large selected app file with lots of inline business copy and configuration.' },
+	{ id: 46, label: 'Reference metric 46', description: 'This is intentionally verbose dashboard reference text number 46 used to simulate a large selected app file with lots of inline business copy and configuration.' },
+	{ id: 47, label: 'Reference metric 47', description: 'This is intentionally verbose dashboard reference text number 47 used to simulate a large selected app file with lots of inline business copy and configuration.' },
+	{ id: 48, label: 'Reference metric 48', description: 'This is intentionally verbose dashboard reference text number 48 used to simulate a large selected app file with lots of inline business copy and configuration.' },
+	{ id: 49, label: 'Reference metric 49', description: 'This is intentionally verbose dashboard reference text number 49 used to simulate a large selected app file with lots of inline business copy and configuration.' },
+	{ id: 50, label: 'Reference metric 50', description: 'This is intentionally verbose dashboard reference text number 50 used to simulate a large selected app file with lots of inline business copy and configuration.' },
+	{ id: 51, label: 'Reference metric 51', description: 'This is intentionally verbose dashboard reference text number 51 used to simulate a large selected app file with lots of inline business copy and configuration.' },
+	{ id: 52, label: 'Reference metric 52', description: 'This is intentionally verbose dashboard reference text number 52 used to simulate a large selected app file with lots of inline business copy and configuration.' },
+	{ id: 53, label: 'Reference metric 53', description: 'This is intentionally verbose dashboard reference text number 53 used to simulate a large selected app file with lots of inline business copy and configuration.' },
+	{ id: 54, label: 'Reference metric 54', description: 'This is intentionally verbose dashboard reference text number 54 used to simulate a large selected app file with lots of inline business copy and configuration.' },
+	{ id: 55, label: 'Reference metric 55', description: 'This is intentionally verbose dashboard reference text number 55 used to simulate a large selected app file with lots of inline business copy and configuration.' },
+	{ id: 56, label: 'Reference metric 56', description: 'This is intentionally verbose dashboard reference text number 56 used to simulate a large selected app file with lots of inline business copy and configuration.' },
+	{ id: 57, label: 'Reference metric 57', description: 'This is intentionally verbose dashboard reference text number 57 used to simulate a large selected app file with lots of inline business copy and configuration.' },
+	{ id: 58, label: 'Reference metric 58', description: 'This is intentionally verbose dashboard reference text number 58 used to simulate a large selected app file with lots of inline business copy and configuration.' },
+	{ id: 59, label: 'Reference metric 59', description: 'This is intentionally verbose dashboard reference text number 59 used to simulate a large selected app file with lots of inline business copy and configuration.' },
+	{ id: 60, label: 'Reference metric 60', description: 'This is intentionally verbose dashboard reference text number 60 used to simulate a large selected app file with lots of inline business copy and configuration.' },
+	{ id: 61, label: 'Reference metric 61', description: 'This is intentionally verbose dashboard reference text number 61 used to simulate a large selected app file with lots of inline business copy and configuration.' },
+	{ id: 62, label: 'Reference metric 62', description: 'This is intentionally verbose dashboard reference text number 62 used to simulate a large selected app file with lots of inline business copy and configuration.' },
+	{ id: 63, label: 'Reference metric 63', description: 'This is intentionally verbose dashboard reference text number 63 used to simulate a large selected app file with lots of inline business copy and configuration.' },
+	{ id: 64, label: 'Reference metric 64', description: 'This is intentionally verbose dashboard reference text number 64 used to simulate a large selected app file with lots of inline business copy and configuration.' },
+	{ id: 65, label: 'Reference metric 65', description: 'This is intentionally verbose dashboard reference text number 65 used to simulate a large selected app file with lots of inline business copy and configuration.' },
+	{ id: 66, label: 'Reference metric 66', description: 'This is intentionally verbose dashboard reference text number 66 used to simulate a large selected app file with lots of inline business copy and configuration.' },
+	{ id: 67, label: 'Reference metric 67', description: 'This is intentionally verbose dashboard reference text number 67 used to simulate a large selected app file with lots of inline business copy and configuration.' },
+	{ id: 68, label: 'Reference metric 68', description: 'This is intentionally verbose dashboard reference text number 68 used to simulate a large selected app file with lots of inline business copy and configuration.' },
+	{ id: 69, label: 'Reference metric 69', description: 'This is intentionally verbose dashboard reference text number 69 used to simulate a large selected app file with lots of inline business copy and configuration.' },
+	{ id: 70, label: 'Reference metric 70', description: 'This is intentionally verbose dashboard reference text number 70 used to simulate a large selected app file with lots of inline business copy and configuration.' },
+	{ id: 71, label: 'Reference metric 71', description: 'This is intentionally verbose dashboard reference text number 71 used to simulate a large selected app file with lots of inline business copy and configuration.' },
+	{ id: 72, label: 'Reference metric 72', description: 'This is intentionally verbose dashboard reference text number 72 used to simulate a large selected app file with lots of inline business copy and configuration.' },
+	{ id: 73, label: 'Reference metric 73', description: 'This is intentionally verbose dashboard reference text number 73 used to simulate a large selected app file with lots of inline business copy and configuration.' },
+	{ id: 74, label: 'Reference metric 74', description: 'This is intentionally verbose dashboard reference text number 74 used to simulate a large selected app file with lots of inline business copy and configuration.' },
+	{ id: 75, label: 'Reference metric 75', description: 'This is intentionally verbose dashboard reference text number 75 used to simulate a large selected app file with lots of inline business copy and configuration.' },
+	{ id: 76, label: 'Reference metric 76', description: 'This is intentionally verbose dashboard reference text number 76 used to simulate a large selected app file with lots of inline business copy and configuration.' },
+	{ id: 77, label: 'Reference metric 77', description: 'This is intentionally verbose dashboard reference text number 77 used to simulate a large selected app file with lots of inline business copy and configuration.' },
+	{ id: 78, label: 'Reference metric 78', description: 'This is intentionally verbose dashboard reference text number 78 used to simulate a large selected app file with lots of inline business copy and configuration.' },
+	{ id: 79, label: 'Reference metric 79', description: 'This is intentionally verbose dashboard reference text number 79 used to simulate a large selected app file with lots of inline business copy and configuration.' },
+	{ id: 80, label: 'Reference metric 80', description: 'This is intentionally verbose dashboard reference text number 80 used to simulate a large selected app file with lots of inline business copy and configuration.' },
+	{ id: 81, label: 'Reference metric 81', description: 'This is intentionally verbose dashboard reference text number 81 used to simulate a large selected app file with lots of inline business copy and configuration.' },
+	{ id: 82, label: 'Reference metric 82', description: 'This is intentionally verbose dashboard reference text number 82 used to simulate a large selected app file with lots of inline business copy and configuration.' },
+	{ id: 83, label: 'Reference metric 83', description: 'This is intentionally verbose dashboard reference text number 83 used to simulate a large selected app file with lots of inline business copy and configuration.' },
+	{ id: 84, label: 'Reference metric 84', description: 'This is intentionally verbose dashboard reference text number 84 used to simulate a large selected app file with lots of inline business copy and configuration.' },
+	{ id: 85, label: 'Reference metric 85', description: 'This is intentionally verbose dashboard reference text number 85 used to simulate a large selected app file with lots of inline business copy and configuration.' },
+	{ id: 86, label: 'Reference metric 86', description: 'This is intentionally verbose dashboard reference text number 86 used to simulate a large selected app file with lots of inline business copy and configuration.' },
+	{ id: 87, label: 'Reference metric 87', description: 'This is intentionally verbose dashboard reference text number 87 used to simulate a large selected app file with lots of inline business copy and configuration.' },
+	{ id: 88, label: 'Reference metric 88', description: 'This is intentionally verbose dashboard reference text number 88 used to simulate a large selected app file with lots of inline business copy and configuration.' },
+	{ id: 89, label: 'Reference metric 89', description: 'This is intentionally verbose dashboard reference text number 89 used to simulate a large selected app file with lots of inline business copy and configuration.' },
+	{ id: 90, label: 'Reference metric 90', description: 'This is intentionally verbose dashboard reference text number 90 used to simulate a large selected app file with lots of inline business copy and configuration.' },
+	{ id: 91, label: 'Reference metric 91', description: 'This is intentionally verbose dashboard reference text number 91 used to simulate a large selected app file with lots of inline business copy and configuration.' },
+	{ id: 92, label: 'Reference metric 92', description: 'This is intentionally verbose dashboard reference text number 92 used to simulate a large selected app file with lots of inline business copy and configuration.' },
+	{ id: 93, label: 'Reference metric 93', description: 'This is intentionally verbose dashboard reference text number 93 used to simulate a large selected app file with lots of inline business copy and configuration.' },
+	{ id: 94, label: 'Reference metric 94', description: 'This is intentionally verbose dashboard reference text number 94 used to simulate a large selected app file with lots of inline business copy and configuration.' },
+	{ id: 95, label: 'Reference metric 95', description: 'This is intentionally verbose dashboard reference text number 95 used to simulate a large selected app file with lots of inline business copy and configuration.' },
+	{ id: 96, label: 'Reference metric 96', description: 'This is intentionally verbose dashboard reference text number 96 used to simulate a large selected app file with lots of inline business copy and configuration.' },
+	{ id: 97, label: 'Reference metric 97', description: 'This is intentionally verbose dashboard reference text number 97 used to simulate a large selected app file with lots of inline business copy and configuration.' },
+	{ id: 98, label: 'Reference metric 98', description: 'This is intentionally verbose dashboard reference text number 98 used to simulate a large selected app file with lots of inline business copy and configuration.' },
+	{ id: 99, label: 'Reference metric 99', description: 'This is intentionally verbose dashboard reference text number 99 used to simulate a large selected app file with lots of inline business copy and configuration.' },
+	{ id: 100, label: 'Reference metric 100', description: 'This is intentionally verbose dashboard reference text number 100 used to simulate a large selected app file with lots of inline business copy and configuration.' },
+	{ id: 101, label: 'Reference metric 101', description: 'This is intentionally verbose dashboard reference text number 101 used to simulate a large selected app file with lots of inline business copy and configuration.' },
+	{ id: 102, label: 'Reference metric 102', description: 'This is intentionally verbose dashboard reference text number 102 used to simulate a large selected app file with lots of inline business copy and configuration.' },
+	{ id: 103, label: 'Reference metric 103', description: 'This is intentionally verbose dashboard reference text number 103 used to simulate a large selected app file with lots of inline business copy and configuration.' },
+	{ id: 104, label: 'Reference metric 104', description: 'This is intentionally verbose dashboard reference text number 104 used to simulate a large selected app file with lots of inline business copy and configuration.' },
+	{ id: 105, label: 'Reference metric 105', description: 'This is intentionally verbose dashboard reference text number 105 used to simulate a large selected app file with lots of inline business copy and configuration.' },
+	{ id: 106, label: 'Reference metric 106', description: 'This is intentionally verbose dashboard reference text number 106 used to simulate a large selected app file with lots of inline business copy and configuration.' },
+	{ id: 107, label: 'Reference metric 107', description: 'This is intentionally verbose dashboard reference text number 107 used to simulate a large selected app file with lots of inline business copy and configuration.' },
+	{ id: 108, label: 'Reference metric 108', description: 'This is intentionally verbose dashboard reference text number 108 used to simulate a large selected app file with lots of inline business copy and configuration.' },
+	{ id: 109, label: 'Reference metric 109', description: 'This is intentionally verbose dashboard reference text number 109 used to simulate a large selected app file with lots of inline business copy and configuration.' },
+	{ id: 110, label: 'Reference metric 110', description: 'This is intentionally verbose dashboard reference text number 110 used to simulate a large selected app file with lots of inline business copy and configuration.' },
+	{ id: 111, label: 'Reference metric 111', description: 'This is intentionally verbose dashboard reference text number 111 used to simulate a large selected app file with lots of inline business copy and configuration.' },
+	{ id: 112, label: 'Reference metric 112', description: 'This is intentionally verbose dashboard reference text number 112 used to simulate a large selected app file with lots of inline business copy and configuration.' },
+	{ id: 113, label: 'Reference metric 113', description: 'This is intentionally verbose dashboard reference text number 113 used to simulate a large selected app file with lots of inline business copy and configuration.' },
+	{ id: 114, label: 'Reference metric 114', description: 'This is intentionally verbose dashboard reference text number 114 used to simulate a large selected app file with lots of inline business copy and configuration.' },
+	{ id: 115, label: 'Reference metric 115', description: 'This is intentionally verbose dashboard reference text number 115 used to simulate a large selected app file with lots of inline business copy and configuration.' },
+	{ id: 116, label: 'Reference metric 116', description: 'This is intentionally verbose dashboard reference text number 116 used to simulate a large selected app file with lots of inline business copy and configuration.' },
+	{ id: 117, label: 'Reference metric 117', description: 'This is intentionally verbose dashboard reference text number 117 used to simulate a large selected app file with lots of inline business copy and configuration.' },
+	{ id: 118, label: 'Reference metric 118', description: 'This is intentionally verbose dashboard reference text number 118 used to simulate a large selected app file with lots of inline business copy and configuration.' },
+	{ id: 119, label: 'Reference metric 119', description: 'This is intentionally verbose dashboard reference text number 119 used to simulate a large selected app file with lots of inline business copy and configuration.' },
+	{ id: 120, label: 'Reference metric 120', description: 'This is intentionally verbose dashboard reference text number 120 used to simulate a large selected app file with lots of inline business copy and configuration.' },
+	{ id: 121, label: 'Reference metric 121', description: 'This is intentionally verbose dashboard reference text number 121 used to simulate a large selected app file with lots of inline business copy and configuration.' },
+	{ id: 122, label: 'Reference metric 122', description: 'This is intentionally verbose dashboard reference text number 122 used to simulate a large selected app file with lots of inline business copy and configuration.' },
+	{ id: 123, label: 'Reference metric 123', description: 'This is intentionally verbose dashboard reference text number 123 used to simulate a large selected app file with lots of inline business copy and configuration.' },
+	{ id: 124, label: 'Reference metric 124', description: 'This is intentionally verbose dashboard reference text number 124 used to simulate a large selected app file with lots of inline business copy and configuration.' },
+	{ id: 125, label: 'Reference metric 125', description: 'This is intentionally verbose dashboard reference text number 125 used to simulate a large selected app file with lots of inline business copy and configuration.' },
+	{ id: 126, label: 'Reference metric 126', description: 'This is intentionally verbose dashboard reference text number 126 used to simulate a large selected app file with lots of inline business copy and configuration.' },
+	{ id: 127, label: 'Reference metric 127', description: 'This is intentionally verbose dashboard reference text number 127 used to simulate a large selected app file with lots of inline business copy and configuration.' },
+	{ id: 128, label: 'Reference metric 128', description: 'This is intentionally verbose dashboard reference text number 128 used to simulate a large selected app file with lots of inline business copy and configuration.' },
+	{ id: 129, label: 'Reference metric 129', description: 'This is intentionally verbose dashboard reference text number 129 used to simulate a large selected app file with lots of inline business copy and configuration.' },
+	{ id: 130, label: 'Reference metric 130', description: 'This is intentionally verbose dashboard reference text number 130 used to simulate a large selected app file with lots of inline business copy and configuration.' },
+	{ id: 131, label: 'Reference metric 131', description: 'This is intentionally verbose dashboard reference text number 131 used to simulate a large selected app file with lots of inline business copy and configuration.' },
+	{ id: 132, label: 'Reference metric 132', description: 'This is intentionally verbose dashboard reference text number 132 used to simulate a large selected app file with lots of inline business copy and configuration.' },
+	{ id: 133, label: 'Reference metric 133', description: 'This is intentionally verbose dashboard reference text number 133 used to simulate a large selected app file with lots of inline business copy and configuration.' },
+	{ id: 134, label: 'Reference metric 134', description: 'This is intentionally verbose dashboard reference text number 134 used to simulate a large selected app file with lots of inline business copy and configuration.' },
+	{ id: 135, label: 'Reference metric 135', description: 'This is intentionally verbose dashboard reference text number 135 used to simulate a large selected app file with lots of inline business copy and configuration.' },
+	{ id: 136, label: 'Reference metric 136', description: 'This is intentionally verbose dashboard reference text number 136 used to simulate a large selected app file with lots of inline business copy and configuration.' },
+	{ id: 137, label: 'Reference metric 137', description: 'This is intentionally verbose dashboard reference text number 137 used to simulate a large selected app file with lots of inline business copy and configuration.' },
+	{ id: 138, label: 'Reference metric 138', description: 'This is intentionally verbose dashboard reference text number 138 used to simulate a large selected app file with lots of inline business copy and configuration.' },
+	{ id: 139, label: 'Reference metric 139', description: 'This is intentionally verbose dashboard reference text number 139 used to simulate a large selected app file with lots of inline business copy and configuration.' },
+]
+
+export default function AnalyticsConsole() {
+	const [summary, setSummary] = useState<string>('Loading analytics summary...')
+	const [filter, setFilter] = useState('')
+
+	useEffect(() => {
+		backend.loadAnalytics({ range: '30d' }).then((result) => {
+			setSummary(result.summary)
+		})
+	}, [])
+
+	const visibleMetrics = useMemo(() => {
+		return referenceMetrics.filter((metric) =>
+			metric.label.toLowerCase().includes(filter.toLowerCase()) ||
+			metric.description.toLowerCase().includes(filter.toLowerCase())
+		)
+	}, [filter])
+
+	return (
+		<main className="min-h-screen bg-slate-950 text-white p-8">
+			<section className="max-w-5xl mx-auto space-y-6">
+				<div>
+					<p className="uppercase tracking-wide text-xs text-cyan-300">Windmill Analytics</p>
+					<h1 className="text-4xl font-bold">Analytics Console</h1>
+					<p className="text-slate-300 mt-2">{summary}</p>
+				</div>
+				<input
+					className="w-full rounded bg-slate-800 border border-slate-700 p-3"
+					placeholder="Filter reference metrics"
+					value={filter}
+					onChange={(event) => setFilter(event.target.value)}
+				/>
+				<div className="grid grid-cols-1 md:grid-cols-2 gap-4">
+					{visibleMetrics.slice(0, 20).map((metric) => (
+						<article key={metric.id} className="rounded border border-slate-800 p-4 bg-slate-900">
+							<h2 className="font-semibold">{metric.label}</h2>
+							<p className="text-sm text-slate-400">{metric.description}</p>
+						</article>
+					))}
+				</div>
+			</section>
+		</main>
+	)
+}

ai_evals/fixtures/frontend/app/initial/token_heavy_datatables/datatables.json

@@ -0,0 +1,695 @@
+[
+  {
+    "datatable_name": "main",
+    "schemas": {
+      "analytics": {
+        "event_log_01": {
+          "column_01_name": "text",
+          "column_02_status": "text?",
+          "column_03_amount": "float8",
+          "column_04_enabled": "bool?",
+          "column_05_created_at": "timestamp=now()",
+          "column_06_metadata": "jsonb?",
+          "column_07_owner": "uuid",
+          "column_08_id": "int4",
+          "column_09_name": "text",
+          "column_10_status": "text?",
+          "column_11_amount": "float8",
+          "column_12_enabled": "bool?",
+          "column_13_created_at": "timestamp=now()",
+          "column_14_metadata": "jsonb?",
+          "column_15_owner": "uuid",
+          "column_16_id": "int4",
+          "column_17_name": "text",
+          "column_18_status": "text?",
+          "column_19_amount": "float8",
+          "column_20_enabled": "bool?",
+          "column_21_created_at": "timestamp=now()",
+          "column_22_metadata": "jsonb?",
+          "column_23_owner": "uuid",
+          "column_24_id": "int4",
+          "column_25_name": "text",
+          "column_26_status": "text?",
+          "column_27_amount": "float8",
+          "column_28_enabled": "bool?",
+          "column_29_created_at": "timestamp=now()",
+          "column_30_metadata": "jsonb?",
+          "column_31_owner": "uuid",
+          "column_32_id": "int4",
+          "id": "int4=nextval('seq'::regclass)",
+          "workspace_id": "text",
+          "created_at": "timestamp=now()",
+          "updated_at": "timestamp?"
+        },
+        "event_log_02": {
+          "column_01_name": "text",
+          "column_02_status": "text?",
+          "column_03_amount": "float8",
+          "column_04_enabled": "bool?",
+          "column_05_created_at": "timestamp=now()",
+          "column_06_metadata": "jsonb?",
+          "column_07_owner": "uuid",
+          "column_08_id": "int4",
+          "column_09_name": "text",
+          "column_10_status": "text?",
+          "column_11_amount": "float8",
+          "column_12_enabled": "bool?",
+          "column_13_created_at": "timestamp=now()",
+          "column_14_metadata": "jsonb?",
+          "column_15_owner": "uuid",
+          "column_16_id": "int4",
+          "column_17_name": "text",
+          "column_18_status": "text?",
+          "column_19_amount": "float8",
+          "column_20_enabled": "bool?",
+          "column_21_created_at": "timestamp=now()",
+          "column_22_metadata": "jsonb?",
+          "column_23_owner": "uuid",
+          "column_24_id": "int4",
+          "column_25_name": "text",
+          "column_26_status": "text?",
+          "column_27_amount": "float8",
+          "column_28_enabled": "bool?",
+          "column_29_created_at": "timestamp=now()",
+          "column_30_metadata": "jsonb?",
+          "column_31_owner": "uuid",
+          "column_32_id": "int4",
+          "id": "int4=nextval('seq'::regclass)",
+          "workspace_id": "text",
+          "created_at": "timestamp=now()",
+          "updated_at": "timestamp?"
+        },
+        "event_log_03": {
+          "column_01_name": "text",
+          "column_02_status": "text?",
+          "column_03_amount": "float8",
+          "column_04_enabled": "bool?",
+          "column_05_created_at": "timestamp=now()",
+          "column_06_metadata": "jsonb?",
+          "column_07_owner": "uuid",
+          "column_08_id": "int4",
+          "column_09_name": "text",
+          "column_10_status": "text?",
+          "column_11_amount": "float8",
+          "column_12_enabled": "bool?",
+          "column_13_created_at": "timestamp=now()",
+          "column_14_metadata": "jsonb?",
+          "column_15_owner": "uuid",
+          "column_16_id": "int4",
+          "column_17_name": "text",
+          "column_18_status": "text?",
+          "column_19_amount": "float8",
+          "column_20_enabled": "bool?",
+          "column_21_created_at": "timestamp=now()",
+          "column_22_metadata": "jsonb?",
+          "column_23_owner": "uuid",
+          "column_24_id": "int4",
+          "column_25_name": "text",
+          "column_26_status": "text?",
+          "column_27_amount": "float8",
+          "column_28_enabled": "bool?",
+          "column_29_created_at": "timestamp=now()",
+          "column_30_metadata": "jsonb?",
+          "column_31_owner": "uuid",
+          "column_32_id": "int4",
+          "id": "int4=nextval('seq'::regclass)",
+          "workspace_id": "text",
+          "created_at": "timestamp=now()",
+          "updated_at": "timestamp?"
+        },
+        "event_log_04": {
+          "column_01_name": "text",
+          "column_02_status": "text?",
+          "column_03_amount": "float8",
+          "column_04_enabled": "bool?",
+          "column_05_created_at": "timestamp=now()",
+          "column_06_metadata": "jsonb?",
+          "column_07_owner": "uuid",
+          "column_08_id": "int4",
+          "column_09_name": "text",
+          "column_10_status": "text?",
+          "column_11_amount": "float8",
+          "column_12_enabled": "bool?",
+          "column_13_created_at": "timestamp=now()",
+          "column_14_metadata": "jsonb?",
+          "column_15_owner": "uuid",
+          "column_16_id": "int4",
+          "column_17_name": "text",
+          "column_18_status": "text?",
+          "column_19_amount": "float8",
+          "column_20_enabled": "bool?",
+          "column_21_created_at": "timestamp=now()",
+          "column_22_metadata": "jsonb?",
+          "column_23_owner": "uuid",
+          "column_24_id": "int4",
+          "column_25_name": "text",
+          "column_26_status": "text?",
+          "column_27_amount": "float8",
+          "column_28_enabled": "bool?",
+          "column_29_created_at": "timestamp=now()",
+          "column_30_metadata": "jsonb?",
+          "column_31_owner": "uuid",
+          "column_32_id": "int4",
+          "id": "int4=nextval('seq'::regclass)",
+          "workspace_id": "text",
+          "created_at": "timestamp=now()",
+          "updated_at": "timestamp?"
+        },
+        "event_log_05": {
+          "column_01_name": "text",
+          "column_02_status": "text?",
+          "column_03_amount": "float8",
+          "column_04_enabled": "bool?",
+          "column_05_created_at": "timestamp=now()",
+          "column_06_metadata": "jsonb?",
+          "column_07_owner": "uuid",
+          "column_08_id": "int4",
+          "column_09_name": "text",
+          "column_10_status": "text?",
+          "column_11_amount": "float8",
+          "column_12_enabled": "bool?",
+          "column_13_created_at": "timestamp=now()",
+          "column_14_metadata": "jsonb?",
+          "column_15_owner": "uuid",
+          "column_16_id": "int4",
+          "column_17_name": "text",
+          "column_18_status": "text?",
+          "column_19_amount": "float8",
+          "column_20_enabled": "bool?",
+          "column_21_created_at": "timestamp=now()",
+          "column_22_metadata": "jsonb?",
+          "column_23_owner": "uuid",
+          "column_24_id": "int4",
+          "column_25_name": "text",
+          "column_26_status": "text?",
+          "column_27_amount": "float8",
+          "column_28_enabled": "bool?",
+          "column_29_created_at": "timestamp=now()",
+          "column_30_metadata": "jsonb?",
+          "column_31_owner": "uuid",
+          "column_32_id": "int4",
+          "id": "int4=nextval('seq'::regclass)",
+          "workspace_id": "text",
+          "created_at": "timestamp=now()",
+          "updated_at": "timestamp?"
+        },
+        "event_log_06": {
+          "column_01_name": "text",
+          "column_02_status": "text?",
+          "column_03_amount": "float8",
+          "column_04_enabled": "bool?",
+          "column_05_created_at": "timestamp=now()",
+          "column_06_metadata": "jsonb?",
+          "column_07_owner": "uuid",
+          "column_08_id": "int4",
+          "column_09_name": "text",
+          "column_10_status": "text?",
+          "column_11_amount": "float8",
+          "column_12_enabled": "bool?",
+          "column_13_created_at": "timestamp=now()",
+          "column_14_metadata": "jsonb?",
+          "column_15_owner": "uuid",
+          "column_16_id": "int4",
+          "column_17_name": "text",
+          "column_18_status": "text?",
+          "column_19_amount": "float8",
+          "column_20_enabled": "bool?",
+          "column_21_created_at": "timestamp=now()",
+          "column_22_metadata": "jsonb?",
+          "column_23_owner": "uuid",
+          "column_24_id": "int4",
+          "column_25_name": "text",
+          "column_26_status": "text?",
+          "column_27_amount": "float8",
+          "column_28_enabled": "bool?",
+          "column_29_created_at": "timestamp=now()",
+          "column_30_metadata": "jsonb?",
+          "column_31_owner": "uuid",
+          "column_32_id": "int4",
+          "id": "int4=nextval('seq'::regclass)",
+          "workspace_id": "text",
+          "created_at": "timestamp=now()",
+          "updated_at": "timestamp?"
+        },
+        "event_log_07": {
+          "column_01_name": "text",
+          "column_02_status": "text?",
+          "column_03_amount": "float8",
+          "column_04_enabled": "bool?",
+          "column_05_created_at": "timestamp=now()",
+          "column_06_metadata": "jsonb?",
+          "column_07_owner": "uuid",
+          "column_08_id": "int4",
+          "column_09_name": "text",
+          "column_10_status": "text?",
+          "column_11_amount": "float8",
+          "column_12_enabled": "bool?",
+          "column_13_created_at": "timestamp=now()",
+          "column_14_metadata": "jsonb?",
+          "column_15_owner": "uuid",
+          "column_16_id": "int4",
+          "column_17_name": "text",
+          "column_18_status": "text?",
+          "column_19_amount": "float8",
+          "column_20_enabled": "bool?",
+          "column_21_created_at": "timestamp=now()",
+          "column_22_metadata": "jsonb?",
+          "column_23_owner": "uuid",
+          "column_24_id": "int4",
+          "column_25_name": "text",
+          "column_26_status": "text?",
+          "column_27_amount": "float8",
+          "column_28_enabled": "bool?",
+          "column_29_created_at": "timestamp=now()",
+          "column_30_metadata": "jsonb?",
+          "column_31_owner": "uuid",
+          "column_32_id": "int4",
+          "id": "int4=nextval('seq'::regclass)",
+          "workspace_id": "text",
+          "created_at": "timestamp=now()",
+          "updated_at": "timestamp?"
+        },
+        "event_log_08": {
+          "column_01_name": "text",
+          "column_02_status": "text?",
+          "column_03_amount": "float8",
+          "column_04_enabled": "bool?",
+          "column_05_created_at": "timestamp=now()",
+          "column_06_metadata": "jsonb?",
+          "column_07_owner": "uuid",
+          "column_08_id": "int4",
+          "column_09_name": "text",
+          "column_10_status": "text?",
+          "column_11_amount": "float8",
+          "column_12_enabled": "bool?",
+          "column_13_created_at": "timestamp=now()",
+          "column_14_metadata": "jsonb?",
+          "column_15_owner": "uuid",
+          "column_16_id": "int4",
+          "column_17_name": "text",
+          "column_18_status": "text?",
+          "column_19_amount": "float8",
+          "column_20_enabled": "bool?",
+          "column_21_created_at": "timestamp=now()",
+          "column_22_metadata": "jsonb?",
+          "column_23_owner": "uuid",
+          "column_24_id": "int4",
+          "column_25_name": "text",
+          "column_26_status": "text?",
+          "column_27_amount": "float8",
+          "column_28_enabled": "bool?",
+          "column_29_created_at": "timestamp=now()",
+          "column_30_metadata": "jsonb?",
+          "column_31_owner": "uuid",
+          "column_32_id": "int4",
+          "id": "int4=nextval('seq'::regclass)",
+          "workspace_id": "text",
+          "created_at": "timestamp=now()",
+          "updated_at": "timestamp?"
+        },
+        "event_log_09": {
+          "column_01_name": "text",
+          "column_02_status": "text?",
+          "column_03_amount": "float8",
+          "column_04_enabled": "bool?",
+          "column_05_created_at": "timestamp=now()",
+          "column_06_metadata": "jsonb?",
+          "column_07_owner": "uuid",
+          "column_08_id": "int4",
+          "column_09_name": "text",
+          "column_10_status": "text?",
+          "column_11_amount": "float8",
+          "column_12_enabled": "bool?",
+          "column_13_created_at": "timestamp=now()",
+          "column_14_metadata": "jsonb?",
+          "column_15_owner": "uuid",
+          "column_16_id": "int4",
+          "column_17_name": "text",
+          "column_18_status": "text?",
+          "column_19_amount": "float8",
+          "column_20_enabled": "bool?",
+          "column_21_created_at": "timestamp=now()",
+          "column_22_metadata": "jsonb?",
+          "column_23_owner": "uuid",
+          "column_24_id": "int4",
+          "column_25_name": "text",
+          "column_26_status": "text?",
+          "column_27_amount": "float8",
+          "column_28_enabled": "bool?",
+          "column_29_created_at": "timestamp=now()",
+          "column_30_metadata": "jsonb?",
+          "column_31_owner": "uuid",
+          "column_32_id": "int4",
+          "id": "int4=nextval('seq'::regclass)",
+          "workspace_id": "text",
+          "created_at": "timestamp=now()",
+          "updated_at": "timestamp?"
+        },
+        "event_log_10": {
+          "column_01_name": "text",
+          "column_02_status": "text?",
+          "column_03_amount": "float8",
+          "column_04_enabled": "bool?",
+          "column_05_created_at": "timestamp=now()",
+          "column_06_metadata": "jsonb?",
+          "column_07_owner": "uuid",
+          "column_08_id": "int4",
+          "column_09_name": "text",
+          "column_10_status": "text?",
+          "column_11_amount": "float8",
+          "column_12_enabled": "bool?",
+          "column_13_created_at": "timestamp=now()",
+          "column_14_metadata": "jsonb?",
+          "column_15_owner": "uuid",
+          "column_16_id": "int4",
+          "column_17_name": "text",
+          "column_18_status": "text?",
+          "column_19_amount": "float8",
+          "column_20_enabled": "bool?",
+          "column_21_created_at": "timestamp=now()",
+          "column_22_metadata": "jsonb?",
+          "column_23_owner": "uuid",
+          "column_24_id": "int4",
+          "column_25_name": "text",
+          "column_26_status": "text?",
+          "column_27_amount": "float8",
+          "column_28_enabled": "bool?",
+          "column_29_created_at": "timestamp=now()",
+          "column_30_metadata": "jsonb?",
+          "column_31_owner": "uuid",
+          "column_32_id": "int4",
+          "id": "int4=nextval('seq'::regclass)",
+          "workspace_id": "text",
+          "created_at": "timestamp=now()",
+          "updated_at": "timestamp?"
+        },
+        "event_log_11": {
+          "column_01_name": "text",
+          "column_02_status": "text?",
+          "column_03_amount": "float8",
+          "column_04_enabled": "bool?",
+          "column_05_created_at": "timestamp=now()",
+          "column_06_metadata": "jsonb?",
+          "column_07_owner": "uuid",
+          "column_08_id": "int4",
+          "column_09_name": "text",
+          "column_10_status": "text?",
+          "column_11_amount": "float8",
+          "column_12_enabled": "bool?",
+          "column_13_created_at": "timestamp=now()",
+          "column_14_metadata": "jsonb?",
+          "column_15_owner": "uuid",
+          "column_16_id": "int4",
+          "column_17_name": "text",
+          "column_18_status": "text?",
+          "column_19_amount": "float8",
+          "column_20_enabled": "bool?",
+          "column_21_created_at": "timestamp=now()",
+          "column_22_metadata": "jsonb?",
+          "column_23_owner": "uuid",
+          "column_24_id": "int4",
+          "column_25_name": "text",
+          "column_26_status": "text?",
+          "column_27_amount": "float8",
+          "column_28_enabled": "bool?",
+          "column_29_created_at": "timestamp=now()",
+          "column_30_metadata": "jsonb?",
+          "column_31_owner": "uuid",
+          "column_32_id": "int4",
+          "id": "int4=nextval('seq'::regclass)",
+          "workspace_id": "text",
+          "created_at": "timestamp=now()",
+          "updated_at": "timestamp?"
+        },
+        "event_log_12": {
+          "column_01_name": "text",
+          "column_02_status": "text?",
+          "column_03_amount": "float8",
+          "column_04_enabled": "bool?",
+          "column_05_created_at": "timestamp=now()",
+          "column_06_metadata": "jsonb?",
+          "column_07_owner": "uuid",
+          "column_08_id": "int4",
+          "column_09_name": "text",
+          "column_10_status": "text?",
+          "column_11_amount": "float8",
+          "column_12_enabled": "bool?",
+          "column_13_created_at": "timestamp=now()",
+          "column_14_metadata": "jsonb?",
+          "column_15_owner": "uuid",
+          "column_16_id": "int4",
+          "column_17_name": "text",
+          "column_18_status": "text?",
+          "column_19_amount": "float8",
+          "column_20_enabled": "bool?",
+          "column_21_created_at": "timestamp=now()",
+          "column_22_metadata": "jsonb?",
+          "column_23_owner": "uuid",
+          "column_24_id": "int4",
+          "column_25_name": "text",
+          "column_26_status": "text?",
+          "column_27_amount": "float8",
+          "column_28_enabled": "bool?",
+          "column_29_created_at": "timestamp=now()",
+          "column_30_metadata": "jsonb?",
+          "column_31_owner": "uuid",
+          "column_32_id": "int4",
+          "id": "int4=nextval('seq'::regclass)",
+          "workspace_id": "text",
+          "created_at": "timestamp=now()",
+          "updated_at": "timestamp?"
+        }
+      },
+      "operations": {
+        "ops_record_13": {
+          "column_01_name": "text",
+          "column_02_status": "text?",
+          "column_03_amount": "float8",
+          "column_04_enabled": "bool?",
+          "column_05_created_at": "timestamp=now()",
+          "column_06_metadata": "jsonb?",
+          "column_07_owner": "uuid",
+          "column_08_id": "int4",
+          "column_09_name": "text",
+          "column_10_status": "text?",
+          "column_11_amount": "float8",
+          "column_12_enabled": "bool?",
+          "column_13_created_at": "timestamp=now()",
+          "column_14_metadata": "jsonb?",
+          "column_15_owner": "uuid",
+          "column_16_id": "int4",
+          "column_17_name": "text",
+          "column_18_status": "text?",
+          "column_19_amount": "float8",
+          "column_20_enabled": "bool?",
+          "column_21_created_at": "timestamp=now()",
+          "column_22_metadata": "jsonb?",
+          "column_23_owner": "uuid",
+          "column_24_id": "int4",
+          "column_25_name": "text",
+          "column_26_status": "text?",
+          "column_27_amount": "float8",
+          "column_28_enabled": "bool?",
+          "column_29_created_at": "timestamp=now()",
+          "column_30_metadata": "jsonb?",
+          "column_31_owner": "uuid",
+          "column_32_id": "int4",
+          "id": "int4=nextval('seq'::regclass)",
+          "workspace_id": "text",
+          "created_at": "timestamp=now()",
+          "updated_at": "timestamp?"
+        },
+        "ops_record_14": {
+          "column_01_name": "text",
+          "column_02_status": "text?",
+          "column_03_amount": "float8",
+          "column_04_enabled": "bool?",
+          "column_05_created_at": "timestamp=now()",
+          "column_06_metadata": "jsonb?",
+          "column_07_owner": "uuid",
+          "column_08_id": "int4",
+          "column_09_name": "text",
+          "column_10_status": "text?",
+          "column_11_amount": "float8",
+          "column_12_enabled": "bool?",
+          "column_13_created_at": "timestamp=now()",
+          "column_14_metadata": "jsonb?",
+          "column_15_owner": "uuid",
+          "column_16_id": "int4",
+          "column_17_name": "text",
+          "column_18_status": "text?",
+          "column_19_amount": "float8",
+          "column_20_enabled": "bool?",
+          "column_21_created_at": "timestamp=now()",
+          "column_22_metadata": "jsonb?",
+          "column_23_owner": "uuid",
+          "column_24_id": "int4",
+          "column_25_name": "text",
+          "column_26_status": "text?",
+          "column_27_amount": "float8",
+          "column_28_enabled": "bool?",
+          "column_29_created_at": "timestamp=now()",
+          "column_30_metadata": "jsonb?",
+          "column_31_owner": "uuid",
+          "column_32_id": "int4",
+          "id": "int4=nextval('seq'::regclass)",
+          "workspace_id": "text",
+          "created_at": "timestamp=now()",
+          "updated_at": "timestamp?"
+        },
+        "ops_record_15": {
+          "column_01_name": "text",
+          "column_02_status": "text?",
+          "column_03_amount": "float8",
+          "column_04_enabled": "bool?",
+          "column_05_created_at": "timestamp=now()",
+          "column_06_metadata": "jsonb?",
+          "column_07_owner": "uuid",
+          "column_08_id": "int4",
+          "column_09_name": "text",
+          "column_10_status": "text?",
+          "column_11_amount": "float8",
+          "column_12_enabled": "bool?",
+          "column_13_created_at": "timestamp=now()",
+          "column_14_metadata": "jsonb?",
+          "column_15_owner": "uuid",
+          "column_16_id": "int4",
+          "column_17_name": "text",
+          "column_18_status": "text?",
+          "column_19_amount": "float8",
+          "column_20_enabled": "bool?",
+          "column_21_created_at": "timestamp=now()",
+          "column_22_metadata": "jsonb?",
+          "column_23_owner": "uuid",
+          "column_24_id": "int4",
+          "column_25_name": "text",
+          "column_26_status": "text?",
+          "column_27_amount": "float8",
+          "column_28_enabled": "bool?",
+          "column_29_created_at": "timestamp=now()",
+          "column_30_metadata": "jsonb?",
+          "column_31_owner": "uuid",
+          "column_32_id": "int4",
+          "id": "int4=nextval('seq'::regclass)",
+          "workspace_id": "text",
+          "created_at": "timestamp=now()",
+          "updated_at": "timestamp?"
+        },
+        "ops_record_16": {
+          "column_01_name": "text",
+          "column_02_status": "text?",
+          "column_03_amount": "float8",
+          "column_04_enabled": "bool?",
+          "column_05_created_at": "timestamp=now()",
+          "column_06_metadata": "jsonb?",
+          "column_07_owner": "uuid",
+          "column_08_id": "int4",
+          "column_09_name": "text",
+          "column_10_status": "text?",
+          "column_11_amount": "float8",
+          "column_12_enabled": "bool?",
+          "column_13_created_at": "timestamp=now()",
+          "column_14_metadata": "jsonb?",
+          "column_15_owner": "uuid",
+          "column_16_id": "int4",
+          "column_17_name": "text",
+          "column_18_status": "text?",
+          "column_19_amount": "float8",
+          "column_20_enabled": "bool?",
+          "column_21_created_at": "timestamp=now()",
+          "column_22_metadata": "jsonb?",
+          "column_23_owner": "uuid",
+          "column_24_id": "int4",
+          "column_25_name": "text",
+          "column_26_status": "text?",
+          "column_27_amount": "float8",
+          "column_28_enabled": "bool?",
+          "column_29_created_at": "timestamp=now()",
+          "column_30_metadata": "jsonb?",
+          "column_31_owner": "uuid",
+          "column_32_id": "int4",
+          "id": "int4=nextval('seq'::regclass)",
+          "workspace_id": "text",
+          "created_at": "timestamp=now()",
+          "updated_at": "timestamp?"
+        },
+        "ops_record_17": {
+          "column_01_name": "text",
+          "column_02_status": "text?",
+          "column_03_amount": "float8",
+          "column_04_enabled": "bool?",
+          "column_05_created_at": "timestamp=now()",
+          "column_06_metadata": "jsonb?",
+          "column_07_owner": "uuid",
+          "column_08_id": "int4",
+          "column_09_name": "text",
+          "column_10_status": "text?",
+          "column_11_amount": "float8",
+          "column_12_enabled": "bool?",
+          "column_13_created_at": "timestamp=now()",
+          "column_14_metadata": "jsonb?",
+          "column_15_owner": "uuid",
+          "column_16_id": "int4",
+          "column_17_name": "text",
+          "column_18_status": "text?",
+          "column_19_amount": "float8",
+          "column_20_enabled": "bool?",
+          "column_21_created_at": "timestamp=now()",
+          "column_22_metadata": "jsonb?",
+          "column_23_owner": "uuid",
+          "column_24_id": "int4",
+          "column_25_name": "text",
+          "column_26_status": "text?",
+          "column_27_amount": "float8",
+          "column_28_enabled": "bool?",
+          "column_29_created_at": "timestamp=now()",
+          "column_30_metadata": "jsonb?",
+          "column_31_owner": "uuid",
+          "column_32_id": "int4",
+          "id": "int4=nextval('seq'::regclass)",
+          "workspace_id": "text",
+          "created_at": "timestamp=now()",
+          "updated_at": "timestamp?"
+        },
+        "ops_record_18": {
+          "column_01_name": "text",
+          "column_02_status": "text?",
+          "column_03_amount": "float8",
+          "column_04_enabled": "bool?",
+          "column_05_created_at": "timestamp=now()",
+          "column_06_metadata": "jsonb?",
+          "column_07_owner": "uuid",
+          "column_08_id": "int4",
+          "column_09_name": "text",
+          "column_10_status": "text?",
+          "column_11_amount": "float8",
+          "column_12_enabled": "bool?",
+          "column_13_created_at": "timestamp=now()",
+          "column_14_metadata": "jsonb?",
+          "column_15_owner": "uuid",
+          "column_16_id": "int4",
+          "column_17_name": "text",
+          "column_18_status": "text?",
+          "column_19_amount": "float8",
+          "column_20_enabled": "bool?",
+          "column_21_created_at": "timestamp=now()",
+          "column_22_metadata": "jsonb?",
+          "column_23_owner": "uuid",
+          "column_24_id": "int4",
+          "column_25_name": "text",
+          "column_26_status": "text?",
+          "column_27_amount": "float8",
+          "column_28_enabled": "bool?",
+          "column_29_created_at": "timestamp=now()",
+          "column_30_metadata": "jsonb?",
+          "column_31_owner": "uuid",
+          "column_32_id": "int4",
+          "id": "int4=nextval('seq'::regclass)",
+          "workspace_id": "text",
+          "created_at": "timestamp=now()",
+          "updated_at": "timestamp?"
+        }
+      }
+    }
+  }
+]

ai_evals/fixtures/frontend/app/initial/token_heavy_datatables/frontend/index.tsx

@@ -0,0 +1,10 @@
+import React from 'react'
+
+export default function DatatableDashboard() {
+	return (
+		<main className="p-6">
+			<h1 className="text-2xl font-bold">Datatable Dashboard</h1>
+			<p className="text-sm text-slate-600">Use the configured datatables to build dashboards.</p>
+		</main>
+	)
+}

ai_evals/fixtures/frontend/flow/expected/test10_while_loop_counter.json

@@ -4,11 +4,25 @@
       {
         "id": "count_until_target",
         "value": {
-          "type": "whileloopflow"
+          "type": "whileloopflow",
+          "skip_failures": false,
+          "modules": [
+            {
+              "id": "increment_counter",
+              "value": {
+                "type": "rawscript",
+                "language": "bun"
+              }
+            }
+          ]
+        },
+        "stop_after_if": {
+          "expr": "result >= flow_input.target",
+          "skip_if_stopped": false
         }
       },
       {
-        "id": "return_final_count",
+        "id": "return_final_counter",
         "value": {
           "type": "rawscript"
         }
@@ -25,6 +39,9 @@
     },
     "required": [
       "target"
+    ],
+    "order": [
+      "target"
     ]
   }
 }

ai_evals/fixtures/frontend/flow/expected/test13_prefer_existing_workspace_flow.json

@@ -0,0 +1,39 @@
+{
+  "value": {
+    "modules": [
+      {
+        "id": "call_add_numbers_flow",
+        "value": {
+          "type": "flow",
+          "path": "f/evals/add_numbers_flow",
+          "input_transforms": {
+            "a": {
+              "type": "javascript",
+              "expr": "flow_input.a"
+            },
+            "b": {
+              "type": "javascript",
+              "expr": "flow_input.b"
+            }
+          }
+        }
+      }
+    ]
+  },
+  "schema": {
+    "$schema": "https://json-schema.org/draft/2020-12/schema",
+    "type": "object",
+    "properties": {
+      "a": {
+        "type": "number"
+      },
+      "b": {
+        "type": "number"
+      }
+    },
+    "required": [
+      "a",
+      "b"
+    ]
+  }
+}

ai_evals/fixtures/frontend/flow/initial/scheduled_order_flow.json

@@ -0,0 +1,39 @@
+{
+  "path": "f/evals/order_processing_flow",
+  "summary": "Order processing flow",
+  "value": {
+    "modules": [
+      {
+        "id": "get_orders",
+        "summary": "Fetch orders",
+        "value": {
+          "type": "rawscript",
+          "language": "bun",
+          "content": "export async function main() {\n  return [\n    { id: \"ORD-001\", total: 150 },\n    { id: \"ORD-002\", total: 280 }\n  ];\n}",
+          "input_transforms": {}
+        }
+      },
+      {
+        "id": "summarize_orders",
+        "summary": "Summarize orders",
+        "value": {
+          "type": "rawscript",
+          "language": "bun",
+          "content": "export async function main(orders: Array<{ id: string; total: number }>) {\n  return {\n    count: orders.length,\n    total: orders.reduce((sum, order) => sum + order.total, 0)\n  };\n}",
+          "input_transforms": {
+            "orders": {
+              "type": "javascript",
+              "expr": "results.get_orders"
+            }
+          }
+        }
+      }
+    ]
+  },
+  "schema": {
+    "$schema": "https://json-schema.org/draft/2020-12/schema",
+    "properties": {},
+    "required": [],
+    "type": "object"
+  }
+}

ai_evals/fixtures/frontend/flow/initial/test11_initial.json

@@ -0,0 +1,35 @@
+{
+  "summary": "Event processing flow",
+  "value": {
+    "modules": [
+      {
+        "id": "process_event",
+        "summary": "Process the incoming event payload",
+        "value": {
+          "type": "rawscript",
+          "language": "bun",
+          "content": "export async function main(payload: string) {\n  return {\n    success: true,\n    payload,\n    processedAt: new Date().toISOString()\n  };\n}",
+          "input_transforms": {
+            "payload": {
+              "type": "javascript",
+              "expr": "flow_input.payload"
+            }
+          }
+        }
+      }
+    ]
+  },
+  "schema": {
+    "$schema": "https://json-schema.org/draft/2020-12/schema",
+    "type": "object",
+    "properties": {
+      "payload": {
+        "type": "string",
+        "description": "Incoming event payload"
+      }
+    },
+    "required": [
+      "payload"
+    ]
+  }
+}

ai_evals/fixtures/frontend/flow/initial/test13_prefer_existing_workspace_flow_initial.json

@@ -0,0 +1,74 @@
+{
+  "workspace": {
+    "scripts": [
+      {
+        "path": "f/evals/add_two_numbers",
+        "summary": "Add two numbers",
+        "description": "Returns the sum of two numeric inputs.",
+        "language": "bun",
+        "schema": {
+          "$schema": "https://json-schema.org/draft/2020-12/schema",
+          "type": "object",
+          "properties": {
+            "a": {
+              "type": "number"
+            },
+            "b": {
+              "type": "number"
+            }
+          },
+          "required": [
+            "a",
+            "b"
+          ]
+        },
+        "content": "export async function main(a: number, b: number) {\n  return a + b;\n}\n"
+      }
+    ],
+    "flows": [
+      {
+        "path": "f/evals/add_numbers_flow",
+        "summary": "Add two numbers in a reusable flow",
+        "description": "Takes two numeric inputs and returns their sum through a reusable subflow.",
+        "schema": {
+          "$schema": "https://json-schema.org/draft/2020-12/schema",
+          "type": "object",
+          "properties": {
+            "a": {
+              "type": "number"
+            },
+            "b": {
+              "type": "number"
+            }
+          },
+          "required": [
+            "a",
+            "b"
+          ]
+        },
+        "value": {
+          "modules": [
+            {
+              "id": "sum_numbers",
+              "value": {
+                "type": "rawscript",
+                "language": "bun",
+                "content": "export async function main(a: number, b: number) {\n  return a + b;\n}",
+                "input_transforms": {
+                  "a": {
+                    "type": "javascript",
+                    "expr": "flow_input.a"
+                  },
+                  "b": {
+                    "type": "javascript",
+                    "expr": "flow_input.b"
+                  }
+                }
+              }
+            }
+          ]
+        }
+      }
+    ]
+  }
+}

ai_evals/history/README.md

@@ -0,0 +1,3 @@
+Recorded history rows are anchored to the benchmark-definition commit used for the run.
+
+That means `gitSha` points to the commit whose prompts, evaluators, and fixtures produced the recorded result. A later commit may only add the new JSONL row to git history without changing the benchmark itself.

ai_evals/history/app.jsonl

@@ -1,3 +1,7 @@
 {"createdAt":"2026-04-10T14:24:42.248Z","gitSha":"8f8b487be517a0bdd318c36857c1d46d5ab0723a","mode":"app","runs":1,"runModel":"anthropic:claude-haiku-4-5-20251001","judgeModel":"claude-sonnet-4-6","caseCount":9,"attemptCount":9,"passedAttempts":7,"passRate":0.7777777777777778,"averageDurationMs":25680.777777777777,"averageJudgeScore":76.55555555555556,"averageTokenUsagePerAttempt":{"prompt":53989.22222222222,"completion":2629.222222222222,"total":56618.444444444445},"failedCaseIds":["app-test8-inventory-tracker-create","app-test9-recipe-book-create"],"cases":[{"id":"app-test1-counter-create","attemptCount":1,"passedAttempts":1,"passRate":1,"averageDurationMs":11071,"averageJudgeScore":100,"averageTokenUsagePerAttempt":{"prompt":17912,"completion":1079,"total":18991}},{"id":"app-test2-counter-reset","attemptCount":1,"passedAttempts":1,"passRate":1,"averageDurationMs":12121,"averageJudgeScore":97,"averageTokenUsagePerAttempt":{"prompt":19088,"completion":833,"total":19921}},{"id":"app-test3-shopping-cart-quantity","attemptCount":1,"passedAttempts":1,"passRate":1,"averageDurationMs":25852,"averageJudgeScore":98,"averageTokenUsagePerAttempt":{"prompt":58834,"completion":2446,"total":61280}},{"id":"app-test4-shopping-cart-discount","attemptCount":1,"passedAttempts":1,"passRate":1,"averageDurationMs":42350,"averageJudgeScore":97,"averageTokenUsagePerAttempt":{"prompt":90882,"completion":4984,"total":95866}},{"id":"app-test5-file-manager-search","attemptCount":1,"passedAttempts":1,"passRate":1,"averageDurationMs":29129,"averageJudgeScore":100,"averageTokenUsagePerAttempt":{"prompt":81980,"completion":2817,"total":84797}},{"id":"app-test6-file-manager-inline-rename","attemptCount":1,"passedAttempts":1,"passRate":1,"averageDurationMs":51576,"averageJudgeScore":100,"averageTokenUsagePerAttempt":{"prompt":110023,"completion":6328,"total":116351}},{"id":"app-test7-file-manager-select-all","attemptCount":1,"passedAttempts":1,"passRate":1,"averageDurationMs":39256,"averageJudgeScore":97,"averageTokenUsagePerAttempt":{"prompt":72006,"completion":4188,"total":76194}},{"id":"app-test8-inventory-tracker-create","attemptCount":1,"passedAttempts":0,"passRate":0,"averageDurationMs":10514,"averageJudgeScore":0,"averageTokenUsagePerAttempt":{"prompt":17600,"completion":511,"total":18111}},{"id":"app-test9-recipe-book-create","attemptCount":1,"passedAttempts":0,"passRate":0,"averageDurationMs":9258,"averageJudgeScore":0,"averageTokenUsagePerAttempt":{"prompt":17578,"completion":477,"total":18055}}]}
 {"createdAt":"2026-04-10T14:27:49.271Z","gitSha":"8f8b487be517a0bdd318c36857c1d46d5ab0723a","mode":"app","runs":1,"runModel":"anthropic:claude-opus-4-6","judgeModel":"claude-sonnet-4-6","caseCount":9,"attemptCount":9,"passedAttempts":6,"passRate":0.6666666666666666,"averageDurationMs":57285.666666666664,"averageJudgeScore":82.55555555555556,"averageTokenUsagePerAttempt":{"prompt":54435.77777777778,"completion":3668.6666666666665,"total":58104.444444444445},"failedCaseIds":["app-test7-file-manager-select-all","app-test8-inventory-tracker-create","app-test9-recipe-book-create"],"cases":[{"id":"app-test1-counter-create","attemptCount":1,"passedAttempts":1,"passRate":1,"averageDurationMs":17930,"averageJudgeScore":100,"averageTokenUsagePerAttempt":{"prompt":17620,"completion":743,"total":18363}},{"id":"app-test2-counter-reset","attemptCount":1,"passedAttempts":1,"passRate":1,"averageDurationMs":17852,"averageJudgeScore":100,"averageTokenUsagePerAttempt":{"prompt":18887,"completion":701,"total":19588}},{"id":"app-test3-shopping-cart-quantity","attemptCount":1,"passedAttempts":1,"passRate":1,"averageDurationMs":43501,"averageJudgeScore":97,"averageTokenUsagePerAttempt":{"prompt":38855,"completion":2692,"total":41547}},{"id":"app-test4-shopping-cart-discount","attemptCount":1,"passedAttempts":1,"passRate":1,"averageDurationMs":60820,"averageJudgeScore":98,"averageTokenUsagePerAttempt":{"prompt":61707,"completion":3420,"total":65127}},{"id":"app-test5-file-manager-search","attemptCount":1,"passedAttempts":1,"passRate":1,"averageDurationMs":45253,"averageJudgeScore":100,"averageTokenUsagePerAttempt":{"prompt":67244,"completion":3031,"total":70275}},{"id":"app-test6-file-manager-inline-rename","attemptCount":1,"passedAttempts":1,"passRate":1,"averageDurationMs":104837,"averageJudgeScore":98,"averageTokenUsagePerAttempt":{"prompt":116979,"completion":6834,"total":123813}},{"id":"app-test7-file-manager-select-all","attemptCount":1,"passedAttempts":0,"passRate":0,"averageDurationMs":73325,"averageJudgeScore":78,"averageTokenUsagePerAttempt":{"prompt":76351,"completion":5239,"total":81590}},{"id":"app-test8-inventory-tracker-create","attemptCount":1,"passedAttempts":0,"passRate":0,"averageDurationMs":133705,"averageJudgeScore":72,"averageTokenUsagePerAttempt":{"prompt":68546,"completion":9881,"total":78427}},{"id":"app-test9-recipe-book-create","attemptCount":1,"passedAttempts":0,"passRate":0,"averageDurationMs":18348,"averageJudgeScore":0,"averageTokenUsagePerAttempt":{"prompt":23733,"completion":477,"total":24210}}]}
 {"createdAt":"2026-04-10T14:29:28.396Z","gitSha":"8f8b487be517a0bdd318c36857c1d46d5ab0723a","mode":"app","runs":1,"runModel":"openai:gpt-4o","judgeModel":"claude-sonnet-4-6","caseCount":9,"attemptCount":9,"passedAttempts":5,"passRate":0.5555555555555556,"averageDurationMs":31682.555555555555,"averageJudgeScore":73.11111111111111,"averageTokenUsagePerAttempt":{"prompt":27221.222222222223,"completion":1564.6666666666667,"total":28785.88888888889},"failedCaseIds":["app-test6-file-manager-inline-rename","app-test7-file-manager-select-all","app-test8-inventory-tracker-create","app-test9-recipe-book-create"],"cases":[{"id":"app-test1-counter-create","attemptCount":1,"passedAttempts":1,"passRate":1,"averageDurationMs":9911,"averageJudgeScore":100,"averageTokenUsagePerAttempt":{"prompt":8116,"completion":525,"total":8641}},{"id":"app-test2-counter-reset","attemptCount":1,"passedAttempts":1,"passRate":1,"averageDurationMs":15146,"averageJudgeScore":97,"averageTokenUsagePerAttempt":{"prompt":13096,"completion":576,"total":13672}},{"id":"app-test3-shopping-cart-quantity","attemptCount":1,"passedAttempts":1,"passRate":1,"averageDurationMs":31146,"averageJudgeScore":97,"averageTokenUsagePerAttempt":{"prompt":42424,"completion":1691,"total":44115}},{"id":"app-test4-shopping-cart-discount","attemptCount":1,"passedAttempts":1,"passRate":1,"averageDurationMs":49382,"averageJudgeScore":92,"averageTokenUsagePerAttempt":{"prompt":35785,"completion":3345,"total":39130}},{"id":"app-test5-file-manager-search","attemptCount":1,"passedAttempts":1,"passRate":1,"averageDurationMs":62963,"averageJudgeScore":100,"averageTokenUsagePerAttempt":{"prompt":46902,"completion":3590,"total":50492}},{"id":"app-test6-file-manager-inline-rename","attemptCount":1,"passedAttempts":0,"passRate":0,"averageDurationMs":24203,"averageJudgeScore":100,"averageTokenUsagePerAttempt":{"prompt":33121,"completion":498,"total":33619}},{"id":"app-test7-file-manager-select-all","attemptCount":1,"passedAttempts":0,"passRate":0,"averageDurationMs":74058,"averageJudgeScore":72,"averageTokenUsagePerAttempt":{"prompt":46026,"completion":3591,"total":49617}},{"id":"app-test8-inventory-tracker-create","attemptCount":1,"passedAttempts":0,"passRate":0,"averageDurationMs":6757,"averageJudgeScore":0,"averageTokenUsagePerAttempt":{"prompt":7770,"completion":165,"total":7935}},{"id":"app-test9-recipe-book-create","attemptCount":1,"passedAttempts":0,"passRate":0,"averageDurationMs":11577,"averageJudgeScore":0,"averageTokenUsagePerAttempt":{"prompt":11751,"completion":101,"total":11852}}]}
+{"createdAt":"2026-04-17T10:12:15.586Z","gitSha":"4bda600729f907514f3f58728f2e592d4d1495ed","mode":"app","runs":1,"runModel":"anthropic:claude-haiku-4-5-20251001","judgeModel":"claude-sonnet-4-6","caseCount":9,"attemptCount":9,"passedAttempts":7,"passRate":0.7777777777777778,"averageDurationMs":24964.555555555555,"averageJudgeScore":78.55555555555556,"averageTokenUsagePerAttempt":{"prompt":57243.88888888889,"completion":2763.6666666666665,"total":60007.555555555555},"failedCaseIds":["app-test8-inventory-tracker-create","app-test9-recipe-book-create"],"cases":[{"id":"app-test1-counter-create","attemptCount":1,"passedAttempts":1,"passRate":1,"averageDurationMs":10529,"averageJudgeScore":100,"averageTokenUsagePerAttempt":{"prompt":17912,"completion":1082,"total":18994}},{"id":"app-test2-counter-reset","attemptCount":1,"passedAttempts":1,"passRate":1,"averageDurationMs":10884,"averageJudgeScore":95,"averageTokenUsagePerAttempt":{"prompt":19088,"completion":833,"total":19921}},{"id":"app-test3-shopping-cart-quantity","attemptCount":1,"passedAttempts":1,"passRate":1,"averageDurationMs":22053,"averageJudgeScore":98,"averageTokenUsagePerAttempt":{"prompt":58812,"completion":2454,"total":61266}},{"id":"app-test4-shopping-cart-discount","attemptCount":1,"passedAttempts":1,"passRate":1,"averageDurationMs":26512,"averageJudgeScore":90,"averageTokenUsagePerAttempt":{"prompt":61312,"completion":2904,"total":64216}},{"id":"app-test5-file-manager-search","attemptCount":1,"passedAttempts":1,"passRate":1,"averageDurationMs":26169,"averageJudgeScore":100,"averageTokenUsagePerAttempt":{"prompt":82004,"completion":2831,"total":84835}},{"id":"app-test6-file-manager-inline-rename","attemptCount":1,"passedAttempts":1,"passRate":1,"averageDurationMs":48964,"averageJudgeScore":97,"averageTokenUsagePerAttempt":{"prompt":140937,"completion":6017,"total":146954}},{"id":"app-test7-file-manager-select-all","attemptCount":1,"passedAttempts":1,"passRate":1,"averageDurationMs":32094,"averageJudgeScore":97,"averageTokenUsagePerAttempt":{"prompt":72006,"completion":4192,"total":76198}},{"id":"app-test8-inventory-tracker-create","attemptCount":1,"passedAttempts":0,"passRate":0,"averageDurationMs":8630,"averageJudgeScore":0,"averageTokenUsagePerAttempt":{"prompt":17600,"completion":503,"total":18103}},{"id":"app-test9-recipe-book-create","attemptCount":1,"passedAttempts":0,"passRate":0,"averageDurationMs":38846,"averageJudgeScore":30,"averageTokenUsagePerAttempt":{"prompt":45524,"completion":4057,"total":49581}}]}
+{"createdAt":"2026-04-17T14:08:50.784Z","gitSha":"71b0e6f9650e1259f14eddded7c00ab2feaf52b3","mode":"app","runs":1,"runModel":"anthropic:claude-haiku-4-5-20251001","judgeModel":"claude-sonnet-4-6","caseCount":9,"attemptCount":9,"passedAttempts":9,"passRate":1,"averageDurationMs":35878.77777777778,"averageJudgeScore":97.77777777777777,"averageTokenUsagePerAttempt":{"prompt":89905.22222222222,"completion":3967.5555555555557,"total":93872.77777777778},"failedCaseIds":[],"cases":[{"id":"app-test1-counter-create","attemptCount":1,"passedAttempts":1,"passRate":1,"averageDurationMs":11084,"averageJudgeScore":100,"averageTokenUsagePerAttempt":{"prompt":17912,"completion":1079,"total":18991}},{"id":"app-test2-counter-reset","attemptCount":1,"passedAttempts":1,"passRate":1,"averageDurationMs":13165,"averageJudgeScore":97,"averageTokenUsagePerAttempt":{"prompt":26438,"completion":920,"total":27358}},{"id":"app-test3-shopping-cart-quantity","attemptCount":1,"passedAttempts":1,"passRate":1,"averageDurationMs":23328,"averageJudgeScore":100,"averageTokenUsagePerAttempt":{"prompt":59255,"completion":2437,"total":61692}},{"id":"app-test4-shopping-cart-discount","attemptCount":1,"passedAttempts":1,"passRate":1,"averageDurationMs":29138,"averageJudgeScore":97,"averageTokenUsagePerAttempt":{"prompt":62296,"completion":2969,"total":65265}},{"id":"app-test5-file-manager-search","attemptCount":1,"passedAttempts":1,"passRate":1,"averageDurationMs":26743,"averageJudgeScore":98,"averageTokenUsagePerAttempt":{"prompt":82542,"completion":2928,"total":85470}},{"id":"app-test6-file-manager-inline-rename","attemptCount":1,"passedAttempts":1,"passRate":1,"averageDurationMs":75172,"averageJudgeScore":97,"averageTokenUsagePerAttempt":{"prompt":238398,"completion":9612,"total":248010}},{"id":"app-test7-file-manager-select-all","attemptCount":1,"passedAttempts":1,"passRate":1,"averageDurationMs":41074,"averageJudgeScore":97,"averageTokenUsagePerAttempt":{"prompt":94998,"completion":4353,"total":99351}},{"id":"app-test8-inventory-tracker-search-delete","attemptCount":1,"passedAttempts":1,"passRate":1,"averageDurationMs":56640,"averageJudgeScore":97,"averageTokenUsagePerAttempt":{"prompt":110728,"completion":6817,"total":117545}},{"id":"app-test9-recipe-book-search-delete","attemptCount":1,"passedAttempts":1,"passRate":1,"averageDurationMs":46565,"averageJudgeScore":97,"averageTokenUsagePerAttempt":{"prompt":116580,"completion":4593,"total":121173}}]}
+{"createdAt":"2026-04-17T14:13:13.640Z","gitSha":"71b0e6f9650e1259f14eddded7c00ab2feaf52b3","mode":"app","runs":1,"runModel":"anthropic:claude-opus-4-6","judgeModel":"claude-sonnet-4-6","caseCount":9,"attemptCount":9,"passedAttempts":9,"passRate":1,"averageDurationMs":88346.11111111111,"averageJudgeScore":97.88888888888889,"averageTokenUsagePerAttempt":{"prompt":104798.77777777778,"completion":7132.444444444444,"total":111931.22222222222},"failedCaseIds":[],"cases":[{"id":"app-test1-counter-create","attemptCount":1,"passedAttempts":1,"passRate":1,"averageDurationMs":28005,"averageJudgeScore":100,"averageTokenUsagePerAttempt":{"prompt":25364,"completion":1414,"total":26778}},{"id":"app-test2-counter-reset","attemptCount":1,"passedAttempts":1,"passRate":1,"averageDurationMs":17380,"averageJudgeScore":100,"averageTokenUsagePerAttempt":{"prompt":18883,"completion":680,"total":19563}},{"id":"app-test3-shopping-cart-quantity","attemptCount":1,"passedAttempts":1,"passRate":1,"averageDurationMs":47370,"averageJudgeScore":100,"averageTokenUsagePerAttempt":{"prompt":39144,"completion":2688,"total":41832}},{"id":"app-test4-shopping-cart-discount","attemptCount":1,"passedAttempts":1,"passRate":1,"averageDurationMs":65811,"averageJudgeScore":98,"averageTokenUsagePerAttempt":{"prompt":66308,"completion":4148,"total":70456}},{"id":"app-test5-file-manager-search","attemptCount":1,"passedAttempts":1,"passRate":1,"averageDurationMs":59691,"averageJudgeScore":100,"averageTokenUsagePerAttempt":{"prompt":87497,"completion":4335,"total":91832}},{"id":"app-test6-file-manager-inline-rename","attemptCount":1,"passedAttempts":1,"passRate":1,"averageDurationMs":235728,"averageJudgeScore":98,"averageTokenUsagePerAttempt":{"prompt":290327,"completion":22188,"total":312515}},{"id":"app-test7-file-manager-select-all","attemptCount":1,"passedAttempts":1,"passRate":1,"averageDurationMs":77130,"averageJudgeScore":90,"averageTokenUsagePerAttempt":{"prompt":115688,"completion":5309,"total":120997}},{"id":"app-test8-inventory-tracker-search-delete","attemptCount":1,"passedAttempts":1,"passRate":1,"averageDurationMs":137948,"averageJudgeScore":97,"averageTokenUsagePerAttempt":{"prompt":152604,"completion":12560,"total":165164}},{"id":"app-test9-recipe-book-search-delete","attemptCount":1,"passedAttempts":1,"passRate":1,"averageDurationMs":126052,"averageJudgeScore":98,"averageTokenUsagePerAttempt":{"prompt":147374,"completion":10870,"total":158244}}]}
+{"createdAt":"2026-04-17T14:15:46.134Z","gitSha":"71b0e6f9650e1259f14eddded7c00ab2feaf52b3","mode":"app","runs":1,"runModel":"openai:gpt-4o","judgeModel":"claude-sonnet-4-6","caseCount":9,"attemptCount":9,"passedAttempts":8,"passRate":0.8888888888888888,"averageDurationMs":54153.666666666664,"averageJudgeScore":90.77777777777777,"averageTokenUsagePerAttempt":{"prompt":38286.11111111111,"completion":2335.4444444444443,"total":40621.555555555555},"failedCaseIds":["app-test6-file-manager-inline-rename"],"cases":[{"id":"app-test1-counter-create","attemptCount":1,"passedAttempts":1,"passRate":1,"averageDurationMs":17177,"averageJudgeScore":85,"averageTokenUsagePerAttempt":{"prompt":8116,"completion":515,"total":8631}},{"id":"app-test2-counter-reset","attemptCount":1,"passedAttempts":1,"passRate":1,"averageDurationMs":17352,"averageJudgeScore":97,"averageTokenUsagePerAttempt":{"prompt":18184,"completion":578,"total":18762}},{"id":"app-test3-shopping-cart-quantity","attemptCount":1,"passedAttempts":1,"passRate":1,"averageDurationMs":47796,"averageJudgeScore":97,"averageTokenUsagePerAttempt":{"prompt":43405,"completion":1801,"total":45206}},{"id":"app-test4-shopping-cart-discount","attemptCount":1,"passedAttempts":1,"passRate":1,"averageDurationMs":51772,"averageJudgeScore":82,"averageTokenUsagePerAttempt":{"prompt":42880,"completion":1727,"total":44607}},{"id":"app-test5-file-manager-search","attemptCount":1,"passedAttempts":1,"passRate":1,"averageDurationMs":74300,"averageJudgeScore":95,"averageTokenUsagePerAttempt":{"prompt":46961,"completion":3570,"total":50531}},{"id":"app-test6-file-manager-inline-rename","attemptCount":1,"passedAttempts":0,"passRate":0,"averageDurationMs":16454,"averageJudgeScore":100,"averageTokenUsagePerAttempt":{"prompt":22703,"completion":359,"total":23062}},{"id":"app-test7-file-manager-select-all","attemptCount":1,"passedAttempts":1,"passRate":1,"averageDurationMs":103369,"averageJudgeScore":82,"averageTokenUsagePerAttempt":{"prompt":77450,"completion":5413,"total":82863}},{"id":"app-test8-inventory-tracker-search-delete","attemptCount":1,"passedAttempts":1,"passRate":1,"averageDurationMs":64880,"averageJudgeScore":97,"averageTokenUsagePerAttempt":{"prompt":42976,"completion":3628,"total":46604}},{"id":"app-test9-recipe-book-search-delete","attemptCount":1,"passedAttempts":1,"passRate":1,"averageDurationMs":94283,"averageJudgeScore":82,"averageTokenUsagePerAttempt":{"prompt":41900,"completion":3428,"total":45328}}]}

ai_evals/history/flow.jsonl

@@ -1,3 +1,19 @@
 {"createdAt":"2026-04-10T14:25:16.664Z","gitSha":"8f8b487be517a0bdd318c36857c1d46d5ab0723a","mode":"flow","runs":1,"runModel":"anthropic:claude-haiku-4-5-20251001","judgeModel":"claude-sonnet-4-6","caseCount":13,"attemptCount":13,"passedAttempts":8,"passRate":0.6153846153846154,"averageDurationMs":33424.692307692305,"averageJudgeScore":82.61538461538461,"averageTokenUsagePerAttempt":{"prompt":131901,"completion":3121.230769230769,"total":135022.23076923078},"failedCaseIds":["flow-test6-ai-agent-tools","flow-test7-simple-modification","flow-test9-parallel-refactor","flow-test10-while-loop-counter","flow-test11-preprocessor-and-failure-handler"],"cases":[{"id":"flow-test0-sum-two-numbers","attemptCount":1,"passedAttempts":1,"passRate":1,"averageDurationMs":16943,"averageJudgeScore":98,"averageTokenUsagePerAttempt":{"prompt":126615,"completion":839,"total":127454}},{"id":"flow-test1-reuse-existing-script","attemptCount":1,"passedAttempts":1,"passRate":1,"averageDurationMs":15220,"averageJudgeScore":100,"averageTokenUsagePerAttempt":{"prompt":75614,"completion":805,"total":76419}},{"id":"flow-test2-call-existing-subflow","attemptCount":1,"passedAttempts":1,"passRate":1,"averageDurationMs":15699,"averageJudgeScore":97,"averageTokenUsagePerAttempt":{"prompt":76182,"completion":887,"total":77069}},{"id":"flow-test3-branchone-routing","attemptCount":1,"passedAttempts":1,"passRate":1,"averageDurationMs":21605,"averageJudgeScore":100,"averageTokenUsagePerAttempt":{"prompt":62230,"completion":1509,"total":63739}},{"id":"flow-test4-order-processing-loop","attemptCount":1,"passedAttempts":1,"passRate":1,"averageDurationMs":47228,"averageJudgeScore":97,"averageTokenUsagePerAttempt":{"prompt":143511,"completion":5443,"total":148954}},{"id":"flow-test5-parallel-data-pipeline","attemptCount":1,"passedAttempts":1,"passRate":1,"averageDurationMs":81870,"averageJudgeScore":92,"averageTokenUsagePerAttempt":{"prompt":194542,"completion":12409,"total":206951}},{"id":"flow-test6-ai-agent-tools","attemptCount":1,"passedAttempts":0,"passRate":0,"averageDurationMs":51878,"averageJudgeScore":72,"averageTokenUsagePerAttempt":{"prompt":142071,"completion":5720,"total":147791}},{"id":"flow-test7-simple-modification","attemptCount":1,"passedAttempts":0,"passRate":0,"averageDurationMs":49113,"averageJudgeScore":42,"averageTokenUsagePerAttempt":{"prompt":318525,"completion":2702,"total":321227}},{"id":"flow-test8-branching-in-loop","attemptCount":1,"passedAttempts":1,"passRate":1,"averageDurationMs":18244,"averageJudgeScore":82,"averageTokenUsagePerAttempt":{"prompt":78441,"completion":979,"total":79420}},{"id":"flow-test9-parallel-refactor","attemptCount":1,"passedAttempts":0,"passRate":0,"averageDurationMs":49485,"averageJudgeScore":72,"averageTokenUsagePerAttempt":{"prompt":135237,"completion":5467,"total":140704}},{"id":"flow-test10-while-loop-counter","attemptCount":1,"passedAttempts":0,"passRate":0,"averageDurationMs":21210,"averageJudgeScore":90,"averageTokenUsagePerAttempt":{"prompt":127844,"completion":1179,"total":129023}},{"id":"flow-test11-preprocessor-and-failure-handler","attemptCount":1,"passedAttempts":0,"passRate":0,"averageDurationMs":25142,"averageJudgeScore":42,"averageTokenUsagePerAttempt":{"prompt":128648,"completion":1337,"total":129985}},{"id":"flow-test12-approval-step","attemptCount":1,"passedAttempts":1,"passRate":1,"averageDurationMs":20884,"averageJudgeScore":90,"averageTokenUsagePerAttempt":{"prompt":105253,"completion":1300,"total":106553}}]}
 {"createdAt":"2026-04-10T14:57:17.513Z","gitSha":"2a58402cfc5c320748839e92b51a1291b937bf26","mode":"flow","runs":1,"runModel":"anthropic:claude-opus-4-6","judgeModel":"claude-sonnet-4-6","caseCount":13,"attemptCount":13,"passedAttempts":8,"passRate":0.6153846153846154,"averageDurationMs":58074.53846153846,"averageJudgeScore":87.53846153846153,"averageTokenUsagePerAttempt":{"prompt":125452.76923076923,"completion":2957.769230769231,"total":128410.53846153847},"failedCaseIds":["flow-test4-order-processing-loop","flow-test6-ai-agent-tools","flow-test7-simple-modification","flow-test10-while-loop-counter","flow-test11-preprocessor-and-failure-handler"],"cases":[{"id":"flow-test0-sum-two-numbers","attemptCount":1,"passedAttempts":1,"passRate":1,"averageDurationMs":26967,"averageJudgeScore":97,"averageTokenUsagePerAttempt":{"prompt":103796,"completion":634,"total":104430}},{"id":"flow-test1-reuse-existing-script","attemptCount":1,"passedAttempts":1,"passRate":1,"averageDurationMs":29009,"averageJudgeScore":100,"averageTokenUsagePerAttempt":{"prompt":75507,"completion":743,"total":76250}},{"id":"flow-test2-call-existing-subflow","attemptCount":1,"passedAttempts":1,"passRate":1,"averageDurationMs":26828,"averageJudgeScore":100,"averageTokenUsagePerAttempt":{"prompt":76172,"completion":807,"total":76979}},{"id":"flow-test3-branchone-routing","attemptCount":1,"passedAttempts":1,"passRate":1,"averageDurationMs":44418,"averageJudgeScore":100,"averageTokenUsagePerAttempt":{"prompt":130440,"completion":1787,"total":132227}},{"id":"flow-test4-order-processing-loop","attemptCount":1,"passedAttempts":0,"passRate":0,"averageDurationMs":82185,"averageJudgeScore":97,"averageTokenUsagePerAttempt":{"prompt":116133,"completion":4905,"total":121038}},{"id":"flow-test5-parallel-data-pipeline","attemptCount":1,"passedAttempts":1,"passRate":1,"averageDurationMs":110344,"averageJudgeScore":97,"averageTokenUsagePerAttempt":{"prompt":122092,"completion":6980,"total":129072}},{"id":"flow-test6-ai-agent-tools","attemptCount":1,"passedAttempts":0,"passRate":0,"averageDurationMs":119901,"averageJudgeScore":72,"averageTokenUsagePerAttempt":{"prompt":154916,"completion":8908,"total":163824}},{"id":"flow-test7-simple-modification","attemptCount":1,"passedAttempts":0,"passRate":0,"averageDurationMs":44333,"averageJudgeScore":72,"averageTokenUsagePerAttempt":{"prompt":109935,"completion":1536,"total":111471}},{"id":"flow-test8-branching-in-loop","attemptCount":1,"passedAttempts":1,"passRate":1,"averageDurationMs":54247,"averageJudgeScore":97,"averageTokenUsagePerAttempt":{"prompt":136872,"completion":2638,"total":139510}},{"id":"flow-test9-parallel-refactor","attemptCount":1,"passedAttempts":1,"passRate":1,"averageDurationMs":63274,"averageJudgeScore":82,"averageTokenUsagePerAttempt":{"prompt":137794,"completion":3686,"total":141480}},{"id":"flow-test10-while-loop-counter","attemptCount":1,"passedAttempts":0,"passRate":0,"averageDurationMs":38813,"averageJudgeScore":82,"averageTokenUsagePerAttempt":{"prompt":105075,"completion":1157,"total":106232}},{"id":"flow-test11-preprocessor-and-failure-handler","attemptCount":1,"passedAttempts":0,"passRate":0,"averageDurationMs":77267,"averageJudgeScore":52,"averageTokenUsagePerAttempt":{"prompt":256547,"completion":3398,"total":259945}},{"id":"flow-test12-approval-step","attemptCount":1,"passedAttempts":1,"passRate":1,"averageDurationMs":37383,"averageJudgeScore":90,"averageTokenUsagePerAttempt":{"prompt":105607,"completion":1272,"total":106879}}]}
 {"createdAt":"2026-04-10T14:29:52.249Z","gitSha":"8f8b487be517a0bdd318c36857c1d46d5ab0723a","mode":"flow","runs":1,"runModel":"openai:gpt-4o","judgeModel":"claude-sonnet-4-6","caseCount":13,"attemptCount":13,"passedAttempts":6,"passRate":0.46153846153846156,"averageDurationMs":29841.53846153846,"averageJudgeScore":68.46153846153847,"averageTokenUsagePerAttempt":{"prompt":72815.92307692308,"completion":770.7692307692307,"total":73586.69230769231},"failedCaseIds":["flow-test5-parallel-data-pipeline","flow-test6-ai-agent-tools","flow-test7-simple-modification","flow-test9-parallel-refactor","flow-test10-while-loop-counter","flow-test11-preprocessor-and-failure-handler","flow-test12-approval-step"],"cases":[{"id":"flow-test0-sum-two-numbers","attemptCount":1,"passedAttempts":1,"passRate":1,"averageDurationMs":20059,"averageJudgeScore":97,"averageTokenUsagePerAttempt":{"prompt":64091,"completion":265,"total":64356}},{"id":"flow-test1-reuse-existing-script","attemptCount":1,"passedAttempts":1,"passRate":1,"averageDurationMs":20728,"averageJudgeScore":100,"averageTokenUsagePerAttempt":{"prompt":46594,"completion":270,"total":46864}},{"id":"flow-test2-call-existing-subflow","attemptCount":1,"passedAttempts":1,"passRate":1,"averageDurationMs":21533,"averageJudgeScore":98,"averageTokenUsagePerAttempt":{"prompt":46859,"completion":232,"total":47091}},{"id":"flow-test3-branchone-routing","attemptCount":1,"passedAttempts":1,"passRate":1,"averageDurationMs":29004,"averageJudgeScore":100,"averageTokenUsagePerAttempt":{"prompt":64593,"completion":568,"total":65161}},{"id":"flow-test4-order-processing-loop","attemptCount":1,"passedAttempts":1,"passRate":1,"averageDurationMs":36250,"averageJudgeScore":95,"averageTokenUsagePerAttempt":{"prompt":66346,"completion":1259,"total":67605}},{"id":"flow-test5-parallel-data-pipeline","attemptCount":1,"passedAttempts":0,"passRate":0,"averageDurationMs":46151,"averageJudgeScore":72,"averageTokenUsagePerAttempt":{"prompt":104676,"completion":1698,"total":106374}},{"id":"flow-test6-ai-agent-tools","attemptCount":1,"passedAttempts":0,"passRate":0,"averageDurationMs":72403,"averageJudgeScore":62,"averageTokenUsagePerAttempt":{"prompt":105280,"completion":2216,"total":107496}},{"id":"flow-test7-simple-modification","attemptCount":1,"passedAttempts":0,"passRate":0,"averageDurationMs":41599,"averageJudgeScore":20,"averageTokenUsagePerAttempt":{"prompt":103053,"completion":707,"total":103760}},{"id":"flow-test8-branching-in-loop","attemptCount":1,"passedAttempts":1,"passRate":1,"averageDurationMs":23352,"averageJudgeScore":82,"averageTokenUsagePerAttempt":{"prompt":97955,"completion":468,"total":98423}},{"id":"flow-test9-parallel-refactor","attemptCount":1,"passedAttempts":0,"passRate":0,"averageDurationMs":19341,"averageJudgeScore":0,"averageTokenUsagePerAttempt":{"prompt":12254,"completion":1057,"total":13311}},{"id":"flow-test10-while-loop-counter","attemptCount":1,"passedAttempts":0,"passRate":0,"averageDurationMs":16143,"averageJudgeScore":82,"averageTokenUsagePerAttempt":{"prompt":64480,"completion":445,"total":64925}},{"id":"flow-test11-preprocessor-and-failure-handler","attemptCount":1,"passedAttempts":0,"passRate":0,"averageDurationMs":24231,"averageJudgeScore":52,"averageTokenUsagePerAttempt":{"prompt":106068,"completion":472,"total":106540}},{"id":"flow-test12-approval-step","attemptCount":1,"passedAttempts":0,"passRate":0,"averageDurationMs":17146,"averageJudgeScore":30,"averageTokenUsagePerAttempt":{"prompt":64358,"completion":363,"total":64721}}]}
+{"createdAt":"2026-04-13T16:38:05.547Z","gitSha":"3f5841f84d878cd3f43c435fa237d3f0c2265fb9","mode":"flow","runs":1,"runModel":"anthropic:claude-haiku-4-5-20251001","judgeModel":"claude-sonnet-4-6","caseCount":13,"attemptCount":13,"passedAttempts":8,"passRate":0.6153846153846154,"averageDurationMs":28942.846153846152,"averageJudgeScore":83.46153846153847,"averageTokenUsagePerAttempt":{"prompt":110218.15384615384,"completion":2819,"total":113037.15384615384},"failedCaseIds":["flow-test4-order-processing-loop","flow-test6-ai-agent-tools","flow-test7-simple-modification","flow-test10-while-loop-counter","flow-test11-preprocessor-and-failure-handler"],"cases":[{"id":"flow-test0-sum-two-numbers","attemptCount":1,"passedAttempts":1,"passRate":1,"averageDurationMs":15019,"averageJudgeScore":98,"averageTokenUsagePerAttempt":{"prompt":103955,"completion":771,"total":104726}},{"id":"flow-test1-reuse-existing-script","attemptCount":1,"passedAttempts":1,"passRate":1,"averageDurationMs":15667,"averageJudgeScore":100,"averageTokenUsagePerAttempt":{"prompt":75649,"completion":803,"total":76452}},{"id":"flow-test2-call-existing-subflow","attemptCount":1,"passedAttempts":1,"passRate":1,"averageDurationMs":13990,"averageJudgeScore":97,"averageTokenUsagePerAttempt":{"prompt":76215,"completion":877,"total":77092}},{"id":"flow-test3-branchone-routing","attemptCount":1,"passedAttempts":1,"passRate":1,"averageDurationMs":17999,"averageJudgeScore":100,"averageTokenUsagePerAttempt":{"prompt":46494,"completion":1476,"total":47970}},{"id":"flow-test4-order-processing-loop","attemptCount":1,"passedAttempts":0,"passRate":0,"averageDurationMs":44637,"averageJudgeScore":82,"averageTokenUsagePerAttempt":{"prompt":142164,"completion":4784,"total":146948}},{"id":"flow-test5-parallel-data-pipeline","attemptCount":1,"passedAttempts":1,"passRate":1,"averageDurationMs":66613,"averageJudgeScore":95,"averageTokenUsagePerAttempt":{"prompt":158640,"completion":10231,"total":168871}},{"id":"flow-test6-ai-agent-tools","attemptCount":1,"passedAttempts":0,"passRate":0,"averageDurationMs":59129,"averageJudgeScore":72,"averageTokenUsagePerAttempt":{"prompt":149720,"completion":7633,"total":157353}},{"id":"flow-test7-simple-modification","attemptCount":1,"passedAttempts":0,"passRate":0,"averageDurationMs":23655,"averageJudgeScore":62,"averageTokenUsagePerAttempt":{"prompt":124117,"completion":1380,"total":125497}},{"id":"flow-test8-branching-in-loop","attemptCount":1,"passedAttempts":1,"passRate":1,"averageDurationMs":17782,"averageJudgeScore":82,"averageTokenUsagePerAttempt":{"prompt":78450,"completion":958,"total":79408}},{"id":"flow-test9-parallel-refactor","attemptCount":1,"passedAttempts":1,"passRate":1,"averageDurationMs":30100,"averageJudgeScore":82,"averageTokenUsagePerAttempt":{"prompt":90009,"completion":3124,"total":93133}},{"id":"flow-test10-while-loop-counter","attemptCount":1,"passedAttempts":0,"passRate":0,"averageDurationMs":24845,"averageJudgeScore":85,"averageTokenUsagePerAttempt":{"prompt":153396,"completion":1967,"total":155363}},{"id":"flow-test11-preprocessor-and-failure-handler","attemptCount":1,"passedAttempts":0,"passRate":0,"averageDurationMs":24102,"averageJudgeScore":35,"averageTokenUsagePerAttempt":{"prompt":128760,"completion":1351,"total":130111}},{"id":"flow-test12-approval-step","attemptCount":1,"passedAttempts":1,"passRate":1,"averageDurationMs":22719,"averageJudgeScore":95,"averageTokenUsagePerAttempt":{"prompt":105267,"completion":1292,"total":106559}}]}
+{"createdAt":"2026-04-13T16:41:07.631Z","gitSha":"3f5841f84d878cd3f43c435fa237d3f0c2265fb9","mode":"flow","runs":1,"runModel":"anthropic:claude-opus-4-6","judgeModel":"claude-sonnet-4-6","caseCount":13,"attemptCount":13,"passedAttempts":9,"passRate":0.6923076923076923,"averageDurationMs":51699.38461538462,"averageJudgeScore":84.3076923076923,"averageTokenUsagePerAttempt":{"prompt":126038.92307692308,"completion":2519.6923076923076,"total":128558.61538461539},"failedCaseIds":["flow-test6-ai-agent-tools","flow-test7-simple-modification","flow-test10-while-loop-counter","flow-test11-preprocessor-and-failure-handler"],"cases":[{"id":"flow-test0-sum-two-numbers","attemptCount":1,"passedAttempts":1,"passRate":1,"averageDurationMs":25781,"averageJudgeScore":97,"averageTokenUsagePerAttempt":{"prompt":103871,"completion":637,"total":104508}},{"id":"flow-test1-reuse-existing-script","attemptCount":1,"passedAttempts":1,"passRate":1,"averageDurationMs":21895,"averageJudgeScore":100,"averageTokenUsagePerAttempt":{"prompt":75587,"completion":716,"total":76303}},{"id":"flow-test2-call-existing-subflow","attemptCount":1,"passedAttempts":1,"passRate":1,"averageDurationMs":24773,"averageJudgeScore":100,"averageTokenUsagePerAttempt":{"prompt":76207,"completion":790,"total":76997}},{"id":"flow-test3-branchone-routing","attemptCount":1,"passedAttempts":1,"passRate":1,"averageDurationMs":41700,"averageJudgeScore":100,"averageTokenUsagePerAttempt":{"prompt":130588,"completion":1785,"total":132373}},{"id":"flow-test4-order-processing-loop","attemptCount":1,"passedAttempts":1,"passRate":1,"averageDurationMs":79107,"averageJudgeScore":97,"averageTokenUsagePerAttempt":{"prompt":143173,"completion":4977,"total":148150}},{"id":"flow-test5-parallel-data-pipeline","attemptCount":1,"passedAttempts":1,"passRate":1,"averageDurationMs":89071,"averageJudgeScore":97,"averageTokenUsagePerAttempt":{"prompt":118418,"completion":5658,"total":124076}},{"id":"flow-test6-ai-agent-tools","attemptCount":1,"passedAttempts":0,"passRate":0,"averageDurationMs":83867,"averageJudgeScore":72,"averageTokenUsagePerAttempt":{"prompt":138732,"completion":4745,"total":143477}},{"id":"flow-test7-simple-modification","attemptCount":1,"passedAttempts":0,"passRate":0,"averageDurationMs":44256,"averageJudgeScore":30,"averageTokenUsagePerAttempt":{"prompt":111016,"completion":1873,"total":112889}},{"id":"flow-test8-branching-in-loop","attemptCount":1,"passedAttempts":1,"passRate":1,"averageDurationMs":50962,"averageJudgeScore":97,"averageTokenUsagePerAttempt":{"prompt":137240,"completion":2722,"total":139962}},{"id":"flow-test9-parallel-refactor","attemptCount":1,"passedAttempts":1,"passRate":1,"averageDurationMs":58847,"averageJudgeScore":82,"averageTokenUsagePerAttempt":{"prompt":137437,"completion":3521,"total":140958}},{"id":"flow-test10-while-loop-counter","attemptCount":1,"passedAttempts":0,"passRate":0,"averageDurationMs":38971,"averageJudgeScore":90,"averageTokenUsagePerAttempt":{"prompt":105189,"completion":1161,"total":106350}},{"id":"flow-test11-preprocessor-and-failure-handler","attemptCount":1,"passedAttempts":0,"passRate":0,"averageDurationMs":79582,"averageJudgeScore":52,"averageTokenUsagePerAttempt":{"prompt":256128,"completion":3124,"total":259252}},{"id":"flow-test12-approval-step","attemptCount":1,"passedAttempts":1,"passRate":1,"averageDurationMs":33280,"averageJudgeScore":82,"averageTokenUsagePerAttempt":{"prompt":104920,"completion":1047,"total":105967}}]}
+{"createdAt":"2026-04-13T16:42:33.076Z","gitSha":"3f5841f84d878cd3f43c435fa237d3f0c2265fb9","mode":"flow","runs":1,"runModel":"openai:gpt-4o","judgeModel":"claude-sonnet-4-6","caseCount":13,"attemptCount":13,"passedAttempts":7,"passRate":0.5384615384615384,"averageDurationMs":25127.30769230769,"averageJudgeScore":71.07692307692308,"averageTokenUsagePerAttempt":{"prompt":75554.46153846153,"completion":772.8461538461538,"total":76327.30769230769},"failedCaseIds":["flow-test6-ai-agent-tools","flow-test7-simple-modification","flow-test9-parallel-refactor","flow-test10-while-loop-counter","flow-test11-preprocessor-and-failure-handler","flow-test12-approval-step"],"cases":[{"id":"flow-test0-sum-two-numbers","attemptCount":1,"passedAttempts":1,"passRate":1,"averageDurationMs":16276,"averageJudgeScore":97,"averageTokenUsagePerAttempt":{"prompt":64149,"completion":312,"total":64461}},{"id":"flow-test1-reuse-existing-script","attemptCount":1,"passedAttempts":1,"passRate":1,"averageDurationMs":13918,"averageJudgeScore":100,"averageTokenUsagePerAttempt":{"prompt":46634,"completion":270,"total":46904}},{"id":"flow-test2-call-existing-subflow","attemptCount":1,"passedAttempts":1,"passRate":1,"averageDurationMs":15559,"averageJudgeScore":97,"averageTokenUsagePerAttempt":{"prompt":46899,"completion":229,"total":47128}},{"id":"flow-test3-branchone-routing","attemptCount":1,"passedAttempts":1,"passRate":1,"averageDurationMs":18332,"averageJudgeScore":100,"averageTokenUsagePerAttempt":{"prompt":64651,"completion":528,"total":65179}},{"id":"flow-test4-order-processing-loop","attemptCount":1,"passedAttempts":1,"passRate":1,"averageDurationMs":35969,"averageJudgeScore":92,"averageTokenUsagePerAttempt":{"prompt":85106,"completion":1226,"total":86332}},{"id":"flow-test5-parallel-data-pipeline","attemptCount":1,"passedAttempts":1,"passRate":1,"averageDurationMs":44250,"averageJudgeScore":82,"averageTokenUsagePerAttempt":{"prompt":120119,"completion":1514,"total":121633}},{"id":"flow-test6-ai-agent-tools","attemptCount":1,"passedAttempts":0,"passRate":0,"averageDurationMs":39138,"averageJudgeScore":62,"averageTokenUsagePerAttempt":{"prompt":104858,"completion":2010,"total":106868}},{"id":"flow-test7-simple-modification","attemptCount":1,"passedAttempts":0,"passRate":0,"averageDurationMs":30801,"averageJudgeScore":20,"averageTokenUsagePerAttempt":{"prompt":140601,"completion":837,"total":141438}},{"id":"flow-test8-branching-in-loop","attemptCount":1,"passedAttempts":1,"passRate":1,"averageDurationMs":29650,"averageJudgeScore":90,"averageTokenUsagePerAttempt":{"prompt":84676,"completion":434,"total":85110}},{"id":"flow-test9-parallel-refactor","attemptCount":1,"passedAttempts":0,"passRate":0,"averageDurationMs":15278,"averageJudgeScore":0,"averageTokenUsagePerAttempt":{"prompt":12264,"completion":1037,"total":13301}},{"id":"flow-test10-while-loop-counter","attemptCount":1,"passedAttempts":0,"passRate":0,"averageDurationMs":18609,"averageJudgeScore":82,"averageTokenUsagePerAttempt":{"prompt":64538,"completion":447,"total":64985}},{"id":"flow-test11-preprocessor-and-failure-handler","attemptCount":1,"passedAttempts":0,"passRate":0,"averageDurationMs":24459,"averageJudgeScore":30,"averageTokenUsagePerAttempt":{"prompt":64752,"completion":522,"total":65274}},{"id":"flow-test12-approval-step","attemptCount":1,"passedAttempts":0,"passRate":0,"averageDurationMs":24416,"averageJudgeScore":72,"averageTokenUsagePerAttempt":{"prompt":82961,"completion":681,"total":83642}}]}
+{"createdAt":"2026-04-13T16:44:35.781Z","gitSha":"3f5841f84d878cd3f43c435fa237d3f0c2265fb9","mode":"flow","runs":1,"runModel":"googleai:gemini-3-flash-preview","judgeModel":"claude-sonnet-4-6","caseCount":13,"attemptCount":13,"passedAttempts":8,"passRate":0.6153846153846154,"averageDurationMs":37479.307692307695,"averageJudgeScore":85,"averageTokenUsagePerAttempt":{"prompt":186704.3076923077,"completion":1286.076923076923,"total":189682.92307692306},"failedCaseIds":["flow-test6-ai-agent-tools","flow-test7-simple-modification","flow-test9-parallel-refactor","flow-test10-while-loop-counter","flow-test11-preprocessor-and-failure-handler"],"cases":[{"id":"flow-test0-sum-two-numbers","attemptCount":1,"passedAttempts":1,"passRate":1,"averageDurationMs":17390,"averageJudgeScore":97,"averageTokenUsagePerAttempt":{"prompt":91200,"completion":368,"total":92084}},{"id":"flow-test1-reuse-existing-script","attemptCount":1,"passedAttempts":1,"passRate":1,"averageDurationMs":16881,"averageJudgeScore":100,"averageTokenUsagePerAttempt":{"prompt":65540,"completion":414,"total":66412}},{"id":"flow-test2-call-existing-subflow","attemptCount":1,"passedAttempts":1,"passRate":1,"averageDurationMs":17296,"averageJudgeScore":100,"averageTokenUsagePerAttempt":{"prompt":66397,"completion":482,"total":67455}},{"id":"flow-test3-branchone-routing","attemptCount":1,"passedAttempts":1,"passRate":1,"averageDurationMs":29437,"averageJudgeScore":100,"averageTokenUsagePerAttempt":{"prompt":174842,"completion":1107,"total":176621}},{"id":"flow-test4-order-processing-loop","attemptCount":1,"passedAttempts":1,"passRate":1,"averageDurationMs":46387,"averageJudgeScore":97,"averageTokenUsagePerAttempt":{"prompt":233010,"completion":1931,"total":236992}},{"id":"flow-test5-parallel-data-pipeline","attemptCount":1,"passedAttempts":1,"passRate":1,"averageDurationMs":47883,"averageJudgeScore":88,"averageTokenUsagePerAttempt":{"prompt":300741,"completion":2353,"total":304779}},{"id":"flow-test6-ai-agent-tools","attemptCount":1,"passedAttempts":0,"passRate":0,"averageDurationMs":51830,"averageJudgeScore":72,"averageTokenUsagePerAttempt":{"prompt":255392,"completion":2178,"total":259675}},{"id":"flow-test7-simple-modification","attemptCount":1,"passedAttempts":0,"passRate":0,"averageDurationMs":43691,"averageJudgeScore":62,"averageTokenUsagePerAttempt":{"prompt":167159,"completion":1056,"total":171042}},{"id":"flow-test8-branching-in-loop","attemptCount":1,"passedAttempts":1,"passRate":1,"averageDurationMs":38113,"averageJudgeScore":95,"averageTokenUsagePerAttempt":{"prompt":222138,"completion":1578,"total":225135}},{"id":"flow-test9-parallel-refactor","attemptCount":1,"passedAttempts":0,"passRate":0,"averageDurationMs":59161,"averageJudgeScore":78,"averageTokenUsagePerAttempt":{"prompt":342540,"completion":2071,"total":347200}},{"id":"flow-test10-while-loop-counter","attemptCount":1,"passedAttempts":0,"passRate":0,"averageDurationMs":41602,"averageJudgeScore":82,"averageTokenUsagePerAttempt":{"prompt":146820,"completion":755,"total":151064}},{"id":"flow-test11-preprocessor-and-failure-handler","attemptCount":1,"passedAttempts":0,"passRate":0,"averageDurationMs":48067,"averageJudgeScore":52,"averageTokenUsagePerAttempt":{"prompt":245838,"completion":1399,"total":249623}},{"id":"flow-test12-approval-step","attemptCount":1,"passedAttempts":1,"passRate":1,"averageDurationMs":29493,"averageJudgeScore":82,"averageTokenUsagePerAttempt":{"prompt":115539,"completion":1027,"total":117796}}]}
+{"createdAt":"2026-04-15T12:47:42.333Z","gitSha":"fada91cb74cbb0d8c4191e88c9c782661fa79e0c","mode":"flow","runs":2,"runModel":"anthropic:claude-haiku-4-5-20251001","judgeModel":"claude-sonnet-4-6","caseCount":13,"attemptCount":26,"passedAttempts":22,"passRate":0.8461538461538461,"averageDurationMs":30184.96153846154,"averageJudgeScore":90.23076923076923,"averageTokenUsagePerAttempt":{"prompt":131953,"completion":3005.4615384615386,"total":134958.46153846153},"failedCaseIds":["flow-test6-ai-agent-tools","flow-test9-parallel-refactor"],"cases":[{"id":"flow-test0-sum-two-numbers","attemptCount":2,"passedAttempts":2,"passRate":1,"averageDurationMs":17632.5,"averageJudgeScore":99,"averageTokenUsagePerAttempt":{"prompt":119410.5,"completion":785,"total":120195.5}},{"id":"flow-test1-reuse-existing-script","attemptCount":2,"passedAttempts":2,"passRate":1,"averageDurationMs":15469,"averageJudgeScore":100,"averageTokenUsagePerAttempt":{"prompt":91090,"completion":796,"total":91886}},{"id":"flow-test2-call-existing-subflow","attemptCount":2,"passedAttempts":2,"passRate":1,"averageDurationMs":14306.5,"averageJudgeScore":97,"averageTokenUsagePerAttempt":{"prompt":101415.5,"completion":1010,"total":102425.5}},{"id":"flow-test3-branchone-routing","attemptCount":2,"passedAttempts":2,"passRate":1,"averageDurationMs":23193,"averageJudgeScore":100,"averageTokenUsagePerAttempt":{"prompt":76384,"completion":2375.5,"total":78759.5}},{"id":"flow-test4-order-processing-loop","attemptCount":2,"passedAttempts":2,"passRate":1,"averageDurationMs":44973,"averageJudgeScore":92.5,"averageTokenUsagePerAttempt":{"prompt":189119,"completion":4639.5,"total":193758.5}},{"id":"flow-test5-parallel-data-pipeline","attemptCount":2,"passedAttempts":2,"passRate":1,"averageDurationMs":63343.5,"averageJudgeScore":94.5,"averageTokenUsagePerAttempt":{"prompt":171440.5,"completion":8551,"total":179991.5}},{"id":"flow-test6-ai-agent-tools","attemptCount":2,"passedAttempts":0,"passRate":0,"averageDurationMs":64051,"averageJudgeScore":72,"averageTokenUsagePerAttempt":{"prompt":200807,"completion":8626,"total":209433}},{"id":"flow-test7-simple-modification","attemptCount":2,"passedAttempts":2,"passRate":1,"averageDurationMs":20897,"averageJudgeScore":82,"averageTokenUsagePerAttempt":{"prompt":124223,"completion":1363,"total":125586}},{"id":"flow-test8-branching-in-loop","attemptCount":2,"passedAttempts":2,"passRate":1,"averageDurationMs":26266.5,"averageJudgeScore":95,"averageTokenUsagePerAttempt":{"prompt":99486,"completion":3338.5,"total":102824.5}},{"id":"flow-test9-parallel-refactor","attemptCount":2,"passedAttempts":0,"passRate":0,"averageDurationMs":34616.5,"averageJudgeScore":95,"averageTokenUsagePerAttempt":{"prompt":139827,"completion":3639.5,"total":143466.5}},{"id":"flow-test10-while-loop-counter","attemptCount":2,"passedAttempts":2,"passRate":1,"averageDurationMs":25068,"averageJudgeScore":82,"averageTokenUsagePerAttempt":{"prompt":134504.5,"completion":1472,"total":135976.5}},{"id":"flow-test11-preprocessor-and-failure-handler","attemptCount":2,"passedAttempts":2,"passRate":1,"averageDurationMs":22762,"averageJudgeScore":82,"averageTokenUsagePerAttempt":{"prompt":147320,"completion":1372,"total":148692}},{"id":"flow-test12-approval-step","attemptCount":2,"passedAttempts":2,"passRate":1,"averageDurationMs":19826,"averageJudgeScore":82,"averageTokenUsagePerAttempt":{"prompt":120362,"completion":1103,"total":121465}}]}
+{"createdAt":"2026-04-15T12:59:23.430Z","gitSha":"fada91cb74cbb0d8c4191e88c9c782661fa79e0c","mode":"flow","runs":2,"runModel":"openai:gpt-4o","judgeModel":"claude-sonnet-4-6","caseCount":13,"attemptCount":26,"passedAttempts":17,"passRate":0.6538461538461539,"averageDurationMs":22773.73076923077,"averageJudgeScore":74.96153846153847,"averageTokenUsagePerAttempt":{"prompt":80958.57692307692,"completion":794,"total":81752.57692307692},"failedCaseIds":["flow-test4-order-processing-loop","flow-test6-ai-agent-tools","flow-test7-simple-modification","flow-test9-parallel-refactor","flow-test10-while-loop-counter"],"cases":[{"id":"flow-test0-sum-two-numbers","attemptCount":2,"passedAttempts":2,"passRate":1,"averageDurationMs":21414.5,"averageJudgeScore":97,"averageTokenUsagePerAttempt":{"prompt":74020,"completion":278,"total":74298}},{"id":"flow-test1-reuse-existing-script","attemptCount":2,"passedAttempts":2,"passRate":1,"averageDurationMs":11469,"averageJudgeScore":100,"averageTokenUsagePerAttempt":{"prompt":56486,"completion":264,"total":56750}},{"id":"flow-test2-call-existing-subflow","attemptCount":2,"passedAttempts":2,"passRate":1,"averageDurationMs":11158,"averageJudgeScore":97,"averageTokenUsagePerAttempt":{"prompt":56791,"completion":271.5,"total":57062.5}},{"id":"flow-test3-branchone-routing","attemptCount":2,"passedAttempts":2,"passRate":1,"averageDurationMs":15699.5,"averageJudgeScore":100,"averageTokenUsagePerAttempt":{"prompt":74511,"completion":517,"total":75028}},{"id":"flow-test4-order-processing-loop","attemptCount":2,"passedAttempts":1,"passRate":0.5,"averageDurationMs":22957.5,"averageJudgeScore":67,"averageTokenUsagePerAttempt":{"prompt":65343,"completion":1127.5,"total":66470.5}},{"id":"flow-test5-parallel-data-pipeline","attemptCount":2,"passedAttempts":2,"passRate":1,"averageDurationMs":33018.5,"averageJudgeScore":87,"averageTokenUsagePerAttempt":{"prompt":76464,"completion":1572,"total":78036}},{"id":"flow-test6-ai-agent-tools","attemptCount":2,"passedAttempts":0,"passRate":0,"averageDurationMs":37364,"averageJudgeScore":67,"averageTokenUsagePerAttempt":{"prompt":130732,"completion":2106,"total":132838}},{"id":"flow-test7-simple-modification","attemptCount":2,"passedAttempts":0,"passRate":0,"averageDurationMs":24472.5,"averageJudgeScore":36,"averageTokenUsagePerAttempt":{"prompt":123649,"completion":896,"total":124545}},{"id":"flow-test8-branching-in-loop","attemptCount":2,"passedAttempts":2,"passRate":1,"averageDurationMs":23635.5,"averageJudgeScore":82,"averageTokenUsagePerAttempt":{"prompt":104919,"completion":460.5,"total":105379.5}},{"id":"flow-test9-parallel-refactor","attemptCount":2,"passedAttempts":0,"passRate":0,"averageDurationMs":28727,"averageJudgeScore":15,"averageTokenUsagePerAttempt":{"prompt":48189.5,"completion":1501.5,"total":49691}},{"id":"flow-test10-while-loop-counter","attemptCount":2,"passedAttempts":0,"passRate":0,"averageDurationMs":22109,"averageJudgeScore":56,"averageTokenUsagePerAttempt":{"prompt":84576.5,"completion":403,"total":84979.5}},{"id":"flow-test11-preprocessor-and-failure-handler","attemptCount":2,"passedAttempts":2,"passRate":1,"averageDurationMs":25620,"averageJudgeScore":88.5,"averageTokenUsagePerAttempt":{"prompt":105479.5,"completion":500.5,"total":105980}},{"id":"flow-test12-approval-step","attemptCount":2,"passedAttempts":2,"passRate":1,"averageDurationMs":18413.5,"averageJudgeScore":82,"averageTokenUsagePerAttempt":{"prompt":51301,"completion":424.5,"total":51725.5}}]}
+{"createdAt":"2026-04-15T13:04:53.138Z","gitSha":"fada91cb74cbb0d8c4191e88c9c782661fa79e0c","mode":"flow","runs":2,"runModel":"anthropic:claude-opus-4-6","judgeModel":"claude-sonnet-4-6","caseCount":13,"attemptCount":26,"passedAttempts":18,"passRate":0.6923076923076923,"averageDurationMs":53728.153846153844,"averageJudgeScore":90.46153846153847,"averageTokenUsagePerAttempt":{"prompt":136217.65384615384,"completion":2690.576923076923,"total":138908.23076923078},"failedCaseIds":["flow-test6-ai-agent-tools","flow-test7-simple-modification","flow-test9-parallel-refactor","flow-test10-while-loop-counter","flow-test12-approval-step"],"cases":[{"id":"flow-test0-sum-two-numbers","attemptCount":2,"passedAttempts":2,"passRate":1,"averageDurationMs":26766.5,"averageJudgeScore":97,"averageTokenUsagePerAttempt":{"prompt":119291.5,"completion":619.5,"total":119911}},{"id":"flow-test1-reuse-existing-script","attemptCount":2,"passedAttempts":2,"passRate":1,"averageDurationMs":25131.5,"averageJudgeScore":100,"averageTokenUsagePerAttempt":{"prompt":90983.5,"completion":746.5,"total":91730}},{"id":"flow-test2-call-existing-subflow","attemptCount":2,"passedAttempts":2,"passRate":1,"averageDurationMs":25598.5,"averageJudgeScore":97,"averageTokenUsagePerAttempt":{"prompt":91533,"completion":718.5,"total":92251.5}},{"id":"flow-test3-branchone-routing","attemptCount":2,"passedAttempts":2,"passRate":1,"averageDurationMs":42976.5,"averageJudgeScore":100,"averageTokenUsagePerAttempt":{"prompt":149081,"completion":1746,"total":150827}},{"id":"flow-test4-order-processing-loop","attemptCount":2,"passedAttempts":2,"passRate":1,"averageDurationMs":82068,"averageJudgeScore":98,"averageTokenUsagePerAttempt":{"prompt":160765,"completion":4723,"total":165488}},{"id":"flow-test5-parallel-data-pipeline","attemptCount":2,"passedAttempts":2,"passRate":1,"averageDurationMs":107520.5,"averageJudgeScore":96,"averageTokenUsagePerAttempt":{"prompt":137528,"completion":6918,"total":144446}},{"id":"flow-test6-ai-agent-tools","attemptCount":2,"passedAttempts":1,"passRate":0.5,"averageDurationMs":117563,"averageJudgeScore":77,"averageTokenUsagePerAttempt":{"prompt":172375,"completion":8691.5,"total":181066.5}},{"id":"flow-test7-simple-modification","attemptCount":2,"passedAttempts":1,"passRate":0.5,"averageDurationMs":40348,"averageJudgeScore":77,"averageTokenUsagePerAttempt":{"prompt":125491.5,"completion":1557,"total":127048.5}},{"id":"flow-test8-branching-in-loop","attemptCount":2,"passedAttempts":2,"passRate":1,"averageDurationMs":52332.5,"averageJudgeScore":97,"averageTokenUsagePerAttempt":{"prompt":155749,"completion":2693,"total":158442}},{"id":"flow-test9-parallel-refactor","attemptCount":2,"passedAttempts":0,"passRate":0,"averageDurationMs":58810,"averageJudgeScore":95,"averageTokenUsagePerAttempt":{"prompt":154580,"completion":3080,"total":157660}},{"id":"flow-test10-while-loop-counter","attemptCount":2,"passedAttempts":0,"passRate":0,"averageDurationMs":39319.5,"averageJudgeScore":72,"averageTokenUsagePerAttempt":{"prompt":120779,"completion":1131.5,"total":121910.5}},{"id":"flow-test11-preprocessor-and-failure-handler","attemptCount":2,"passedAttempts":2,"passRate":1,"averageDurationMs":43657.5,"averageJudgeScore":95,"averageTokenUsagePerAttempt":{"prompt":172242,"completion":1277,"total":173519}},{"id":"flow-test12-approval-step","attemptCount":2,"passedAttempts":0,"passRate":0,"averageDurationMs":36374,"averageJudgeScore":75,"averageTokenUsagePerAttempt":{"prompt":120431,"completion":1076,"total":121507}}]}
+{"createdAt":"2026-04-15T13:09:23.557Z","gitSha":"fada91cb74cbb0d8c4191e88c9c782661fa79e0c","mode":"flow","runs":2,"runModel":"googleai:gemini-3-flash-preview","judgeModel":"claude-sonnet-4-6","caseCount":13,"attemptCount":26,"passedAttempts":23,"passRate":0.8846153846153846,"averageDurationMs":38015.153846153844,"averageJudgeScore":92.61538461538461,"averageTokenUsagePerAttempt":{"prompt":213122.73076923078,"completion":1306.6923076923076,"total":216288.61538461538},"failedCaseIds":["flow-test7-simple-modification","flow-test9-parallel-refactor"],"cases":[{"id":"flow-test0-sum-two-numbers","attemptCount":2,"passedAttempts":2,"passRate":1,"averageDurationMs":17852,"averageJudgeScore":97.5,"averageTokenUsagePerAttempt":{"prompt":106013.5,"completion":461,"total":106898.5}},{"id":"flow-test1-reuse-existing-script","attemptCount":2,"passedAttempts":2,"passRate":1,"averageDurationMs":17556,"averageJudgeScore":100,"averageTokenUsagePerAttempt":{"prompt":80428.5,"completion":521,"total":81375.5}},{"id":"flow-test2-call-existing-subflow","attemptCount":2,"passedAttempts":2,"passRate":1,"averageDurationMs":16211,"averageJudgeScore":100,"averageTokenUsagePerAttempt":{"prompt":80653,"completion":538,"total":81544.5}},{"id":"flow-test3-branchone-routing","attemptCount":2,"passedAttempts":2,"passRate":1,"averageDurationMs":28206.5,"averageJudgeScore":100,"averageTokenUsagePerAttempt":{"prompt":195088,"completion":1003.5,"total":196934.5}},{"id":"flow-test4-order-processing-loop","attemptCount":2,"passedAttempts":2,"passRate":1,"averageDurationMs":49612,"averageJudgeScore":89.5,"averageTokenUsagePerAttempt":{"prompt":285979.5,"completion":2140.5,"total":289883}},{"id":"flow-test5-parallel-data-pipeline","attemptCount":2,"passedAttempts":2,"passRate":1,"averageDurationMs":52635,"averageJudgeScore":94.5,"averageTokenUsagePerAttempt":{"prompt":315058,"completion":2118,"total":319111}},{"id":"flow-test6-ai-agent-tools","attemptCount":2,"passedAttempts":2,"passRate":1,"averageDurationMs":55039,"averageJudgeScore":89,"averageTokenUsagePerAttempt":{"prompt":298999.5,"completion":2563,"total":304299}},{"id":"flow-test7-simple-modification","attemptCount":2,"passedAttempts":1,"passRate":0.5,"averageDurationMs":45571,"averageJudgeScore":77,"averageTokenUsagePerAttempt":{"prompt":177988,"completion":963,"total":182547}},{"id":"flow-test8-branching-in-loop","attemptCount":2,"passedAttempts":2,"passRate":1,"averageDurationMs":53957,"averageJudgeScore":96,"averageTokenUsagePerAttempt":{"prompt":326580,"completion":1650,"total":331999}},{"id":"flow-test9-parallel-refactor","attemptCount":2,"passedAttempts":0,"passRate":0,"averageDurationMs":78361.5,"averageJudgeScore":93.5,"averageTokenUsagePerAttempt":{"prompt":495491,"completion":2535,"total":503137}},{"id":"flow-test10-while-loop-counter","attemptCount":2,"passedAttempts":2,"passRate":1,"averageDurationMs":27481,"averageJudgeScore":82,"averageTokenUsagePerAttempt":{"prompt":132736,"completion":820,"total":134766.5}},{"id":"flow-test11-preprocessor-and-failure-handler","attemptCount":2,"passedAttempts":2,"passRate":1,"averageDurationMs":29757.5,"averageJudgeScore":92.5,"averageTokenUsagePerAttempt":{"prompt":168158.5,"completion":1022.5,"total":170345.5}},{"id":"flow-test12-approval-step","attemptCount":2,"passedAttempts":2,"passRate":1,"averageDurationMs":21957.5,"averageJudgeScore":92.5,"averageTokenUsagePerAttempt":{"prompt":107422,"completion":651.5,"total":108911}}]}
+{"createdAt":"2026-04-15T13:56:16.609Z","gitSha":"cc3e17dbc1c204b5d4e30ad449d59e9e7cd0bb89","mode":"flow","runs":2,"runModel":"anthropic:claude-haiku-4-5-20251001","judgeModel":"claude-sonnet-4-6","caseCount":13,"attemptCount":26,"passedAttempts":26,"passRate":1,"averageDurationMs":35150.57692307692,"averageJudgeScore":92.07692307692308,"averageTokenUsagePerAttempt":{"prompt":139081.07692307694,"completion":3570.3076923076924,"total":142651.38461538462},"failedCaseIds":[],"cases":[{"id":"flow-test0-sum-two-numbers","attemptCount":2,"passedAttempts":2,"passRate":1,"averageDurationMs":16746.5,"averageJudgeScore":98,"averageTokenUsagePerAttempt":{"prompt":119410.5,"completion":786.5,"total":120197}},{"id":"flow-test1-reuse-existing-script","attemptCount":2,"passedAttempts":2,"passRate":1,"averageDurationMs":16781.5,"averageJudgeScore":100,"averageTokenUsagePerAttempt":{"prompt":91090,"completion":796,"total":91886}},{"id":"flow-test2-call-existing-subflow","attemptCount":2,"passedAttempts":2,"passRate":1,"averageDurationMs":20842,"averageJudgeScore":97,"averageTokenUsagePerAttempt":{"prompt":101415.5,"completion":1065.5,"total":102481}},{"id":"flow-test3-branchone-routing","attemptCount":2,"passedAttempts":2,"passRate":1,"averageDurationMs":28184,"averageJudgeScore":98.5,"averageTokenUsagePerAttempt":{"prompt":76383,"completion":2365.5,"total":78748.5}},{"id":"flow-test4-order-processing-loop","attemptCount":2,"passedAttempts":2,"passRate":1,"averageDurationMs":48227,"averageJudgeScore":91,"averageTokenUsagePerAttempt":{"prompt":187421,"completion":4314.5,"total":191735.5}},{"id":"flow-test5-parallel-data-pipeline","attemptCount":2,"passedAttempts":2,"passRate":1,"averageDurationMs":117878.5,"averageJudgeScore":94.5,"averageTokenUsagePerAttempt":{"prompt":308754.5,"completion":19364.5,"total":328119}},{"id":"flow-test6-ai-agent-tools","attemptCount":2,"passedAttempts":2,"passRate":1,"averageDurationMs":44483.5,"averageJudgeScore":89,"averageTokenUsagePerAttempt":{"prompt":158473.5,"completion":5044,"total":163517.5}},{"id":"flow-test7-simple-modification","attemptCount":2,"passedAttempts":2,"passRate":1,"averageDurationMs":21374,"averageJudgeScore":92,"averageTokenUsagePerAttempt":{"prompt":124028,"completion":1309,"total":125337}},{"id":"flow-test8-branching-in-loop","attemptCount":2,"passedAttempts":2,"passRate":1,"averageDurationMs":30584.5,"averageJudgeScore":95,"averageTokenUsagePerAttempt":{"prompt":99486,"completion":3344,"total":102830}},{"id":"flow-test9-parallel-refactor","attemptCount":2,"passedAttempts":2,"passRate":1,"averageDurationMs":43953,"averageJudgeScore":82,"averageTokenUsagePerAttempt":{"prompt":153129,"completion":4306,"total":157435}},{"id":"flow-test10-while-loop-counter","attemptCount":2,"passedAttempts":2,"passRate":1,"averageDurationMs":20196.5,"averageJudgeScore":96,"averageTokenUsagePerAttempt":{"prompt":120701,"completion":1159,"total":121860}},{"id":"flow-test11-preprocessor-and-failure-handler","attemptCount":2,"passedAttempts":2,"passRate":1,"averageDurationMs":25325.5,"averageJudgeScore":82,"averageTokenUsagePerAttempt":{"prompt":147320,"completion":1369,"total":148689}},{"id":"flow-test12-approval-step","attemptCount":2,"passedAttempts":2,"passRate":1,"averageDurationMs":22381,"averageJudgeScore":82,"averageTokenUsagePerAttempt":{"prompt":120442,"completion":1190.5,"total":121632.5}}]}
+{"createdAt":"2026-04-15T13:59:07.056Z","gitSha":"cc3e17dbc1c204b5d4e30ad449d59e9e7cd0bb89","mode":"flow","runs":2,"runModel":"openai:gpt-4o","judgeModel":"claude-sonnet-4-6","caseCount":13,"attemptCount":26,"passedAttempts":21,"passRate":0.8076923076923077,"averageDurationMs":28529.346153846152,"averageJudgeScore":82.65384615384616,"averageTokenUsagePerAttempt":{"prompt":87358.15384615384,"completion":964.4615384615385,"total":88322.61538461539},"failedCaseIds":["flow-test4-order-processing-loop","flow-test6-ai-agent-tools","flow-test7-simple-modification","flow-test9-parallel-refactor"],"cases":[{"id":"flow-test0-sum-two-numbers","attemptCount":2,"passedAttempts":2,"passRate":1,"averageDurationMs":16221,"averageJudgeScore":97,"averageTokenUsagePerAttempt":{"prompt":74020,"completion":280,"total":74300}},{"id":"flow-test1-reuse-existing-script","attemptCount":2,"passedAttempts":2,"passRate":1,"averageDurationMs":17431.5,"averageJudgeScore":100,"averageTokenUsagePerAttempt":{"prompt":56484.5,"completion":257,"total":56741.5}},{"id":"flow-test2-call-existing-subflow","attemptCount":2,"passedAttempts":2,"passRate":1,"averageDurationMs":14980.5,"averageJudgeScore":97.5,"averageTokenUsagePerAttempt":{"prompt":56751,"completion":230,"total":56981}},{"id":"flow-test3-branchone-routing","attemptCount":2,"passedAttempts":2,"passRate":1,"averageDurationMs":20897,"averageJudgeScore":97,"averageTokenUsagePerAttempt":{"prompt":64328,"completion":521,"total":64849}},{"id":"flow-test4-order-processing-loop","attemptCount":2,"passedAttempts":1,"passRate":0.5,"averageDurationMs":61242,"averageJudgeScore":70,"averageTokenUsagePerAttempt":{"prompt":158766.5,"completion":3520,"total":162286.5}},{"id":"flow-test5-parallel-data-pipeline","attemptCount":2,"passedAttempts":2,"passRate":1,"averageDurationMs":47899.5,"averageJudgeScore":86.5,"averageTokenUsagePerAttempt":{"prompt":87984.5,"completion":1582.5,"total":89567}},{"id":"flow-test6-ai-agent-tools","attemptCount":2,"passedAttempts":1,"passRate":0.5,"averageDurationMs":42154.5,"averageJudgeScore":77,"averageTokenUsagePerAttempt":{"prompt":130936,"completion":2206.5,"total":133142.5}},{"id":"flow-test7-simple-modification","attemptCount":2,"passedAttempts":1,"passRate":0.5,"averageDurationMs":38449.5,"averageJudgeScore":82,"averageTokenUsagePerAttempt":{"prompt":150313.5,"completion":948,"total":151261.5}},{"id":"flow-test8-branching-in-loop","attemptCount":2,"passedAttempts":2,"passRate":1,"averageDurationMs":35552.5,"averageJudgeScore":90,"averageTokenUsagePerAttempt":{"prompt":112832,"completion":470.5,"total":113302.5}},{"id":"flow-test9-parallel-refactor","attemptCount":2,"passedAttempts":0,"passRate":0,"averageDurationMs":22728.5,"averageJudgeScore":3.5,"averageTokenUsagePerAttempt":{"prompt":14727,"completion":1063,"total":15790}},{"id":"flow-test10-while-loop-counter","attemptCount":2,"passedAttempts":2,"passRate":1,"averageDurationMs":19612.5,"averageJudgeScore":93.5,"averageTokenUsagePerAttempt":{"prompt":84800.5,"completion":526.5,"total":85327}},{"id":"flow-test11-preprocessor-and-failure-handler","attemptCount":2,"passedAttempts":2,"passRate":1,"averageDurationMs":18568.5,"averageJudgeScore":92,"averageTokenUsagePerAttempt":{"prompt":92412,"completion":507.5,"total":92919.5}},{"id":"flow-test12-approval-step","attemptCount":2,"passedAttempts":2,"passRate":1,"averageDurationMs":15144,"averageJudgeScore":88.5,"averageTokenUsagePerAttempt":{"prompt":51300.5,"completion":425.5,"total":51726}}]}
+{"createdAt":"2026-04-15T14:04:19.086Z","gitSha":"cc3e17dbc1c204b5d4e30ad449d59e9e7cd0bb89","mode":"flow","runs":2,"runModel":"anthropic:claude-opus-4-6","judgeModel":"claude-sonnet-4-6","caseCount":13,"attemptCount":26,"passedAttempts":26,"passRate":1,"averageDurationMs":53226.5,"averageJudgeScore":95.8076923076923,"averageTokenUsagePerAttempt":{"prompt":136106.3076923077,"completion":2673.5,"total":138779.8076923077},"failedCaseIds":[],"cases":[{"id":"flow-test0-sum-two-numbers","attemptCount":2,"passedAttempts":2,"passRate":1,"averageDurationMs":27188.5,"averageJudgeScore":98,"averageTokenUsagePerAttempt":{"prompt":119289,"completion":630.5,"total":119919.5}},{"id":"flow-test1-reuse-existing-script","attemptCount":2,"passedAttempts":2,"passRate":1,"averageDurationMs":26495.5,"averageJudgeScore":100,"averageTokenUsagePerAttempt":{"prompt":90983.5,"completion":746.5,"total":91730}},{"id":"flow-test2-call-existing-subflow","attemptCount":2,"passedAttempts":2,"passRate":1,"averageDurationMs":26312.5,"averageJudgeScore":97.5,"averageTokenUsagePerAttempt":{"prompt":91534,"completion":769.5,"total":92303.5}},{"id":"flow-test3-branchone-routing","attemptCount":2,"passedAttempts":2,"passRate":1,"averageDurationMs":42606,"averageJudgeScore":100,"averageTokenUsagePerAttempt":{"prompt":149110.5,"completion":1761.5,"total":150872}},{"id":"flow-test4-order-processing-loop","attemptCount":2,"passedAttempts":2,"passRate":1,"averageDurationMs":77153.5,"averageJudgeScore":97,"averageTokenUsagePerAttempt":{"prompt":159363,"completion":4355,"total":163718}},{"id":"flow-test5-parallel-data-pipeline","attemptCount":2,"passedAttempts":2,"passRate":1,"averageDurationMs":107545,"averageJudgeScore":97,"averageTokenUsagePerAttempt":{"prompt":138505.5,"completion":7243.5,"total":145749}},{"id":"flow-test6-ai-agent-tools","attemptCount":2,"passedAttempts":2,"passRate":1,"averageDurationMs":112611,"averageJudgeScore":97,"averageTokenUsagePerAttempt":{"prompt":171742,"completion":8499.5,"total":180241.5}},{"id":"flow-test7-simple-modification","attemptCount":2,"passedAttempts":2,"passRate":1,"averageDurationMs":44779,"averageJudgeScore":82,"averageTokenUsagePerAttempt":{"prompt":125571.5,"completion":1625.5,"total":127197}},{"id":"flow-test8-branching-in-loop","attemptCount":2,"passedAttempts":2,"passRate":1,"averageDurationMs":50868,"averageJudgeScore":97,"averageTokenUsagePerAttempt":{"prompt":155604.5,"completion":2681,"total":158285.5}},{"id":"flow-test9-parallel-refactor","attemptCount":2,"passedAttempts":2,"passRate":1,"averageDurationMs":59752,"averageJudgeScore":92.5,"averageTokenUsagePerAttempt":{"prompt":154274.5,"completion":2961,"total":157235.5}},{"id":"flow-test10-while-loop-counter","attemptCount":2,"passedAttempts":2,"passRate":1,"averageDurationMs":36922.5,"averageJudgeScore":97,"averageTokenUsagePerAttempt":{"prompt":120778,"completion":1121,"total":121899}},{"id":"flow-test11-preprocessor-and-failure-handler","attemptCount":2,"passedAttempts":2,"passRate":1,"averageDurationMs":44307.5,"averageJudgeScore":93.5,"averageTokenUsagePerAttempt":{"prompt":172195,"completion":1285,"total":173480}},{"id":"flow-test12-approval-step","attemptCount":2,"passedAttempts":2,"passRate":1,"averageDurationMs":35403.5,"averageJudgeScore":97,"averageTokenUsagePerAttempt":{"prompt":120431,"completion":1076,"total":121507}}]}
+{"createdAt":"2026-04-15T14:09:26.896Z","gitSha":"cc3e17dbc1c204b5d4e30ad449d59e9e7cd0bb89","mode":"flow","runs":2,"runModel":"googleai:gemini-3-flash-preview","judgeModel":"claude-sonnet-4-6","caseCount":13,"attemptCount":26,"passedAttempts":26,"passRate":1,"averageDurationMs":43444.88461538462,"averageJudgeScore":93.73076923076923,"averageTokenUsagePerAttempt":{"prompt":209953.38461538462,"completion":1267.2307692307693,"total":213042.65384615384},"failedCaseIds":[],"cases":[{"id":"flow-test0-sum-two-numbers","attemptCount":2,"passedAttempts":2,"passRate":1,"averageDurationMs":18405.5,"averageJudgeScore":97,"averageTokenUsagePerAttempt":{"prompt":106013.5,"completion":466,"total":106954.5}},{"id":"flow-test1-reuse-existing-script","attemptCount":2,"passedAttempts":2,"passRate":1,"averageDurationMs":18034.5,"averageJudgeScore":100,"averageTokenUsagePerAttempt":{"prompt":80428.5,"completion":524.5,"total":81372.5}},{"id":"flow-test2-call-existing-subflow","attemptCount":2,"passedAttempts":2,"passRate":1,"averageDurationMs":17393,"averageJudgeScore":100,"averageTokenUsagePerAttempt":{"prompt":80653,"completion":538,"total":81544.5}},{"id":"flow-test3-branchone-routing","attemptCount":2,"passedAttempts":2,"passRate":1,"averageDurationMs":28979,"averageJudgeScore":100,"averageTokenUsagePerAttempt":{"prompt":195088,"completion":1003.5,"total":196934.5}},{"id":"flow-test4-order-processing-loop","attemptCount":2,"passedAttempts":2,"passRate":1,"averageDurationMs":47315,"averageJudgeScore":87,"averageTokenUsagePerAttempt":{"prompt":264983,"completion":1909.5,"total":268753}},{"id":"flow-test5-parallel-data-pipeline","attemptCount":2,"passedAttempts":2,"passRate":1,"averageDurationMs":55034.5,"averageJudgeScore":96,"averageTokenUsagePerAttempt":{"prompt":315058,"completion":2118,"total":319111}},{"id":"flow-test6-ai-agent-tools","attemptCount":2,"passedAttempts":2,"passRate":1,"averageDurationMs":53794.5,"averageJudgeScore":88.5,"averageTokenUsagePerAttempt":{"prompt":278794,"completion":2275,"total":283175}},{"id":"flow-test7-simple-modification","attemptCount":2,"passedAttempts":2,"passRate":1,"averageDurationMs":43680,"averageJudgeScore":91,"averageTokenUsagePerAttempt":{"prompt":177988,"completion":963,"total":182547}},{"id":"flow-test8-branching-in-loop","attemptCount":2,"passedAttempts":2,"passRate":1,"averageDurationMs":65355.5,"averageJudgeScore":97,"averageTokenUsagePerAttempt":{"prompt":326580,"completion":1650,"total":331999}},{"id":"flow-test9-parallel-refactor","attemptCount":2,"passedAttempts":2,"passRate":1,"averageDurationMs":99143,"averageJudgeScore":82,"averageTokenUsagePerAttempt":{"prompt":495491,"completion":2535,"total":503137}},{"id":"flow-test10-while-loop-counter","attemptCount":2,"passedAttempts":2,"passRate":1,"averageDurationMs":33126.5,"averageJudgeScore":94.5,"averageTokenUsagePerAttempt":{"prompt":132736,"completion":820,"total":134766.5}},{"id":"flow-test11-preprocessor-and-failure-handler","attemptCount":2,"passedAttempts":2,"passRate":1,"averageDurationMs":52696.5,"averageJudgeScore":87,"averageTokenUsagePerAttempt":{"prompt":168158.5,"completion":1022.5,"total":170345.5}},{"id":"flow-test12-approval-step","attemptCount":2,"passedAttempts":2,"passRate":1,"averageDurationMs":31826,"averageJudgeScore":98.5,"averageTokenUsagePerAttempt":{"prompt":107422.5,"completion":649,"total":108914.5}}]}
+{"createdAt":"2026-04-16T01:02:51.812Z","gitSha":"cdc185556bed51323a2c726711d0d0bec34f0f42","mode":"flow","runs":1,"runModel":"anthropic:claude-haiku-4-5-20251001","judgeModel":"claude-sonnet-4-6","caseCount":15,"attemptCount":15,"passedAttempts":15,"passRate":1,"averageDurationMs":32733.066666666666,"averageJudgeScore":92.53333333333333,"averageTokenUsagePerAttempt":{"prompt":164523.86666666667,"completion":3247.8,"total":167771.66666666666},"failedCaseIds":[],"cases":[{"id":"flow-test0-sum-two-numbers","attemptCount":1,"passedAttempts":1,"passRate":1,"averageDurationMs":16668,"averageJudgeScore":98,"averageTokenUsagePerAttempt":{"prompt":117359,"completion":772,"total":118131}},{"id":"flow-test1-reuse-existing-script","attemptCount":1,"passedAttempts":1,"passRate":1,"averageDurationMs":13641,"averageJudgeScore":100,"averageTokenUsagePerAttempt":{"prompt":88826,"completion":694,"total":89520}},{"id":"flow-test2-call-existing-subflow","attemptCount":1,"passedAttempts":1,"passRate":1,"averageDurationMs":15970,"averageJudgeScore":100,"averageTokenUsagePerAttempt":{"prompt":89671,"completion":893,"total":90564}},{"id":"flow-test3-branchone-routing","attemptCount":1,"passedAttempts":1,"passRate":1,"averageDurationMs":19647,"averageJudgeScore":100,"averageTokenUsagePerAttempt":{"prompt":73131,"completion":1517,"total":74648}},{"id":"flow-test4-order-processing-loop","attemptCount":1,"passedAttempts":1,"passRate":1,"averageDurationMs":139058,"averageJudgeScore":95,"averageTokenUsagePerAttempt":{"prompt":715701,"completion":19959,"total":735660}},{"id":"flow-test5-parallel-data-pipeline","attemptCount":1,"passedAttempts":1,"passRate":1,"averageDurationMs":68666,"averageJudgeScore":87,"averageTokenUsagePerAttempt":{"prompt":256741,"completion":7646,"total":264387}},{"id":"flow-test6-ai-agent-tools","attemptCount":1,"passedAttempts":1,"passRate":1,"averageDurationMs":51789,"averageJudgeScore":90,"averageTokenUsagePerAttempt":{"prompt":189053,"completion":6419,"total":195472}},{"id":"flow-test7-simple-modification","attemptCount":1,"passedAttempts":1,"passRate":1,"averageDurationMs":32950,"averageJudgeScore":90,"averageTokenUsagePerAttempt":{"prompt":205558,"completion":2051,"total":207609}},{"id":"flow-test8-branching-in-loop","attemptCount":1,"passedAttempts":1,"passRate":1,"averageDurationMs":19320,"averageJudgeScore":90,"averageTokenUsagePerAttempt":{"prompt":90399,"completion":965,"total":91364}},{"id":"flow-test9-parallel-refactor","attemptCount":1,"passedAttempts":1,"passRate":1,"averageDurationMs":29521,"averageJudgeScore":82,"averageTokenUsagePerAttempt":{"prompt":120530,"completion":2938,"total":123468}},{"id":"flow-test10-while-loop-counter","attemptCount":1,"passedAttempts":1,"passRate":1,"averageDurationMs":21553,"averageJudgeScore":95,"averageTokenUsagePerAttempt":{"prompt":143648,"completion":1016,"total":144664}},{"id":"flow-test11-preprocessor-and-failure-handler","attemptCount":1,"passedAttempts":1,"passRate":1,"averageDurationMs":22783,"averageJudgeScore":82,"averageTokenUsagePerAttempt":{"prompt":144835,"completion":1400,"total":146235}},{"id":"flow-test12-approval-step","attemptCount":1,"passedAttempts":1,"passRate":1,"averageDurationMs":13423,"averageJudgeScore":97,"averageTokenUsagePerAttempt":{"prompt":71103,"completion":918,"total":72021}},{"id":"flow-test13-loop-resilience-toggle","attemptCount":1,"passedAttempts":1,"passRate":1,"averageDurationMs":7087,"averageJudgeScore":100,"averageTokenUsagePerAttempt":{"prompt":35384,"completion":505,"total":35889}},{"id":"flow-test14-modify-existing-special-modules","attemptCount":1,"passedAttempts":1,"passRate":1,"averageDurationMs":18920,"averageJudgeScore":82,"averageTokenUsagePerAttempt":{"prompt":125919,"completion":1024,"total":126943}}]}
+{"createdAt":"2026-04-16T01:05:41.621Z","gitSha":"cdc185556bed51323a2c726711d0d0bec34f0f42","mode":"flow","runs":1,"runModel":"anthropic:claude-opus-4-6","judgeModel":"claude-sonnet-4-6","caseCount":15,"attemptCount":15,"passedAttempts":15,"passRate":1,"averageDurationMs":47620.26666666667,"averageJudgeScore":95.13333333333334,"averageTokenUsagePerAttempt":{"prompt":128621.13333333333,"completion":2339,"total":130960.13333333333},"failedCaseIds":[],"cases":[{"id":"flow-test0-sum-two-numbers","attemptCount":1,"passedAttempts":1,"passRate":1,"averageDurationMs":24723,"averageJudgeScore":98,"averageTokenUsagePerAttempt":{"prompt":117007,"completion":559,"total":117566}},{"id":"flow-test1-reuse-existing-script","attemptCount":1,"passedAttempts":1,"passRate":1,"averageDurationMs":24774,"averageJudgeScore":100,"averageTokenUsagePerAttempt":{"prompt":88921,"completion":752,"total":89673}},{"id":"flow-test2-call-existing-subflow","attemptCount":1,"passedAttempts":1,"passRate":1,"averageDurationMs":24118,"averageJudgeScore":100,"averageTokenUsagePerAttempt":{"prompt":89537,"completion":767,"total":90304}},{"id":"flow-test3-branchone-routing","attemptCount":1,"passedAttempts":1,"passRate":1,"averageDurationMs":40090,"averageJudgeScore":100,"averageTokenUsagePerAttempt":{"prompt":146657,"completion":1736,"total":148393}},{"id":"flow-test4-order-processing-loop","attemptCount":1,"passedAttempts":1,"passRate":1,"averageDurationMs":78076,"averageJudgeScore":98,"averageTokenUsagePerAttempt":{"prompt":157763,"completion":4602,"total":162365}},{"id":"flow-test5-parallel-data-pipeline","attemptCount":1,"passedAttempts":1,"passRate":1,"averageDurationMs":93797,"averageJudgeScore":95,"averageTokenUsagePerAttempt":{"prompt":133791,"completion":6294,"total":140085}},{"id":"flow-test6-ai-agent-tools","attemptCount":1,"passedAttempts":1,"passRate":1,"averageDurationMs":130350,"averageJudgeScore":97,"averageTokenUsagePerAttempt":{"prompt":249710,"completion":9813,"total":259523}},{"id":"flow-test7-simple-modification","attemptCount":1,"passedAttempts":1,"passRate":1,"averageDurationMs":35901,"averageJudgeScore":95,"averageTokenUsagePerAttempt":{"prompt":121075,"completion":1166,"total":122241}},{"id":"flow-test8-branching-in-loop","attemptCount":1,"passedAttempts":1,"passRate":1,"averageDurationMs":39271,"averageJudgeScore":100,"averageTokenUsagePerAttempt":{"prompt":123150,"completion":1749,"total":124899}},{"id":"flow-test9-parallel-refactor","attemptCount":1,"passedAttempts":1,"passRate":1,"averageDurationMs":57122,"averageJudgeScore":82,"averageTokenUsagePerAttempt":{"prompt":148102,"completion":2846,"total":150948}},{"id":"flow-test10-while-loop-counter","attemptCount":1,"passedAttempts":1,"passRate":1,"averageDurationMs":39623,"averageJudgeScore":82,"averageTokenUsagePerAttempt":{"prompt":118709,"completion":1166,"total":119875}},{"id":"flow-test11-preprocessor-and-failure-handler","attemptCount":1,"passedAttempts":1,"passRate":1,"averageDurationMs":34068,"averageJudgeScore":98,"averageTokenUsagePerAttempt":{"prompt":118318,"completion":1045,"total":119363}},{"id":"flow-test12-approval-step","attemptCount":1,"passedAttempts":1,"passRate":1,"averageDurationMs":34094,"averageJudgeScore":97,"averageTokenUsagePerAttempt":{"prompt":118383,"completion":1073,"total":119456}},{"id":"flow-test13-loop-resilience-toggle","attemptCount":1,"passedAttempts":1,"passRate":1,"averageDurationMs":16041,"averageJudgeScore":100,"averageTokenUsagePerAttempt":{"prompt":53039,"completion":369,"total":53408}},{"id":"flow-test14-modify-existing-special-modules","attemptCount":1,"passedAttempts":1,"passRate":1,"averageDurationMs":42256,"averageJudgeScore":85,"averageTokenUsagePerAttempt":{"prompt":145155,"completion":1148,"total":146303}}]}
+{"createdAt":"2026-04-16T01:06:51.528Z","gitSha":"cdc185556bed51323a2c726711d0d0bec34f0f42","mode":"flow","runs":1,"runModel":"openai:gpt-4o","judgeModel":"claude-sonnet-4-6","caseCount":15,"attemptCount":15,"passedAttempts":12,"passRate":0.8,"averageDurationMs":18615.733333333334,"averageJudgeScore":84.26666666666667,"averageTokenUsagePerAttempt":{"prompt":67076,"completion":633.4666666666667,"total":67709.46666666666},"failedCaseIds":["flow-test6-ai-agent-tools","flow-test9-parallel-refactor","flow-test10-while-loop-counter"],"cases":[{"id":"flow-test0-sum-two-numbers","attemptCount":1,"passedAttempts":1,"passRate":1,"averageDurationMs":12800,"averageJudgeScore":98,"averageTokenUsagePerAttempt":{"prompt":72506,"completion":286,"total":72792}},{"id":"flow-test1-reuse-existing-script","attemptCount":1,"passedAttempts":1,"passRate":1,"averageDurationMs":11602,"averageJudgeScore":100,"averageTokenUsagePerAttempt":{"prompt":54967,"completion":244,"total":55211}},{"id":"flow-test2-call-existing-subflow","attemptCount":1,"passedAttempts":1,"passRate":1,"averageDurationMs":11120,"averageJudgeScore":98,"averageTokenUsagePerAttempt":{"prompt":55243,"completion":266,"total":55509}},{"id":"flow-test3-branchone-routing","attemptCount":1,"passedAttempts":1,"passRate":1,"averageDurationMs":16791,"averageJudgeScore":100,"averageTokenUsagePerAttempt":{"prompt":72966,"completion":485,"total":73451}},{"id":"flow-test4-order-processing-loop","attemptCount":1,"passedAttempts":1,"passRate":1,"averageDurationMs":29028,"averageJudgeScore":82,"averageTokenUsagePerAttempt":{"prompt":53858,"completion":1195,"total":55053}},{"id":"flow-test5-parallel-data-pipeline","attemptCount":1,"passedAttempts":1,"passRate":1,"averageDurationMs":35788,"averageJudgeScore":91,"averageTokenUsagePerAttempt":{"prompt":53990,"completion":1267,"total":55257}},{"id":"flow-test6-ai-agent-tools","attemptCount":1,"passedAttempts":0,"passRate":0,"averageDurationMs":38589,"averageJudgeScore":72,"averageTokenUsagePerAttempt":{"prompt":117435,"completion":2065,"total":119500}},{"id":"flow-test7-simple-modification","attemptCount":1,"passedAttempts":1,"passRate":1,"averageDurationMs":19563,"averageJudgeScore":97,"averageTokenUsagePerAttempt":{"prompt":87775,"completion":494,"total":88269}},{"id":"flow-test8-branching-in-loop","attemptCount":1,"passedAttempts":1,"passRate":1,"averageDurationMs":17346,"averageJudgeScore":95,"averageTokenUsagePerAttempt":{"prompt":108036,"completion":418,"total":108454}},{"id":"flow-test9-parallel-refactor","attemptCount":1,"passedAttempts":0,"passRate":0,"averageDurationMs":12960,"averageJudgeScore":5,"averageTokenUsagePerAttempt":{"prompt":13963,"completion":1065,"total":15028}},{"id":"flow-test10-while-loop-counter","attemptCount":1,"passedAttempts":0,"passRate":0,"averageDurationMs":15586,"averageJudgeScore":72,"averageTokenUsagePerAttempt":{"prompt":72867,"completion":374,"total":73241}},{"id":"flow-test11-preprocessor-and-failure-handler","attemptCount":1,"passedAttempts":1,"passRate":1,"averageDurationMs":18644,"averageJudgeScore":90,"averageTokenUsagePerAttempt":{"prompt":92741,"completion":473,"total":93214}},{"id":"flow-test12-approval-step","attemptCount":1,"passedAttempts":1,"passRate":1,"averageDurationMs":15050,"averageJudgeScore":82,"averageTokenUsagePerAttempt":{"prompt":41317,"completion":370,"total":41687}},{"id":"flow-test13-loop-resilience-toggle","attemptCount":1,"passedAttempts":1,"passRate":1,"averageDurationMs":12388,"averageJudgeScore":100,"averageTokenUsagePerAttempt":{"prompt":55238,"completion":133,"total":55371}},{"id":"flow-test14-modify-existing-special-modules","attemptCount":1,"passedAttempts":1,"passRate":1,"averageDurationMs":11981,"averageJudgeScore":82,"averageTokenUsagePerAttempt":{"prompt":53238,"completion":367,"total":53605}}]}
+{"createdAt":"2026-04-16T01:09:35.850Z","gitSha":"cdc185556bed51323a2c726711d0d0bec34f0f42","mode":"flow","runs":1,"runModel":"googleai:gemini-3-flash-preview","judgeModel":"claude-sonnet-4-6","caseCount":15,"attemptCount":15,"passedAttempts":15,"passRate":1,"averageDurationMs":42658.26666666667,"averageJudgeScore":96,"averageTokenUsagePerAttempt":{"prompt":181623.4,"completion":1251.0666666666666,"total":184458.6},"failedCaseIds":[],"cases":[{"id":"flow-test0-sum-two-numbers","attemptCount":1,"passedAttempts":1,"passRate":1,"averageDurationMs":15259,"averageJudgeScore":98,"averageTokenUsagePerAttempt":{"prompt":103856,"completion":394,"total":104816}},{"id":"flow-test1-reuse-existing-script","attemptCount":1,"passedAttempts":1,"passRate":1,"averageDurationMs":14271,"averageJudgeScore":100,"averageTokenUsagePerAttempt":{"prompt":78118,"completion":555,"total":79034}},{"id":"flow-test2-call-existing-subflow","attemptCount":1,"passedAttempts":1,"passRate":1,"averageDurationMs":15106,"averageJudgeScore":100,"averageTokenUsagePerAttempt":{"prompt":78732,"completion":519,"total":79650}},{"id":"flow-test3-branchone-routing","attemptCount":1,"passedAttempts":1,"passRate":1,"averageDurationMs":154431,"averageJudgeScore":100,"averageTokenUsagePerAttempt":{"prompt":201301,"completion":1242,"total":203738}},{"id":"flow-test4-order-processing-loop","attemptCount":1,"passedAttempts":1,"passRate":1,"averageDurationMs":97282,"averageJudgeScore":97,"averageTokenUsagePerAttempt":{"prompt":239645,"completion":2368,"total":244070}},{"id":"flow-test5-parallel-data-pipeline","attemptCount":1,"passedAttempts":1,"passRate":1,"averageDurationMs":47206,"averageJudgeScore":97,"averageTokenUsagePerAttempt":{"prompt":292614,"completion":2478,"total":297114}},{"id":"flow-test6-ai-agent-tools","attemptCount":1,"passedAttempts":1,"passRate":1,"averageDurationMs":46492,"averageJudgeScore":97,"averageTokenUsagePerAttempt":{"prompt":284592,"completion":2530,"total":288626}},{"id":"flow-test7-simple-modification","attemptCount":1,"passedAttempts":1,"passRate":1,"averageDurationMs":27878,"averageJudgeScore":97,"averageTokenUsagePerAttempt":{"prompt":112452,"completion":663,"total":115732}},{"id":"flow-test8-branching-in-loop","attemptCount":1,"passedAttempts":1,"passRate":1,"averageDurationMs":40458,"averageJudgeScore":98,"averageTokenUsagePerAttempt":{"prompt":258728,"completion":1905,"total":263106}},{"id":"flow-test9-parallel-refactor","attemptCount":1,"passedAttempts":1,"passRate":1,"averageDurationMs":60399,"averageJudgeScore":95,"averageTokenUsagePerAttempt":{"prompt":430591,"completion":2620,"total":436961}},{"id":"flow-test10-while-loop-counter","attemptCount":1,"passedAttempts":1,"passRate":1,"averageDurationMs":25597,"averageJudgeScore":90,"averageTokenUsagePerAttempt":{"prompt":132065,"completion":858,"total":134550}},{"id":"flow-test11-preprocessor-and-failure-handler","attemptCount":1,"passedAttempts":1,"passRate":1,"averageDurationMs":29488,"averageJudgeScore":92,"averageTokenUsagePerAttempt":{"prompt":177880,"completion":789,"total":180269}},{"id":"flow-test12-approval-step","attemptCount":1,"passedAttempts":1,"passRate":1,"averageDurationMs":19587,"averageJudgeScore":97,"averageTokenUsagePerAttempt":{"prompt":105921,"completion":861,"total":107605}},{"id":"flow-test13-loop-resilience-toggle","attemptCount":1,"passedAttempts":1,"passRate":1,"averageDurationMs":10940,"averageJudgeScore":100,"averageTokenUsagePerAttempt":{"prompt":47133,"completion":181,"total":47914}},{"id":"flow-test14-modify-existing-special-modules","attemptCount":1,"passedAttempts":1,"passRate":1,"averageDurationMs":35480,"averageJudgeScore":82,"averageTokenUsagePerAttempt":{"prompt":180723,"completion":803,"total":183694}}]}

ai_evals/modes/app.ts

@@ -1,32 +1,47 @@
 import { loadAppFixture } from "../adapters/frontend/core/app/appFixtureLoader";
-import type { AppFiles } from "../../frontend/src/lib/components/copilot/chat/app/core";
+import { buildAppArtifacts } from "../core/appArtifacts";
+import type { AppValidationSpec } from "../core/types";
 import type { FrontendEvalModelConfig } from "../core/models";
 import { validateAppState, type AppFilesState } from "../core/validators";
 import type { BenchmarkArtifactFile, ModeRunner } from "../core/types";
 import { runAppEval } from "../adapters/frontend/core/app/appEvalRunner";
-import { DEFAULT_FRONTEND_EVAL_MODEL, getFrontendApiKey } from "./frontendCommon";
+import {
+  DEFAULT_FRONTEND_EVAL_MODEL,
+  getFrontendApiKey,
+} from "./frontendCommon";
+import type { FrontendEvalTransportSettings } from "../core/frontendTransport";
 
 export function createAppModeRunner(
-  modelConfig: FrontendEvalModelConfig = DEFAULT_FRONTEND_EVAL_MODEL
+  modelConfig: FrontendEvalModelConfig = DEFAULT_FRONTEND_EVAL_MODEL,
+  transportSettings?: FrontendEvalTransportSettings,
 ): ModeRunner<AppFilesState, AppFilesState, AppFilesState> {
   return {
     mode: "app",
     concurrency: 5,
     judgeThreshold: 80,
     async loadInitial(path) {
-      return path ? (await loadAppFixture(path)) : undefined;
+      return path ? await loadAppFixture(path) : undefined;
     },
     async loadExpected(path) {
-      return path ? (await loadAppFixture(path)) : undefined;
+      return path ? await loadAppFixture(path) : undefined;
     },
     async run(prompt, initial, context) {
-      const result = await runAppEval(prompt, getFrontendApiKey(modelConfig.provider), {
-        initialFrontend: initial?.frontend,
-        initialBackend: initial?.backend as AppFiles["backend"] | undefined,
-        provider: modelConfig.provider,
-        model: modelConfig.model,
-        runContext: context,
-      });
+      const result = await runAppEval(
+        prompt,
+        getFrontendApiKey(modelConfig.provider),
+        {
+          initialFrontend: initial?.frontend,
+          initialBackend: initial?.backend,
+          initialDatatables: initial?.datatables,
+          maxIterations: context.evalCase?.runtime?.maxTurns,
+          appContext: context.evalCase?.runtime?.appContext,
+          provider: modelConfig.provider,
+          model: modelConfig.model,
+          transport: transportSettings?.transport,
+          backend: transportSettings?.backend,
+          runContext: context,
+        },
+      );
 
       return {
         success: result.success,
@@ -39,41 +54,17 @@ export function createAppModeRunner(
         tokenUsage: result.tokenUsage,
       };
     },
-    validate({ actual, initial, expected }) {
-      return validateAppState({ actual, initial, expected });
+    validate({ evalCase, actual, initial, expected, run }) {
+      return validateAppState({
+        actual,
+        initial,
+        expected,
+        validate: evalCase.validate as AppValidationSpec | undefined,
+        toolsUsed: run.toolsUsed,
+      });
     },
     buildArtifacts(actual): BenchmarkArtifactFile[] {
-      const artifacts: BenchmarkArtifactFile[] = [
-        {
-          path: "app.json",
-          content: JSON.stringify(actual, null, 2) + "\n",
-        },
-      ];
-
-      for (const [filePath, content] of Object.entries(actual.frontend)) {
-        artifacts.push({
-          path: `frontend${filePath.startsWith("/") ? filePath : `/${filePath}`}`,
-          content,
-        });
-      }
-
-      for (const [key, runnable] of Object.entries(actual.backend)) {
-        artifacts.push({
-          path: `backend/${key}/meta.json`,
-          content: JSON.stringify(runnable, null, 2) + "\n",
-        });
-
-        const inlineContent = runnable.inlineScript?.content;
-        if (inlineContent) {
-          const extension = runnable.inlineScript?.language === "python3" ? "py" : "ts";
-          artifacts.push({
-            path: `backend/${key}/main.${extension}`,
-            content: inlineContent,
-          });
-        }
-      }
-
-      return artifacts;
+      return buildAppArtifacts(actual);
     },
   };
 }

ai_evals/modes/cli.ts

@@ -13,9 +13,16 @@ import {
 } from "../adapters/cli/runtime";
 import { copyDirectory, readDirectoryFiles } from "../core/files";
 import { validateCliWorkspace } from "../core/validators";
-import type { BenchmarkArtifactFile, ModeRunner } from "../core/types";
+import type { BenchmarkArtifactFile, CliTrace, ModeRunner } from "../core/types";
 
-const IGNORE_WORKSPACE_FILES = new Set([".claude", "AGENTS.md", "CLAUDE.md", "rt.d.ts"]);
+const IGNORE_WORKSPACE_FILES = new Set([
+  ".claude",
+  "AGENTS.md",
+  "CLAUDE.md",
+  "rt.d.ts",
+  ".wmill-benchmark-bin",
+  ".wmill-benchmark-wmill-invocations.log",
+]);
 
 interface CliWorkspaceFixture {
   sourceDir: string;
@@ -25,6 +32,7 @@ interface CliWorkspaceFixture {
 interface CliRunActual {
   assistantOutput: string;
   workspaceFiles: Record<string, string>;
+  trace: CliTrace;
 }
 
 const CLAUDE_PROJECT_PREAMBLE = [
@@ -62,7 +70,7 @@ export function createCliModeRunner(
           }
         : undefined;
     },
-    async run(prompt, initial, _context) {
+    async run(prompt, initial, context) {
       const workspaceDir = await mkdtemp(join(tmpdir(), "wmill-cli-benchmark-"));
 
       try {
@@ -78,7 +86,12 @@ export function createCliModeRunner(
         await writeFile(join(workspaceDir, "rt.d.ts"), "export namespace RT {}\n", "utf8");
 
         const renderedPrompt = await renderPrompt(prompt, workspaceDir);
-        const run = await runPromptAndCapture(renderedPrompt, workspaceDir, 6, modelConfig);
+        const run = await runPromptAndCapture(
+          renderedPrompt,
+          workspaceDir,
+          context.evalCase?.runtime?.maxTurns ?? 6,
+          modelConfig
+        );
         const workspaceFiles = await readDirectoryFiles(workspaceDir, { ignore: IGNORE_WORKSPACE_FILES });
 
         return {
@@ -86,11 +99,12 @@ export function createCliModeRunner(
           actual: {
             assistantOutput: run.output,
             workspaceFiles,
+            trace: run.trace,
           },
-          assistantMessageCount: run.assistantMessageCount,
-          toolCallCount: run.toolsUsed.length,
-          toolsUsed: run.toolsUsed.map((entry) => entry.tool),
-          skillsInvoked: run.skillsInvoked,
+          assistantMessageCount: run.trace.assistantMessageCount,
+          toolCallCount: run.trace.toolsUsed.length,
+          toolsUsed: run.trace.toolsUsed.map((entry) => entry.tool),
+          skillsInvoked: run.trace.skillsInvoked,
           tokenUsage: run.tokenUsage ?? null,
         };
       } catch (error) {
@@ -100,6 +114,7 @@ export function createCliModeRunner(
           actual: {
             assistantOutput: "",
             workspaceFiles: {},
+            trace: emptyCliTrace(),
           },
           error: message,
           assistantMessageCount: 0,
@@ -112,11 +127,14 @@ export function createCliModeRunner(
         await rm(workspaceDir, { recursive: true, force: true });
       }
     },
-    validate({ actual, initial, expected }) {
+    validate({ evalCase, actual, initial, expected }) {
       return validateCliWorkspace({
         actualFiles: actual.workspaceFiles,
         expectedFiles: expected?.files,
         initialFiles: initial?.files,
+        assistantOutput: actual.assistantOutput,
+        trace: actual.trace,
+        cliExpect: evalCase.cliExpect,
       });
     },
     buildArtifacts(actual): BenchmarkArtifactFile[] {
@@ -125,6 +143,17 @@ export function createCliModeRunner(
           path: "assistant-output.txt",
           content: `${actual.assistantOutput}\n`,
         },
+        {
+          path: "trace.json",
+          content: JSON.stringify(actual.trace, null, 2) + "\n",
+        },
+        {
+          path: "wmill-invocations.jsonl",
+          content:
+            actual.trace.wmillInvocations
+              .map((entry) => JSON.stringify(entry))
+              .join("\n") + (actual.trace.wmillInvocations.length > 0 ? "\n" : ""),
+        },
       ];
 
       for (const [filePath, content] of Object.entries(actual.workspaceFiles)) {
@@ -139,6 +168,19 @@ export function createCliModeRunner(
   };
 }
 
+function emptyCliTrace(): CliTrace {
+  return {
+    toolsUsed: [],
+    skillsInvoked: [],
+    assistantMessageCount: 0,
+    bashCommands: [],
+    proposedCommands: [],
+    executedWmillCommands: [],
+    wmillInvocations: [],
+    firstMutationToolIndex: null,
+  };
+}
+
 export function getCliRunModelLabel(
   modelConfig: CliEvalModelConfig = DEFAULT_CLI_EVAL_MODEL
 ): string {

ai_evals/modes/flow.ts

@@ -1,24 +1,27 @@
 import { readJsonFile } from "../core/files";
 import type { BackendValidationSettings } from "../core/backendValidation";
 import type { FrontendEvalModelConfig } from "../core/models";
+import type { FlowValidationSpec } from "../core/types";
 import { validateFlowState, type FlowState } from "../core/validators";
 import type { BenchmarkArtifactFile, ModeRunner } from "../core/types";
-import {
-  runFlowEval,
-  type FlowFixture,
-} from "../adapters/frontend/core/flow/flowEvalRunner";
+import { runFlowEval } from "../adapters/frontend/core/flow/flowEvalRunner";
 import type { FlowWorkspaceFixtures } from "../adapters/frontend/core/flow/fileHelpers";
 import { BackendPreviewClient } from "../adapters/frontend/backendPreview";
-import { DEFAULT_FRONTEND_EVAL_MODEL, getFrontendApiKey } from "./frontendCommon";
-
-interface FlowInitialFixture {
-  flow?: FlowFixture;
-  workspace?: FlowWorkspaceFixtures;
-}
+import {
+  DEFAULT_FRONTEND_EVAL_MODEL,
+  getFrontendApiKey,
+} from "./frontendCommon";
+import type { FrontendEvalTransportSettings } from "../core/frontendTransport";
+import {
+  normalizeFlowInitialFixture,
+  normalizeFlowStateFixture,
+  type FlowInitialFixture,
+} from "./flowFixtures";
 
 export function createFlowModeRunner(
   modelConfig: FrontendEvalModelConfig = DEFAULT_FRONTEND_EVAL_MODEL,
-  backendValidation?: BackendValidationSettings
+  backendValidation?: BackendValidationSettings,
+  transportSettings?: FrontendEvalTransportSettings,
 ): ModeRunner<FlowInitialFixture, FlowState, FlowState> {
   return {
     mode: "flow",
@@ -37,13 +40,20 @@ export function createFlowModeRunner(
       return normalizeFlowStateFixture(await readJsonFile<unknown>(path));
     },
     async run(prompt, initial, context) {
-      const result = await runFlowEval(prompt, getFrontendApiKey(modelConfig.provider), {
-        initialFlow: initial?.flow,
-        workspaceFixtures: initial?.workspace,
-        provider: modelConfig.provider,
-        model: modelConfig.model,
-        runContext: context,
-      });
+      const result = await runFlowEval(
+        prompt,
+        getFrontendApiKey(modelConfig.provider),
+        {
+          initialFlow: initial?.flowFixture,
+          workspaceFixtures: initial?.workspace,
+          maxIterations: context.evalCase?.runtime?.maxTurns,
+          provider: modelConfig.provider,
+          model: modelConfig.model,
+          transport: transportSettings?.transport,
+          backend: transportSettings?.backend,
+          runContext: context,
+        },
+      );
 
       return {
         success: result.success,
@@ -52,6 +62,7 @@ export function createFlowModeRunner(
         assistantMessageCount: result.assistantMessageCount,
         toolCallCount: result.toolCallCount,
         toolsUsed: result.toolsUsed,
+        toolCallDetails: result.toolCallDetails,
         skillsInvoked: [],
         tokenUsage: result.tokenUsage,
       };
@@ -59,13 +70,16 @@ export function createFlowModeRunner(
     validate({ evalCase, actual, initial, expected }) {
       return validateFlowState({
         actual,
-        initial: initial?.flow,
+        initial: initial?.flowState,
         expected,
-        validate: evalCase.validate,
+        validate: evalCase.validate as FlowValidationSpec | undefined,
       });
     },
     async backendValidate({ evalCase, initial, actual, context }) {
-      if (backendValidation?.mode !== "preview" || !evalCase.runtime?.backendPreview) {
+      if (
+        backendValidation?.mode !== "preview" ||
+        !evalCase.runtime?.backendPreview
+      ) {
         return null;
       }
 
@@ -82,46 +96,54 @@ export function createFlowModeRunner(
       }
 
       const previewClient = new BackendPreviewClient(backendValidation);
-      return await previewClient.withWorkspace(evalCase.id, context.attempt, async (workspaceId) => {
-        await seedWorkspaceFixtures(previewClient, workspaceId, initial?.workspace);
+      return await previewClient.withWorkspace(
+        evalCase.id,
+        context.attempt,
+        async (workspaceId) => {
+          await seedWorkspaceFixtures(
+            previewClient,
+            workspaceId,
+            initial?.workspace,
+          );
 
-        const completedJob = await previewClient.runFlowPreview({
-          workspaceId,
-          value: actual.value as Record<string, unknown>,
-          args: evalCase.runtime?.backendPreview?.args ?? {},
-          timeoutSeconds: evalCase.runtime?.backendPreview?.timeoutSeconds,
-        });
+          const completedJob = await previewClient.runFlowPreview({
+            workspaceId,
+            value: actual.value as Record<string, unknown>,
+            args: evalCase.runtime?.backendPreview?.args ?? {},
+            timeoutSeconds: evalCase.runtime?.backendPreview?.timeoutSeconds,
+          });
 
-        return {
-          checks: [
-            {
-              name: "backend flow preview succeeded",
-              passed: completedJob.success,
-              details: completedJob.success
-                ? `workspace=${workspaceId}`
-                : `workspace=${workspaceId}; job=${completedJob.id}`,
-            },
-          ],
-          artifactFiles: [
-            {
-              path: "backend-preview.json",
-              content:
-                JSON.stringify(
-                  {
-                    workspaceId,
-                    jobId: completedJob.id,
-                    success: completedJob.success,
-                    result: completedJob.result,
-                    logs: completedJob.logs,
-                    completedJob: completedJob.raw,
-                  },
-                  null,
-                  2
-                ) + "\n",
-            },
-          ],
-        };
-      });
+          return {
+            checks: [
+              {
+                name: "backend flow preview succeeded",
+                passed: completedJob.success,
+                details: completedJob.success
+                  ? `workspace=${workspaceId}`
+                  : `workspace=${workspaceId}; job=${completedJob.id}`,
+              },
+            ],
+            artifactFiles: [
+              {
+                path: "backend-preview.json",
+                content:
+                  JSON.stringify(
+                    {
+                      workspaceId,
+                      jobId: completedJob.id,
+                      success: completedJob.success,
+                      result: completedJob.result,
+                      logs: completedJob.logs,
+                      completedJob: completedJob.raw,
+                    },
+                    null,
+                    2,
+                  ) + "\n",
+              },
+            ],
+          };
+        },
+      );
     },
     buildArtifacts(actual): BenchmarkArtifactFile[] {
       return [
@@ -134,41 +156,10 @@ export function createFlowModeRunner(
   };
 }
 
-function normalizeFlowInitialFixture(value: unknown): FlowInitialFixture {
-  if (isObject(value) && ("flow" in value || "workspace" in value)) {
-    const fixture = value as {
-      flow?: FlowFixture;
-      workspace?: FlowWorkspaceFixtures;
-    };
-    return {
-      flow: fixture.flow,
-      workspace: fixture.workspace,
-    };
-  }
-
-  return {
-    flow: normalizeFlowStateFixture(value),
-  };
-}
-
-function normalizeFlowStateFixture(value: unknown): FlowState {
-  if (!isObject(value)) {
-    return {};
-  }
-  if ("flow" in value && isObject((value as { flow?: unknown }).flow)) {
-    return (value as { flow: FlowState }).flow;
-  }
-  return value as FlowState;
-}
-
-function isObject(value: unknown): value is Record<string, unknown> {
-  return typeof value === "object" && value !== null && !Array.isArray(value);
-}
-
 async function seedWorkspaceFixtures(
   previewClient: BackendPreviewClient,
   workspaceId: string,
-  fixtures?: FlowWorkspaceFixtures
+  fixtures?: FlowWorkspaceFixtures,
 ): Promise<void> {
   for (const script of fixtures?.scripts ?? []) {
     await previewClient.createScript({

ai_evals/modes/flowFixtures.test.ts

@@ -0,0 +1,40 @@
+import { describe, expect, it } from "bun:test";
+import { validateFlowState } from "../core/validators";
+import { normalizeFlowInitialFixture } from "./flowFixtures";
+
+describe("normalizeFlowInitialFixture", () => {
+  it("preserves initial flow metadata for validation", () => {
+    const initialFlow = {
+      summary: "existing summary",
+      value: {
+        modules: [
+          {
+            id: "a",
+            value: {
+              type: "rawscript",
+              language: "bun",
+              content: "export async function main() { return 1; }",
+              input_transforms: {},
+            },
+          },
+        ],
+      },
+      schema: {
+        $schema: "https://json-schema.org/draft/2020-12/schema",
+        properties: {},
+        required: [],
+        type: "object",
+      },
+    };
+
+    const initial = normalizeFlowInitialFixture(initialFlow);
+    const checks = validateFlowState({
+      actual: structuredClone(initialFlow),
+      initial: initial.flowState,
+    });
+
+    const differsFromInitial = checks.find((check) => check.name === "flow differs from initial");
+    expect(initial.flowState?.summary).toBe("existing summary");
+    expect(differsFromInitial?.passed).toBe(false);
+  });
+});

ai_evals/modes/flowFixtures.ts

@@ -0,0 +1,59 @@
+import type { FlowModule } from "../../frontend/src/lib/gen";
+import type { FlowFixture } from "../adapters/frontend/core/flow/flowEvalRunner";
+import type { FlowWorkspaceFixtures } from "../adapters/frontend/core/flow/fileHelpers";
+import type { FlowState } from "../core/validators";
+
+export interface FlowInitialFixture {
+  flowFixture?: FlowFixture;
+  flowState?: FlowState;
+  workspace?: FlowWorkspaceFixtures;
+}
+
+export function normalizeFlowInitialFixture(value: unknown): FlowInitialFixture {
+  if (isObject(value) && ("flow" in value || "workspace" in value)) {
+    const fixture = value as {
+      flow?: unknown;
+      workspace?: FlowWorkspaceFixtures;
+    };
+    const flowState = fixture.flow ? normalizeFlowStateFixture(fixture.flow) : undefined;
+    return {
+      flowState,
+      flowFixture: flowState ? normalizeFlowFixture(flowState) : undefined,
+      workspace: fixture.workspace,
+    };
+  }
+
+  const flowState = normalizeFlowStateFixture(value);
+  return {
+    flowState,
+    flowFixture: normalizeFlowFixture(flowState),
+  };
+}
+
+export function normalizeFlowStateFixture(value: unknown): FlowState {
+  if (!isObject(value)) {
+    return {};
+  }
+  if ("flow" in value && isObject((value as { flow?: unknown }).flow)) {
+    return (value as { flow: FlowState }).flow;
+  }
+  return value as FlowState;
+}
+
+export function normalizeFlowFixture(value: FlowState): FlowFixture {
+  return {
+    path: value.path,
+    schema: value.schema,
+    value: value.value
+      ? {
+          modules: value.value.modules as FlowModule[] | undefined,
+          preprocessor_module: value.value.preprocessor_module as FlowModule | undefined,
+          failure_module: value.value.failure_module as FlowModule | undefined,
+        }
+      : undefined,
+  };
+}
+
+function isObject(value: unknown): value is Record<string, unknown> {
+  return typeof value === "object" && value !== null && !Array.isArray(value);
+}

ai_evals/modes/script.ts

@@ -6,11 +6,16 @@ import type { BenchmarkArtifactFile, ModeRunner } from "../core/types";
 import { BackendPreviewClient } from "../adapters/frontend/backendPreview";
 import { runScriptEval } from "../adapters/frontend/core/script/scriptEvalRunner";
 import type { ScriptEvalState } from "../adapters/frontend/core/script/fileHelpers";
-import { DEFAULT_FRONTEND_EVAL_MODEL, getFrontendApiKey } from "./frontendCommon";
+import {
+  DEFAULT_FRONTEND_EVAL_MODEL,
+  getFrontendApiKey,
+} from "./frontendCommon";
+import type { FrontendEvalTransportSettings } from "../core/frontendTransport";
 
 export function createScriptModeRunner(
   modelConfig: FrontendEvalModelConfig = DEFAULT_FRONTEND_EVAL_MODEL,
-  backendValidation?: BackendValidationSettings
+  backendValidation?: BackendValidationSettings,
+  transportSettings?: FrontendEvalTransportSettings,
 ): ModeRunner<ScriptEvalState, ScriptEvalState, ScriptEvalState> {
   return {
     mode: "script",
@@ -27,12 +32,19 @@ export function createScriptModeRunner(
         throw new Error("Script evals require an initial script fixture");
       }
 
-      const result = await runScriptEval(prompt, getFrontendApiKey(modelConfig.provider), {
-        initialScript: initial,
-        provider: modelConfig.provider,
-        model: modelConfig.model,
-        runContext: context,
-      });
+      const result = await runScriptEval(
+        prompt,
+        getFrontendApiKey(modelConfig.provider),
+        {
+          initialScript: initial,
+          maxIterations: context.evalCase?.runtime?.maxTurns,
+          provider: modelConfig.provider,
+          model: modelConfig.model,
+          transport: transportSettings?.transport,
+          backend: transportSettings?.backend,
+          runContext: context,
+        },
+      );
 
       return {
         success: result.success,
@@ -41,6 +53,7 @@ export function createScriptModeRunner(
         assistantMessageCount: result.assistantMessageCount,
         toolCallCount: result.toolCallCount,
         toolsUsed: result.toolsUsed,
+        toolCallDetails: result.toolCallDetails,
         skillsInvoked: [],
         tokenUsage: result.tokenUsage,
       };
@@ -54,50 +67,56 @@ export function createScriptModeRunner(
       }
 
       const previewClient = new BackendPreviewClient(backendValidation);
-      return await previewClient.withWorkspace(evalCase.id, context.attempt, async (workspaceId) => {
-        const completedJob = await previewClient.runScriptPreview({
-          workspaceId,
-          content: actual.code,
-          args:
-            (evalCase.runtime?.backendPreview?.args as Record<string, unknown> | undefined) ??
-            actual.args ??
-            initial?.args ??
-            {},
-          language: normalizePreviewLanguage(actual.lang),
-          path: toPreviewScriptPath(actual.path),
-          timeoutSeconds: evalCase.runtime?.backendPreview?.timeoutSeconds,
-        });
+      return await previewClient.withWorkspace(
+        evalCase.id,
+        context.attempt,
+        async (workspaceId) => {
+          const completedJob = await previewClient.runScriptPreview({
+            workspaceId,
+            content: actual.code,
+            args:
+              (evalCase.runtime?.backendPreview?.args as
+                | Record<string, unknown>
+                | undefined) ??
+              actual.args ??
+              initial?.args ??
+              {},
+            language: normalizePreviewLanguage(actual.lang),
+            path: toPreviewScriptPath(actual.path),
+            timeoutSeconds: evalCase.runtime?.backendPreview?.timeoutSeconds,
+          });
 
-        return {
-          checks: [
-            {
-              name: "backend script preview succeeded",
-              passed: completedJob.success,
-              details: completedJob.success
-                ? `workspace=${workspaceId}`
-                : `workspace=${workspaceId}; job=${completedJob.id}`,
-            },
-          ],
-          artifactFiles: [
-            {
-              path: "backend-preview.json",
-              content:
-                JSON.stringify(
-                  {
-                    workspaceId,
-                    jobId: completedJob.id,
-                    success: completedJob.success,
-                    result: completedJob.result,
-                    logs: completedJob.logs,
-                    completedJob: completedJob.raw,
-                  },
-                  null,
-                  2
-                ) + "\n",
-            },
-          ],
-        };
-      });
+          return {
+            checks: [
+              {
+                name: "backend script preview succeeded",
+                passed: completedJob.success,
+                details: completedJob.success
+                  ? `workspace=${workspaceId}`
+                  : `workspace=${workspaceId}; job=${completedJob.id}`,
+              },
+            ],
+            artifactFiles: [
+              {
+                path: "backend-preview.json",
+                content:
+                  JSON.stringify(
+                    {
+                      workspaceId,
+                      jobId: completedJob.id,
+                      success: completedJob.success,
+                      result: completedJob.result,
+                      logs: completedJob.logs,
+                      completedJob: completedJob.raw,
+                    },
+                    null,
+                    2,
+                  ) + "\n",
+              },
+            ],
+          };
+        },
+      );
     },
     buildArtifacts(actual): BenchmarkArtifactFile[] {
       return [

ai_evals/package.json

@@ -3,7 +3,8 @@
   "private": true,
   "type": "module",
   "scripts": {
-    "cli": "bun cli/index.ts"
+    "cli": "bun cli/index.ts",
+    "typecheck": "tsc -p tsconfig.json"
   },
   "dependencies": {
     "@anthropic-ai/claude-agent-sdk": "^0.2.25",

ai_evals/tsconfig.json

@@ -0,0 +1,19 @@
+{
+  "extends": "../frontend/tsconfig.json",
+  "compilerOptions": {
+    "noEmit": true,
+    "types": [],
+    "allowImportingTsExtensions": true
+  },
+  "include": [
+    "./adapters/**/*.ts",
+    "./cli/**/*.ts",
+    "./core/**/*.ts",
+    "./modes/**/*.ts",
+    "../frontend/src/app.d.ts"
+  ],
+  "exclude": [
+    "./**/*.test.ts",
+    "./adapters/frontend/vitest.config.ts"
+  ]
+}

backend/.sqlx/query-0010ef26da16facd1c2c832601ac687c4c27de46a90f45496b8446af1a9d0578.json

@@ -0,0 +1,45 @@
+{
+  "db_name": "PostgreSQL",
+  "query": "\n        SELECT\n            oauth_data as \"oauth_data: sqlx::types::Json<WorkspaceOAuthConfig>\",\n            service_name as \"service_name!: ServiceName\",\n            resource_path\n        FROM\n            workspace_integrations\n        WHERE\n            workspace_id = $1\n        ",
+  "describe": {
+    "columns": [
+      {
+        "ordinal": 0,
+        "name": "oauth_data: sqlx::types::Json<WorkspaceOAuthConfig>",
+        "type_info": "Jsonb"
+      },
+      {
+        "ordinal": 1,
+        "name": "service_name!: ServiceName",
+        "type_info": {
+          "Custom": {
+            "name": "native_trigger_service",
+            "kind": {
+              "Enum": [
+                "nextcloud",
+                "google",
+                "github"
+              ]
+            }
+          }
+        }
+      },
+      {
+        "ordinal": 2,
+        "name": "resource_path",
+        "type_info": "Text"
+      }
+    ],
+    "parameters": {
+      "Left": [
+        "Text"
+      ]
+    },
+    "nullable": [
+      true,
+      false,
+      true
+    ]
+  },
+  "hash": "0010ef26da16facd1c2c832601ac687c4c27de46a90f45496b8446af1a9d0578"
+}

backend/.sqlx/query-002d68d7c4437522a6dae95af007a356217bbae06b8453f0c32046f0cbf20dcb.json

@@ -0,0 +1,23 @@
+{
+  "db_name": "PostgreSQL",
+  "query": "SELECT created_by FROM v2_job WHERE id = $1 AND workspace_id = $2",
+  "describe": {
+    "columns": [
+      {
+        "ordinal": 0,
+        "name": "created_by",
+        "type_info": "Varchar"
+      }
+    ],
+    "parameters": {
+      "Left": [
+        "Uuid",
+        "Text"
+      ]
+    },
+    "nullable": [
+      false
+    ]
+  },
+  "hash": "002d68d7c4437522a6dae95af007a356217bbae06b8453f0c32046f0cbf20dcb"
+}

backend/.sqlx/query-00588a40dde5189ac1c61505f17acb0f4c244c60477427505bf5bd1b104d3bf9.json

@@ -0,0 +1,14 @@
+{
+  "db_name": "PostgreSQL",
+  "query": "UPDATE alerts SET acknowledged_workspace = true, acknowledged = true WHERE workspace_id = $1",
+  "describe": {
+    "columns": [],
+    "parameters": {
+      "Left": [
+        "Text"
+      ]
+    },
+    "nullable": []
+  },
+  "hash": "00588a40dde5189ac1c61505f17acb0f4c244c60477427505bf5bd1b104d3bf9"
+}

backend/.sqlx/query-005b9255699e73600c579f74b529caf531b2312b6e405b4d35efd2f7ca663143.json

@@ -0,0 +1,22 @@
+{
+  "db_name": "PostgreSQL",
+  "query": "SELECT email FROM password WHERE email = $1",
+  "describe": {
+    "columns": [
+      {
+        "ordinal": 0,
+        "name": "email",
+        "type_info": "Varchar"
+      }
+    ],
+    "parameters": {
+      "Left": [
+        "Text"
+      ]
+    },
+    "nullable": [
+      false
+    ]
+  },
+  "hash": "005b9255699e73600c579f74b529caf531b2312b6e405b4d35efd2f7ca663143"
+}

backend/.sqlx/query-006f03e979abdf8055b1c598bc9806337216a6abf74db4eb64b0acb918a0de08.json

@@ -0,0 +1,16 @@
+{
+  "db_name": "PostgreSQL",
+  "query": "UPDATE usr SET disabled = $1 WHERE username = $2 AND workspace_id = $3",
+  "describe": {
+    "columns": [],
+    "parameters": {
+      "Left": [
+        "Bool",
+        "Text",
+        "Text"
+      ]
+    },
+    "nullable": []
+  },
+  "hash": "006f03e979abdf8055b1c598bc9806337216a6abf74db4eb64b0acb918a0de08"
+}