docs: align global ai with user draft model

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

.agents/skills/pr/SKILL.md

@@ -55,55 +55,18 @@ The body MUST be explicit about what changed. Structure:
 
 The harness/tooling that invoked the skill may add its own attribution trailer; the skill itself does not prescribe one.
 
-## Screenshots (required for frontend changes)
-
-If `git diff main...HEAD --name-only` matches `^frontend/`, the PR body **must** include
-screenshots of the affected UI. Skip only when there is no visible UI effect (types,
-tests, build config) — and say so in the body.
-
-1. Verify the change in the browser (AGENTS.md → "Verifying Frontend Changes").
-2. Screenshot each affected page with `mcp__playwright__browser_take_screenshot` (save to a file).
-3. Host each image and get its Markdown embed by pushing to the public
-   `windmill-labs/agent-screenshots-internal` repo. **Pipe base64 through stdin** —
-   passing it as `-f content=…` fails with `argument list too long` on real images:
-
-   ```bash
-   REPO=windmill-labs/agent-screenshots-internal
-   IMG=screenshot.png                                          # repeat per page
-   DEST="shots/$(git branch --show-current)/$(date +%s)-$(basename "$IMG")"
-   base64 -w0 "$IMG" | jq -Rs --arg m "add $DEST" '{message:$m, content:.}' \
-     | gh api -X PUT "repos/$REPO/contents/$DEST" --input - >/dev/null
-   echo "![$(basename "$IMG" .png)](https://raw.githubusercontent.com/$REPO/main/$DEST)"
-   ```
-   Derive `$DEST` from the file name (as above) so distinct pages never collide — a
-   fixed name would make same-second uploads reuse one path, and the second `PUT`
-   then 422s (the Contents API needs the existing file's `sha` to overwrite).
-4. Put the printed `![…](…)` lines under a `## Screenshots` heading in the PR body.
-
-Requires `gh` (`repo` scope), `jq`, `base64` — all in the devShell. The host repo is
-public (so the raw URLs render for reviewers without a token) and its history is
-permanent — **never screenshot pages that show secrets or sensitive values** (workspace
-variables, resource values, instance settings, OAuth/SMTP config); deleting the file
-can't undo an accidental capture. (GitHub's drag-and-drop uploader needs a browser
-session and can't be driven from a token.)
-
-If `gh` can't push to the host repo (e.g. a CI token scoped only to `windmill`), do
-**not** fail the PR or skip silently — hand the upload to the user, who has push access,
-and continue once they confirm it's done.
-
 ## Execution Steps
 
 1. Run `git status` to check for uncommitted changes
 2. Run `git log main..HEAD --oneline` to see all commits in this branch
 3. Run `git diff main...HEAD` to see the full diff against main
 4. **Invoke the `local-review` skill** before creating the PR (`/local-review` in Claude Code, `$local-review` in Codex, `pi --skill local-review` / `/skill:local-review` in Pi). If issues are found, fix them and commit before proceeding. Do not skip this step.
-5. **Screenshots for frontend changes**: if `git diff main...HEAD --name-only` matches `^frontend/`, capture and embed screenshots of the affected UI per "Screenshots" above before writing the PR body (skip only if there is no visible UI effect).
-6. Check if remote branch exists and is up to date:
+5. Check if remote branch exists and is up to date:
    ```bash
    git rev-parse --abbrev-ref --symbolic-full-name @{u} 2>/dev/null || echo "no upstream"
    ```
-7. Push to remote if needed: `git push -u origin HEAD`
-8. Create draft PR using gh CLI:
+6. Push to remote if needed: `git push -u origin HEAD`
+7. Create draft PR using gh CLI:
    ```bash
    gh pr create --draft --title "<type>: <description>" --body "$(cat <<'EOF'
    ## Summary
@@ -119,7 +82,7 @@ and continue once they confirm it's done.
    EOF
    )"
    ```
-9. Return the PR URL to the user
+8. Return the PR URL to the user
 
 ## EE Companion PR (when `*_ee.rs` files were modified)
 

.agents/skills/svelte-frontend/SKILL.md

@@ -78,7 +78,3 @@ Use the Svelte MCP tools when working on Svelte code:
 2. **get-documentation**: Fetch relevant sections based on use_cases
 3. **svelte-autofixer**: MUST use on all Svelte code before finalizing — keep calling until no issues
 4. **playground-link**: Only after user confirms and code was NOT written to project files
-
-## Verifying in the Browser
-
-After changing Svelte code, use the **Playwright MCP** (`mcp__playwright__*`) to drive the running frontend and confirm the change works. See AGENTS.md → "Verifying Frontend Changes" for the full flow. Use `playwright` (headless) on devboxes; `playwright-headed` when a display is available.

.claude/hooks/guard-main-branch.sh

@@ -16,23 +16,6 @@ command="$(echo "$input" | jq -r '.tool_input.command // empty')"
 if [[ "$command" =~ ^git\ (push|reset|revert|checkout|merge|rebase|commit|add) ]]; then
   branch="$(git rev-parse --abbrev-ref HEAD 2>/dev/null || true)"
   if [[ "$branch" == "main" ]]; then
-    echo "BLOCK: You are on the main branch. Create or switch to a feature branch first." >&2
-    exit 2
-  fi
-fi
-
-# Block force-push targeting main from any branch.
-if [[ "$command" =~ ^git[[:space:]]+push([[:space:]]|$) ]]; then
-  has_force=false
-  if [[ "$command" =~ (--force([[:space:]]|=|$)|--force-with-lease|[[:space:]]-f([[:space:]]|$)) ]]; then
-    has_force=true
-  fi
-  # `+ref` refspec syntax is also a force push.
-  if [[ "$command" =~ [[:space:]]\+[A-Za-z] ]]; then
-    has_force=true
-  fi
-  if $has_force && [[ "$command" =~ (^|[[:space:]:])\+?main([[:space:]]|$) ]]; then
-    echo "BLOCK: Force-push to main is not allowed via Claude. Run it yourself if you really mean to." >&2
-    exit 2
+    echo "BLOCK: You are on the main branch. Create or switch to a feature branch first."
   fi
 fi

.claude/settings.json

@@ -44,25 +44,7 @@
       "Bash(git merge:*)",
       "Bash(git rebase:*)",
       "Bash(git add:*)",
-      "Bash(git commit:*)",
-      "Read(/tmp/**)",
-      "Write(/tmp/**)",
-      "Edit(/tmp/**)",
-      "Bash(rm:/tmp/*)",
-      "Bash(rm:/tmp/**)",
-      "Bash(rmdir:/tmp/*)",
-      "Bash(mkdir:/tmp/*)",
-      "Bash(mkdir:/tmp/**)",
-      "Bash(cp:/tmp/*)",
-      "Bash(cp:/tmp/**)",
-      "Bash(mv:/tmp/*)",
-      "Bash(mv:/tmp/**)",
-      "Bash(touch:/tmp/*)",
-      "Bash(touch:/tmp/**)",
-      "Bash(chmod:/tmp/*)",
-      "Bash(chmod:/tmp/**)",
-      "Bash(tar * /tmp/*)",
-      "Bash(unzip * /tmp/*)"
+      "Bash(git commit:*)"
     ],
     "deny": [
       "Read(.env)",
@@ -90,13 +72,7 @@
       "Bash(chown:*)",
       "Bash(truncate:*)",
       "Bash(shred:*)",
-      "Bash(unlink:*)",
-      "mcp__claude_ai_Stripe",
-      "mcp__claude_ai_Gmail",
-      "mcp__claude_ai_Google_Calendar",
-      "mcp__claude_ai_Google_Drive",
-      "mcp__claude_ai_Slack",
-      "mcp__claude_ai_Linear"
+      "Bash(unlink:*)"
     ]
   },
   "enableAllProjectMcpServers": true,

.github/DockerfileBackendTests

@@ -28,7 +28,7 @@ ENV PATH="${PATH}:/usr/local/go/bin"
 ENV GO_PATH=/usr/local/go/bin/go
 
 # UV
-RUN curl --proto '=https' --tlsv1.2 -LsSf https://github.com/astral-sh/uv/releases/download/0.9.25/uv-installer.sh | sh && mv /usr/local/cargo/bin/uv /usr/local/bin/uv
+RUN curl --proto '=https' --tlsv1.2 -LsSf https://github.com/astral-sh/uv/releases/download/0.9.24/uv-installer.sh | sh && mv /usr/local/cargo/bin/uv /usr/local/bin/uv
 
 ENV TZ=Etc/UTC
 

.github/change-versions-mac.sh

@@ -7,7 +7,7 @@ VERSION=$1
 echo "Updating versions to: $VERSION"
 
 sed -i '' -e "/^version =/s/= .*/= \"$VERSION\"/" ${root_dirpath}/backend/Cargo.toml
-sed -i '' -e "/^export const VERSION =/s/= .*/= \"v$VERSION\";/"  ${root_dirpath}/cli/src/core/constants.ts
+sed -i '' -e "/^export const VERSION =/s/= .*/= \"v$VERSION\";/"  ${root_dirpath}/cli/src/main.ts
 sed -i '' -e "/^export const VERSION =/s/= .*/= \"v$VERSION\";/" ${root_dirpath}/benchmarks/lib.ts
 sed -i '' -e "/version: /s/: .*/: $VERSION/" ${root_dirpath}/backend/windmill-api/openapi.yaml
 sed -i '' -e "/version: /s/: .*/: $VERSION/" ${root_dirpath}/openflow.openapi.yaml

.github/change-versions.sh

@@ -7,7 +7,7 @@ VERSION=$1
 echo "Updating versions to: $VERSION"
 
 sed -i -e "/^version =/s/= .*/= \"$VERSION\"/" ${root_dirpath}/backend/Cargo.toml
-sed -i -e "/^export const VERSION =/s/= .*/= \"$VERSION\";/" ${root_dirpath}/cli/src/core/constants.ts
+sed -i -e "/^export const VERSION =/s/= .*/= \"$VERSION\";/" ${root_dirpath}/cli/src/main.ts
 sed -i -e "/^export const VERSION =/s/= .*/= \"v$VERSION\";/" ${root_dirpath}/benchmarks/lib.ts
 sed -i -e "/version: /s/: .*/: $VERSION/" ${root_dirpath}/backend/windmill-api/openapi.yaml
 sed -i -e "/version: /s/: .*/: $VERSION/" ${root_dirpath}/openflow.openapi.yaml

.github/workflows/ai-agent-tests.yml

@@ -1,132 +0,0 @@
-name: AI Agent Integration Tests
-
-# Exercises the AI agent flow path (preview_flow with `aiagent` modules) against
-# real LLM providers. Runs only when AI-agent backend code or the tests change,
-# because each run makes real (paid) LLM calls. To avoid spending on every commit,
-# the PR side triggers only when a PR is marked ready for review (out of draft) —
-# not on `synchronize` — plus push to main and manual dispatch.
-on:
-  workflow_dispatch:
-  push:
-    branches: [main]
-    paths:
-      - "integration_tests/ai_agent_tests/**"
-      - "backend/windmill-ai/**"
-      - "backend/windmill-api/src/ai.rs"
-      - "backend/windmill-worker/src/ai_executor.rs"
-      - "backend/windmill-worker/src/ai/**"
-      - "backend/windmill-worker/src/memory_common.rs"
-      - "backend/windmill-common/src/flow_conversations.rs"
-      - ".github/workflows/ai-agent-tests.yml"
-  pull_request:
-    types: [opened, reopened, ready_for_review]
-    paths:
-      - "integration_tests/ai_agent_tests/**"
-      - "backend/windmill-ai/**"
-      - "backend/windmill-api/src/ai.rs"
-      - "backend/windmill-worker/src/ai_executor.rs"
-      - "backend/windmill-worker/src/ai/**"
-      - "backend/windmill-worker/src/memory_common.rs"
-      - "backend/windmill-common/src/flow_conversations.rs"
-      - ".github/workflows/ai-agent-tests.yml"
-
-concurrency:
-  group: ai-agent-tests-${{ github.ref }}
-  cancel-in-progress: true
-
-jobs:
-  ai_agent_e2e:
-    # Skip draft PRs; the `opened`/`reopened` types would otherwise fire while
-    # still a draft. `ready_for_review` always arrives non-draft.
-    if: github.event_name != 'pull_request' || github.event.pull_request.draft == false
-    runs-on: ubicloud-standard-16
-    services:
-      postgres:
-        image: postgres:16
-        ports:
-          - 5432:5432
-        env:
-          POSTGRES_DB: windmill
-          POSTGRES_PASSWORD: changeme
-        options: >-
-          --health-cmd pg_isready --health-interval 10s --health-timeout 5s
-          --health-retries 5
-    steps:
-      - uses: actions/checkout@v4
-
-      - uses: actions-rust-lang/setup-rust-toolchain@v1
-        with:
-          cache-workspaces: backend
-          toolchain: 1.93.0
-
-      - uses: oven-sh/setup-bun@v2
-        with:
-          bun-version: 1.3.10
-
-      - uses: actions/setup-node@v4
-        with:
-          node-version: "20"
-
-      - uses: actions/setup-python@v5
-        with:
-          python-version: "3.11"
-
-      # CE build (no enterprise/license needed for AI agents). `quickjs` powers
-      # flow input-transform JS eval; `mcp` is required by the deepwiki MCP tool
-      # test. Bun tool scripts run via the always-on worker (BUN_PATH).
-      - name: Build Windmill
-        working-directory: ./backend
-        env:
-          SQLX_OFFLINE: true
-          CARGO_BUILD_JOBS: 12
-          RUSTFLAGS: ""
-        run: cargo build --features quickjs,mcp
-
-      - name: Start Windmill
-        working-directory: ./backend
-        env:
-          DATABASE_URL: postgres://postgres:changeme@localhost:5432/windmill
-          BUN_PATH: bun
-          NODE_BIN_PATH: node
-          RUST_LOG: info
-        run: |
-          mkdir -p ../integration_tests/logs
-          ./target/debug/windmill > ../integration_tests/logs/windmill.log 2>&1 &
-          echo "Waiting for Windmill to be ready..."
-          for i in $(seq 1 60); do
-            if curl -sf http://localhost:8000/api/version > /dev/null 2>&1; then
-              echo "Windmill is ready"
-              break
-            fi
-            sleep 2
-          done
-          curl -sf http://localhost:8000/api/version > /dev/null || { echo "Windmill failed to start"; tail -50 ../integration_tests/logs/windmill.log; exit 1; }
-
-      - name: Run AI agent integration tests
-        timeout-minutes: 20
-        working-directory: ./integration_tests/ai_agent_tests
-        env:
-          WINDMILL_URL: http://localhost:8000
-          # Only the providers we have org secrets for. Other providers
-          # (Azure, Bedrock, OpenRouter) are skipped by conftest when their
-          # keys are absent — see skip_provider_without_credentials.
-          ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }}
-          OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
-          GOOGLE_AI_API_KEY: ${{ secrets.GOOGLE_API_KEY }}
-        run: |
-          python -m venv .venv
-          .venv/bin/pip install -r requirements.txt
-          # The S3/vision-attachment tests need MinIO large-file storage and
-          # image-capable provider setup; out of scope for this cost-controlled
-          # smoke. Add MinIO secrets + a storage service to enable them.
-          .venv/bin/python -m pytest -v \
-            --ignore=test_user_attachments.py \
-            --ignore=test_user_images.py \
-            --ignore=test_image_output.py
-
-      - name: Archive Windmill logs
-        uses: actions/upload-artifact@v4
-        if: always()
-        with:
-          name: ai-agent-tests-windmill-logs
-          path: integration_tests/logs

.github/workflows/ai-evals-test.yml

@@ -1,166 +0,0 @@
-name: AI Evals (global mode)
-
-# Smoke-tests the production global AI chat proxy/frontend execution path via
-# the ai_evals harness, one case across one cheap model per provider. Runs only
-# when the eval harness or the global chat code change, since each run makes real
-# (paid) LLM calls. The backend is built from source purely as the AI proxy the
-# harness routes model calls through; the global tools/drafts run in-process in
-# the Vitest bridge against production frontend code. To avoid spending on every
-# commit, the PR side triggers only when a PR is marked ready for review (out of
-# draft) — not on `synchronize` — plus push to main and manual dispatch.
-on:
-  workflow_dispatch:
-  push:
-    branches: [main]
-    paths:
-      - "ai_evals/**"
-      - "backend/windmill-api/src/ai.rs"
-      - "backend/windmill-ai/**"
-      - "frontend/src/lib/components/copilot/**"
-      # The eval harness runs production frontend code in-process; these are the
-      # AI/draft-specific deps outside copilot/ that the global smoke exercises.
-      - "frontend/src/lib/userDraft.svelte.ts"
-      - "frontend/src/lib/userDraftDbSyncer.svelte.ts"
-      - "frontend/src/lib/infer.ts"
-      - ".github/workflows/ai-evals-test.yml"
-  pull_request:
-    types: [opened, reopened, ready_for_review]
-    paths:
-      - "ai_evals/**"
-      - "backend/windmill-api/src/ai.rs"
-      - "backend/windmill-ai/**"
-      - "frontend/src/lib/components/copilot/**"
-      # The eval harness runs production frontend code in-process; these are the
-      # AI/draft-specific deps outside copilot/ that the global smoke exercises.
-      - "frontend/src/lib/userDraft.svelte.ts"
-      - "frontend/src/lib/userDraftDbSyncer.svelte.ts"
-      - "frontend/src/lib/infer.ts"
-      - ".github/workflows/ai-evals-test.yml"
-
-concurrency:
-  group: ai-evals-test-${{ github.ref }}
-  cancel-in-progress: true
-
-jobs:
-  ai_evals_global:
-    # Provider secrets are unavailable to forked and Dependabot PRs.
-    if: >-
-      github.event_name != 'pull_request' ||
-      (
-        github.event.pull_request.draft == false &&
-        github.event.pull_request.head.repo.full_name == github.repository &&
-        github.event.pull_request.user.login != 'dependabot[bot]'
-      )
-    runs-on: ubicloud-standard-16
-    services:
-      postgres:
-        image: postgres:16
-        ports:
-          - 5432:5432
-        env:
-          POSTGRES_DB: windmill
-          POSTGRES_PASSWORD: changeme
-        options: >-
-          --health-cmd pg_isready --health-interval 10s --health-timeout 5s
-          --health-retries 5
-    steps:
-      - uses: actions/checkout@v4
-
-      - uses: actions-rust-lang/setup-rust-toolchain@v1
-        with:
-          cache-workspaces: backend
-          toolchain: 1.93.0
-
-      - uses: oven-sh/setup-bun@v2
-        with:
-          bun-version: 1.3.10
-
-      - uses: actions/setup-node@v4
-        with:
-          # Node 22.19+ is required by the frontend's undici 8.x, which the
-          # Vitest bridge loads; Node 20 fails with markAsUncloneable.
-          node-version: "22"
-
-      # CE build used only as the AI proxy (login, workspace, provider resource,
-      # /ai/proxy). No worker execution or MCP needed — global tools/drafts run
-      # in the Vitest bridge. quickjs matches the standard CE feature set.
-      - name: Build Windmill (AI proxy)
-        working-directory: ./backend
-        env:
-          SQLX_OFFLINE: true
-          CARGO_BUILD_JOBS: 12
-          RUSTFLAGS: ""
-        run: cargo build --features quickjs
-
-      - name: Start Windmill
-        working-directory: ./backend
-        env:
-          DATABASE_URL: postgres://postgres:changeme@localhost:5432/windmill
-          RUST_LOG: info
-        run: |
-          mkdir -p ../ai_evals/logs
-          ./target/debug/windmill > ../ai_evals/logs/windmill.log 2>&1 &
-          echo "Waiting for Windmill to be ready..."
-          for i in $(seq 1 60); do
-            if curl -sf http://localhost:8000/api/version > /dev/null 2>&1; then
-              echo "Windmill is ready"
-              break
-            fi
-            sleep 2
-          done
-          curl -sf http://localhost:8000/api/version > /dev/null || { echo "Windmill failed to start"; tail -50 ../ai_evals/logs/windmill.log; exit 1; }
-
-      - name: Install frontend deps + generate client
-        working-directory: ./frontend
-        run: |
-          npm ci
-          npm run generate-backend-client
-
-      - name: Run global AI evals
-        timeout-minutes: 20
-        working-directory: ./ai_evals
-        env:
-          WMILL_AI_EVAL_BACKEND_URL: http://localhost:8000
-          WMILL_AI_EVAL_BACKEND_WORKSPACE: integration-tests
-          # Anthropic backs the haiku model. Google AI uses GEMINI_API_KEY.
-          ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }}
-          OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
-          GEMINI_API_KEY: ${{ secrets.GOOGLE_API_KEY }}
-          DEEPSEEK_API_KEY: ${{ secrets.DEEPSEEK_API_KEY }}
-        run: |
-          bun install
-          mkdir -p results
-          # One cheap model per provider (anthropic/openai/googleai/deepseek).
-          fail=0
-          for m in haiku 4o gemini-3-flash-preview deepseek-v4-flash; do
-            echo "::group::global-test1-script-create ($m)"
-            if ! bun run cli -- run global global-test1-script-create \
-                  --model "$m" --execution-only --output "$PWD/results/ci-$m.json"; then
-              echo "$m: harness/proxy errored"
-              fail=1
-              echo "::endgroup::"
-              continue
-            fi
-            # The CLI exits 0 when the harness records failed attempts, so gate
-            # on execution-only pass counts while ignoring model output quality.
-            if jq -e \
-              '.attemptCount > 0 and .passedAttempts == .attemptCount' \
-              "results/ci-$m.json" > /dev/null; then
-              echo "$m: OK — proxy/frontend execution completed"
-            else
-              echo "$m: FAILED proxy/frontend execution"
-              jq -c '.cases[0].attempts[0].checks' "results/ci-$m.json" || true
-              fail=1
-            fi
-            echo "::endgroup::"
-          done
-          [ "$fail" = 0 ] || { echo "ai_evals global smoke failed"; exit 1; }
-
-      - name: Archive logs and results
-        uses: actions/upload-artifact@v4
-        if: always()
-        with:
-          name: ai-evals-global-logs
-          path: |
-            ai_evals/logs
-            ai_evals/results

.github/workflows/backend-test-windows.yml

@@ -74,7 +74,7 @@ jobs:
 
       - uses: astral-sh/setup-uv@v6.2.1
         with:
-          version: "0.9.25"
+          version: "0.9.24"
 
       - uses: shivammathur/setup-php@v2
         with:
@@ -98,21 +98,6 @@ jobs:
           vcpkg.exe install openssl:x64-windows-static
           vcpkg.exe integrate install
 
-      - name: Free disk space (post-vcpkg)
-        shell: pwsh
-        run: |
-          # vcpkg leaves multi-GB of buildtrees/downloads after installing openssl;
-          # we only need the installed/ dir for linking.
-          $vcpkgRoot = $env:VCPKG_INSTALLATION_ROOT
-          foreach ($sub in @("buildtrees", "downloads", "packages")) {
-            $path = Join-Path $vcpkgRoot $sub
-            if (Test-Path $path) {
-              Write-Host "Removing $path"
-              Remove-Item -Recurse -Force -ErrorAction SilentlyContinue $path
-            }
-          }
-          Get-PSDrive C | Select-Object Used,Free | Format-Table -AutoSize
-
       - name: Get runtime paths
         id: runtime-paths
         shell: pwsh
@@ -134,10 +119,6 @@ jobs:
           cargo build --release -p windmill_duckdb_ffi_internal
           New-Item -ItemType Directory -Path ..\target\debug -Force
           Copy-Item target\release\windmill_duckdb_ffi_internal.dll ..\target\debug\
-          # duckdb is bundled (~2GB of build artifacts); the DLL is the only
-          # thing we need from this excluded-crate target dir.
-          Remove-Item -Recurse -Force -ErrorAction SilentlyContinue target
-          Get-PSDrive C | Select-Object Used,Free | Format-Table -AutoSize
 
       - name: Print runtime versions and env
         shell: pwsh
@@ -155,10 +136,6 @@ jobs:
           echo "USERPROFILE=$env:USERPROFILE"
           echo "HOME=$env:HOME"
 
-      - name: Disk space before cargo test
-        shell: pwsh
-        run: Get-PSDrive C | Select-Object Used,Free | Format-Table -AutoSize
-
       - name: cargo test
         working-directory: backend
         timeout-minutes: 60
@@ -167,16 +144,13 @@ jobs:
           RUST_LOG: "off"
           RUST_LOG_STYLE: never
           CARGO_NET_GIT_FETCH_WITH_CLI: true
-          # 16-vcpu runners with disabled PDB still hit LNK1180 ("insufficient
-          # disk space") at link time with 12 parallel link jobs: each test
-          # binary link spikes several hundred MB of transient I/O. Capping at
-          # 8 trades ~25% wall time for headroom on the ~75GB runner disk.
-          CARGO_BUILD_JOBS: 8
+          CARGO_BUILD_JOBS: 12
           # backend/Cargo.toml sets split-debuginfo = "unpacked", which on
           # windows-msvc is coerced to "packed": every test-binary link spawns
-          # the mspdbsrv.exe PDB type server and writes a large .pdb. CI needs
-          # no debug info, so disable PDB generation for the dev/test profiles
-          # here (avoids both LNK1318 type-server limit and PDB disk usage).
+          # the mspdbsrv.exe PDB type server and writes a large .pdb. With 12
+          # parallel link jobs this races the type-server cap (LNK1318 "LIMIT
+          # (12)") and exhausts the runner disk (LNK1180). CI needs no debug
+          # info, so disable PDB generation for the dev/test profiles here.
           CARGO_PROFILE_DEV_SPLIT_DEBUGINFO: "off"
           CARGO_PROFILE_TEST_SPLIT_DEBUGINFO: "off"
           # Tests' poll-time stack frames (deep nested async fn chains in

.github/workflows/backend-test.yml

@@ -62,7 +62,7 @@ jobs:
           node-version: "20"
       - uses: astral-sh/setup-uv@v6.2.1
         with:
-          version: "0.9.25"
+          version: "0.9.24"
       - uses: shivammathur/setup-php@v2
         with:
           php-version: "8.3"

.github/workflows/check-empty-fixture.yml

@@ -1,19 +0,0 @@
-name: Check fixture is empty
-
-on:
-  push:
-    branches: [main]
-    paths:
-      - "fixtures/**"
-  pull_request:
-    paths:
-      - "fixtures/**"
-
-jobs:
-  check-empty-fixture:
-    runs-on: ubuntu-latest
-    steps:
-      - uses: actions/checkout@v4
-
-      - name: Ensure fixtures/cli-sync/ has no committed snapshot
-        run: bash fixtures/check-empty.sh

.github/workflows/claude-plan.yml

@@ -45,7 +45,7 @@ jobs:
           allowed_bots: 'windmill-internal-app[bot]'
           trigger_phrase: '/plan'
           claude_args: |
-            --model claude-opus-4-8
+            --model opus
             --system-prompt "# Claude Planning Mode
 
             You are operating in PLANNING MODE ONLY. Your role is to create detailed, structured plans without making any code changes.

.github/workflows/claude.yml

@@ -51,4 +51,4 @@ jobs:
             }
           claude_args: |
             --allowedTools "Bash,WebFetch,WebSearch"
-            --model claude-opus-4-8
+            --model opus

.github/workflows/git-sync-test.yml

@@ -8,7 +8,6 @@ on:
       - "backend/windmill-git-sync/**"
       - "backend/windmill-api-integration-tests/tests/git_sync*"
       - "backend/ee-repo-ref.txt"
-      - "backend/windmill-common/src/workspaces.rs"
       - "integration_tests/test/git_sync_test.py"
       - ".github/workflows/git-sync-test.yml"
   pull_request:
@@ -17,7 +16,6 @@ on:
       - "backend/windmill-git-sync/**"
       - "backend/windmill-api-integration-tests/tests/git_sync*"
       - "backend/ee-repo-ref.txt"
-      - "backend/windmill-common/src/workspaces.rs"
       - "integration_tests/test/git_sync_test.py"
       - ".github/workflows/git-sync-test.yml"
 
@@ -51,7 +49,7 @@ jobs:
           echo "$CHANGED_FILES"
 
           # Direct git sync file changes — always relevant
-          if echo "$CHANGED_FILES" | grep -qE '^(backend/windmill-git-sync/|backend/windmill-api-integration-tests/tests/git_sync|backend/windmill-common/src/workspaces\.rs|integration_tests/test/git_sync|\.github/workflows/git-sync-test\.yml)'; then
+          if echo "$CHANGED_FILES" | grep -qE '^(backend/windmill-git-sync/|backend/windmill-api-integration-tests/tests/git_sync|integration_tests/test/git_sync|\.github/workflows/git-sync-test\.yml)'; then
             echo "should_run=true" >> "$GITHUB_OUTPUT"
             echo "Relevant: direct git sync file changes"
             exit 0

.github/workflows/pr-ready-review.yml

@@ -160,4 +160,4 @@ jobs:
             ${{ env.REVIEW_PROMPT }}
           claude_args: |
             --allowedTools "mcp__github_inline_comment__create_inline_comment,Bash(gh pr comment:*),Bash(gh pr diff:*),Bash(gh pr view:*)"
-            --model claude-opus-4-8
+            --model opus

.github/workflows/spawn-ephemeral-backend.yml

@@ -0,0 +1,126 @@
+name: Spawn Ephemeral Backend
+
+on:
+  issue_comment:
+    types: [created]
+  pull_request_review_comment:
+    types: [created]
+  workflow_dispatch:
+    inputs:
+      pr_number:
+        description: "PR number"
+        required: true
+        type: number
+
+jobs:
+  check-membership:
+    if: |
+      (github.event_name == 'issue_comment' && contains(github.event.comment.body, '/spawnbackend')) ||
+      (github.event_name == 'pull_request_review_comment' && contains(github.event.comment.body, '/spawnbackend'))
+    uses: ./.github/workflows/check-org-membership.yml
+    secrets:
+      access_token: ${{ secrets.ORG_ACCESS_TOKEN }}
+
+  spawn-backend:
+    needs: check-membership
+    # Only run on PR comments that contain /spawn-backend, or manual dispatch
+    if: |
+      github.event_name == 'workflow_dispatch' ||
+      (github.event.issue.pull_request && needs.check-membership.outputs.is_member == 'true')
+    runs-on: ubuntu-latest
+    permissions:
+      pull-requests: write
+      contents: read
+
+    steps:
+      - name: Get PR details
+        id: pr-details
+        uses: actions/github-script@v7
+        with:
+          script: |
+            const prNumber = context.eventName === 'workflow_dispatch'
+              ? context.payload.inputs.pr_number
+              : context.issue.number;
+
+            const pr = await github.rest.pulls.get({
+              owner: context.repo.owner,
+              repo: context.repo.repo,
+              pull_number: prNumber
+            });
+
+            // Get branch name and format it for Cloudflare Pages
+            // Replace '/' with '-' for the URL
+            const branchName = pr.data.head.ref;
+            const formattedBranch = branchName.replace(/\//g, '-');
+            const cfFrontendUrl = `https://${formattedBranch}.windmill.pages.dev`;
+
+            core.setOutput('commit_hash', pr.data.head.sha);
+            core.setOutput('pr_number', prNumber);
+            core.setOutput('branch_name', branchName);
+            core.setOutput('cf_frontend_url', cfFrontendUrl);
+
+      - name: Check manager URL
+        id: check-manager-url
+        run: |
+          if [ -z "${{ secrets.EPHEMERAL_BACKEND_QUEUE_URL }}" ]; then
+            echo "manager_url_set=false" >> $GITHUB_OUTPUT
+          else
+            echo "manager_url_set=true" >> $GITHUB_OUTPUT
+          fi
+
+      - name: Post error comment if manager not running
+        if: steps.check-manager-url.outputs.manager_url_set == 'false'
+        uses: actions/github-script@v7
+        with:
+          script: |
+            const prNumber = context.eventName === 'workflow_dispatch'
+              ? Number(context.payload.inputs.pr_number)
+              : context.issue.number;
+
+            await github.rest.issues.createComment({
+              owner: context.repo.owner,
+              repo: context.repo.repo,
+              issue_number: prNumber,
+              body: `❌ Manager URL not set (did you start the ephemeral backend manager?)\n\nThe ephemeral backend manager needs to be running to spawn backends. Please start the manager first.`
+            });
+
+      - name: Fail if manager not running
+        if: steps.check-manager-url.outputs.manager_url_set == 'false'
+        run: |
+          echo "Error: EPHEMERAL_BACKEND_QUEUE_URL secret is not set"
+          exit 1
+
+      - name: Trigger Windmill flow
+        if: steps.check-manager-url.outputs.manager_url_set == 'true'
+        id: trigger-flow
+        run: |
+          JOB_UUID=$(curl -s -X POST "https://app.windmill.dev/api/w/windmill-labs/jobs/run/f/f/all/run_ephemeral_backend" \
+            -H "Authorization: Bearer ${{ secrets.WINDMILL_RUN_FLOW_TOKEN }}" \
+            -H "Content-Type: application/json" \
+            -d '{
+              "manager_url": "${{ secrets.EPHEMERAL_BACKEND_QUEUE_URL }}",
+              "commit_hash": "${{ steps.pr-details.outputs.commit_hash }}",
+              "pr_number": ${{ steps.pr-details.outputs.pr_number }},
+              "cf_frontend_url": "${{ steps.pr-details.outputs.cf_frontend_url }}"
+            }' | tr -d '"')
+
+          echo "Job UUID: $JOB_UUID"
+          echo "job_uuid=$JOB_UUID" >> $GITHUB_OUTPUT
+
+      - name: Post comment with job link
+        if: steps.check-manager-url.outputs.manager_url_set == 'true'
+        uses: actions/github-script@v7
+        with:
+          script: |
+            const jobUuid = '${{ steps.trigger-flow.outputs.job_uuid }}';
+            const appUrl = `https://app.windmill.dev/public/windmill-labs/a106bad0256c1dfa7a4f9279c42b1a4b#${jobUuid}`;
+            const prNumber = context.eventName === 'workflow_dispatch'
+              ? Number(context.payload.inputs.pr_number)
+              : context.issue.number;
+
+            await github.rest.issues.createComment({
+              owner: context.repo.owner,
+              repo: context.repo.repo,
+              issue_number: prNumber,
+              body: `🚀 Spawning new ephemeral backend!\n\n${appUrl}`
+            });

.gitignore

@@ -33,5 +33,4 @@ backend/chrome_profiler.json
 .fast-check/
 __pycache__/
 .playwright-mcp/
-.codex
-.claude/scheduled_tasks.lock
+.codex

.mcp.json

@@ -3,16 +3,6 @@
     "svelte": {
       "type": "http",
       "url": "https://mcp.svelte.dev/mcp"
-    },
-    "playwright": {
-      "type": "stdio",
-      "command": "npx",
-      "args": ["-y", "@playwright/mcp@latest", "--browser", "chromium", "--headless"]
-    },
-    "playwright-headed": {
-      "type": "stdio",
-      "command": "npx",
-      "args": ["-y", "@playwright/mcp@latest", "--browser", "chromium"]
     }
   }
 }

.webmux.yaml

@@ -47,7 +47,6 @@ profiles:
       For this window specifically, backend is running on: ${BACKEND_PORT} and frontend is running on: ${FRONTEND_PORT}.
       To connect to the database, use this connection string: ${DATABASE_URL}
       Because we are running backend with cargo watch, to verify your changes, just check the logs in the backend pane. No need for cargo check.
-      For UI verification, use the Playwright MCP (`mcp__playwright__*`) — the `playwright` server is headless and works without a display. Navigate to http://localhost:${FRONTEND_PORT}, log in as admin@windmill.dev / changeme.
       IMPORTANT: Read docs/autonomous-mode.md before starting any work.
     panes:
       - id: agent
@@ -77,7 +76,6 @@ profiles:
       On this window specifically, frontend is running on: ${FRONTEND_PORT}.
       To connect to the database, use this connection string: ${DATABASE_URL}
       Because we are running frontend with npm run dev, to verify your changes, just check the logs in the frontend pane. No need for npm run build.
-      For UI verification, use the Playwright MCP (`mcp__playwright__*`) — the `playwright` server is headless and works without a display. Navigate to http://localhost:${FRONTEND_PORT}, log in as admin@windmill.dev / changeme.
       IMPORTANT: Read docs/autonomous-mode.md before starting any work.
     panes:
       - id: agent
@@ -102,55 +100,10 @@ profiles:
 
 integrations:
   github:
-    autoRemoveOnMerge: true
     linkedRepos:
       - repo: windmill-labs/windmill-ee-private
         alias: ee-private
         dir: ../windmill-ee-private__worktrees
   linear:
     enabled: true
-    autoCreateWorktrees: true
     watchTeams: [WIN,GIT]
-
-oneshot:
-  systemPrompt: |
-    You are running in webmux ONESHOT mode.
-
-    # No interactive user
-    There is NO interactive user — nobody is watching the chat or will respond
-    to questions, approvals, or status checks. Any message asking the user to
-    review, approve, confirm, take a look, or "let you know" is wasted output:
-    it will not be answered.
-
-    # Your job
-    Take the task to its real conclusion without pausing:
-      1. Make the change.
-      2. Validate it (run the relevant tests, typecheck, build, or quick
-         manual check). For UI changes, drive the running frontend with
-         the Playwright MCP (`mcp__playwright__*`, headless) and confirm
-         the change works end-to-end before moving on.
-      3. Commit.
-      4. Push.
-      5. Open a pull request.
-    Only then are you done.
-
-    # Decisions
-    When something is ambiguous, pick the most reasonable default and proceed.
-    When you would normally ask "should I X or Y?", just pick one and continue
-    — note the choice in the PR description if it matters.
-
-    # PR readiness
-    Default to opening the PR as a draft. If you are highly confident in the
-    change — the scope is small and well-understood, validation passed
-    cleanly, and you would not change anything if a reviewer pushed back —
-    open the PR as ready-for-review directly (omit `--draft` when invoking
-    `gh pr create`, or call `gh pr ready <number>` after creation). Err on
-    the side of draft when validation was partial, the change touches
-    public APIs or shared infrastructure, or you made a non-obvious judgment
-    call.
-
-    # Ending your turn
-    Never end your turn with a question, a suggestion to "take a look", or a
-    request for approval. Stop only when the PR is open, or when you hit a
-    technical error you cannot recover from yourself (in which case clearly
-    state the blocker).

AGENTS.md

@@ -15,7 +15,6 @@ Open-source platform for internal tools, workflows, API integrations, background
 - **Enterprise**: `docs/enterprise.md` — EE file conventions and PR workflow
 - **Backend patterns**: use the `rust-backend` skill when writing Rust code
 - **Frontend patterns**: use the `svelte-frontend` skill when writing Svelte code. Do NOT edit svelte files unless you have read that skill.
-- **Frontend UUIDs**: do not call `crypto.randomUUID()` in frontend code. Import `randomUUID` from `$lib/utils/uuid` instead.
 - **Code review**: review the current PR or branch against the shared review policy in `REVIEW.md` (severity triage, public-surface checklist, AGENTS.md compliance, test-coverage assessment). The skill at `.agents/skills/local-review/SKILL.md` orchestrates it. All three CLIs auto-discover the same SKILL — Claude reads `.claude/skills/` (symlinked to the canonical `.agents/skills/` file), Codex and Pi read `.agents/skills/` directly. Invoke with `/local-review` in Claude Code, `$local-review` (or `/skills` selector) in Codex, or `pi --skill local-review` / `/skill:local-review` in Pi.
 - **Domain guides**: `.claude/skills/native-trigger/` and `frontend/tutorial-system-guide.mdc`
 - **Brand/UI guidelines**: `frontend/brand-guidelines.md`
@@ -30,28 +29,6 @@ Open-source platform for internal tools, workflows, API integrations, background
 - **Instance settings**: navigate to `/#superadmin-settings`
 - **Migrations**: use `cargo sqlx migrate add -r <name>` from `backend/` to create new migrations (never generate timestamps manually)
 
-## Verifying Frontend Changes
-
-After modifying frontend code, drive the running dev server with the **Playwright MCP** to verify the change in a real browser — don't claim a UI change works without exercising it.
-
-Two MCP servers are registered in `.mcp.json`:
-- `playwright` — headless Chromium, default for devboxes (no display required)
-- `playwright-headed` — windowed Chromium, when a display is available
-
-**One-time setup:** run `npx playwright install chromium` to download the browser binary (Playwright won't fetch it automatically on first use).
-
-Typical flow:
-1. Ensure backend (`cargo run`) and frontend (`REMOTE=http://localhost:8000 npm run dev`) are running
-2. `mcp__playwright__browser_navigate` to the relevant page (login at `admin@windmill.dev` / `changeme`)
-3. `mcp__playwright__browser_snapshot` to inspect the accessibility tree (preferred over screenshots for reading the DOM)
-4. `mcp__playwright__browser_click` / `browser_fill_form` / `browser_type` to interact
-5. `mcp__playwright__browser_take_screenshot` for visual confirmation
-6. `mcp__playwright__browser_console_messages` / `browser_network_requests` to surface errors
-
-**Attach the screenshots to the PR.** For any change under `frontend/`, embed screenshots of the affected UI in the PR body — the `pr` skill requires this and carries the upload recipe.
-
-If you cannot exercise a UI change (no dev server, etc.), say so explicitly rather than claiming success.
-
 ## Banned Patterns
 
 ### `$bindable(default_value)` on optional props
@@ -109,5 +86,3 @@ $NAV --root backend callees "X"                           # what does X call?
 - Search for existing code to reuse before writing new code
 - Follow established patterns in the codebase
 - Keep changes focused — don't refactor beyond what's asked
-- **Comments record constraints, not narration.** Write a comment only for what the code can't show: why a non-obvious approach is required, what breaks if it's "simplified" away. State each invariant once, at the place where someone would break it, in ≤4 lines. Don't describe what the next line does, don't repeat the same rationale at multiple sites, and don't address the PR reviewer (justifying a change belongs in the PR description, not the code). Describe the code as it is, never its drafting history: "we no longer do X", "unchanged behavior", "instead of the previous approach" are meaningless to a reader who never saw the earlier iteration — before finishing, reread your comments as if the current state is the only state that ever existed.
-- **Never attribute work to a specific customer, account, or "requested by a customer" in repo-tracked content** (PR descriptions, commit messages, code comments, docs). Describe changes by their technical motivation instead.

CHANGELOG.md

@@ -1,679 +1,5 @@
 # Changelog
 
-## [1.737.0](https://github.com/windmill-labs/windmill/compare/v1.736.0...v1.737.0) (2026-06-23)
-
-
-### Features
-
-* **apps:** opt-in sandbox isolation for published & raw apps (alpha) ([#9420](https://github.com/windmill-labs/windmill/issues/9420)) ([2879cbb](https://github.com/windmill-labs/windmill/commit/2879cbb65a4122c86b4a472206d74d9009b07904))
-
-
-### Bug Fixes
-
-* allow SQL args in managed // materialize scripts ([#9733](https://github.com/windmill-labs/windmill/issues/9733)) ([fa35968](https://github.com/windmill-labs/windmill/commit/fa3596885bf2d7ee8859f295e820ee362c756911))
-* bound orphan-cleanup drain rate with capped multi-batch loop ([#9730](https://github.com/windmill-labs/windmill/issues/9730)) ([31d9215](https://github.com/windmill-labs/windmill/commit/31d9215e5a61f19662cc87be8147007e1d47ebb6))
-* **ext-jwt:** reject external JWT auth for non-existent workspaces ([#9723](https://github.com/windmill-labs/windmill/issues/9723)) ([c644311](https://github.com/windmill-labs/windmill/commit/c644311eca4bcaf4b68058cf1d5d79d4078aee1a))
-* optimize cleanup_job_perms_orphaned and job_result_stream cleanup queries ([#9727](https://github.com/windmill-labs/windmill/issues/9727)) ([6d94865](https://github.com/windmill-labs/windmill/commit/6d9486510933af9109f52011d93b13847dbdbb39))
-* prevent silent audit-partition outage via monitor watchdog + alert ([#9729](https://github.com/windmill-labs/windmill/issues/9729)) ([8dea383](https://github.com/windmill-labs/windmill/commit/8dea38383f884f59b2956c39f1424005a21265bd))
-
-
-### Performance Improvements
-
-* **monitor:** hash active-root exclusion in retention delete (WIN-2088) ([#9732](https://github.com/windmill-labs/windmill/issues/9732)) ([75bafab](https://github.com/windmill-labs/windmill/commit/75bafabeeec76cf6da33eef41f588e37071df011))
-
-## [1.736.0](https://github.com/windmill-labs/windmill/compare/v1.735.0...v1.736.0) (2026-06-23)
-
-
-### Features
-
-* **ai-chat:** workspace AI chat skills (SKILL.md upload + read_skill tool) ([#9648](https://github.com/windmill-labs/windmill/issues/9648)) ([6f4017d](https://github.com/windmill-labs/windmill/commit/6f4017d694494a158ebd0579c93336c378cba0fd))
-
-
-### Bug Fixes
-
-* **drafts:** stop mis-filing workspace-blind legacy drafts on migration ([#9725](https://github.com/windmill-labs/windmill/issues/9725)) ([3bf5b72](https://github.com/windmill-labs/windmill/commit/3bf5b72afab3241ea41a261a40c2434764bdaf72))
-* **frontend:** destroy old WebsocketProvider on workspace switch in MultiplayerMenu ([#9719](https://github.com/windmill-labs/windmill/issues/9719)) ([6e96f90](https://github.com/windmill-labs/windmill/commit/6e96f90065dfe2f6ccc5eb4f85f4facd3515c70c))
-* **frontend:** ensure type:object in test_run_flow tool schema for Anthropic ([#9721](https://github.com/windmill-labs/windmill/issues/9721)) ([d5cb944](https://github.com/windmill-labs/windmill/commit/d5cb944cf92f074b2ee42c876595eacdfa2f4d76))
-* **health:** detect read-only replica via pg_is_in_recovery() ([#9722](https://github.com/windmill-labs/windmill/issues/9722)) ([e16061d](https://github.com/windmill-labs/windmill/commit/e16061df06babeae935a9396de5bdcd46e8119a9))
-* re-enforce scoped API token boundaries across handlers ([#9712](https://github.com/windmill-labs/windmill/issues/9712)) ([e19594d](https://github.com/windmill-labs/windmill/commit/e19594df2ad015a0336ade95e04562f5562ec3f6))
-
-## [1.735.0](https://github.com/windmill-labs/windmill/compare/v1.734.0...v1.735.0) (2026-06-22)
-
-
-### Features
-
-* clarify session draft bar tracks all workspace draft changes ([#9714](https://github.com/windmill-labs/windmill/issues/9714)) ([ed016a5](https://github.com/windmill-labs/windmill/commit/ed016a5edb4527877bf7e6bf92feafc2710669c2))
-* **copilot:** improve global-mode path selection + add path-selection evals ([#9698](https://github.com/windmill-labs/windmill/issues/9698)) ([74a2329](https://github.com/windmill-labs/windmill/commit/74a2329d2e8395141807c43acef07ef132490039))
-* link files & folders to the global AI chat ([#9520](https://github.com/windmill-labs/windmill/issues/9520)) ([84cc043](https://github.com/windmill-labs/windmill/commit/84cc043406d63a1e1472165cf24ce8c09905fc5f))
-* scope default instance db name to workspace (dt_/dl_) ([#9699](https://github.com/windmill-labs/windmill/issues/9699)) ([4a8a724](https://github.com/windmill-labs/windmill/commit/4a8a724895dcecb835e1eb1e4fd7d1bbc8b3e0fb))
-
-
-### Bug Fixes
-
-* enforce job_dir containment when writing module files ([#9703](https://github.com/windmill-labs/windmill/issues/9703)) ([e403f92](https://github.com/windmill-labs/windmill/commit/e403f92d7e84cebc78709dce1a0928048ba2506d))
-* **frontend:** deploy full script/flow draft from AI chat via shared module ([#9642](https://github.com/windmill-labs/windmill/issues/9642)) ([23bf6bf](https://github.com/windmill-labs/windmill/commit/23bf6bf3da01d552d2dfab6dbcfd30f758ed34d2))
-* **frontend:** strip raw-app post-deploy diff noise (raw_app/lock/data) ([#9706](https://github.com/windmill-labs/windmill/issues/9706)) ([e20a277](https://github.com/windmill-labs/windmill/commit/e20a27745a08d552e6d2c5a8bbaf08ccfe89c68f))
-* ignore NotFound errors when deleting log files from object store ([#9707](https://github.com/windmill-labs/windmill/issues/9707)) ([8a0b0ab](https://github.com/windmill-labs/windmill/commit/8a0b0abead71320c4f69eb3007739a19f76d4126))
-* **oauth:** restore bring-your-own CC token URL override ([#9711](https://github.com/windmill-labs/windmill/issues/9711)) ([ef4962e](https://github.com/windmill-labs/windmill/commit/ef4962e52aba0bc79bf72523de9101853c660654))
-* sanitize git credentials from ansible executor errors and logs ([#9697](https://github.com/windmill-labs/windmill/issues/9697)) ([ace7b68](https://github.com/windmill-labs/windmill/commit/ace7b68a28b00d715298ffcb6ae907c1974a74b8))
-
-## [1.734.0](https://github.com/windmill-labs/windmill/compare/v1.733.1...v1.734.0) (2026-06-20)
-
-
-### Features
-
-* ducklake materialization for data pipelines ([#9689](https://github.com/windmill-labs/windmill/issues/9689)) ([3ebf243](https://github.com/windmill-labs/windmill/commit/3ebf24359d66048d6361ce65cd879cdc04b737ed))
-
-
-### Bug Fixes
-
-* **frontend:** clear branch step state when switching outer loop iterations ([#9650](https://github.com/windmill-labs/windmill/issues/9650)) ([09a8004](https://github.com/windmill-labs/windmill/commit/09a80040ca268a4379d5302e3401435ba93247e0))
-
-## [1.733.1](https://github.com/windmill-labs/windmill/compare/v1.733.0...v1.733.1) (2026-06-19)
-
-
-### Bug Fixes
-
-* **backend:** validate ansible vault_id entries before config generation ([#9681](https://github.com/windmill-labs/windmill/issues/9681)) ([c1f31c0](https://github.com/windmill-labs/windmill/commit/c1f31c0e4777bf0cfed0dd7f03249e9a61cd8cb9))
-* **frontend:** group live pipeline runs in the activity panel ([#9684](https://github.com/windmill-labs/windmill/issues/9684)) ([1be4df9](https://github.com/windmill-labs/windmill/commit/1be4df9acb935250d4cc12e83cf67e366d870d5a))
-* require super admin for object storage config test endpoint ([#9683](https://github.com/windmill-labs/windmill/issues/9683)) ([fb44fe7](https://github.com/windmill-labs/windmill/commit/fb44fe7af2b8ebe8ef64ffb0e5acce8580bf4200))
-* validate websocket trigger urls and gate trigger test route ([#9682](https://github.com/windmill-labs/windmill/issues/9682)) ([c39ee07](https://github.com/windmill-labs/windmill/commit/c39ee07c0bcd2249dd19ffa5cd988125eefc6c9f))
-
-## [1.733.0](https://github.com/windmill-labs/windmill/compare/v1.732.0...v1.733.0) (2026-06-19)
-
-
-### Features
-
-* **ai-chat:** cap read_app_file + search_app grep tool to bound context in large raw apps ([#9653](https://github.com/windmill-labs/windmill/issues/9653)) ([4296a6a](https://github.com/windmill-labs/windmill/commit/4296a6ae1f73564de4df54fe1df0a03c1df05dfd))
-* **python, windows:** enable S3 to cache wheels ([#5199](https://github.com/windmill-labs/windmill/issues/5199)) ([ab3bc97](https://github.com/windmill-labs/windmill/commit/ab3bc97cd92b6480327029bcf018280442462af7))
-
-
-### Bug Fixes
-
-* allow users to always discard their own drafts without write permission ([#9659](https://github.com/windmill-labs/windmill/issues/9659)) ([6833a55](https://github.com/windmill-labs/windmill/commit/6833a554aeddb3e63173d3c3140b490c0bf2822b))
-* **backend:** clean up unique_ext_jwt_token on workspace deletion ([#9676](https://github.com/windmill-labs/windmill/issues/9676)) ([9add719](https://github.com/windmill-labs/windmill/commit/9add719d936cdcfb2c4062629e3e1f792694dafe))
-* **backend:** strip NUL bytes from draft values on write ([#9673](https://github.com/windmill-labs/windmill/issues/9673)) ([924f9c7](https://github.com/windmill-labs/windmill/commit/924f9c7e8d8863d9af40aee246a519b4be0e1ea2))
-* **python:** split PIP_TRUSTED_HOST by whitespace to support multiple hosts ([#9675](https://github.com/windmill-labs/windmill/issues/9675)) ([cafb473](https://github.com/windmill-labs/windmill/commit/cafb473494d9cff3a8b2aeaf9f18b015f966e7b3))
-
-## [1.732.0](https://github.com/windmill-labs/windmill/compare/v1.731.0...v1.732.0) (2026-06-19)
-
-
-### Features
-
-* **ansible:** add AI chat and editor bar buttons for ansible ([#9671](https://github.com/windmill-labs/windmill/issues/9671)) ([017c3d3](https://github.com/windmill-labs/windmill/commit/017c3d3343c2577501103be4b2dd8dac9727d80d))
-
-
-### Bug Fixes
-
-* **ai:** emit token usage in gemini proxy streaming translation ([#9669](https://github.com/windmill-labs/windmill/issues/9669)) ([0cc2257](https://github.com/windmill-labs/windmill/commit/0cc2257596a3965cf6db21a0090edcce6e1b8419))
-* **backend:** grant script_trigger access to windmill roles ([#9674](https://github.com/windmill-labs/windmill/issues/9674)) ([3361736](https://github.com/windmill-labs/windmill/commit/33617367d09537667d2ab3f91135c736194b9e7e))
-* **frontend:** ignore hash/assets in script diffs and drafts (WIN-2071) ([#9664](https://github.com/windmill-labs/windmill/issues/9664)) ([3371265](https://github.com/windmill-labs/windmill/commit/33712653821e83f2562dd5f271dbec0188d5d2f8))
-
-## [1.731.0](https://github.com/windmill-labs/windmill/compare/v1.730.0...v1.731.0) (2026-06-19)
-
-
-### Features
-
-* **backend:** auto-reconnect postgres trigger listener with backoff (WIN-2073) ([#9666](https://github.com/windmill-labs/windmill/issues/9666)) ([a425431](https://github.com/windmill-labs/windmill/commit/a425431e9067bcf85474fdc7b7ef7f73e41b9071))
-
-
-### Bug Fixes
-
-* **backend:** grant notify_event access to windmill roles ([#9665](https://github.com/windmill-labs/windmill/issues/9665)) ([a682d02](https://github.com/windmill-labs/windmill/commit/a682d02311a2110bfc0d5e0a5b52e96147fe0dd7))
-* **mcp:** repair invalid type keywords in tool JSON schemas ([#9667](https://github.com/windmill-labs/windmill/issues/9667)) ([c30bdec](https://github.com/windmill-labs/windmill/commit/c30bdecea77ff9b4d74d52961f3101201099b683))
-* trigger flow error handler on unrecoverable (OOM/zombie) step failures ([#9662](https://github.com/windmill-labs/windmill/issues/9662)) ([7e4df02](https://github.com/windmill-labs/windmill/commit/7e4df02bd60c4d6ee8c92d3dfd19f4e587ff9632))
-
-## [1.730.0](https://github.com/windmill-labs/windmill/compare/v1.729.0...v1.730.0) (2026-06-18)
-
-
-### Features
-
-* **ai-chat:** summary-based conversation compaction ([#9645](https://github.com/windmill-labs/windmill/issues/9645)) ([5d553b8](https://github.com/windmill-labs/windmill/commit/5d553b81c06664aab61131a93b198575c088d12d))
-* Data Pipelines alpha ([#9193](https://github.com/windmill-labs/windmill/issues/9193)) ([7155a0b](https://github.com/windmill-labs/windmill/commit/7155a0bb96cf30bd878272a0f4c3c3b02341b261))
-
-
-### Bug Fixes
-
-* **ai-chat:** stop echoing app draft value in global chat write tool results ([#9658](https://github.com/windmill-labs/windmill/issues/9658)) ([2fed808](https://github.com/windmill-labs/windmill/commit/2fed808b9e716d9a44b34c7a073ec0d37374be05))
-* **backend:** include raw_app drafts in list_apps draft_users ([#9647](https://github.com/windmill-labs/windmill/issues/9647)) ([19bc005](https://github.com/windmill-labs/windmill/commit/19bc0052f1069d732231950a0ec958f675d57417))
-* **frontend:** keep ?new_draft flag until first save is confirmed ([#9656](https://github.com/windmill-labs/windmill/issues/9656)) ([9b6b7c3](https://github.com/windmill-labs/windmill/commit/9b6b7c3862d9988e5e91eaab2b967a23f41cdc0d))
-* **frontend:** re-key raw-app autosave on post-deploy navigation ([#9646](https://github.com/windmill-labs/windmill/issues/9646)) ([1058bde](https://github.com/windmill-labs/windmill/commit/1058bdeccdc4c403ef4599db0ee74a65a66c715f))
-* gate agent-worker global setting reads with a blocklist ([#9623](https://github.com/windmill-labs/windmill/issues/9623)) ([fdd82f0](https://github.com/windmill-labs/windmill/commit/fdd82f0c48f29805cd9e219649f27fba45c7fd92))
-* **workspaces:** add instance setting to disable workspace invite/add emails ([#9643](https://github.com/windmill-labs/windmill/issues/9643)) ([796230d](https://github.com/windmill-labs/windmill/commit/796230d90a7e6d1debc15e139ab708881e527862))
-
-## [1.729.0](https://github.com/windmill-labs/windmill/compare/v1.728.1...v1.729.0) (2026-06-18)
-
-
-### Features
-
-* add ducklake schema support to the database manager ([#9633](https://github.com/windmill-labs/windmill/issues/9633)) ([3eeccaf](https://github.com/windmill-labs/windmill/commit/3eeccaf9682b7803fdf5be8dcbc4d243e0ba2e49))
-* **ai-chat:** self-hosted docs tools via windmill.dev llms.txt + ask benchmark ([#9578](https://github.com/windmill-labs/windmill/issues/9578)) ([f4425fc](https://github.com/windmill-labs/windmill/commit/f4425fca9fb0d02b845bd72888ade54905c5a30b))
-* **frontend:** View Diff and in-place Load for other users' drafts ([#9621](https://github.com/windmill-labs/windmill/issues/9621)) ([5508f1d](https://github.com/windmill-labs/windmill/commit/5508f1da9cd04c2583eb3f7ee6bce19d067f2227))
-* per-user draft review & deploy page (gating, badges, rename, raw-app deploy fixes) ([#9625](https://github.com/windmill-labs/windmill/issues/9625)) ([e09cd58](https://github.com/windmill-labs/windmill/commit/e09cd5862cb636e143027fe8d9a5be9c7097b031))
-* queue messages typed while ai chat is streaming ([#9525](https://github.com/windmill-labs/windmill/issues/9525)) ([51bd869](https://github.com/windmill-labs/windmill/commit/51bd8692a482850f7ac8b04dd16db5876336b5b9))
-* zero-setup oauth client credentials for registry providers ([#9559](https://github.com/windmill-labs/windmill/issues/9559)) ([e26a923](https://github.com/windmill-labs/windmill/commit/e26a9239a62a25abf90ef06ade4dde7f36e791bb))
-
-
-### Bug Fixes
-
-* **ai_evals:** adapt global eval harness to DB-backed user drafts ([#9641](https://github.com/windmill-labs/windmill/issues/9641)) ([e87ff79](https://github.com/windmill-labs/windmill/commit/e87ff79ecf6a6e0958916ed1b3756fb3addf719f))
-* **drafts:** preserve original timestamp when migrating localStorage drafts ([#9638](https://github.com/windmill-labs/windmill/issues/9638)) ([8021775](https://github.com/windmill-labs/windmill/commit/8021775f5f961ef6fd01b022639b85855326a1da))
-* **frontend:** don't save drafts on leave when auto-save is off, warn instead ([#9630](https://github.com/windmill-labs/windmill/issues/9630)) ([2523465](https://github.com/windmill-labs/windmill/commit/252346500945a9571af744c839ac0c7d6870504f))
-* **frontend:** render Modal2 dialogs above the AI chat panel ([#9636](https://github.com/windmill-labs/windmill/issues/9636)) ([b67c8cf](https://github.com/windmill-labs/windmill/commit/b67c8cf42b477575fc1bc448058ec0d3b7e54fee))
-* **frontend:** show AI sessions when AI unconfigured, with disabled chat ([#9644](https://github.com/windmill-labs/windmill/issues/9644)) ([ba69d81](https://github.com/windmill-labs/windmill/commit/ba69d8147b615e160cf3d2885fc65a0777b78b71))
-* **git-sync:** bump default sync script to hub/28719 (windmill-cli 1.728.1) for WAC modules ([#9649](https://github.com/windmill-labs/windmill/issues/9649)) ([3c0e38b](https://github.com/windmill-labs/windmill/commit/3c0e38b5890d77983cb5cf5f422a62a73e7a4f22))
-
-## [1.728.1](https://github.com/windmill-labs/windmill/compare/v1.728.0...v1.728.1) (2026-06-17)
-
-
-### Bug Fixes
-
-* **backend:** purge workspace_diff cache on workspace delete ([#9627](https://github.com/windmill-labs/windmill/issues/9627)) ([8a3f69d](https://github.com/windmill-labs/windmill/commit/8a3f69dda8f2088fb859ed8ed6e54458940423d0))
-* **cli:** fall back to esbuild-wasm on native host/binary mismatch ([#9629](https://github.com/windmill-labs/windmill/issues/9629)) ([86d1d16](https://github.com/windmill-labs/windmill/commit/86d1d160f0d3bd9faabdafada07e2956dd98445d))
-* **frontend:** persist session-editor draft path/summary edits + per-line diff tooltips ([#9622](https://github.com/windmill-labs/windmill/issues/9622)) ([e4bfeb2](https://github.com/windmill-labs/windmill/commit/e4bfeb29bc4e89669863b5f6396904a331167658))
-
-## [1.728.0](https://github.com/windmill-labs/windmill/compare/v1.727.0...v1.728.0) (2026-06-16)
-
-
-### Features
-
-* **frontend:** adapt AI-chat/sessions drafts to DB-backed model ([#9601](https://github.com/windmill-labs/windmill/issues/9601)) ([611c70a](https://github.com/windmill-labs/windmill/commit/611c70acd211cf4b8f8308da4a264c670a2f5f43))
-* **frontend:** consolidate draft-migration errors into a single toast + modal ([#9612](https://github.com/windmill-labs/windmill/issues/9612)) ([bc0d5bf](https://github.com/windmill-labs/windmill/commit/bc0d5bf241df3633921bd9d43d171e91034fbfcf))
-* **frontend:** dedup user drafts against the deployed baseline ([#9618](https://github.com/windmill-labs/windmill/issues/9618)) ([a2ce446](https://github.com/windmill-labs/windmill/commit/a2ce44645fdbfa98bf250fac2d15d2b5b26c4b47))
-
-
-### Bug Fixes
-
-* **frontend:** reset deleteWorkspaceForkModal on confirm in SidebarContent ([#9619](https://github.com/windmill-labs/windmill/issues/9619)) ([7cb5c6e](https://github.com/windmill-labs/windmill/commit/7cb5c6e749b2020dee5ee1499f0dc69c5109a6d8))
-* **frontend:** session Drafts drawer uses raw_app kind for the raw-app diff ([#9617](https://github.com/windmill-labs/windmill/issues/9617)) ([46288b6](https://github.com/windmill-labs/windmill/commit/46288b6143efae4dfdf6fe068b97a1e8831fce6a))
-* **nativets:** respect custom CA certs in in-process fetch runtime ([#9615](https://github.com/windmill-labs/windmill/issues/9615)) ([41562c7](https://github.com/windmill-labs/windmill/commit/41562c7d7c708d7d056d9b3d0c39b994a6f4a016))
-* **ResourceForm:** initialize JSON editor when resource type schema is unavailable ([#9611](https://github.com/windmill-labs/windmill/issues/9611)) ([5a24057](https://github.com/windmill-labs/windmill/commit/5a2405743b4622fc1021109114d007057abd5dfd))
-* show folder labels in the folder list table ([#9620](https://github.com/windmill-labs/windmill/issues/9620)) ([651fa13](https://github.com/windmill-labs/windmill/commit/651fa13ee80ff76e5a53ef1ed545b03ce6792294))
-* show last updated date per user in other-users-drafts modal ([#9614](https://github.com/windmill-labs/windmill/issues/9614)) ([f6104ce](https://github.com/windmill-labs/windmill/commit/f6104ce05c4005ffb9fe8112782d1ef6d3065300))
-
-## [1.727.0](https://github.com/windmill-labs/windmill/compare/v1.726.1...v1.727.0) (2026-06-16)
-
-
-### Features
-
-* support temp_script_refs in wmill dev for local relative imports ([#9554](https://github.com/windmill-labs/windmill/issues/9554)) ([33ac287](https://github.com/windmill-labs/windmill/commit/33ac287065742df53f363a5fe09f54f5584a85a6))
-
-
-### Bug Fixes
-
-* **cli:** harden legacy flow lock migration ordering and collision guard ([#9557](https://github.com/windmill-labs/windmill/issues/9557)) ([cd09870](https://github.com/windmill-labs/windmill/commit/cd098700c2cd8d7e9150f760938f4eaf34d188ec))
-* **cli:** include __mod/ folder in gitSyncIncludePattern for scripts ([#9606](https://github.com/windmill-labs/windmill/issues/9606)) ([252c1b3](https://github.com/windmill-labs/windmill/commit/252c1b35fc716c3486109d89615127c588bbe90a))
-* **frontend:** allow same-origin redirects in isValidLogoutRedirect ([#9568](https://github.com/windmill-labs/windmill/issues/9568)) ([8500435](https://github.com/windmill-labs/windmill/commit/8500435e82231e13a1b8a874fd0545f0a0a73fee))
-* **frontend:** make UserDraft read-after-write work without live entry ([#9609](https://github.com/windmill-labs/windmill/issues/9609)) ([51e82d7](https://github.com/windmill-labs/windmill/commit/51e82d7c6d30c66c84236feb743c09929934e564))
-* **frontend:** seed detached user-draft handles so new-item drawers render ([#9608](https://github.com/windmill-labs/windmill/issues/9608)) ([9e3c0de](https://github.com/windmill-labs/windmill/commit/9e3c0decf95378c66055d82215c15cd3bf4a69cb))
-* **frontend:** strip server-managed fields from value diffs ([#9599](https://github.com/windmill-labs/windmill/issues/9599)) ([c213801](https://github.com/windmill-labs/windmill/commit/c213801b5aee54d801c14b9eb31422f2a312ef7e))
-
-## [1.726.1](https://github.com/windmill-labs/windmill/compare/v1.726.0...v1.726.1) (2026-06-15)
-
-
-### Bug Fixes
-
-* **apps:** prevent decision tree graph editor crash on missing graph context ([#9602](https://github.com/windmill-labs/windmill/issues/9602))  ([24f3259](https://github.com/windmill-labs/windmill/commit/24f32596e9ca39c963c19af4a8b14fbcd04e3a78))
-* db-backed draft fixes — review-page UX, legacy drafts, session restore ([#9600](https://github.com/windmill-labs/windmill/issues/9600)) ([4e4b224](https://github.com/windmill-labs/windmill/commit/4e4b2247ef471dada1b8c894974fe921d14b3947))
-
-## [1.726.0](https://github.com/windmill-labs/windmill/compare/v1.725.1...v1.726.0) (2026-06-15)
-
-
-### Features
-
-* **audit:** record workspace archive/unarchive/delete in instance audit log ([#9596](https://github.com/windmill-labs/windmill/issues/9596)) ([9de5708](https://github.com/windmill-labs/windmill/commit/9de57086086bb5626d175c7f926915d1d6ac67ca))
-* **frontend:** add user-level toggle to disable Windmill AI ([#9585](https://github.com/windmill-labs/windmill/issues/9585)) ([5709a56](https://github.com/windmill-labs/windmill/commit/5709a564fbafd9aa91943572ecd8c3e0c45c20b1))
-
-
-### Bug Fixes
-
-* **embeddings:** retry HuggingFace model downloads with backoff ([#9597](https://github.com/windmill-labs/windmill/issues/9597)) ([6a62959](https://github.com/windmill-labs/windmill/commit/6a6295921d681359155d814507908792be405679))
-* resolve release CI failures (pypi bundle, flow serde test, cli windows) ([#9595](https://github.com/windmill-labs/windmill/issues/9595)) ([5ccaae8](https://github.com/windmill-labs/windmill/commit/5ccaae8ab36f2be18b67863ea069763455908029))
-
-## [1.725.1](https://github.com/windmill-labs/windmill/compare/v1.725.0...v1.725.1) (2026-06-15)
-
-
-### Bug Fixes
-
-* **apps:** apply scope-path predicate to app list/search endpoints ([#9581](https://github.com/windmill-labs/windmill/issues/9581)) ([3bf6e10](https://github.com/windmill-labs/windmill/commit/3bf6e102afbdad41e558617bc812012eaaaecd9b))
-* **auth:** add scope checks to scripts/flows list_tokens endpoints ([#9582](https://github.com/windmill-labs/windmill/issues/9582)) ([36c9f86](https://github.com/windmill-labs/windmill/commit/36c9f8612b5778aa2c981454729590b71671ce8d))
-* **cli:** preserve committed script.lock on transient NULL lock during git-sync deploy ([#9593](https://github.com/windmill-labs/windmill/issues/9593)) ([6b916ac](https://github.com/windmill-labs/windmill/commit/6b916ac688e0305284e6cf819bf28803bcca0118))
-* expose parent_hash in MCP createScript tool for updates ([#9586](https://github.com/windmill-labs/windmill/issues/9586)) ([a69505d](https://github.com/windmill-labs/windmill/commit/a69505df9bf25d7c4f11d0528a7450c08dbb422c))
-* **flows:** stop serializing default retry/stop_after_if fields ([#9583](https://github.com/windmill-labs/windmill/issues/9583)) ([e1e2a24](https://github.com/windmill-labs/windmill/commit/e1e2a24b6a6752b3ac779cb0db38061cbc54425e))
-* **security:** sanitize dependency names & connection strings against command/SQL injection ([#9590](https://github.com/windmill-labs/windmill/issues/9590)) ([aff0a4e](https://github.com/windmill-labs/windmill/commit/aff0a4ec189cd8e315282e878bb858ef00635b90))
-
-## [1.725.0](https://github.com/windmill-labs/windmill/compare/v1.724.0...v1.725.0) (2026-06-15)
-
-
-### Features
-
-* Db-backed user drafts ([#9351](https://github.com/windmill-labs/windmill/issues/9351)) ([1fc3557](https://github.com/windmill-labs/windmill/commit/1fc355709c025fd256c5a4035356e15a5a05b23d))
-* scope AI session storage per user, session list in IndexedDB ([#9518](https://github.com/windmill-labs/windmill/issues/9518)) ([aa26c4d](https://github.com/windmill-labs/windmill/commit/aa26c4d9b22b3a353a6c0605eb9a4193e34aa18c))
-
-
-### Bug Fixes
-
-* **powershell:** sanitize module names to prevent command injection (CWE-78) ([#9587](https://github.com/windmill-labs/windmill/issues/9587)) ([6acce7a](https://github.com/windmill-labs/windmill/commit/6acce7a88733683153db534cb18752b31d93af82))
-
-## [1.724.0](https://github.com/windmill-labs/windmill/compare/v1.723.0...v1.724.0) (2026-06-15)
-
-
-### Features
-
-* **cli:** add --yes, --secret/--no-secret and --description to variable add ([#9548](https://github.com/windmill-labs/windmill/issues/9548)) ([4e9e0c0](https://github.com/windmill-labs/windmill/commit/4e9e0c024b4b95f9676b1646591d8f0c662e84ab))
-* **frontend:** improve AI chat cancel and interrupted-turn handling ([#9539](https://github.com/windmill-labs/windmill/issues/9539)) ([114c412](https://github.com/windmill-labs/windmill/commit/114c41251a8c738b58a1a3dd9434d09d33feb6f1))
-* **frontend:** precise AI chat context usage tracking + indicator ([#9551](https://github.com/windmill-labs/windmill/issues/9551)) ([2b47180](https://github.com/windmill-labs/windmill/commit/2b471805bf1c92bb210cfacda217a4341e1f989c))
-* wire chat reasoning effort through gemini and bedrock proxies ([#9545](https://github.com/windmill-labs/windmill/issues/9545)) ([aaf0563](https://github.com/windmill-labs/windmill/commit/aaf05635cedadc73455dc522474b673719f9fd5c))
-
-
-### Bug Fixes
-
-* actually isolate windows job children from CTRL_BREAK_EVENT + reap on worker death ([#9563](https://github.com/windmill-labs/windmill/issues/9563)) ([61f3291](https://github.com/windmill-labs/windmill/commit/61f3291b240bdb5c26bee8947351a9590bc3bd45))
-* **ai:** enforce resource authz when loading MCP tools in agent worker ([#9571](https://github.com/windmill-labs/windmill/issues/9571)) ([317a862](https://github.com/windmill-labs/windmill/commit/317a8629d1c8436d2a6f3443bd25b81d606ce283))
-* append system CA bundle to tracing proxy cert file ([#9549](https://github.com/windmill-labs/windmill/issues/9549)) ([3cf4083](https://github.com/windmill-labs/windmill/commit/3cf40839602e5c3d1df51f0a29b01736bade09da))
-* **cli:** consistent flow inline lock filenames for compound extensions ([#9555](https://github.com/windmill-labs/windmill/issues/9555)) ([f0659a7](https://github.com/windmill-labs/windmill/commit/f0659a755a161420833e3bfdbe04befc6ebeb977))
-* **flows:** skip_if evaluates wrong previous_result during retry ([#9547](https://github.com/windmill-labs/windmill/issues/9547)) ([2aab352](https://github.com/windmill-labs/windmill/commit/2aab35245c362c2f911c60ea435f29bfb1369ebf))
-* **folders:** allow hyphens in folder names ([#9566](https://github.com/windmill-labs/windmill/issues/9566)) ([84df111](https://github.com/windmill-labs/windmill/commit/84df11177f2009bff007e9b722b55a9a5a63c06a)), closes [#8474](https://github.com/windmill-labs/windmill/issues/8474)
-* **frontend:** load resource value in JSON editor when resource type is missing ([#9574](https://github.com/windmill-labs/windmill/issues/9574)) ([251266c](https://github.com/windmill-labs/windmill/commit/251266cd8119dbef314314daed43aa43ab92f1c9))
-* isolate windows job children from worker CTRL_BREAK_EVENT ([#9562](https://github.com/windmill-labs/windmill/issues/9562)) ([1d6191e](https://github.com/windmill-labs/windmill/commit/1d6191ebb75843917eec6c76a4be347f9ac4cb72))
-* stop sending temperature for AI chat across all providers ([#9553](https://github.com/windmill-labs/windmill/issues/9553)) ([3585716](https://github.com/windmill-labs/windmill/commit/358571687296fbe5c378533b3c1662707955c64a))
-
-## [1.723.0](https://github.com/windmill-labs/windmill/compare/v1.722.0...v1.723.0) (2026-06-11)
-
-
-### Features
-
-* add get_app_runtime_logs tool to global chat ([#9502](https://github.com/windmill-labs/windmill/issues/9502)) ([f86d0d7](https://github.com/windmill-labs/windmill/commit/f86d0d79fc6aa23119fd59761330ab79592d0a2a))
-* **cli:** improve agent prompts/skills and workspace fork workflow ([#9531](https://github.com/windmill-labs/windmill/issues/9531)) ([5bdc4f8](https://github.com/windmill-labs/windmill/commit/5bdc4f83ce37302a2c375d0ff73763acfee2aadb))
-* enable native web search in copilot ([#9522](https://github.com/windmill-labs/windmill/issues/9522)) ([d3f5fe1](https://github.com/windmill-labs/windmill/commit/d3f5fe1c8c39ff07d05f0922b8f40aa95756a707))
-
-
-### Bug Fixes
-
-* **frontend:** stop live activity flickering when user has multiple tabs ([#9543](https://github.com/windmill-labs/windmill/issues/9543)) ([57e627e](https://github.com/windmill-labs/windmill/commit/57e627eabf7c4144ce1c07214ad44d026b82f0b4))
-* omit temperature for claude fable 5 ([#9540](https://github.com/windmill-labs/windmill/issues/9540)) ([bd00bee](https://github.com/windmill-labs/windmill/commit/bd00beeac54dbcfa9ab86fd336fca1a8fa289341))
-* refetch license key from settings when in-memory key is invalid ([#9534](https://github.com/windmill-labs/windmill/issues/9534)) ([38c0ccd](https://github.com/windmill-labs/windmill/commit/38c0ccdf563d3655a4cba390ce9372bc9f9c4a9b))
-
-## [1.722.0](https://github.com/windmill-labs/windmill/compare/v1.721.0...v1.722.0) (2026-06-11)
-
-
-### Features
-
-* add reasoning effort control and thinking display to AI chat ([#9511](https://github.com/windmill-labs/windmill/issues/9511)) ([7f987e8](https://github.com/windmill-labs/windmill/commit/7f987e8c9807b72d9cc3901b6e4d02a24c423f50))
-* **ai-chat:** collapse big pastes, cap input height, escape HTML ([#9487](https://github.com/windmill-labs/windmill/issues/9487)) ([365e204](https://github.com/windmill-labs/windmill/commit/365e20410ed528d5b4e967b64fb282bcc1e03ccd))
-* **ai-chat:** quick access to AI prompt settings from chat ([#9508](https://github.com/windmill-labs/windmill/issues/9508)) ([b894f78](https://github.com/windmill-labs/windmill/commit/b894f783f183ab795eab6f57da8654275d9ad82d))
-* **ai:** add list_runs and get_job_logs tools to global chat mode ([#9488](https://github.com/windmill-labs/windmill/issues/9488)) ([cfe5119](https://github.com/windmill-labs/windmill/commit/cfe51190356a9e922f6398dd875abc368accbd15))
-* clear conflict error + force delete when reusing a fork workspace id ([#9499](https://github.com/windmill-labs/windmill/issues/9499)) ([fddabe9](https://github.com/windmill-labs/windmill/commit/fddabe9c5c6f178b4b09854dc11e50adafd44c87))
-* **flow:** support worker tag override on AI agent steps ([#9513](https://github.com/windmill-labs/windmill/issues/9513)) ([a6a5600](https://github.com/windmill-labs/windmill/commit/a6a5600833063e36b35aac7f90dcbdcd3db7c627))
-* folder-level label inheritance for scripts, flows and jobs ([#9524](https://github.com/windmill-labs/windmill/issues/9524)) ([765f50c](https://github.com/windmill-labs/windmill/commit/765f50c474f8abf76550664ed15418ddb3c0b221))
-* **frontend:** show AI sessions in narrow-screen burger menu ([#9523](https://github.com/windmill-labs/windmill/issues/9523)) ([a8f1062](https://github.com/windmill-labs/windmill/commit/a8f1062f37228e7a8257aac1cd3295bd5084ff90))
-* prefer idle worker pods on k8s autoscaling scale-in via pod-deletion-cost ([#9515](https://github.com/windmill-labs/windmill/issues/9515)) ([3c3f157](https://github.com/windmill-labs/windmill/commit/3c3f15722fd22713cc8baa744d9a81207943a33c))
-* prompt browser confirmation on page exit with unsaved changes ([#9503](https://github.com/windmill-labs/windmill/issues/9503)) ([3119e16](https://github.com/windmill-labs/windmill/commit/3119e16ed8df0daec0e019ae2efb2176d7e9e953))
-* **worker:** #ssh directive to run a bash script on a remote SSH host ([#9479](https://github.com/windmill-labs/windmill/issues/9479)) ([afddfe8](https://github.com/windmill-labs/windmill/commit/afddfe84452357b06f4fb815566d4c8269ceafbf))
-* workspace protection rule to restrict anonymous app deployment ([#9509](https://github.com/windmill-labs/windmill/issues/9509)) ([cf9ad54](https://github.com/windmill-labs/windmill/commit/cf9ad54181d38c5d3c3aa05208e17d8d97eae4ef))
-
-
-### Bug Fixes
-
-* **cli:** include lock-relevant script content in lock cache key ([#9528](https://github.com/windmill-labs/windmill/issues/9528)) ([dc60e1a](https://github.com/windmill-labs/windmill/commit/dc60e1aa174f2a7616d2d4a37117a463ef25a9d6))
-* **frontend:** allow copy/paste shortcuts inside ConfirmationModal ([#9505](https://github.com/windmill-labs/windmill/issues/9505)) ([7fc5340](https://github.com/windmill-labs/windmill/commit/7fc5340da39c049d059aa0ae15ceb6a98ca58053))
-* **frontend:** clarify trigger filters match the message parsed as JSON ([#9516](https://github.com/windmill-labs/windmill/issues/9516)) ([0b17843](https://github.com/windmill-labs/windmill/commit/0b178437ce8cd8295be21c0c6077c39e957a3337))
-* **frontend:** enable Apply button when env vars change in worker group config ([#9501](https://github.com/windmill-labs/windmill/issues/9501)) ([c80c6d8](https://github.com/windmill-labs/windmill/commit/c80c6d8fcdfc1de8fe841e9699452f67a25cad23))
-* **frontend:** improve AI chat markdown and typing dots in dark mode ([#9497](https://github.com/windmill-labs/windmill/issues/9497)) ([4e86806](https://github.com/windmill-labs/windmill/commit/4e868062d4c22c5574149d35235b7ad1abe805f3))
-* **frontend:** stop echoing draft values in global chat write tool results ([#9530](https://github.com/windmill-labs/windmill/issues/9530)) ([ce6e2f7](https://github.com/windmill-labs/windmill/commit/ce6e2f7ade25ca91c375f0de5f1be3f1c82ccb47))
-* inherit container NO_PROXY into MITM tracing proxy job exclusions ([#9492](https://github.com/windmill-labs/windmill/issues/9492)) ([4c22e3b](https://github.com/windmill-labs/windmill/commit/4c22e3b712a74828cf654ea7d89aeab5b50cfbd7))
-* make default chat model optional in AI settings ([#9514](https://github.com/windmill-labs/windmill/issues/9514)) ([1d43288](https://github.com/windmill-labs/windmill/commit/1d4328877fcc87352fb56de63f0570739d5a9dc4))
-* **nsjail:** make ansible collections mount non-mandatory ([#9510](https://github.com/windmill-labs/windmill/issues/9510)) ([08da7a1](https://github.com/windmill-labs/windmill/commit/08da7a121b4b835500dfc2bd943c4bdce912c63e))
-* **nsjail:** make ansible uv tools mount non-mandatory ([#9507](https://github.com/windmill-labs/windmill/issues/9507)) ([dc368a9](https://github.com/windmill-labs/windmill/commit/dc368a9669e812c593ad848266f645b6223501e6))
-
-## [1.721.0](https://github.com/windmill-labs/windmill/compare/v1.720.0...v1.721.0) (2026-06-09)
-
-
-### Features
-
-* deployed↔draft compare + AI-session draft bar ([#9435](https://github.com/windmill-labs/windmill/issues/9435)) ([b0b330c](https://github.com/windmill-labs/windmill/commit/b0b330c7864d0159af4b0f17dbb3c09bd015145b))
-
-
-### Bug Fixes
-
-* **cli:** reconcile case-only path drift during sync on case-insensitive filesystems (WIN-2020) ([#9485](https://github.com/windmill-labs/windmill/issues/9485)) ([c258928](https://github.com/windmill-labs/windmill/commit/c258928ab62adc1327913c21556520dfd1e5c24c))
-* drop archived items from fork compare (spurious 'not visible' warning) ([#9481](https://github.com/windmill-labs/windmill/issues/9481)) ([92c21bb](https://github.com/windmill-labs/windmill/commit/92c21bbe6586f3c285796a98692e976515a629d5))
-* require auth to view approval details when user_auth_required ([#9482](https://github.com/windmill-labs/windmill/issues/9482)) ([5f41ddd](https://github.com/windmill-labs/windmill/commit/5f41ddd3a592bcd504f94fc99060ca5d79c36190))
-
-## [1.720.0](https://github.com/windmill-labs/windmill/compare/v1.719.0...v1.720.0) (2026-06-08)
-
-
-### Features
-
-* allow private MCP server URLs ([#9470](https://github.com/windmill-labs/windmill/issues/9470)) ([3bc5800](https://github.com/windmill-labs/windmill/commit/3bc5800197db383ae6f708415701a4bdbe2e3345))
-* **api:** add endpoint to update token label ([#9474](https://github.com/windmill-labs/windmill/issues/9474)) ([e8e0701](https://github.com/windmill-labs/windmill/commit/e8e0701a360d0614c4c5a74f6410ba6ac0638caa))
-* **frontend:** use unified drill picker for AI chat @-mention dropdown ([#9159](https://github.com/windmill-labs/windmill/issues/9159)) ([64b089c](https://github.com/windmill-labs/windmill/commit/64b089cd23cca4601abb09f092a32becb80d9394))
-
-
-### Bug Fixes
-
-* center auth0/okta icons and respect currentColor ([#9457](https://github.com/windmill-labs/windmill/issues/9457)) ([5d0ef7d](https://github.com/windmill-labs/windmill/commit/5d0ef7dfd91b3021d125a1b34f81f0788f173786))
-* **forks:** keep trigger/schedule operational state owned by the parent - WIN-2019 ([#9476](https://github.com/windmill-labs/windmill/issues/9476)) ([192574a](https://github.com/windmill-labs/windmill/commit/192574ab8f98d9521a232fc8a4935d407b00cb3a))
-* **frontend:** respect forced column order for numeric column names ([#9463](https://github.com/windmill-labs/windmill/issues/9463)) ([44f5dd6](https://github.com/windmill-labs/windmill/commit/44f5dd6636d4b23aa55383b8b8abe4c2f73bc88d))
-* **frontend:** use ban icon for canceled jobs instead of hourglass ([#9478](https://github.com/windmill-labs/windmill/issues/9478)) ([fa86c62](https://github.com/windmill-labs/windmill/commit/fa86c62b6600e7d47dadf4706d7002706333d919))
-* gate native integration pickers behind non-operator check ([#9465](https://github.com/windmill-labs/windmill/issues/9465)) ([6156e23](https://github.com/windmill-labs/windmill/commit/6156e2372a785ccd0c6f29cb74e90bee69e76483))
-* **oauth:** persist refreshed token through configured secret backend ([#9471](https://github.com/windmill-labs/windmill/issues/9471)) ([76c0d97](https://github.com/windmill-labs/windmill/commit/76c0d970a18bf28ddc48dd746570e73486542606))
-* refresh session editor preview on breadcrumb target switch ([#9475](https://github.com/windmill-labs/windmill/issues/9475)) ([6d522b3](https://github.com/windmill-labs/windmill/commit/6d522b3989ace1f214bd049d910bc0d2a2a6893e))
-
-## [1.719.0](https://github.com/windmill-labs/windmill/compare/v1.718.0...v1.719.0) (2026-06-06)
-
-
-### Features
-
-* **otel:** connect jobs to the inbound distributed trace ([#9456](https://github.com/windmill-labs/windmill/issues/9456)) ([fad1a54](https://github.com/windmill-labs/windmill/commit/fad1a549d95c00d0746a48163c4f95fc69733e1a))
-
-
-### Bug Fixes
-
-* authenticate slack callback payload with per-workspace hmac ([#9461](https://github.com/windmill-labs/windmill/issues/9461)) ([fbdf81b](https://github.com/windmill-labs/windmill/commit/fbdf81ba5f77d282c025360ecee14138dd4cb4a2))
-* prevent token label collision bypassing job read access control ([#9462](https://github.com/windmill-labs/windmill/issues/9462)) ([e1e7af6](https://github.com/windmill-labs/windmill/commit/e1e7af6a25a44eb06b67332ce1efeae2a21e0c6d))
-* **python:** escape reserved-keyword step ids in wrapper codegen ([#9460](https://github.com/windmill-labs/windmill/issues/9460)) ([6a15a9b](https://github.com/windmill-labs/windmill/commit/6a15a9b152ad20be4b5c3de6000516da231e41e0)), closes [#8893](https://github.com/windmill-labs/windmill/issues/8893)
-
-## [1.718.0](https://github.com/windmill-labs/windmill/compare/v1.717.1...v1.718.0) (2026-06-05)
-
-
-### Features
-
-* **flows:** opt-in to include the stopping step's result in early-stop errors ([#9446](https://github.com/windmill-labs/windmill/issues/9446)) ([f2f0812](https://github.com/windmill-labs/windmill/commit/f2f0812a04c9256cfc8eba5e0dcf38d71d971410))
-* make C# dotnet target framework configurable via DOTNET_TARGET_FRAMEWORK ([#9454](https://github.com/windmill-labs/windmill/issues/9454)) ([9a609bf](https://github.com/windmill-labs/windmill/commit/9a609bf08ac1b6157dbdfb827fc01e771d71262e))
-* sandboxed daemonless container runtime via '# sandbox <image>' ([#9453](https://github.com/windmill-labs/windmill/issues/9453)) ([1727271](https://github.com/windmill-labs/windmill/commit/1727271e197b34026efeaf1b6561bb404a440baa))
-* **sandbox:** pull/extract images with crane instead of podman ([#9455](https://github.com/windmill-labs/windmill/issues/9455)) ([7590b28](https://github.com/windmill-labs/windmill/commit/7590b281085afd1fc2774e8fb37a4c0af3aedbad))
-
-
-### Bug Fixes
-
-* distinguish canceled jobs in runs ([#9452](https://github.com/windmill-labs/windmill/issues/9452)) ([9067787](https://github.com/windmill-labs/windmill/commit/90677872f6185eb0c81e0e84a426a54653818457))
-
-## [1.717.1](https://github.com/windmill-labs/windmill/compare/v1.717.0...v1.717.1) (2026-06-04)
-
-
-### Bug Fixes
-
-* invalidate relative-import cache when imported script changes ([#9443](https://github.com/windmill-labs/windmill/issues/9443)) ([f595787](https://github.com/windmill-labs/windmill/commit/f595787409a3fcda9278bbcf2cfcc80092f16460))
-
-## [1.717.0](https://github.com/windmill-labs/windmill/compare/v1.716.0...v1.717.0) (2026-06-04)
-
-
-### Features
-
-* let flow AI chat create and edit sticky notes ([#9412](https://github.com/windmill-labs/windmill/issues/9412)) ([e4e0984](https://github.com/windmill-labs/windmill/commit/e4e0984e55afd3c73f1c365cd0608493a9fd87ed))
-
-
-### Bug Fixes
-
-* **cli:** push whole raw app instead of treating frontend files as scripts ([#9442](https://github.com/windmill-labs/windmill/issues/9442)) ([b5a6a1e](https://github.com/windmill-labs/windmill/commit/b5a6a1eeab663c2d6aaec2c89eab7a550cb0bb6b))
-* read latest db draft for scripts/flows in global mode read tool ([#9441](https://github.com/windmill-labs/windmill/issues/9441)) ([819ba5e](https://github.com/windmill-labs/windmill/commit/819ba5e150ec9f5199919fbea50874fc156d0189))
-
-## [1.716.0](https://github.com/windmill-labs/windmill/compare/v1.715.0...v1.716.0) (2026-06-03)
-
-
-### Features
-
-* add metadata generation model setting ([#9418](https://github.com/windmill-labs/windmill/issues/9418)) ([cf5fefb](https://github.com/windmill-labs/windmill/commit/cf5fefb521479170b9dc64b884630c4dac789931))
-* auto-generate AI session names ([#9399](https://github.com/windmill-labs/windmill/issues/9399)) ([26b7270](https://github.com/windmill-labs/windmill/commit/26b727041830c9b741668a9ab73e2eb90c7cec74))
-* support $f/ and $u/ import path aliases for scripts ([#9378](https://github.com/windmill-labs/windmill/issues/9378)) ([220cd35](https://github.com/windmill-labs/windmill/commit/220cd35cf799c42ebf588bc97a6d8e6f4e97c2e3))
-* use metadata model for small AI tasks ([#9431](https://github.com/windmill-labs/windmill/issues/9431)) ([79178f6](https://github.com/windmill-labs/windmill/commit/79178f6f5a7c606a2e05677c6efcbdd84c608325))
-
-
-### Bug Fixes
-
-* **apps:** relock no longer reverts raw app to a stale version ([#9432](https://github.com/windmill-labs/windmill/issues/9432)) ([073857a](https://github.com/windmill-labs/windmill/commit/073857ac0a9ed54bdeac8f373f7c855fe34eb0ac))
-* **security:** scope variable and resource value caches by caller identity ([#9427](https://github.com/windmill-labs/windmill/issues/9427)) ([0ba128a](https://github.com/windmill-labs/windmill/commit/0ba128afe797bd016da60563949ac3abbbfe1978))
-
-## [1.715.0](https://github.com/windmill-labs/windmill/compare/v1.714.1...v1.715.0) (2026-06-03)
-
-
-### Features
-
-* **frontend:** add rebuild dependency map button to workspace settings ([#9424](https://github.com/windmill-labs/windmill/issues/9424)) ([3b2e748](https://github.com/windmill-labs/windmill/commit/3b2e748daf0a8ec4447c30423068df803f3f9ca2))
-
-
-### Bug Fixes
-
-* **auth:** filter script/flow listings by token scope (GHSA-2ppx-66jv-wpw5) ([#9426](https://github.com/windmill-labs/windmill/issues/9426)) ([7edf3f0](https://github.com/windmill-labs/windmill/commit/7edf3f02122e20fde1e95e0252e7bda641075326))
-* **backend:** authorize single-job read endpoints by job/flow visibility ([#9416](https://github.com/windmill-labs/windmill/issues/9416)) ([89a7a37](https://github.com/windmill-labs/windmill/commit/89a7a377764086911db18252f2478f42f0e1e3ea))
-* **mcp:** resolve MCP resource token via caller RLS + SSRF-guard url ([#9428](https://github.com/windmill-labs/windmill/issues/9428)) ([8053266](https://github.com/windmill-labs/windmill/commit/8053266f88bd4c94fc86278412df5a0beeed5e77))
-* **nsjail:** precompile python stdlib + raise download rlimit_as ([#9429](https://github.com/windmill-labs/windmill/issues/9429)) ([7031744](https://github.com/windmill-labs/windmill/commit/7031744a199f0bf8b8e35043afa959977e5ecdbd))
-* omit temperature for gpt-5+ and o-series models on all providers ([#9422](https://github.com/windmill-labs/windmill/issues/9422)) ([11d1ad9](https://github.com/windmill-labs/windmill/commit/11d1ad9a872d2ec2f14cde35708c84a0c7bdc172))
-
-## [1.714.1](https://github.com/windmill-labs/windmill/compare/v1.714.0...v1.714.1) (2026-06-02)
-
-
-### Bug Fixes
-
-* **backend:** route //native TypeScript previews to native workers (WIN-2007) ([#9407](https://github.com/windmill-labs/windmill/issues/9407)) ([73edebc](https://github.com/windmill-labs/windmill/commit/73edebc833a981488a8ea116f4f13c020a011a6f))
-* **nsjail:** raise python download fd limit for --compile-bytecode (WIN-2009) ([#9414](https://github.com/windmill-labs/windmill/issues/9414)) ([9e6559a](https://github.com/windmill-labs/windmill/commit/9e6559a6f688cc8d982277b19920219ea6d0fd8e))
-* **triggers:** prevent Zoom challenge handler from being used as a signing oracle ([#9413](https://github.com/windmill-labs/windmill/issues/9413)) ([ab2a15b](https://github.com/windmill-labs/windmill/commit/ab2a15b2a859096eabde718bf6e60289ae187118))
-
-## [1.714.0](https://github.com/windmill-labs/windmill/compare/v1.713.1...v1.714.0) (2026-06-02)
-
-
-### Features
-
-* add global ai chat test tools ([#9391](https://github.com/windmill-labs/windmill/issues/9391)) ([5c20d6b](https://github.com/windmill-labs/windmill/commit/5c20d6b4f79f2ccc1987ce7fdaf74e6b8f697846))
-* add workspace datatable tools to global AI chat mode ([#9395](https://github.com/windmill-labs/windmill/issues/9395)) ([943ef6e](https://github.com/windmill-labs/windmill/commit/943ef6eb2089f4b744cfa7945ce47f7f3b361ec7))
-* **flow-ai:** constrain flow-group colors to the NoteColor palette ([#9343](https://github.com/windmill-labs/windmill/issues/9343)) ([e4213c1](https://github.com/windmill-labs/windmill/commit/e4213c1ab8c448f492f372580f5c9df37e33fffc))
-* **frontend:** surface local drafts in drawer editors with an unsaved-changes banner ([#9335](https://github.com/windmill-labs/windmill/issues/9335)) ([075faab](https://github.com/windmill-labs/windmill/commit/075faabf3bba16a10a02ae3973008e5a13473085))
-* handle CTRL_BREAK_EVENT for graceful shutdown on Windows ([#9400](https://github.com/windmill-labs/windmill/issues/9400)) ([2e14456](https://github.com/windmill-labs/windmill/commit/2e1445616a412c5112ad2247b4087c7ddc218845))
-* refine ask-user-question chat display and keyboard nav ([#9392](https://github.com/windmill-labs/windmill/issues/9392)) ([1275487](https://github.com/windmill-labs/windmill/commit/1275487f028d4c74a9eeb18981ed05c225505be0))
-* sessions page with isolated AI chat + flow editor ([#9034](https://github.com/windmill-labs/windmill/issues/9034)) ([eadeac2](https://github.com/windmill-labs/windmill/commit/eadeac248bd022c2796cfe638eb617c6143b8fc4))
-
-
-### Bug Fixes
-
-* **cli:** make encryption key push non-interactive-safe + add --skip-reencrypt-on-key-change ([#9402](https://github.com/windmill-labs/windmill/issues/9402)) ([e356bb1](https://github.com/windmill-labs/windmill/commit/e356bb1f5df92eca3fbb0ca2114b9f4c32d4c496))
-* **cli:** stop git-sync promotion deploys from dropping triggers/schedules ([#9403](https://github.com/windmill-labs/windmill/issues/9403)) ([24e3ef2](https://github.com/windmill-labs/windmill/commit/24e3ef27be8498fb820c228a52febf6a0a91b487))
-* **frontend:** align Monaco editor font size with text-xs ([#9161](https://github.com/windmill-labs/windmill/issues/9161)) ([de76668](https://github.com/windmill-labs/windmill/commit/de76668c10c04abe8771a8ca7bba7b2259819a1c))
-* resolve username rename failing on apps with runnable deps ([#9401](https://github.com/windmill-labs/windmill/issues/9401)) ([e8ad53d](https://github.com/windmill-labs/windmill/commit/e8ad53dae92597f5a1a8b76f38a7d8c24f578a47))
-
-
-### Performance Improvements
-
-* **python:** add --compile-bytecode to uv pip install ([#9393](https://github.com/windmill-labs/windmill/issues/9393)) ([c19441b](https://github.com/windmill-labs/windmill/commit/c19441bc8cb2da064e4ad44d77dc04ab8bbb22ec))
-
-## [1.713.1](https://github.com/windmill-labs/windmill/compare/v1.713.0...v1.713.1) (2026-06-01)
-
-
-### Bug Fixes
-
-* **api:** handle multi-version scripts when removing granular ACL ([#9388](https://github.com/windmill-labs/windmill/issues/9388)) ([9d9c503](https://github.com/windmill-labs/windmill/commit/9d9c5038ce8b0016320a670c434ef9063cb40441))
-
-## [1.713.0](https://github.com/windmill-labs/windmill/compare/v1.712.0...v1.713.0) (2026-05-31)
-
-
-### Features
-
-* **flows:** preserve step/subflow worker tags under a custom-tagged flow ([#9375](https://github.com/windmill-labs/windmill/issues/9375)) ([f0301b1](https://github.com/windmill-labs/windmill/commit/f0301b1605cee5fba4024803555333e6fa5c40ee))
-* **oauth:** support per-provider sandbox URLs ([#9358](https://github.com/windmill-labs/windmill/issues/9358)) ([2bf11dc](https://github.com/windmill-labs/windmill/commit/2bf11dcb15540c538ea2ac3cf70dcbe589060b4e))
-
-
-### Bug Fixes
-
-* **ai:** validate token_url for SSRF in OAuth credentials flow ([#9385](https://github.com/windmill-labs/windmill/issues/9385)) ([4b06881](https://github.com/windmill-labs/windmill/commit/4b06881918b76c5a411cc70b318e46efcc1393a7))
-* **api:** authorize and harden log-file reading endpoints ([#9368](https://github.com/windmill-labs/windmill/issues/9368)) ([bb90f4c](https://github.com/windmill-labs/windmill/commit/bb90f4ce83a0e60af219b11c12ab4fe1d13f47a4))
-* **apps:** make public apps opt into cross-origin isolation via wm_coep (GIT-884) ([#9374](https://github.com/windmill-labs/windmill/issues/9374)) ([2c0c2c4](https://github.com/windmill-labs/windmill/commit/2c0c2c467f163cd24c14c7be2db07af9cf2ce020))
-* **auth:** enforce monotonic privilege on user token lifecycle endpoints ([#9371](https://github.com/windmill-labs/windmill/issues/9371)) ([2ddf93d](https://github.com/windmill-labs/windmill/commit/2ddf93de96622b2a1b2b6f59398a7a1f59360efd))
-* batch encryption-key rotation into one git-sync job ([#9355](https://github.com/windmill-labs/windmill/issues/9355)) ([04a0897](https://github.com/windmill-labs/windmill/commit/04a08976aec4ba9b0516350316df303e9f96bfd3))
-* **cli:** preserve user drafts on sync push and permissioned-as ([#9381](https://github.com/windmill-labs/windmill/issues/9381)) ([b0c3b01](https://github.com/windmill-labs/windmill/commit/b0c3b01d31b0ab3a6566e1f5fec60e3e230cfadb))
-* **frontend:** sanitize user markdown to prevent stored XSS ([#9386](https://github.com/windmill-labs/windmill/issues/9386)) ([def01b8](https://github.com/windmill-labs/windmill/commit/def01b8ff6f331cc36ce02b947adc31c766042c4))
-* **security:** re-pin cached hub scripts to CVE-patched versions (+ HUB_BASE_URL override for cache mode) ([#9387](https://github.com/windmill-labs/windmill/issues/9387)) ([edf340c](https://github.com/windmill-labs/windmill/commit/edf340c4d4f18b16b142cb7deb67afa586f10946))
-
-## [1.712.0](https://github.com/windmill-labs/windmill/compare/v1.711.0...v1.712.0) (2026-05-28)
-
-
-### Features
-
-* add deepseek fim support ([#9365](https://github.com/windmill-labs/windmill/issues/9365)) ([2553fbf](https://github.com/windmill-labs/windmill/commit/2553fbfe31417bd985e7994eac695bf918f97ce2))
-* deploy raw apps from global chat ([#9349](https://github.com/windmill-labs/windmill/issues/9349)) ([dec58e6](https://github.com/windmill-labs/windmill/commit/dec58e6c4f55062b42a752c43c89ef05903e713a))
-* inject active editor into global chat ([#9361](https://github.com/windmill-labs/windmill/issues/9361)) ([9e7eaf3](https://github.com/windmill-labs/windmill/commit/9e7eaf36847ad3a004ec84e8b7d4784771b7b451))
-* **queue:** duration-weighted fairness admission ([#9334](https://github.com/windmill-labs/windmill/issues/9334)) ([045d120](https://github.com/windmill-labs/windmill/commit/045d12043e7c99830ef90bc0da798c94e2094711))
-* warn when custom instance db is shared across workspaces ([#9359](https://github.com/windmill-labs/windmill/issues/9359)) ([a9e5140](https://github.com/windmill-labs/windmill/commit/a9e514099585e5ee72df21bd551a223cceb20fb0))
-
-
-### Bug Fixes
-
-* **cli:** redact encryption_key diff in stdout by default ([#9347](https://github.com/windmill-labs/windmill/issues/9347)) ([88056f8](https://github.com/windmill-labs/windmill/commit/88056f8d4c91c1d14d85a08851ecf0bd97e2260d))
-* **cli:** stop re-prompting on wmill refresh prompts ([#9357](https://github.com/windmill-labs/windmill/issues/9357)) ([c2b5ba8](https://github.com/windmill-labs/windmill/commit/c2b5ba8871abbbcff6de69c90e2f09fee70586c1))
-* **frontend:** close other sidebar menus when hovering Help ([#9354](https://github.com/windmill-labs/windmill/issues/9354)) ([da882c5](https://github.com/windmill-labs/windmill/commit/da882c54b21e3eaf2c1d1abccd0996b243d96dce))
-* **frontend:** prevent duplicate asset node ids crashing flow graph ([#9367](https://github.com/windmill-labs/windmill/issues/9367)) ([9a659b6](https://github.com/windmill-labs/windmill/commit/9a659b636d713ee8fdfbdad41c58bb3d7c79e0d9))
-* **frontend:** prevent MultiSelect crash on undefined value ([#9364](https://github.com/windmill-labs/windmill/issues/9364)) ([aea0061](https://github.com/windmill-labs/windmill/commit/aea00611c41379be2afdad0eedd608c9537d03f7))
-* **git-sync:** publish fork branch on only_create_branch from the CLI ([#9366](https://github.com/windmill-labs/windmill/issues/9366)) ([2fdc51e](https://github.com/windmill-labs/windmill/commit/2fdc51e62985fc755884436130bdd58e294247c8))
-* infer script arg schema when deploying via AI chat ([#9356](https://github.com/windmill-labs/windmill/issues/9356)) ([4efc372](https://github.com/windmill-labs/windmill/commit/4efc37212a98571214aba135b0fbb10dc263fd4f))
-* **monitor:** cleanup stale server_heartbeat background_task_state rows ([#9338](https://github.com/windmill-labs/windmill/issues/9338)) ([59ab038](https://github.com/windmill-labs/windmill/commit/59ab038d7718d8a4c25efa5928f42e1393ebbf40))
-
-## [1.711.0](https://github.com/windmill-labs/windmill/compare/v1.710.1...v1.711.0) (2026-05-26)
-
-
-### Features
-
-* **cli:** add object-storage commands and flow test-step ([#9326](https://github.com/windmill-labs/windmill/issues/9326)) ([36f574f](https://github.com/windmill-labs/windmill/commit/36f574ff951198a4d40ee068a27d74c41ce32154))
-
-
-### Bug Fixes
-
-* **cli:** handle __flow suffix when deriving the flow's Windmill path ([#9333](https://github.com/windmill-labs/windmill/issues/9333)) ([6f77034](https://github.com/windmill-labs/windmill/commit/6f770346fb330997a836c39fba347df4c088a83c))
-* **queue:** duration-weighted workspace fairness signal ([#9329](https://github.com/windmill-labs/windmill/issues/9329)) ([42d2121](https://github.com/windmill-labs/windmill/commit/42d2121af925de50f549ecb72ffb5132f5c41079))
-
-## [1.710.1](https://github.com/windmill-labs/windmill/compare/v1.710.0...v1.710.1) (2026-05-26)
-
-
-### Bug Fixes
-
-* improve workspace fairness ([896add0](https://github.com/windmill-labs/windmill/commit/896add0350f4de31f5674d6be0907a582c5ec17e))
-
-## [1.710.0](https://github.com/windmill-labs/windmill/compare/v1.709.0...v1.710.0) (2026-05-26)
-
-
-### Features
-
-* **queue:** stochastic admission + EE availability of workspace fairness algorithm ([#9321](https://github.com/windmill-labs/windmill/issues/9321)) ([8bf7fd2](https://github.com/windmill-labs/windmill/commit/8bf7fd2c921c48861b71731a085b18ea8f72fb68))
-
-
-### Bug Fixes
-
-* **websocket-trigger:** honor HTTPS_PROXY/HTTP_PROXY/NO_PROXY ([#9324](https://github.com/windmill-labs/windmill/issues/9324)) ([6f36316](https://github.com/windmill-labs/windmill/commit/6f363163df9cd15f5af7d56cf34a01b70d236830))
-
-## [1.709.0](https://github.com/windmill-labs/windmill/compare/v1.708.0...v1.709.0) (2026-05-25)
-
-
-### Features
-
-* add copy button to Path component ([#9311](https://github.com/windmill-labs/windmill/issues/9311)) ([98bd5e7](https://github.com/windmill-labs/windmill/commit/98bd5e7f2a437b8b534028838b6ed0d7c59f7011))
-* **ai-chat:** align footer bar + DropdownV2 mode/autonomy selectors ([#9308](https://github.com/windmill-labs/windmill/issues/9308)) ([2f50e8b](https://github.com/windmill-labs/windmill/commit/2f50e8bab0b5ae9ae297c79abfe96df441f405e2))
-* **ai-chat:** expand chat question answers ([#9310](https://github.com/windmill-labs/windmill/issues/9310)) ([3f219ae](https://github.com/windmill-labs/windmill/commit/3f219aed98d93158aefce01bb51ed12dcb4711a1))
-* plug global chat drafts into userdraft ([#9291](https://github.com/windmill-labs/windmill/issues/9291)) ([1eef531](https://github.com/windmill-labs/windmill/commit/1eef53170b1b2afb75b9812e33787d1f28cf50dd))
-* **raw_apps:** surface UI Builder build errors over the preview pane ([#9316](https://github.com/windmill-labs/windmill/issues/9316)) ([90a196d](https://github.com/windmill-labs/windmill/commit/90a196d8d81993ffc2377d7088ab98f7b0f5ddcc))
-* **raw_apps:** tab-based editor surface with split-with-preview ([#9273](https://github.com/windmill-labs/windmill/issues/9273)) ([368e677](https://github.com/windmill-labs/windmill/commit/368e6774194a58058f28d1b4a42f8f4a7ec4ab63))
-* **service-accounts:** allow choosing role at creation time ([#9307](https://github.com/windmill-labs/windmill/issues/9307)) ([b125eca](https://github.com/windmill-labs/windmill/commit/b125eca7628b07c071bd102b161d389259fd6c62))
-
-
-### Bug Fixes
-
-* **auth:** filter resource/variable listings by token scope (WIN-1981) ([#9302](https://github.com/windmill-labs/windmill/issues/9302)) ([b5a0d46](https://github.com/windmill-labs/windmill/commit/b5a0d46695fdfe692d64573d1cfa06511e3b33f5))
-* **jobs:** authorization bypass in only_result job updates (WIN-1980) ([#9301](https://github.com/windmill-labs/windmill/issues/9301)) ([108a88a](https://github.com/windmill-labs/windmill/commit/108a88a1801548c8570d56aa3e1eb80246367bf4))
-
-## [1.708.0](https://github.com/windmill-labs/windmill/compare/v1.707.0...v1.708.0) (2026-05-24)
-
-
-### Features
-
-* **queue:** per-workspace fairness cap on the shared cloud worker pool ([#9303](https://github.com/windmill-labs/windmill/issues/9303)) ([de2e243](https://github.com/windmill-labs/windmill/commit/de2e243313ee34348675dec600cb412b475d1b4b))
-
-## [1.707.0](https://github.com/windmill-labs/windmill/compare/v1.706.1...v1.707.0) (2026-05-22)
-
-
-### Features
-
-* add wmill job rerun subcommand ([#9275](https://github.com/windmill-labs/windmill/issues/9275)) ([e0ffea2](https://github.com/windmill-labs/windmill/commit/e0ffea2deb5acf30815edd3669f4fc4c818b6e19))
-* **github-app:** hide cloud-only UI on self-managed + admin assignment UI ([#9299](https://github.com/windmill-labs/windmill/issues/9299)) ([dcee8cc](https://github.com/windmill-labs/windmill/commit/dcee8cc0d3dd71c3a12f1720e3ce4eb86cdacf4f))
-* **typescript-client:** add deleteS3File + optional workspace arg on S3 helpers ([#9300](https://github.com/windmill-labs/windmill/issues/9300)) ([daab561](https://github.com/windmill-labs/windmill/commit/daab561ec0763468d93e42e8f7f0796dc77be74d))
-
-
-### Bug Fixes
-
-* **auth:** tighten token-owner fallback for unscoped tokens (WIN-1978) ([#9293](https://github.com/windmill-labs/windmill/issues/9293)) ([7003998](https://github.com/windmill-labs/windmill/commit/7003998a575d76c272c6abd0789a1d1f7b722076))
-* **cli:** wmill sync pull updates wmill-lock.yaml for raw apps ([#9289](https://github.com/windmill-labs/windmill/issues/9289)) ([486e5f9](https://github.com/windmill-labs/windmill/commit/486e5f947b1649c17d32e3b214c50d4be701a4e8))
-* flow recording teardown crash + rename package to @windmill-labs/components ([#9288](https://github.com/windmill-labs/windmill/issues/9288)) ([13a2fae](https://github.com/windmill-labs/windmill/commit/13a2fae745ba4862006db5ee0811475c1d27fd1d))
-* **flows:** restore Variables and Resources in flow editor prop picker ([#9290](https://github.com/windmill-labs/windmill/issues/9290)) ([5566c7b](https://github.com/windmill-labs/windmill/commit/5566c7b3ff2d5a6b15cb9187aa15ce1c7245b3fb))
-* **ResourceEditor:** don't reset state when `selected` reverts to undefined ([#9295](https://github.com/windmill-labs/windmill/issues/9295)) ([1f2d2c1](https://github.com/windmill-labs/windmill/commit/1f2d2c11493db20b87615d41c21e5e1c35564739))
-* **secret-backend:** pass DB to Vault migrations + show failure details ([#9292](https://github.com/windmill-labs/windmill/issues/9292)) ([ace2291](https://github.com/windmill-labs/windmill/commit/ace22910c40585a6a2c9abd0c46f7e5e0214e78e))
-
-## [1.706.1](https://github.com/windmill-labs/windmill/compare/v1.706.0...v1.706.1) (2026-05-22)
-
-
-### Bug Fixes
-
-* fork compare visibility for non-admins and stale-token superadmins ([#9283](https://github.com/windmill-labs/windmill/issues/9283)) ([8272244](https://github.com/windmill-labs/windmill/commit/82722449e79da0b4b0ad4142aec7e7965e9ff236))
-* **git-sync:** bump to hub/28234 with stateless gpg.program wrapper (WIN-1974) ([#9282](https://github.com/windmill-labs/windmill/issues/9282)) ([89a2f07](https://github.com/windmill-labs/windmill/commit/89a2f07218818b95238b4a4484deab3138099672))
-* **nsjail:** gate unix-symlink test behind cfg(unix) for Windows build ([#9280](https://github.com/windmill-labs/windmill/issues/9280)) ([72e2c3a](https://github.com/windmill-labs/windmill/commit/72e2c3a6b3e0cb0f5bddf8291ae18bb8cf55ec28))
-
-## [1.706.0](https://github.com/windmill-labs/windmill/compare/v1.705.0...v1.706.0) (2026-05-21)
-
-
-### Features
-
-* add userdraft listing primitives ([#9268](https://github.com/windmill-labs/windmill/issues/9268)) ([d0ee697](https://github.com/windmill-labs/windmill/commit/d0ee697e8b8de58085ea0b2ecde1af2b2441428d))
-* add UV_PYTHON_INSTALL_MIRROR env and instance setting ([#9271](https://github.com/windmill-labs/windmill/issues/9271)) ([1169371](https://github.com/windmill-labs/windmill/commit/1169371d4885bdc18c76d03c6caae71f0e440235))
-* add yolo mode for ai chat tools ([#9258](https://github.com/windmill-labs/windmill/issues/9258)) ([ac26aa4](https://github.com/windmill-labs/windmill/commit/ac26aa4e4c7cc2d493f136b59738c0708803cc6d))
-* CLI datatable serve / psql ([#9267](https://github.com/windmill-labs/windmill/issues/9267)) ([28c8b5c](https://github.com/windmill-labs/windmill/commit/28c8b5c60fd46f961ae11b363b9be834fad6ee68))
-* **cli:** add `wmill init prompts` and custom override slot ([#9266](https://github.com/windmill-labs/windmill/issues/9266)) ([1ba8ed8](https://github.com/windmill-labs/windmill/commit/1ba8ed8abd827313ce0f7728d9f84357417206ee))
-* **nsjail:** optional disk-backed /tmp via instance setting ([#9272](https://github.com/windmill-labs/windmill/issues/9272)) ([b656dc6](https://github.com/windmill-labs/windmill/commit/b656dc6cdc8c50ef9740240447f119cceed18547))
-
-
-### Bug Fixes
-
-* **ai:** enforce RLS and scope check on user-supplied X-Resource-Path ([#9276](https://github.com/windmill-labs/windmill/issues/9276)) ([0692b97](https://github.com/windmill-labs/windmill/commit/0692b97c8a3818549d7050ea3e057e9cbf1ddb44))
-* **debugger:** add non-root user support to Dockerfile ([#9277](https://github.com/windmill-labs/windmill/issues/9277)) ([0bdb6a9](https://github.com/windmill-labs/windmill/commit/0bdb6a9d5d5fb28a27af1b6eda9fde7172308faf))
-* **indexer:** tell admins when ingress routes search to wrong pod ([#9274](https://github.com/windmill-labs/windmill/issues/9274)) ([d29a561](https://github.com/windmill-labs/windmill/commit/d29a5612fcd17eb4197468289e955a1209127cc1))
-
-## [1.705.0](https://github.com/windmill-labs/windmill/compare/v1.704.1...v1.705.0) (2026-05-20)
-
-
-### Features
-
-* add flow_user_state(key) to QuickJS input transform sandbox (WIN-1947) ([#9093](https://github.com/windmill-labs/windmill/issues/9093)) ([88c1493](https://github.com/windmill-labs/windmill/commit/88c149314576789f46feb5c7e1af3225b061f0c6))
-* add wmill protection-rules pull/push CLI commands ([#9240](https://github.com/windmill-labs/windmill/issues/9240)) ([01bad16](https://github.com/windmill-labs/windmill/commit/01bad16c0cc40fa64b2a72ccb8ded487c729cf35))
-* **chat:** visual redesign — input, streaming indicator, scroll polish ([#9232](https://github.com/windmill-labs/windmill/issues/9232)) ([31a0469](https://github.com/windmill-labs/windmill/commit/31a046973af960764ee4e153b68c020cbd4690ce))
-* **chat:** waiting-for-user indicator + scroll-to-latest polish ([#9252](https://github.com/windmill-labs/windmill/issues/9252)) ([7909878](https://github.com/windmill-labs/windmill/commit/790987831380611b5bd19a760b0a5433492d7796))
-* **cli:** add datatable and ducklake list/run commands ([#9257](https://github.com/windmill-labs/windmill/issues/9257)) ([1d04904](https://github.com/windmill-labs/windmill/commit/1d04904a47245062e50c2cc6362bbbb21a6987aa))
-* **debug:** show ghost breakpoint and tooltip on gutter hover ([#9150](https://github.com/windmill-labs/windmill/issues/9150)) ([271f0cb](https://github.com/windmill-labs/windmill/commit/271f0cbd087851fca86ed1530618ddbf728f13f3))
-* **editors:** responsive top-bars + collapsible raw-app sidebar ([#9237](https://github.com/windmill-labs/windmill/issues/9237)) ([b0ed270](https://github.com/windmill-labs/windmill/commit/b0ed27096d9e918e946cf8a8a04af8ff1892b50f))
-* export audit logs to a dedicated object store folder ([#9207](https://github.com/windmill-labs/windmill/issues/9207)) ([ba6fb70](https://github.com/windmill-labs/windmill/commit/ba6fb7021b5a720bff8e86b4741031902cf1c267))
-* **frontend:** new path component ([#9017](https://github.com/windmill-labs/windmill/issues/9017)) ([9c28bbf](https://github.com/windmill-labs/windmill/commit/9c28bbfd694a5047b4a8a9fe5cc2e54309f8f067))
-* **frontend:** sync home search bar state to URL ([#9256](https://github.com/windmill-labs/windmill/issues/9256)) ([31b7810](https://github.com/windmill-labs/windmill/commit/31b781000e62384af6b8e1ba0172e45ac6ab591f))
-* **git-sync:** hidden `sync git-deploy` owns wm_deploy branch + e2e regression tests ([#9230](https://github.com/windmill-labs/windmill/issues/9230)) ([07202fd](https://github.com/windmill-labs/windmill/commit/07202fd048c999c9d32f3feee94a08625050283d))
-* **indexer:** observability for unavailable search index (WIN-1956) ([#9239](https://github.com/windmill-labs/windmill/issues/9239)) ([285a787](https://github.com/windmill-labs/windmill/commit/285a78752a23aa467f9a82868d784599793d3a1f))
-* **nsjail:** make tmpfs size configurable via instance setting ([#9261](https://github.com/windmill-labs/windmill/issues/9261)) ([9111f89](https://github.com/windmill-labs/windmill/commit/9111f8908de82e9032a63711158dff9c6bca255b))
-* open ai chat path links in drawers ([#9220](https://github.com/windmill-labs/windmill/issues/9220)) ([f6fcdb5](https://github.com/windmill-labs/windmill/commit/f6fcdb5599c28b4890d6f775f657bfafeea1d380))
-* persistent in-editor drafts via UserDraft ([#9121](https://github.com/windmill-labs/windmill/issues/9121)) ([0f7dd86](https://github.com/windmill-labs/windmill/commit/0f7dd86e5c3a43bc62c4c0501efec34226b6e279))
-* resolve relative imports from local content in script/flow preview ([#9233](https://github.com/windmill-labs/windmill/issues/9233)) ([2a780ad](https://github.com/windmill-labs/windmill/commit/2a780ad87af69358f241536697f694612fe92d93))
-* **snowflake:** derive public key from private key when omitted (WIN-1959) ([#9251](https://github.com/windmill-labs/windmill/issues/9251)) ([aa12c66](https://github.com/windmill-labs/windmill/commit/aa12c66c25e68eefec22c213e8f228fd0699d8ce))
-* **vault:** optional KV secret path prefix setting (WIN-1960) ([#9249](https://github.com/windmill-labs/windmill/issues/9249)) ([d08f72b](https://github.com/windmill-labs/windmill/commit/d08f72b3e1ef194b5d656cafa15bc88e8b6ba731))
-
-
-### Bug Fixes
-
-* **autoscaling:** count custom worker groups by row, divide only native by NUM_WORKERS ([#9255](https://github.com/windmill-labs/windmill/issues/9255)) ([76d949e](https://github.com/windmill-labs/windmill/commit/76d949e7bc30eb8cadfdc52e031fcdf5ad97d2ed))
-* **autoscaling:** full-scale below min_workers on large backlog ([#9234](https://github.com/windmill-labs/windmill/issues/9234)) ([a4d59a8](https://github.com/windmill-labs/windmill/commit/a4d59a81dfb6fffbd185a3aa009eb90c160bf42b))
-* bound resource/variable interpolation recursion depth (WIN-1957) ([#9243](https://github.com/windmill-labs/windmill/issues/9243)) ([26f3cbe](https://github.com/windmill-labs/windmill/commit/26f3cbef259e643c6d79be893701eec70b7c0501))
-* cgroup-aware DuckDB memory_limit + allocator memory release ([#9245](https://github.com/windmill-labs/windmill/issues/9245)) ([0022112](https://github.com/windmill-labs/windmill/commit/00221128cbf0801a45bad40246e50beceaba0a7e))
-* collapse successful ai tool details ([#9265](https://github.com/windmill-labs/windmill/issues/9265)) ([413404a](https://github.com/windmill-labs/windmill/commit/413404a788bbe6b5c9df387a2db3000ffec74083))
-* early return should consider failure_module result ([#9241](https://github.com/windmill-labs/windmill/issues/9241)) ([2db1c0a](https://github.com/windmill-labs/windmill/commit/2db1c0a1fcfdcad94cae97dcffa090ffb91494f7))
-* enable jemalloc background_thread to prevent worker RSS growth ([#9236](https://github.com/windmill-labs/windmill/issues/9236)) ([a974ff6](https://github.com/windmill-labs/windmill/commit/a974ff68e00278ccaf441b0567cd46e2b5067fdd))
-* enforce auth guards on app component preview execution ([#9235](https://github.com/windmill-labs/windmill/issues/9235)) ([4b1bea8](https://github.com/windmill-labs/windmill/commit/4b1bea8aed51eb9e24940d89d984ce32f375ab0c))
-* **flows:** flag noLogs jobs and lazily resolve them in log panel ([#9099](https://github.com/windmill-labs/windmill/issues/9099)) ([740a35b](https://github.com/windmill-labs/windmill/commit/740a35bf7b20f0bd8cb94c3d703dd353f0711b0a))
-* **frontend:** flow progress bar for early-stop completion and error handler (WIN-1961) ([#9254](https://github.com/windmill-labs/windmill/issues/9254)) ([cc141ef](https://github.com/windmill-labs/windmill/commit/cc141effa3b1019f70f4b7230fe7ebc7017632a0))
-* **frontend:** open customer portal in popup synchronously to bypass Safari blocker ([#9242](https://github.com/windmill-labs/windmill/issues/9242)) ([f51b51a](https://github.com/windmill-labs/windmill/commit/f51b51a9a1aee5183fa597cf93ff14fbaddaffa9))
-* prevent undefined user flickering in multiplayer presence list ([#9231](https://github.com/windmill-labs/windmill/issues/9231)) ([8c1f6cc](https://github.com/windmill-labs/windmill/commit/8c1f6ccc5d22e657a83831eb5f37a9516cdec10a))
-* **s3:** sandbox stored XSS via download response headers ([#9263](https://github.com/windmill-labs/windmill/issues/9263)) ([bb78b1c](https://github.com/windmill-labs/windmill/commit/bb78b1c06de5b73b951691460f81a3a2ec6e7f80))
-* **saml:** preserve deep links from /a/[...path] across SAML round-trip ([#9259](https://github.com/windmill-labs/windmill/issues/9259)) ([78cf6c7](https://github.com/windmill-labs/windmill/commit/78cf6c7f8181ad431cffebd18c45a3a802b3a601))
-* scope VSCode webview clipboard paste to focused editor ([#9221](https://github.com/windmill-labs/windmill/issues/9221)) ([bd06282](https://github.com/windmill-labs/windmill/commit/bd062825a255da364c1590820ead65172e835d13))
-
 ## [1.704.1](https://github.com/windmill-labs/windmill/compare/v1.704.0...v1.704.1) (2026-05-19)
 
 

Dockerfile

@@ -66,7 +66,6 @@ RUN npm ci
 COPY frontend .
 RUN mkdir /backend
 COPY /backend/windmill-api/openapi.yaml /backend/windmill-api/openapi.yaml
-COPY /backend/oauth_connect.json /backend/oauth_connect.json
 COPY /openflow.openapi.yaml /openflow.openapi.yaml
 COPY /backend/windmill-api/build_openapi.sh /backend/windmill-api/build_openapi.sh
 COPY /system_prompts/auto-generated /system_prompts/auto-generated
@@ -233,14 +232,11 @@ ENV PATH="${PATH}:/usr/local/go/bin"
 ENV GO_PATH=/usr/local/go/bin/go
 
 # Install UV
-RUN curl --proto '=https' --tlsv1.2 -LsSf https://github.com/astral-sh/uv/releases/download/0.9.25/uv-installer.sh | sh && mv /root/.local/bin/uv /usr/local/bin/uv
+RUN curl --proto '=https' --tlsv1.2 -LsSf https://github.com/astral-sh/uv/releases/download/0.9.24/uv-installer.sh | sh && mv /root/.local/bin/uv /usr/local/bin/uv
 
 # Preinstall python runtimes to temp build location (will copy with world-writable perms later)
-# --compile-bytecode precompiles the stdlib to .pyc so jobs don't recompile it on every run
-# under the read-only nsjail runtime mount (uv >= 0.9.25). The copy below MUST preserve
-# timestamps or Python's mtime-based .pyc invalidation discards these compiled files.
-RUN UV_CACHE_DIR=/tmp/build_cache/uv UV_PYTHON_INSTALL_DIR=/tmp/build_cache/py_runtime uv python install 3.11 --compile-bytecode
-RUN UV_CACHE_DIR=/tmp/build_cache/uv UV_PYTHON_INSTALL_DIR=/tmp/build_cache/py_runtime uv python install $LATEST_STABLE_PY --compile-bytecode
+RUN UV_CACHE_DIR=/tmp/build_cache/uv UV_PYTHON_INSTALL_DIR=/tmp/build_cache/py_runtime uv python install 3.11
+RUN UV_CACHE_DIR=/tmp/build_cache/uv UV_PYTHON_INSTALL_DIR=/tmp/build_cache/py_runtime uv python install $LATEST_STABLE_PY
 
 
 RUN curl -sL https://deb.nodesource.com/setup_20.x | bash -
@@ -262,7 +258,7 @@ RUN export GOCACHE=/tmp/build_cache/go && \
 # chmod a+rw adds read+write WITHOUT removing execute bits (755->777, 644->666)
 # Note: uv python install only creates py_runtime, not uv cache - we create uv/go dirs for runtime
 RUN mkdir -p /tmp/windmill/cache && \
-    cp -r --preserve=timestamps /tmp/build_cache/* /tmp/windmill/cache/ && \
+    cp -r /tmp/build_cache/* /tmp/windmill/cache/ && \
     chmod -R a+rw /tmp/windmill/cache && \
     rm -rf /tmp/build_cache && \
     mkdir -p -m 777 /tmp/windmill/cache/uv /tmp/windmill/cache/go /tmp/windmill/cache/rustup /tmp/windmill/cache/cargo
@@ -303,20 +299,10 @@ ENV CARGO_HOME="/tmp/windmill/cache/cargo"
 ENV LD_LIBRARY_PATH="."
 
 # nsjail runtime deps and binary
-RUN apt-get update && apt-get install -y --no-install-recommends libprotobuf32 libnl-route-3-200 libnl-3-200 \
+RUN apt-get update && apt-get install -y libprotobuf-dev libnl-route-3-dev \
     && apt-get clean && rm -rf /var/lib/apt/lists/*
 COPY --from=nsjail /nsjail/nsjail /bin/nsjail
 
-# crane: pulls + flattens images for the sandboxed container runtime (`# sandbox <image>`).
-# Single static binary — no daemon/store/root needed. See docs/docker-v2-runtime.md.
-ARG CRANE_VERSION=v0.20.6
-RUN arch="$(dpkg --print-architecture)"; \
-    case "$arch" in amd64) crane_arch=x86_64 ;; arm64) crane_arch=arm64 ;; *) echo >&2 "error: unsupported arch '$arch' for crane"; exit 1 ;; esac; \
-    wget -O /tmp/crane.tgz "https://github.com/google/go-containerregistry/releases/download/${CRANE_VERSION}/go-containerregistry_Linux_${crane_arch}.tar.gz" \
-    && tar -xzf /tmp/crane.tgz -C /usr/local/bin crane \
-    && rm /tmp/crane.tgz \
-    && chmod +x /usr/local/bin/crane
-
 WORKDIR ${APP}
 
 RUN ln -s ${APP}/windmill /usr/local/bin/windmill

ai_evals/AGENTS.md

@@ -86,31 +86,6 @@ Global prompts should exercise workspace-level drafting behavior:
 
 Keep deterministic validation focused on the draft contract: required draft type/path, required content snippets, forbidden draft paths, and forbidden mutating tools such as deploy/delete unless the case explicitly asks for them.
 
-Datatable cases should set `skipJudge: true` and validate through tool-use
-(`requiredToolsUsed` / `forbiddenToolsUsed`) and SQL-argument assertions
-(`toolCallArgs` with `stringIncludesAnyOf`, e.g. `['select']`, `['create table']`,
-`['update', 'insert into']`). Two reasons the judge is unreliable here:
-
-- `list_datatables`, `get_datatable_table_schema`, and `exec_datatable_sql`
-  produce no drafts, and the global judge only sees the drafts artifact — it
-  scores a no-draft conversational answer as empty (same as the
-  `askUserQuestion` cases).
-- Even a case that *does* produce a draft (a script reading the data table via
-  `wmill.datatable()` at runtime) is mis-judged: the judge has no datatable SDK
-  reference and penalizes correct `wmill.datatable()` usage as wrong. Verify the
-  SDK call deterministically instead — `requiredDrafts.valueIncludes: ['wmill.datatable(']`
-  plus forbidding `exec_datatable_sql` (keeping chat-time SQL distinct from
-  runtime SDK use).
-
-`stringIncludesAnyOf` is existential over calls (at least one matching call), so a
-mutation case still passes when the model mixes its UPDATE/INSERT with
-verification SELECTs. The in-memory engine (`datatableSqlEngine.ts`) is stateful
-within a case — writes persist, so a model that re-queries to verify its
-CREATE/UPDATE sees the change and does not loop. But the engine is best-effort
-(SELECT returns all rows of the referenced/first table with no WHERE/projection),
-so still never assert specific returned row values. Seed data via
-`workspace.datatables` in the `initial` fixture (see README).
-
 ## Deterministic validation
 
 Use deterministic validation only for hard failures such as:

ai_evals/README.md

@@ -56,7 +56,7 @@ bun run cli -- run flow flow-test4-order-processing-loop --model opus
 bun run cli -- run flow flow-test0-sum-two-numbers --models haiku,opus,4o
 bun run cli -- run flow flow-test0-sum-two-numbers --runs 3 --verbose
 bun run cli -- run flow --record
-GEMINI_API_KEY=... bun run cli -- run app app-test1-counter-create --model gemini-3-flash-preview
+GEMINI_API_KEY=... bun run cli -- run app app-test1-counter-create --model gemini-pro
 WMILL_AI_EVAL_BACKEND_URL=http://127.0.0.1:8000 bun run cli -- run flow --backend-validation preview
 bun run cli -- run global global-test1-script-create
 bun run cli -- run cli bun-hello-script
@@ -75,8 +75,6 @@ Public CLI surface:
 - `--model <alias>`: choose the model under test
 - `--models <a,b,c>`: run the same cases sequentially against several model aliases
 - `--verbose`: stream assistant output for frontend runs
-- `--skip-judge`: skip LLM judge scoring for the run
-- `--execution-only`: only require the model/proxy/frontend loop to complete; skip validators, tool expectations, backend artifact validation, and judge scoring
 - `--record`: append a compact tracked summary line to `ai_evals/history/<mode>.jsonl` for full-suite runs only
 - `--backend-validation <mode>`: optional backend smoke validation (`off` or `preview`) for `script` and `flow` evals
 
@@ -90,18 +88,17 @@ Today:
 - `sonnet`
 - `opus`
 - `4o`
-- `gpt-5.5`
+- `gemini-flash`
+- `gemini-pro`
 - `gemini-3-flash-preview`
 - `gemini-3.1-pro-preview`
-- `deepseek-v4-flash`
-- `deepseek-v4-pro`
 
 Notes:
 
-- the command also prints accepted alias spellings such as `gpt-4o`, `gpt-55`, `claude-opus-4.6`, and `claude-haiku-4.5`
-- frontend modes (`flow`, `script`, `app`, `global`) can use Anthropic, OpenAI, Gemini, and DeepSeek-backed aliases
+- the command also prints accepted alias spellings such as `gpt-4o`, `claude-opus-4.6`, and `claude-haiku-4.5`
+- frontend modes (`flow`, `script`, `app`, `global`) can use Anthropic, OpenAI, and Gemini-backed aliases
 - `cli` mode always uses the Anthropic agent SDK, so only Anthropic aliases are valid there
-- the judge model is separate and currently defaults to `claude-sonnet-4-6`; use `--skip-judge` for deterministic-only runs
+- the judge model is separate and currently defaults to `claude-sonnet-4-6`
 
 ## Case Format
 
@@ -145,32 +142,6 @@ For `global` mode, `validate` can express draft-level requirements such as:
 - required or forbidden draft counts
 - forbidden draft paths
 
-Global initial fixtures can also seed `liveEditorDrafts` with `type`,
-`storagePath`, `effectivePath`, and `value` fields. These drafts emulate the
-currently open script, flow, or raw app editor so cases can test prompts that
-refer to "this" or the "current" item.
-
-Global (and flow) initial fixtures can seed `workspace.datatables` so the
-`list_datatables`, `get_datatable_table_schema`, and `exec_datatable_sql` tools
-return seeded data during evals. Each entry is
-`{ datatable_name, schemas: { <schema>: { <table>: { columns, rows? } } } }`.
-SQL runs through a small in-memory engine (`datatableSqlEngine.ts`), not a real
-database. Writes are **stateful within a case**: `CREATE`/`DROP`/`INSERT`/`UPDATE`/
-`DELETE` mutate the seeded datatable in place, so a later `list_datatables`,
-`get_datatable_table_schema`, `SELECT`, or `information_schema` query reflects them
-— this is what stops a model from looping when it re-queries to verify a write.
-The engine is best-effort: `SELECT` returns all rows of the referenced (or first)
-table with no WHERE filtering/projection/joins, `WHERE` on UPDATE/DELETE supports
-`col = value` predicates joined by `AND`, and anything unparseable is a no-op
-success. So validate datatable cases through tool-use and SQL-argument assertions
-(`requiredToolsUsed`, `stringIncludesAnyOf`) — not through exact returned row
-values. An empty/absent `datatables` seed makes `list_datatables` return `[]`,
-which is what the "no datatable configured" blocking cases rely on.
-
-Set `WMILL_AI_EVAL_DISABLE_ACTIVE_EDITOR_CONTEXT=1` to run those cases with
-the old behavior where the live editor is only discoverable through
-`list_workspace_items`.
-
 App fixtures can also include an optional `datatables.json` file at the fixture root.
 
 For `flow` mode, an `initial` fixture can also include a benchmark workspace catalog of
@@ -218,15 +189,11 @@ If `--record` is used, the CLI also appends one compact JSON line to:
 Each recorded line contains:
 
 - run metadata (`createdAt`, `gitSha`, `mode`, `runModel`, `judgeModel`)
-- suite totals (`caseCount`, `attemptCount`, `passedAttempts`, `passRate`, `averageDurationMs`, `averagePassedDurationMs`, `averageJudgeScore`)
-- average token usage (`averageTokenUsagePerAttempt`, `averageTokenUsagePerPassedAttempt`)
-- per-case metrics under `cases[]` (`averageDurationMs`, `averagePassedDurationMs`, `averageJudgeScore`, `averageTokenUsagePerAttempt`, `averageTokenUsagePerPassedAttempt`, pass rate)
+- suite totals (`caseCount`, `attemptCount`, `passedAttempts`, `passRate`, `averageDurationMs`, `averageJudgeScore`)
+- average token usage (`averageTokenUsagePerAttempt`)
+- per-case metrics under `cases[]` (`averageDurationMs`, `averageJudgeScore`, `averageTokenUsagePerAttempt`, pass rate)
 - `failedCaseIds`
 
-The CLI headline duration and token averages use passed attempts only.
-All-attempt averages are still recorded to make failures auditable without
-letting failed attempts skew success cost comparisons.
-
 Example:
 
 - summary: `ai_evals/results/2026-04-09T09-40-33.051Z__flow.json`

ai_evals/adapters/cli/runtime.ts

@@ -16,9 +16,6 @@ export interface PromptRunResult {
   output: string;
   durationMs: number;
   tokenUsage: BenchmarkTokenUsage | null;
-  // Input tokens on the last assistant turn. The SDK `result` message reports
-  // usage cumulatively, so the final context size comes from per-turn usage.
-  finalContextTokens: number | null;
   trace: CliTrace;
 }
 
@@ -147,7 +144,6 @@ export async function runPromptAndCapture(
   let output = "";
   let assistantMessageCount = 0;
   let tokenUsage: BenchmarkTokenUsage | null = null;
-  let finalContextTokens: number | null = null;
   const startedAt = Date.now();
   const stubBinDir = join(cwd, WMILL_STUB_DIR_NAME);
   const wmillLogPath = join(cwd, WMILL_LOG_FILE_NAME);
@@ -170,12 +166,6 @@ export async function runPromptAndCapture(
   for await (const message of query({ prompt, options })) {
     if (message.type === "assistant") {
       assistantMessageCount += 1;
-      const turnContext = anthropicUsageToBenchmarkTokenUsage(
-        message.message?.usage
-      )?.prompt;
-      if (turnContext && turnContext > 0) {
-        finalContextTokens = turnContext;
-      }
       const content = message.message?.content;
       if (Array.isArray(content)) {
         for (const block of content) {
@@ -220,7 +210,6 @@ export async function runPromptAndCapture(
     output,
     durationMs: Date.now() - startedAt,
     tokenUsage,
-    finalContextTokens,
     trace: {
       toolsUsed,
       skillsInvoked,

ai_evals/adapters/frontend/benchmarkRunner.ts

@@ -10,6 +10,10 @@ import { runSuite } from "../../core/runSuite";
 import type { BenchmarkRunResult, ModeRunner } from "../../core/types";
 import { resolveWindmillBackendSettings } from "../../core/windmillBackendSettings";
 import { emitFrontendBenchmarkProgress } from "./progress";
+import { createAppModeRunner } from "../../modes/app";
+import { createFlowModeRunner } from "../../modes/flow";
+import { createGlobalModeRunner } from "../../modes/global";
+import { createScriptModeRunner } from "../../modes/script";
 import { DEFAULT_JUDGE_MODEL } from "../../core/judge";
 
 export type FrontendBenchmarkMode = "flow" | "app" | "script" | "global";
@@ -25,12 +29,6 @@ export async function runFrontendBenchmarkFromEnv(): Promise<BenchmarkRunResult>
   );
   const emitProgress = process.env.WMILL_FRONTEND_AI_EVAL_PROGRESS === "1";
   const verbose = process.env.WMILL_FRONTEND_AI_EVAL_VERBOSE === "1";
-  const executionOnly =
-    process.env.WMILL_FRONTEND_AI_EVAL_EXECUTION_ONLY === "1";
-  const judgeModel =
-    process.env.WMILL_FRONTEND_AI_EVAL_SKIP_JUDGE === "1" || executionOnly
-      ? null
-      : DEFAULT_JUDGE_MODEL;
   const model = resolveEvalModel(
     mode,
     process.env.WMILL_FRONTEND_AI_EVAL_MODEL,
@@ -42,7 +40,7 @@ export async function runFrontendBenchmarkFromEnv(): Promise<BenchmarkRunResult>
   const backendSettings = resolveWindmillBackendSettings();
 
   const selectedCases = await loadSelectedCases(mode, caseIds);
-  const modeRunner = await getModeRunner(
+  const modeRunner = getModeRunner(
     mode,
     getFrontendEvalModel(model),
     backendValidation,
@@ -54,8 +52,7 @@ export async function runFrontendBenchmarkFromEnv(): Promise<BenchmarkRunResult>
     cases: selectedCases,
     runs,
     runModel,
-    judgeModel,
-    executionOnly,
+    judgeModel: DEFAULT_JUDGE_MODEL,
     concurrency: verbose ? 1 : undefined,
     verbose,
     onProgress: emitProgress
@@ -67,48 +64,35 @@ export async function runFrontendBenchmarkFromEnv(): Promise<BenchmarkRunResult>
     mode,
     runs,
     runModel,
-    judgeModel,
+    judgeModel: DEFAULT_JUDGE_MODEL,
     caseResults,
   });
 }
 
-async function getModeRunner(
+function getModeRunner(
   mode: FrontendBenchmarkMode,
   model: ReturnType<typeof getFrontendEvalModel>,
   backendValidation: ReturnType<typeof resolveBackendValidationSettings>,
   backendSettings: ReturnType<typeof resolveWindmillBackendSettings>,
-): Promise<ModeRunner<any, any, any>> {
+): ModeRunner<any, any, any> {
   switch (mode) {
-    case "flow": {
-      const { createFlowModeRunner } = await import("../../modes/flow");
+    case "flow":
       return createFlowModeRunner(model, backendValidation, backendSettings);
-    }
-    case "app": {
-      const { createAppModeRunner } = await import("../../modes/app");
+    case "app":
       return createAppModeRunner(model, backendSettings);
-    }
-    case "script": {
-      const { createScriptModeRunner } = await import("../../modes/script");
+    case "script":
       return createScriptModeRunner(
         model,
         backendValidation,
         backendSettings,
       );
-    }
-    case "global": {
-      const { createGlobalModeRunner } = await import("../../modes/global");
+    case "global":
       return createGlobalModeRunner(model, backendSettings);
-    }
   }
 }
 
 function parseMode(value: string | undefined): FrontendBenchmarkMode {
-  if (
-    value === "flow" ||
-    value === "app" ||
-    value === "script" ||
-    value === "global"
-  ) {
+  if (value === "flow" || value === "app" || value === "script" || value === "global") {
     return value;
   }
   throw new Error(`Unsupported frontend benchmark mode: ${String(value)}`);

ai_evals/adapters/frontend/core/app/appEvalRunner.ts

@@ -38,7 +38,6 @@ export interface AppEvalResult {
   toolCallCount: number;
   toolsUsed: string[];
   tokenUsage: TokenUsage;
-  finalContextTokens: number | null;
 }
 
 export interface AppEvalOptions {
@@ -114,7 +113,6 @@ export async function runAppEval(
       toolCallCount: rawResult.toolCallsCount,
       toolsUsed: rawResult.toolsCalled,
       tokenUsage: rawResult.tokenUsage,
-      finalContextTokens: rawResult.finalContextTokens,
     };
   } finally {
     await cleanup();

ai_evals/adapters/frontend/core/flow/flowEvalRunner.ts

@@ -39,7 +39,6 @@ export interface FlowEvalResult {
   toolsUsed: string[];
   toolCallDetails: ToolCallDetail[];
   tokenUsage: TokenUsage;
-  finalContextTokens: number | null;
 }
 
 export interface FlowEvalOptions {
@@ -114,7 +113,6 @@ export async function runFlowEval(
       toolsUsed: rawResult.toolsCalled,
       toolCallDetails: rawResult.toolCallDetails,
       tokenUsage: rawResult.tokenUsage,
-      finalContextTokens: rawResult.finalContextTokens,
     };
   } finally {
     await cleanup();

ai_evals/adapters/frontend/core/global/globalEvalRunner.ts

@@ -7,19 +7,13 @@ import {
   prepareGlobalSystemMessage,
   prepareGlobalUserMessage,
 } from "../../../../../frontend/src/lib/components/copilot/chat/global/core";
-import {
-  clearGlobalDrafts,
-  getGlobalDraft,
-  listGlobalDrafts,
-} from "../../../../../frontend/src/lib/components/copilot/chat/global/userDraftAdapter";
+import { globalDraftStore } from "../../../../../frontend/src/lib/components/copilot/chat/global/draftStore.svelte";
 import type { Tool as ProductionTool } from "../../../../../frontend/src/lib/components/copilot/chat/shared";
-import { UserDraft } from "../../../../../frontend/src/lib/userDraft.svelte";
 import type { ModeRunContext } from "../../../../core/types";
 import type { GlobalDraftState } from "../../../../core/validators";
 import type { WindmillBackendSettings } from "../../../../core/windmillBackendSettings";
 import {
   registerBenchmarkWorkspaceRunnables,
-  seedBenchmarkDraft,
   unregisterBenchmarkWorkspaceRunnables,
   type BenchmarkWorkspaceRunnables,
 } from "../../mockBackend";
@@ -30,39 +24,6 @@ const MUTATING_GLOBAL_TOOLS = new Set([
   "deploy_workspace_item",
   "delete_workspace_item",
 ]);
-const DISABLE_ACTIVE_EDITOR_CONTEXT_ENV =
-  "WMILL_AI_EVAL_DISABLE_ACTIVE_EDITOR_CONTEXT";
-// A/B gate for the search_app read tool: set to "1" to run the baseline arm
-// (toolset without search_app) so its token cost can be compared against the arm
-// that offers it.
-const DISABLE_SEARCH_APP_ENV = "WMILL_AI_EVAL_DISABLE_SEARCH_APP";
-
-const LIVE_EDITOR_ITEM_KINDS = {
-  script: "script",
-  flow: "flow",
-  app: "raw_app",
-} as const;
-
-export interface GlobalLiveEditorDraftFixture {
-  type: keyof typeof LIVE_EDITOR_ITEM_KINDS;
-  storagePath?: string;
-  effectivePath?: string;
-  value?: unknown;
-}
-
-// Identity the global system prompt builds paths from. Production reads
-// `userStore` (whoami) to fill `u/{username}/...`; the eval harness never logs
-// in, so without this the prompt sees an empty username (`u//...`) and no
-// path-selection case is meaningful. Seeded per-case via the initial fixture and
-// passed straight to `prepareGlobalSystemMessage` (no global-store mutation).
-export interface GlobalUserFixture {
-  username: string;
-  is_admin?: boolean;
-  /** Folders the user can write to (the writable set whoami returns). */
-  folders?: string[];
-  /** Folders the user can read; read-only folders = folders_read \ folders. */
-  folders_read?: string[];
-}
 
 export interface GlobalEvalResult {
   success: boolean;
@@ -73,13 +34,10 @@ export interface GlobalEvalResult {
   toolsUsed: string[];
   toolCallDetails: ToolCallDetail[];
   tokenUsage: TokenUsage;
-  finalContextTokens: number | null;
 }
 
 export interface GlobalEvalOptions {
   workspaceFixtures?: BenchmarkWorkspaceRunnables;
-  liveEditorDrafts?: GlobalLiveEditorDraftFixture[];
-  user?: GlobalUserFixture;
   model?: string;
   maxIterations?: number;
   provider?: AIProvider;
@@ -97,28 +55,19 @@ export async function runGlobalEval(
     options.workspaceRoot ??
     (await mkdtemp(join(tmpdir(), "wmill-frontend-global-benchmark-")));
 
-  clearGlobalDrafts(workspaceRoot);
+  globalDraftStore.clearDrafts(workspaceRoot);
   registerBenchmarkWorkspaceRunnables(workspaceRoot, options.workspaceFixtures ?? {});
-  seedLiveEditorDrafts(workspaceRoot, options.liveEditorDrafts ?? []);
 
   try {
     const model = options.model ?? "claude-haiku-4-5-20251001";
-    const injectActiveEditorContext =
-      process.env[DISABLE_ACTIVE_EDITOR_CONTEXT_ENV] !== "1";
-    // Pass the seeded identity straight to the prompt builder rather than mutating
-    // the process-global `userStore`, so concurrent cases never race on it.
     const rawResult = await runEval({
       userPrompt,
-      systemMessage: prepareGlobalSystemMessage(undefined, { user: options.user }),
-      userMessage: prepareGlobalUserMessage(
-        userPrompt,
-        [],
-        injectActiveEditorContext ? { workspace: workspaceRoot } : {},
-      ),
+      systemMessage: prepareGlobalSystemMessage(),
+      userMessage: prepareGlobalUserMessage(userPrompt),
       tools: getGlobalEvalTools(),
       helpers: {},
       apiKey,
-      getOutput: () => collectGlobalDraftState(workspaceRoot),
+      getOutput: () => ({ drafts: globalDraftStore.listDrafts(workspaceRoot) }),
       onAssistantMessageStart: options.runContext?.onAssistantMessageStart,
       onAssistantToken: options.runContext?.onAssistantChunk,
       onAssistantMessageEnd: options.runContext?.onAssistantMessageEnd,
@@ -143,11 +92,9 @@ export async function runGlobalEval(
       toolsUsed: rawResult.toolsCalled,
       toolCallDetails: rawResult.toolCallDetails,
       tokenUsage: rawResult.tokenUsage,
-      finalContextTokens: rawResult.finalContextTokens,
     };
   } finally {
-    clearGlobalDrafts(workspaceRoot);
-    clearLiveEditorDrafts(workspaceRoot, options.liveEditorDrafts ?? []);
+    globalDraftStore.clearDrafts(workspaceRoot);
     unregisterBenchmarkWorkspaceRunnables(workspaceRoot);
     if (!options.workspaceRoot) {
       await rm(workspaceRoot, { recursive: true, force: true });
@@ -155,87 +102,26 @@ export async function runGlobalEval(
   }
 }
 
-// Build the harness output from the DB-backed drafts. `listGlobalDrafts` returns
-// metadata-only rows for backend drafts (the model's `write_script` etc. persist
-// straight to the backend with no in-tab editor cell), so re-read each such row
-// with `getGlobalDraft` to attach the full value the validators assert on. A row
-// that already carries a value (the production in-tab cell overlay) is kept as-is.
-async function collectGlobalDraftState(
-  workspace: string,
-): Promise<GlobalDraftState> {
-  const items = await listGlobalDrafts(workspace);
-  const drafts = await Promise.all(
-    items.map(async (item) => {
-      if (item.value !== undefined) {
-        return item;
-      }
-      const full = await getGlobalDraft(
-        workspace,
-        item.type,
-        item.path,
-        item.triggerKind,
-      );
-      return full ?? item;
-    }),
-  );
-  return { drafts: drafts as GlobalDraftState["drafts"] };
-}
-
-function seedLiveEditorDrafts(
-  workspace: string,
-  fixtures: GlobalLiveEditorDraftFixture[],
-): void {
-  for (const fixture of fixtures) {
-    const itemKind = LIVE_EDITOR_ITEM_KINDS[fixture.type];
-    const storagePath = fixture.storagePath ?? fixture.effectivePath ?? "";
-    if (fixture.value !== undefined) {
-      // Seed as a backend draft row, not an in-tab cell: a cell would shadow the
-      // model's DB-backed edit when the output is read back via listGlobalDrafts.
-      seedBenchmarkDraft(workspace, itemKind, storagePath, fixture.value);
-    }
-    UserDraft.setLiveEditorDraft({
-      workspace,
-      itemKind,
-      storagePath,
-      effectivePath: fixture.effectivePath ?? fixture.storagePath,
-    });
-  }
-}
-
-function clearLiveEditorDrafts(
-  workspace: string,
-  fixtures: GlobalLiveEditorDraftFixture[],
-): void {
-  for (const fixture of fixtures) {
-    const itemKind = LIVE_EDITOR_ITEM_KINDS[fixture.type];
-    const storagePath = fixture.storagePath ?? fixture.effectivePath ?? "";
-    UserDraft.clearLiveEditorDraft(itemKind, { workspace, storagePath });
-  }
-}
-
 function getGlobalEvalTools(): ProductionTool<{}>[] {
-  const disableSearchApp = process.env[DISABLE_SEARCH_APP_ENV] === "1";
-  return (globalTools as ProductionTool<{}>[])
-    .filter((tool) => !(disableSearchApp && tool.def.function.name === "search_app"))
-    .map((tool) => {
-      if (!MUTATING_GLOBAL_TOOLS.has(tool.def.function.name)) {
-        return tool;
-      }
+  return (globalTools as ProductionTool<{}>[]).map((tool) => {
+    if (!MUTATING_GLOBAL_TOOLS.has(tool.def.function.name)) {
+      return tool;
+    }
 
-      return {
-        ...tool,
-        requiresConfirmation: false,
-        validateBeforeConfirmation: undefined,
-        fn: async () =>
-          JSON.stringify(
-            {
-              success: false,
-              error:
-                "This mutating workspace tool is disabled during ai_evals global mode.",
-            },
-            null,
-            2,
-          ),
-      };
-    });
+    return {
+      ...tool,
+      requiresConfirmation: false,
+      validateBeforeConfirmation: undefined,
+      fn: async () =>
+        JSON.stringify(
+          {
+            success: false,
+            error:
+              "This mutating workspace tool is disabled during ai_evals global mode.",
+          },
+          null,
+          2,
+        ),
+    };
+  });
 }

ai_evals/adapters/frontend/core/script/scriptEvalRunner.ts

@@ -25,7 +25,6 @@ export interface ScriptEvalResult {
   toolsUsed: string[];
   toolCallDetails: ToolCallDetail[];
   tokenUsage: TokenUsage;
-  finalContextTokens: number | null;
 }
 
 export interface ScriptEvalOptions {
@@ -112,7 +111,6 @@ export async function runScriptEval(
       toolsUsed: rawResult.toolsCalled,
       toolCallDetails: rawResult.toolCallDetails,
       tokenUsage: rawResult.tokenUsage,
-      finalContextTokens: rawResult.finalContextTokens,
     };
   } finally {
     await cleanup();

ai_evals/adapters/frontend/core/shared/baseEvalRunner.ts

@@ -38,9 +38,8 @@ export interface RunEvalParams<THelpers, TOutput> {
   helpers: THelpers;
   /** API key for the provider */
   apiKey: string;
-  /** Function to get the current output state. May be async — global mode reads
-   * DB-backed drafts back through the (mocked) backend to build its output. */
-  getOutput: () => TOutput | Promise<TOutput>;
+  /** Function to get the current output state */
+  getOutput: () => TOutput;
   /** Model and Windmill backend configuration */
   options: EvalRunnerOptions;
   onAssistantMessageStart?: () => void;
@@ -155,10 +154,9 @@ export async function runEval<THelpers, TOutput>(
       if (result.hitMaxIterations) {
         return {
           success: false,
-          output: (await getOutput()) as TOutput,
+          output: getOutput(),
           error: `Reached max turns (${maxIterations})`,
           tokenUsage: result.tokenUsage,
-          finalContextTokens: result.lastIterationUsage?.prompt ?? null,
           toolCallsCount,
           toolsCalled,
           toolCallDetails,
@@ -172,9 +170,8 @@ export async function runEval<THelpers, TOutput>(
 
       return {
         success: true,
-        output: (await getOutput()) as TOutput,
+        output: getOutput(),
         tokenUsage: result.tokenUsage,
-        finalContextTokens: result.lastIterationUsage?.prompt ?? null,
         toolCallsCount,
         toolsCalled,
         toolCallDetails,
@@ -194,10 +191,9 @@ export async function runEval<THelpers, TOutput>(
 
       return {
         success: false,
-        output: (await getOutput()) as TOutput,
+        output: getOutput(),
         error: errorMessage,
         tokenUsage: { prompt: 0, completion: 0, total: 0 },
-        finalContextTokens: null,
         toolCallsCount,
         toolsCalled,
         toolCallDetails,

ai_evals/adapters/frontend/core/shared/providerConfig.test.ts

@@ -21,9 +21,9 @@ describe("proxy helpers", () => {
 
 describe("resolveEvalModelProvider", () => {
   it("infers googleai from Gemini model ids", () => {
-    expect(resolveEvalModelProvider("gemini-3-flash-preview")).toEqual({
+    expect(resolveEvalModelProvider("gemini-2.5-flash")).toEqual({
       provider: "googleai",
-      model: "gemini-3-flash-preview",
+      model: "gemini-2.5-flash",
     });
   });
 
@@ -35,11 +35,9 @@ describe("resolveEvalModelProvider", () => {
   });
 
   it("preserves an explicit provider", () => {
-    expect(
-      resolveEvalModelProvider("gemini-3.1-pro-preview", "googleai"),
-    ).toEqual({
+    expect(resolveEvalModelProvider("gemini-2.5-pro", "googleai")).toEqual({
       provider: "googleai",
-      model: "gemini-3.1-pro-preview",
+      model: "gemini-2.5-pro",
     });
   });
 });

ai_evals/adapters/frontend/core/shared/types.ts

@@ -28,8 +28,6 @@ export interface RawEvalResult<TOutput> {
   output: TOutput;
   error?: string;
   tokenUsage: TokenUsage;
-  /** Input tokens on the last model request of the loop (see BenchmarkAttemptResult.finalContextTokens). */
-  finalContextTokens: number | null;
   toolCallsCount: number;
   toolsCalled: string[];
   toolCallDetails: ToolCallDetail[];

ai_evals/adapters/frontend/datatableSqlEngine.test.ts

@@ -1,262 +0,0 @@
-import { describe, expect, it } from 'bun:test'
-import { applyDatatableSql, type BenchmarkDatatableSeed } from './datatableSqlEngine'
-
-function makeDatatable(): BenchmarkDatatableSeed {
-	return {
-		datatable_name: 'main',
-		schemas: {
-			public: {
-				orders: {
-					columns: { id: 'int4', customer_id: 'int4', total: 'numeric', status: 'text' },
-					rows: [
-						{ id: 1, customer_id: 1, total: 42.5, status: 'shipped' },
-						{ id: 2, customer_id: 2, total: 19.99, status: 'pending' },
-						{ id: 3, customer_id: 1, total: 88, status: 'shipped' }
-					]
-				},
-				customers: {
-					columns: { id: 'int4', name: 'text' },
-					rows: [{ id: 1, name: 'Alice' }]
-				}
-			}
-		}
-	}
-}
-
-describe('SELECT', () => {
-	it('returns the referenced table rows', () => {
-		const dt = makeDatatable()
-		expect(applyDatatableSql(dt, 'SELECT id, name FROM customers').rows).toEqual([
-			{ id: 1, name: 'Alice' }
-		])
-	})
-
-	it('falls back to the first table when no known table is referenced', () => {
-		const dt = makeDatatable()
-		expect(applyDatatableSql(dt, 'select 1').rows).toHaveLength(3)
-	})
-
-	it('resolves a schema-qualified table', () => {
-		const dt = makeDatatable()
-		expect(applyDatatableSql(dt, 'SELECT * FROM public.customers').rows).toEqual([
-			{ id: 1, name: 'Alice' }
-		])
-	})
-})
-
-describe('CREATE TABLE', () => {
-	it('adds a table with parsed columns, skipping table constraints and FK clauses', () => {
-		const dt = makeDatatable()
-		const result = applyDatatableSql(
-			dt,
-			'CREATE TABLE public.refunds (\n  order_id int4 NOT NULL REFERENCES public.orders(id),\n  amount numeric(10,2),\n  PRIMARY KEY (order_id)\n)'
-		)
-		expect(result.rows).toEqual([])
-		expect(dt.schemas.public.refunds).toEqual({
-			columns: { order_id: 'int4', amount: 'numeric(10,2)' },
-			rows: []
-		})
-	})
-
-	it('defaults an unqualified table to the public schema', () => {
-		const dt = makeDatatable()
-		applyDatatableSql(dt, 'CREATE TABLE notes (id int4, body text)')
-		expect(dt.schemas.public.notes.columns).toEqual({ id: 'int4', body: 'text' })
-	})
-
-	it('is a no-op for an existing table with IF NOT EXISTS', () => {
-		const dt = makeDatatable()
-		applyDatatableSql(dt, 'CREATE TABLE IF NOT EXISTS public.orders (x int4)')
-		expect(Object.keys(dt.schemas.public.orders.columns)).toContain('status')
-	})
-})
-
-describe('DROP TABLE', () => {
-	it('removes the table', () => {
-		const dt = makeDatatable()
-		applyDatatableSql(dt, 'DROP TABLE IF EXISTS public.customers')
-		expect(dt.schemas.public.customers).toBeUndefined()
-	})
-})
-
-describe('INSERT', () => {
-	it('appends a row using an explicit column list', () => {
-		const dt = makeDatatable()
-		applyDatatableSql(dt, "INSERT INTO customers (id, name) VALUES (2, 'Bob')")
-		expect(dt.schemas.public.customers.rows).toContainEqual({ id: 2, name: 'Bob' })
-	})
-
-	it('infers columns from the table when none are given, and appends multiple tuples', () => {
-		const dt = makeDatatable()
-		applyDatatableSql(dt, "INSERT INTO customers VALUES (2, 'Bob'), (3, 'Carol')")
-		expect(dt.schemas.public.customers.rows).toHaveLength(3)
-	})
-
-	it('returns the inserted rows when RETURNING is present', () => {
-		const dt = makeDatatable()
-		const result = applyDatatableSql(
-			dt,
-			"INSERT INTO customers (id, name) VALUES (2, 'Bob') RETURNING *"
-		)
-		expect(result.rows).toEqual([{ id: 2, name: 'Bob' }])
-	})
-})
-
-describe('UPDATE', () => {
-	it('updates only the rows matching an equality WHERE', () => {
-		const dt = makeDatatable()
-		const result = applyDatatableSql(
-			dt,
-			"UPDATE public.orders SET status = 'shipped' WHERE id = 2"
-		)
-		expect(result.rows).toEqual([])
-		expect(dt.schemas.public.orders.rows?.find((r) => r.id === 2)?.status).toBe('shipped')
-		expect(dt.schemas.public.orders.rows?.find((r) => r.id === 1)?.status).toBe('shipped')
-	})
-
-	it('strips a Postgres cast in the WHERE value', () => {
-		const dt = makeDatatable()
-		applyDatatableSql(dt, "UPDATE orders SET status = 'done' WHERE id = 2::int4")
-		expect(dt.schemas.public.orders.rows?.find((r) => r.id === 2)?.status).toBe('done')
-	})
-
-	it('matches multiple AND predicates including a numeric literal', () => {
-		const dt = makeDatatable()
-		applyDatatableSql(
-			dt,
-			"UPDATE orders SET status = 'done' WHERE customer_id = 2 AND total = 19.99"
-		)
-		expect(dt.schemas.public.orders.rows?.find((r) => r.id === 2)?.status).toBe('done')
-		expect(dt.schemas.public.orders.rows?.find((r) => r.id === 1)?.status).toBe('shipped')
-	})
-
-	it('updates every row when there is no WHERE', () => {
-		const dt = makeDatatable()
-		applyDatatableSql(dt, "UPDATE orders SET status = 'archived'")
-		expect(dt.schemas.public.orders.rows?.every((r) => r.status === 'archived')).toBe(true)
-	})
-
-	it('returns the affected rows when RETURNING is present', () => {
-		const dt = makeDatatable()
-		const result = applyDatatableSql(
-			dt,
-			"UPDATE orders SET status = 'shipped' WHERE id = 2 RETURNING *"
-		)
-		expect(result.rows).toHaveLength(1)
-		expect(result.rows[0]).toMatchObject({ id: 2, status: 'shipped' })
-	})
-
-	it('affects no rows when the WHERE clause cannot be parsed', () => {
-		const dt = makeDatatable()
-		applyDatatableSql(dt, "UPDATE orders SET status = 'x' WHERE total > 20")
-		expect(dt.schemas.public.orders.rows?.some((r) => r.status === 'x')).toBe(false)
-	})
-})
-
-describe('DELETE', () => {
-	it('removes only the matching rows', () => {
-		const dt = makeDatatable()
-		applyDatatableSql(dt, 'DELETE FROM orders WHERE id = 2')
-		expect(dt.schemas.public.orders.rows?.map((r) => r.id)).toEqual([1, 3])
-	})
-
-	it('returns the removed rows when RETURNING is present', () => {
-		const dt = makeDatatable()
-		const result = applyDatatableSql(dt, 'DELETE FROM orders WHERE id = 2 RETURNING *')
-		expect(result.rows).toEqual([{ id: 2, customer_id: 2, total: 19.99, status: 'pending' }])
-	})
-})
-
-describe('writes are reflected by later reads', () => {
-	it('UPDATE then SELECT sees the new value (the verify-loop fix)', () => {
-		const dt = makeDatatable()
-		applyDatatableSql(dt, "UPDATE orders SET status = 'shipped' WHERE id = 2")
-		const seen = applyDatatableSql(dt, 'SELECT * FROM orders').rows
-		expect(seen.find((r) => r.id === 2)?.status).toBe('shipped')
-	})
-
-	it('INSERT then SELECT sees the new row', () => {
-		const dt = makeDatatable()
-		applyDatatableSql(dt, "INSERT INTO customers (id, name) VALUES (9, 'Zed')")
-		const seen = applyDatatableSql(dt, 'SELECT * FROM customers').rows
-		expect(seen).toContainEqual({ id: 9, name: 'Zed' })
-	})
-
-	it('CREATE then SELECT on the new table returns its (empty) rows', () => {
-		const dt = makeDatatable()
-		applyDatatableSql(dt, 'CREATE TABLE public.refunds (order_id int4, amount numeric)')
-		expect(applyDatatableSql(dt, 'SELECT * FROM refunds').rows).toEqual([])
-	})
-})
-
-describe('system-catalog queries reflect the current tables/columns', () => {
-	it('lists current tables (including a freshly created one) via information_schema.tables', () => {
-		const dt = makeDatatable()
-		applyDatatableSql(dt, 'CREATE TABLE public.refunds (order_id int4)')
-		const rows = applyDatatableSql(
-			dt,
-			"SELECT table_name FROM information_schema.tables WHERE table_name = 'refunds'"
-		).rows
-		expect(rows.map((r) => r.table_name)).toContain('refunds')
-	})
-
-	it('does not list a dropped table', () => {
-		const dt = makeDatatable()
-		applyDatatableSql(dt, 'DROP TABLE public.customers')
-		const rows = applyDatatableSql(dt, 'SELECT table_name FROM information_schema.tables').rows
-		expect(rows.map((r) => r.table_name)).not.toContain('customers')
-	})
-
-	it('reports columns via information_schema.columns', () => {
-		const dt = makeDatatable()
-		const rows = applyDatatableSql(
-			dt,
-			"SELECT column_name FROM information_schema.columns WHERE table_name = 'orders'"
-		).rows
-		expect(rows.map((r) => r.column_name)).toContain('status')
-	})
-})
-
-describe('parser robustness (string/paren-aware splitting)', () => {
-	it('does not treat the word "returning" inside a string value as a RETURNING clause', () => {
-		const dt = makeDatatable()
-		const result = applyDatatableSql(
-			dt,
-			"INSERT INTO customers (id, name) VALUES (5, 'is returning soon')"
-		)
-		expect(result.rows).toEqual([])
-		expect(dt.schemas.public.customers.rows).toContainEqual({ id: 5, name: 'is returning soon' })
-	})
-
-	it('does not split on the word "where" inside a SET string value', () => {
-		const dt = makeDatatable()
-		applyDatatableSql(dt, "UPDATE orders SET status = 'ship where ordered' WHERE id = 2")
-		expect(dt.schemas.public.orders.rows?.find((r) => r.id === 2)?.status).toBe('ship where ordered')
-		expect(dt.schemas.public.orders.rows?.find((r) => r.id === 1)?.status).toBe('shipped')
-	})
-
-	it('keeps INSERT tuples intact when a value contains a function call', () => {
-		const dt = makeDatatable()
-		applyDatatableSql(dt, "INSERT INTO customers (id, name) VALUES (6, coalesce(NULL, 'x'))")
-		expect(dt.schemas.public.customers.rows).toHaveLength(2)
-		expect(dt.schemas.public.customers.rows?.[1]).toMatchObject({ id: 6 })
-	})
-
-	it('CREATE TABLE ignores a trailing semicolon-separated statement', () => {
-		const dt = makeDatatable()
-		applyDatatableSql(
-			dt,
-			'CREATE TABLE public.refunds (id int4, amount numeric); INSERT INTO refunds VALUES (1, 5)'
-		)
-		expect(dt.schemas.public.refunds.columns).toEqual({ id: 'int4', amount: 'numeric' })
-		expect(dt.schemas.public.refunds.rows).toEqual([])
-	})
-})
-
-describe('unparseable statements are a safe no-op', () => {
-	it('returns [] and does not throw', () => {
-		const dt = makeDatatable()
-		expect(applyDatatableSql(dt, 'VACUUM ANALYZE').rows).toEqual([])
-		expect(applyDatatableSql(dt, 'GRANT SELECT ON orders TO someone').rows).toEqual([])
-	})
-})

ai_evals/adapters/frontend/datatableSqlEngine.ts

@@ -1,541 +0,0 @@
-/**
- * A deliberately small, best-effort SQL engine for the benchmark datatable mock.
- *
- * This is NOT a real SQL implementation — it exists only so that writes a model
- * issues during an eval (`CREATE TABLE`, `INSERT`, `UPDATE`, `DELETE`, `DROP`)
- * become visible to its later reads (`list_datatables`, `get_datatable_table_schema`,
- * `SELECT`). Without that, a model that re-queries to verify a write sees stale
- * seed data, concludes the write failed, and loops until it exhausts its turns.
- *
- * It parses only the common statement shapes models produce. Anything it cannot
- * parse is a no-op success (it never throws) — behavioral evals assert that the
- * right statement was issued, not its exact data effects. Notable limits:
- *  - `SELECT` returns all rows of the referenced (or first) table — no WHERE
- *    filtering, projection, joins, or aggregation.
- *  - `WHERE` supports `col = value` predicates joined by `AND` only; an
- *    unparseable WHERE on UPDATE/DELETE affects zero rows (never the whole table).
- */
-
-/** One seeded datatable table: its columns (col -> compact_type) and optional rows. */
-export interface BenchmarkDatatableTableSeed {
-	columns: Record<string, string>
-	rows?: Record<string, unknown>[]
-}
-
-/** A seeded datatable: `datatable_name` plus a `schema -> table -> seed` map. */
-export interface BenchmarkDatatableSeed {
-	datatable_name: string
-	schemas: {
-		[schema: string]: {
-			[table: string]: BenchmarkDatatableTableSeed
-		}
-	}
-}
-
-export interface DatatableSqlResult {
-	rows: Record<string, unknown>[]
-}
-
-const DEFAULT_SCHEMA = 'public'
-
-type ParsedRef = { schema: string; table: string }
-type Predicate = { column: string; value: unknown }
-
-/**
- * Apply one SQL statement to `datatable` IN PLACE and return the result rows.
- * SELECT returns the referenced/first table's rows; a mutation returns its
- * affected rows when it has a RETURNING clause, otherwise `[]`.
- */
-export function applyDatatableSql(
-	datatable: BenchmarkDatatableSeed,
-	sql: string
-): DatatableSqlResult {
-	const statement = stripTrailingSemicolon(sql.trim())
-	if (/^\s*(with|select)\b/i.test(statement)) {
-		return { rows: selectRows(datatable, statement) }
-	}
-	if (/^\s*create\s+table\b/i.test(statement)) {
-		return { rows: applyCreateTable(datatable, statement) }
-	}
-	if (/^\s*drop\s+table\b/i.test(statement)) {
-		return { rows: applyDropTable(datatable, statement) }
-	}
-	if (/^\s*insert\s+into\b/i.test(statement)) {
-		return { rows: applyInsert(datatable, statement) }
-	}
-	if (/^\s*update\b/i.test(statement)) {
-		return { rows: applyUpdate(datatable, statement) }
-	}
-	if (/^\s*delete\s+from\b/i.test(statement)) {
-		return { rows: applyDelete(datatable, statement) }
-	}
-	return { rows: [] }
-}
-
-// ============= Reads =============
-
-function selectRows(
-	datatable: BenchmarkDatatableSeed,
-	sql: string
-): Record<string, unknown>[] {
-	const fromRef = sql.match(/\bfrom\s+([a-zA-Z_"][\w."]*)/i)?.[1]
-	if (fromRef) {
-		const catalog = catalogRows(datatable, fromRef)
-		if (catalog) {
-			return catalog
-		}
-	}
-	const table = fromRef ? resolveTable(datatable, fromRef) : undefined
-	const seed = table ?? firstTable(datatable)
-	return seed?.rows ?? []
-}
-
-/**
- * Synthesize rows for a system-catalog query so a model verifying a `CREATE`/`DROP`
- * via `information_schema.tables` / `.columns` (or `pg_tables`) sees the current
- * tables/columns instead of fallback data. WHERE is not applied, so the model gets
- * the full set and finds (or no longer finds) the table it just changed.
- * Returns `undefined` for non-catalog refs so normal table resolution proceeds.
- */
-function catalogRows(
-	datatable: BenchmarkDatatableSeed,
-	ref: string
-): Record<string, unknown>[] | undefined {
-	const normalized = ref.toLowerCase().replace(/"/g, '')
-	const name = normalized.split('.').pop()
-	const isCatalog = normalized.includes('information_schema.') || normalized.startsWith('pg_')
-	if (!isCatalog) {
-		return undefined
-	}
-	const tables = allTables(datatable)
-	if (name === 'tables' || name === 'pg_tables') {
-		return tables.map(({ schema, table }) => ({
-			table_schema: schema,
-			table_name: table,
-			schemaname: schema,
-			tablename: table
-		}))
-	}
-	if (name === 'columns') {
-		return tables.flatMap(({ schema, table, seed }) =>
-			Object.entries(seed.columns).map(([column, type]) => ({
-				table_schema: schema,
-				table_name: table,
-				column_name: column,
-				data_type: type
-			}))
-		)
-	}
-	return undefined
-}
-
-function allTables(
-	datatable: BenchmarkDatatableSeed
-): { schema: string; table: string; seed: BenchmarkDatatableTableSeed }[] {
-	return Object.entries(datatable.schemas).flatMap(([schema, tables]) =>
-		Object.entries(tables).map(([table, seed]) => ({ schema, table, seed }))
-	)
-}
-
-// ============= DDL =============
-
-function applyCreateTable(
-	datatable: BenchmarkDatatableSeed,
-	sql: string
-): Record<string, unknown>[] {
-	const head = sql.match(
-		/^\s*create\s+table\s+(?:if\s+not\s+exists\s+)?([a-zA-Z_"][\w."]*)/i
-	)
-	// The first top-level paren group is the column-definition list; using it (rather
-	// than a greedy `(...)` capture) ignores any trailing `;`-separated statement.
-	const columnText = extractParenGroups(sql)[0]
-	if (!head || columnText === undefined) {
-		return []
-	}
-	const { schema, table } = parseRef(head[1])
-	const existing = datatable.schemas[schema]?.[table]
-	if (existing) {
-		return []
-	}
-	const columns: Record<string, string> = {}
-	for (const rawDef of splitTopLevel(columnText)) {
-		const def = rawDef.trim()
-		if (!def || isTableConstraint(def)) {
-			continue
-		}
-		const tokens = def.split(/\s+/)
-		const column = unquoteIdentifier(tokens[0])
-		if (!column) {
-			continue
-		}
-		columns[column] = tokens[1] ?? 'text'
-	}
-	if (!datatable.schemas[schema]) {
-		datatable.schemas[schema] = {}
-	}
-	datatable.schemas[schema][table] = { columns, rows: [] }
-	return []
-}
-
-function applyDropTable(
-	datatable: BenchmarkDatatableSeed,
-	sql: string
-): Record<string, unknown>[] {
-	const match = sql.match(
-		/^\s*drop\s+table\s+(?:if\s+exists\s+)?([a-zA-Z_"][\w."]*)/i
-	)
-	if (!match) {
-		return []
-	}
-	const { schema, table } = parseRef(match[1])
-	if (datatable.schemas[schema]?.[table]) {
-		delete datatable.schemas[schema][table]
-	}
-	return []
-}
-
-// ============= DML =============
-
-function applyInsert(
-	datatable: BenchmarkDatatableSeed,
-	sql: string
-): Record<string, unknown>[] {
-	const { body, returning } = splitOffReturning(sql)
-	const match = body.match(
-		/^\s*insert\s+into\s+([a-zA-Z_"][\w."]*)\s*(?:\(([^)]*)\))?\s*values\s*([\s\S]+)$/i
-	)
-	if (!match) {
-		return []
-	}
-	const table = resolveTable(datatable, match[1])
-	if (!table) {
-		return []
-	}
-	const columns = match[2]
-		? splitTopLevel(match[2]).map((entry) => unquoteIdentifier(entry.trim()))
-		: Object.keys(table.columns)
-	const inserted: Record<string, unknown>[] = []
-	for (const tuple of extractParenGroups(match[3])) {
-		const values = splitTopLevel(tuple).map((entry) => parseValue(entry))
-		const row: Record<string, unknown> = {}
-		columns.forEach((column, index) => {
-			row[column] = values[index]
-		})
-		inserted.push(row)
-	}
-	table.rows ??= []
-	table.rows.push(...inserted)
-	return returning ? inserted : []
-}
-
-function applyUpdate(
-	datatable: BenchmarkDatatableSeed,
-	sql: string
-): Record<string, unknown>[] {
-	const { body, returning } = splitOffReturning(sql)
-	const match = body.match(/^\s*update\s+([a-zA-Z_"][\w."]*)\s+set\s+([\s\S]+)$/i)
-	if (!match) {
-		return []
-	}
-	const table = resolveTable(datatable, match[1])
-	if (!table) {
-		return []
-	}
-	let assignmentText = match[2]
-	let whereText: string | undefined
-	const whereMatch = maskForClauseScan(assignmentText).match(/\swhere\s/i)
-	if (whereMatch && whereMatch.index !== undefined) {
-		whereText = assignmentText.slice(whereMatch.index + whereMatch[0].length)
-		assignmentText = assignmentText.slice(0, whereMatch.index)
-	}
-	const predicates = parsePredicates(whereText)
-	if (predicates === null) {
-		return []
-	}
-	const assignments: Record<string, unknown> = {}
-	for (const entry of splitTopLevel(assignmentText)) {
-		const pair = entry.match(/^\s*([a-zA-Z_"][\w."]*)\s*=\s*([\s\S]+?)\s*$/)
-		if (pair) {
-			assignments[lastIdentifier(pair[1])] = parseValue(pair[2])
-		}
-	}
-	const affected = (table.rows ?? []).filter((row) => rowMatches(row, predicates))
-	for (const row of affected) {
-		Object.assign(row, assignments)
-	}
-	return returning ? affected : []
-}
-
-function applyDelete(
-	datatable: BenchmarkDatatableSeed,
-	sql: string
-): Record<string, unknown>[] {
-	const { body, returning } = splitOffReturning(sql)
-	const match = body.match(/^\s*delete\s+from\s+([a-zA-Z_"][\w."]*)\s*([\s\S]*)$/i)
-	if (!match) {
-		return []
-	}
-	const table = resolveTable(datatable, match[1])
-	if (!table) {
-		return []
-	}
-	const whereText = match[2].replace(/^\s*where\s+/i, '').trim() || undefined
-	const predicates = parsePredicates(whereText)
-	if (predicates === null) {
-		return []
-	}
-	const rows = table.rows ?? []
-	const removed = rows.filter((row) => rowMatches(row, predicates))
-	table.rows = rows.filter((row) => !rowMatches(row, predicates))
-	return returning ? removed : []
-}
-
-// ============= Parsing helpers =============
-
-function resolveTable(
-	datatable: BenchmarkDatatableSeed,
-	ref: string
-): BenchmarkDatatableTableSeed | undefined {
-	const { schema, table } = parseRef(ref)
-	const direct = datatable.schemas[schema]?.[table]
-	if (direct) {
-		return direct
-	}
-	// Bare table name: fall back to searching every schema for a matching table.
-	if (!ref.includes('.')) {
-		for (const tables of Object.values(datatable.schemas)) {
-			if (tables[table]) {
-				return tables[table]
-			}
-		}
-	}
-	return undefined
-}
-
-function firstTable(
-	datatable: BenchmarkDatatableSeed
-): BenchmarkDatatableTableSeed | undefined {
-	for (const tables of Object.values(datatable.schemas)) {
-		for (const seed of Object.values(tables)) {
-			return seed
-		}
-	}
-	return undefined
-}
-
-function parseRef(ref: string): ParsedRef {
-	const parts = ref.split('.').map(unquoteIdentifier)
-	if (parts.length >= 2) {
-		return { schema: parts[parts.length - 2], table: parts[parts.length - 1] }
-	}
-	return { schema: DEFAULT_SCHEMA, table: parts[0] }
-}
-
-/** A WHERE clause with no parseable form returns `null`; absent WHERE returns `[]` (match all). */
-function parsePredicates(whereText: string | undefined): Predicate[] | null {
-	if (whereText === undefined || whereText.trim() === '') {
-		return []
-	}
-	const predicates: Predicate[] = []
-	for (const part of whereText.split(/\s+and\s+/i)) {
-		const match = part.match(/^\s*([a-zA-Z_"][\w."]*)\s*=\s*([\s\S]+?)\s*$/)
-		if (!match) {
-			return null
-		}
-		predicates.push({ column: lastIdentifier(match[1]), value: parseValue(match[2]) })
-	}
-	return predicates
-}
-
-function rowMatches(row: Record<string, unknown>, predicates: Predicate[]): boolean {
-	return predicates.every((predicate) => looseEquals(row[predicate.column], predicate.value))
-}
-
-function looseEquals(left: unknown, right: unknown): boolean {
-	if (left === null || left === undefined) {
-		return right === null || right === undefined
-	}
-	if (typeof left === 'number' && typeof right === 'number') {
-		return left === right
-	}
-	return String(left) === String(right)
-}
-
-function parseValue(raw: string): unknown {
-	// Drop a trailing Postgres cast (e.g. `2::int4`) before interpreting the literal.
-	const token = raw.trim().replace(/::\s*[a-zA-Z_][\w]*(\([^)]*\))?\s*$/, '').trim()
-	const stringMatch = token.match(/^'([\s\S]*)'$/)
-	if (stringMatch) {
-		return stringMatch[1].replace(/''/g, "'")
-	}
-	if (/^-?\d+(\.\d+)?$/.test(token)) {
-		return Number(token)
-	}
-	if (/^true$/i.test(token)) {
-		return true
-	}
-	if (/^false$/i.test(token)) {
-		return false
-	}
-	if (/^null$/i.test(token)) {
-		return null
-	}
-	return token
-}
-
-function splitOffReturning(sql: string): { body: string; returning: boolean } {
-	const match = maskForClauseScan(sql).match(/\sreturning\s/i)
-	if (!match || match.index === undefined) {
-		return { body: sql, returning: false }
-	}
-	return { body: sql.slice(0, match.index), returning: true }
-}
-
-/**
- * A same-length copy of `sql` with the contents of single-quoted strings and
- * parenthesized groups blanked to spaces, so a top-level keyword scan
- * (WHERE / RETURNING) cannot match inside a string literal or a subquery. Index
- * positions in the result map 1:1 back onto the original.
- */
-function maskForClauseScan(sql: string): string {
-	let masked = ''
-	let depth = 0
-	let inString = false
-	for (let i = 0; i < sql.length; i++) {
-		const char = sql[i]
-		if (inString) {
-			if (char === "'") {
-				if (sql[i + 1] === "'") {
-					masked += '  '
-					i++
-					continue
-				}
-				inString = false
-			}
-			masked += ' '
-			continue
-		}
-		if (char === "'") {
-			inString = true
-			masked += ' '
-		} else if (char === '(') {
-			depth++
-			masked += ' '
-		} else if (char === ')') {
-			depth = Math.max(0, depth - 1)
-			masked += ' '
-		} else {
-			masked += depth > 0 ? ' ' : char
-		}
-	}
-	return masked
-}
-
-/**
- * Inner text of each top-level `( ... )` group in `input`, honoring nested parens
- * (e.g. `now()`, `numeric(10,2)`) and single-quoted strings. Used for the CREATE
- * column-definition group and INSERT value tuples.
- */
-function extractParenGroups(input: string): string[] {
-	const groups: string[] = []
-	let depth = 0
-	let inString = false
-	let current = ''
-	for (let i = 0; i < input.length; i++) {
-		const char = input[i]
-		if (inString) {
-			current += char
-			if (char === "'") {
-				if (input[i + 1] === "'") {
-					current += input[++i]
-				} else {
-					inString = false
-				}
-			}
-			continue
-		}
-		if (char === "'") {
-			inString = true
-			current += char
-		} else if (char === '(') {
-			depth++
-			if (depth === 1) {
-				current = ''
-			} else {
-				current += char
-			}
-		} else if (char === ')') {
-			depth = Math.max(0, depth - 1)
-			if (depth === 0) {
-				groups.push(current)
-				current = ''
-			} else {
-				current += char
-			}
-		} else if (depth > 0) {
-			current += char
-		}
-	}
-	return groups
-}
-
-/** Split on commas that are not inside parentheses or single-quoted strings. */
-function splitTopLevel(input: string): string[] {
-	const parts: string[] = []
-	let depth = 0
-	let inString = false
-	let current = ''
-	for (let i = 0; i < input.length; i++) {
-		const char = input[i]
-		if (inString) {
-			current += char
-			if (char === "'") {
-				if (input[i + 1] === "'") {
-					current += input[++i]
-				} else {
-					inString = false
-				}
-			}
-			continue
-		}
-		if (char === "'") {
-			inString = true
-			current += char
-		} else if (char === '(') {
-			depth++
-			current += char
-		} else if (char === ')') {
-			depth = Math.max(0, depth - 1)
-			current += char
-		} else if (char === ',' && depth === 0) {
-			parts.push(current)
-			current = ''
-		} else {
-			current += char
-		}
-	}
-	if (current.trim() !== '') {
-		parts.push(current)
-	}
-	return parts
-}
-
-function isTableConstraint(def: string): boolean {
-	return /^(primary\s+key|foreign\s+key|constraint|unique|check|exclude|like)\b/i.test(def)
-}
-
-function unquoteIdentifier(identifier: string): string {
-	const trimmed = identifier.trim()
-	const quoted = trimmed.match(/^"([\s\S]*)"$/)
-	return quoted ? quoted[1] : trimmed
-}
-
-/** For a qualified reference like `orders.id`, keep only the final identifier. */
-function lastIdentifier(reference: string): string {
-	const parts = reference.split('.')
-	return unquoteIdentifier(parts[parts.length - 1])
-}
-
-function stripTrailingSemicolon(sql: string): string {
-	return sql.replace(/;\s*$/, '')
-}

ai_evals/adapters/frontend/mockBackend.ts

@@ -1,25 +1,7 @@
 import { randomUUID } from 'node:crypto'
-import type {
-	AppWithLastVersion,
-	CompletedJob,
-	Flow,
-	Job,
-	ListableApp,
-	Script
-} from '../../../frontend/src/lib/gen'
-import type {
-	DataTableTables,
-	DataTableTableSchema,
-	GetDraftForUserResponse,
-	ListDraftsResponse,
-	ScriptLang,
-	UpdateDraftResponse,
-	UserDraftItemKind
-} from '../../../frontend/src/lib/gen/types.gen'
+import type { CompletedJob, Flow, Script } from '../../../frontend/src/lib/gen'
+import type { ScriptLang } from '../../../frontend/src/lib/gen/types.gen'
 import { buildScriptLintResult } from './core/script/preview'
-import { applyDatatableSql, type BenchmarkDatatableSeed } from './datatableSqlEngine'
-
-export type { BenchmarkDatatableSeed, BenchmarkDatatableTableSeed } from './datatableSqlEngine'
 
 const BENCHMARK_TIMESTAMP = '1970-01-01T00:00:00.000Z'
 
@@ -40,54 +22,21 @@ export interface BenchmarkWorkspaceFlow {
 	value: Flow['value']
 }
 
-export interface BenchmarkWorkspaceApp {
-	path: string
-	summary: string
-	value: {
-		files: Record<string, string>
-		runnables: Record<string, unknown>
-		data?: unknown
-		policy?: unknown
-		custom_path?: unknown
-	}
-}
-
-export interface BenchmarkWorkspaceJob {
-	/** Stable id so a case prompt can reference a specific run (e.g. for get_job_logs). */
-	id?: string
-	jobKind?: CompletedJob['job_kind']
-	scriptPath?: string
-	createdBy?: string
-	label?: string
-	success?: boolean
-	logs?: string
-}
-
 export interface BenchmarkWorkspaceRunnables {
 	scripts?: BenchmarkWorkspaceScript[]
 	flows?: BenchmarkWorkspaceFlow[]
-	apps?: BenchmarkWorkspaceApp[]
-	datatables?: BenchmarkDatatableSeed[]
-	jobs?: BenchmarkWorkspaceJob[]
 }
 
 type BenchmarkCompletedJob = CompletedJob & { type: 'CompletedJob' }
 
 const benchmarkWorkspaces = new Set<string>()
 const benchmarkWorkspaceRunnables = new Map<string, BenchmarkWorkspaceRunnables>()
-// Keyed by `${workspace}::${jobId}` so concurrent attempts (or distinct cases)
-// can seed the same fixed job id without clobbering each other's entry.
 const benchmarkJobs = new Map<string, { workspace: string; job: BenchmarkCompletedJob }>()
 
-function benchmarkJobKey(workspace: string, jobId: string): string {
-	return `${workspace}::${jobId}`
-}
-
 export function resetBenchmarkMockBackend(): void {
 	benchmarkWorkspaces.clear()
 	benchmarkWorkspaceRunnables.clear()
 	benchmarkJobs.clear()
-	benchmarkDrafts.clear()
 }
 
 export function registerBenchmarkWorkspace(workspace: string): void {
@@ -99,33 +48,12 @@ export function registerBenchmarkWorkspaceRunnables(
 	runnables: BenchmarkWorkspaceRunnables
 ): void {
 	benchmarkWorkspaces.add(workspace)
-	// Fresh case: drop any drafts left from a prior run on this workspace id.
-	clearBenchmarkDrafts(workspace)
-	// Datatables are mutated in place by exec_datatable_sql (a write must be visible
-	// to later reads), so store an isolated deep copy — never mutate the caller's seed.
-	benchmarkWorkspaceRunnables.set(workspace, {
-		...runnables,
-		datatables: runnables.datatables ? structuredClone(runnables.datatables) : undefined
-	})
-	// Seed any fixture jobs so list_runs / get_job_logs have data to return.
-	for (const seed of runnables.jobs ?? []) {
-		createBenchmarkCompletedJob({
-			workspace,
-			id: seed.id,
-			jobKind: seed.jobKind ?? 'script',
-			success: seed.success,
-			scriptPath: seed.scriptPath,
-			createdBy: seed.createdBy,
-			label: seed.label,
-			logs: seed.logs
-		})
-	}
+	benchmarkWorkspaceRunnables.set(workspace, runnables)
 }
 
 export function unregisterBenchmarkWorkspace(workspace: string): void {
 	benchmarkWorkspaces.delete(workspace)
 	benchmarkWorkspaceRunnables.delete(workspace)
-	clearBenchmarkDrafts(workspace)
 	for (const [jobId, entry] of benchmarkJobs.entries()) {
 		if (entry.workspace === workspace) {
 			benchmarkJobs.delete(jobId)
@@ -181,22 +109,6 @@ export function getBenchmarkFlowByPath(workspace: string, path: string): Flow |
 	return flow ? buildBenchmarkFlow(flow) : null
 }
 
-export function listBenchmarkApps(workspace: string): ListableApp[] | null {
-	const runnables = benchmarkWorkspaceRunnables.get(workspace)
-	if (!runnables) {
-		return null
-	}
-	return (runnables.apps ?? []).map(buildBenchmarkListableApp)
-}
-
-export function getBenchmarkAppByPath(workspace: string, path: string): AppWithLastVersion | null {
-	const app = benchmarkWorkspaceRunnables
-		.get(workspace)
-		?.apps?.find((entry) => entry.path === path)
-
-	return app ? buildBenchmarkApp(app) : null
-}
-
 export function createBenchmarkCompletedJob(input: {
 	workspace: string
 	jobKind: CompletedJob['job_kind']
@@ -206,17 +118,14 @@ export function createBenchmarkCompletedJob(input: {
 	scriptPath?: string
 	scriptHash?: string
 	args?: Record<string, unknown>
-	id?: string
-	createdBy?: string
-	label?: string
 }): string {
-	const jobId = input.id ?? `benchmark-job-${randomUUID()}`
+	const jobId = `benchmark-job-${randomUUID()}`
 	const now = new Date().toISOString()
 	const job: BenchmarkCompletedJob = {
 		type: 'CompletedJob',
 		id: jobId,
 		workspace_id: input.workspace,
-		created_by: input.createdBy ?? 'ai-evals',
+		created_by: 'ai-evals',
 		created_at: now,
 		started_at: now,
 		completed_at: now,
@@ -234,11 +143,10 @@ export function createBenchmarkCompletedJob(input: {
 		is_skipped: false,
 		email: 'ai-evals@local',
 		visible_to_owner: true,
-		tag: 'benchmark',
-		labels: input.label ? [input.label] : undefined
+		tag: 'benchmark'
 	}
 
-	benchmarkJobs.set(benchmarkJobKey(input.workspace, jobId), { workspace: input.workspace, job })
+	benchmarkJobs.set(jobId, { workspace: input.workspace, job })
 	return jobId
 }
 
@@ -246,239 +154,13 @@ export function getBenchmarkCompletedJob(
 	workspace: string,
 	jobId: string
 ): BenchmarkCompletedJob | null {
-	const entry = benchmarkJobs.get(benchmarkJobKey(workspace, jobId))
-	if (!entry) {
+	const entry = benchmarkJobs.get(jobId)
+	if (!entry || entry.workspace !== workspace) {
 		return null
 	}
 	return structuredClone(entry.job)
 }
 
-/**
- * List seeded/recorded jobs for a benchmark workspace, most recent first —
- * the shape `JobService.listJobs` returns. Returns `null` for a non-benchmark
- * workspace so the caller can fall through to the real backend. Server-side
- * filters (path/creator/status/limit) are intentionally not applied: global
- * eval cases assert on the recorded `list_runs` tool call, not on filtering.
- */
-export function listBenchmarkJobs(workspace: string): Job[] | null {
-	if (!hasBenchmarkWorkspace(workspace)) {
-		return null
-	}
-	return [...benchmarkJobs.values()]
-		.filter((entry) => entry.workspace === workspace)
-		.map((entry) => structuredClone(entry.job) as Job)
-		.sort((a, b) => (b.created_at ?? '').localeCompare(a.created_at ?? ''))
-}
-
-/**
- * Mirror `JobService.getJobLogs` (response is the raw log string). Throws a
- * "not found" error for an unknown id, matching the backend 404.
- */
-export function getBenchmarkJobLogs(workspace: string, jobId: string): string {
-	const job = getBenchmarkCompletedJob(workspace, jobId)
-	if (!job) {
-		throw new Error(`Job Logs not found for "${jobId}"`)
-	}
-	return job.logs ?? ''
-}
-
-// ============= Drafts (per-user, DB-backed in production) =============
-
-/**
- * In-memory stand-in for the per-user draft backend (`DraftService`). The global
- * AI chat now persists and reads drafts through the backend DB instead of an
- * in-tab `UserDraft` cell, so the eval mocks the three draft endpoints it
- * exercises (`updateDraft` / `getDraftForUser` / `listDrafts`) and keeps the
- * saved values here, keyed by workspace + draft kind + storage path. Mirrors the
- * semantics of the production unit test's mock in
- * `frontend/src/lib/components/copilot/chat/global/core.test.ts`.
- */
-const benchmarkDrafts = new Map<
-	string,
-	{ workspace: string; kind: UserDraftItemKind; path: string; value: unknown }
->()
-
-// Fixed timestamp so artifacts stay deterministic. No eval simulates a
-// concurrent writer, so every save is accepted and the conflict branch is
-// never taken — the syncer just records this as its `last_sync` baseline.
-const BENCHMARK_DRAFT_TIMESTAMP = '1970-01-01T00:00:00.000Z'
-
-function benchmarkDraftKey(workspace: string, kind: string, path: string): string {
-	return `${workspace}::${kind}::${path}`
-}
-
-export function clearBenchmarkDrafts(workspace: string): void {
-	for (const [key, entry] of benchmarkDrafts.entries()) {
-		if (entry.workspace === workspace) {
-			benchmarkDrafts.delete(key)
-		}
-	}
-}
-
-/**
- * Seed a draft straight into the store — used by the eval's live-editor draft
- * fixtures, which model "the user already has this draft open/saved". Writing it
- * here (instead of through `UserDraft.save`) keeps it a backend draft row with no
- * shadowing in-tab cell, so a model edit that persists to the backend is what the
- * output read-back captures — not the stale seed.
- */
-export function seedBenchmarkDraft(
-	workspace: string,
-	kind: UserDraftItemKind,
-	path: string,
-	value: unknown
-): void {
-	benchmarkDrafts.set(benchmarkDraftKey(workspace, kind, path), {
-		workspace,
-		kind,
-		path,
-		value
-	})
-}
-
-/** Mirror `DraftService.updateDraft`: a `null`/omitted value deletes the row. */
-export function updateBenchmarkDraft(input: {
-	workspace: string
-	kind: UserDraftItemKind
-	path: string
-	requestBody?: { value?: unknown }
-}): UpdateDraftResponse {
-	const key = benchmarkDraftKey(input.workspace, input.kind, input.path)
-	const value = input.requestBody?.value
-	if (value == null) {
-		benchmarkDrafts.delete(key)
-	} else {
-		benchmarkDrafts.set(key, {
-			workspace: input.workspace,
-			kind: input.kind,
-			path: input.path,
-			value
-		})
-	}
-	return { status: 'saved', current_timestamp: BENCHMARK_DRAFT_TIMESTAMP }
-}
-
-/** Mirror `DraftService.getDraftForUser`: 404-shaped throw when absent so the
- * adapter's narrowed catch treats it as "no draft" instead of re-throwing. */
-export function getBenchmarkDraftForUser(input: {
-	workspace: string
-	kind: UserDraftItemKind
-	path: string
-}): GetDraftForUserResponse {
-	const entry = benchmarkDrafts.get(benchmarkDraftKey(input.workspace, input.kind, input.path))
-	if (!entry) {
-		throw Object.assign(new Error(`no draft for "${input.path}"`), { status: 404 })
-	}
-	return { value: entry.value, created_at: BENCHMARK_DRAFT_TIMESTAMP }
-}
-
-/** Mirror `DraftService.listDrafts`: metadata rows (no value) for a workspace. */
-export function listBenchmarkDrafts(workspace: string): ListDraftsResponse {
-	return [...benchmarkDrafts.values()]
-		.filter((entry) => entry.workspace === workspace)
-		.map((entry) => ({
-			kind: entry.kind,
-			path: entry.path,
-			summary: (entry.value as { summary?: string } | null)?.summary,
-			draft_only: true,
-			legacy_draft: false,
-			created_at: BENCHMARK_DRAFT_TIMESTAMP
-		}))
-}
-
-// ============= Datatables (best-effort in-memory SQL) =============
-
-/**
- * Project the seeded datatables down to the `list_datatable_tables` response:
- * `datatable_name` + `schema -> table_names`, with no column detail.
- * Returns `null` for a non-benchmark workspace so callers can fall through to
- * the real backend; an empty seed yields `[]`.
- */
-export function listBenchmarkDatatables(workspace: string): DataTableTables[] | null {
-	const runnables = benchmarkWorkspaceRunnables.get(workspace)
-	if (!runnables) {
-		return null
-	}
-	return (runnables.datatables ?? []).map((datatable) => ({
-		datatable_name: datatable.datatable_name,
-		schemas: Object.fromEntries(
-			Object.entries(datatable.schemas).map(([schema, tables]) => [schema, Object.keys(tables)])
-		)
-	}))
-}
-
-export function getBenchmarkDatatableSchema(input: {
-	workspace: string
-	datatableName: string
-	schemaName: string
-	tableName: string
-}): DataTableTableSchema {
-	const runnables = benchmarkWorkspaceRunnables.get(input.workspace)
-	const datatable = (runnables?.datatables ?? []).find(
-		(entry) => entry.datatable_name === input.datatableName
-	)
-	if (!datatable) {
-		// Message MUST match the production `isDatatableNotConfiguredError` regex
-		// (/datatable\s+\S+\s+not found/i in datatableTools.ts) so the
-		// get_datatable_table_schema not-configured mapping is actually exercised.
-		throw new Error(`datatable "${input.datatableName}" not found`)
-	}
-	const table = datatable.schemas?.[input.schemaName]?.[input.tableName]
-	if (!table) {
-		throw new Error(
-			`table "${input.schemaName}.${input.tableName}" not found in datatable "${input.datatableName}"`
-		)
-	}
-	return {
-		datatable_name: input.datatableName,
-		schema_name: input.schemaName,
-		table_name: input.tableName,
-		columns: table.columns
-	}
-}
-
-/**
- * Execute SQL against a seeded datatable through the best-effort in-memory engine
- * (`applyDatatableSql`). Writes (CREATE/INSERT/UPDATE/DELETE/DROP) mutate the
- * stored datatable in place so a later list/schema/SELECT reflects them; SELECT
- * (and RETURNING) yield rows, other statements yield `[]`. Creates a benchmark
- * completed job and returns its id, like `runBenchmarkScriptPreview`.
- */
-export function runBenchmarkDatatableSql(input: {
-	workspace: string
-	datatableName: string
-	sql: string
-}): string {
-	const runnables = benchmarkWorkspaceRunnables.get(input.workspace)
-	const datatable = (runnables?.datatables ?? []).find(
-		(entry) => entry.datatable_name === input.datatableName
-	)
-	const rows = datatable ? applyDatatableSql(datatable, input.sql).rows : []
-	return createBenchmarkCompletedJob({
-		workspace: input.workspace,
-		jobKind: 'preview',
-		success: true,
-		args: { database: `datatable://${input.datatableName}` },
-		result: rows
-	})
-}
-
-/**
- * Mirror `JobService.getCompletedJobResultMaybe` for benchmark workspaces — the
- * shape `pollJobResult` consumes. The job is created synchronously before
- * polling, so it is always present and completed.
- */
-export function getBenchmarkCompletedJobResultMaybe(input: {
-	workspace: string
-	id: string
-}): { success: boolean; completed: boolean; result: unknown } {
-	const job = getBenchmarkCompletedJob(input.workspace, input.id)
-	if (!job) {
-		throw new Error(`Job "${input.id}" not found in benchmark workspace`)
-	}
-	return { success: job.success, completed: true, result: job.result }
-}
-
 export function runBenchmarkScriptPreview(input: {
 	workspace: string
 	requestBody: {
@@ -640,35 +322,3 @@ function buildBenchmarkFlow(flow: BenchmarkWorkspaceFlow): Flow {
 		extra_perms: {}
 	} as Flow
 }
-
-function buildBenchmarkListableApp(app: BenchmarkWorkspaceApp): ListableApp {
-	return {
-		id: 0,
-		workspace_id: 'benchmark',
-		path: app.path,
-		summary: app.summary,
-		version: 1,
-		extra_perms: {},
-		edited_at: BENCHMARK_TIMESTAMP,
-		execution_mode: 'viewer',
-		raw_app: true
-	}
-}
-
-function buildBenchmarkApp(app: BenchmarkWorkspaceApp): AppWithLastVersion {
-	return {
-		id: 0,
-		workspace_id: 'benchmark',
-		path: app.path,
-		summary: app.summary,
-		versions: [1],
-		created_by: 'benchmark',
-		created_at: BENCHMARK_TIMESTAMP,
-		value: app.value,
-		policy: (app.value.policy ?? {}) as AppWithLastVersion['policy'],
-		execution_mode: 'viewer',
-		extra_perms: {},
-		custom_path: app.value.custom_path as string | undefined,
-		raw_app: true
-	}
-}

ai_evals/adapters/frontend/mockBackendDatatables.test.ts

@@ -1,175 +0,0 @@
-import { afterEach, beforeEach, describe, expect, it } from 'bun:test'
-import {
-	getBenchmarkCompletedJobResultMaybe,
-	getBenchmarkDatatableSchema,
-	listBenchmarkDatatables,
-	registerBenchmarkWorkspaceRunnables,
-	resetBenchmarkMockBackend,
-	runBenchmarkDatatableSql,
-	type BenchmarkWorkspaceRunnables
-} from './mockBackend'
-
-const WORKSPACE = 'benchmark-datatable-ws'
-
-// Mirrors the production `isDatatableNotConfiguredError` regex in
-// datatableTools.ts. The schema mock's "not configured" message MUST match it,
-// otherwise the not-configured mapping in get_datatable_table_schema is silently
-// untested.
-const NOT_CONFIGURED_RE = /datatable\s+\S+\s+not found/i
-
-const SEED: BenchmarkWorkspaceRunnables = {
-	datatables: [
-		{
-			datatable_name: 'main',
-			schemas: {
-				public: {
-					orders: {
-						columns: { id: 'int', total: 'numeric' },
-						rows: [
-							{ id: 1, total: 10 },
-							{ id: 2, total: 20 }
-						]
-					},
-					customers: {
-						columns: { id: 'int', name: 'text' },
-						rows: [{ id: 1, name: 'alice' }]
-					}
-				}
-			}
-		}
-	]
-}
-
-beforeEach(() => resetBenchmarkMockBackend())
-afterEach(() => resetBenchmarkMockBackend())
-
-describe('listBenchmarkDatatables', () => {
-	it('returns null for a non-benchmark workspace (caller falls through to real backend)', () => {
-		expect(listBenchmarkDatatables('unregistered')).toBeNull()
-	})
-
-	it('returns [] for a registered workspace with no datatables seed', () => {
-		registerBenchmarkWorkspaceRunnables(WORKSPACE, {})
-		expect(listBenchmarkDatatables(WORKSPACE)).toEqual([])
-	})
-
-	it('projects seeded datatables to schema -> table names only (no columns)', () => {
-		registerBenchmarkWorkspaceRunnables(WORKSPACE, SEED)
-		expect(listBenchmarkDatatables(WORKSPACE)).toEqual([
-			{ datatable_name: 'main', schemas: { public: ['orders', 'customers'] } }
-		])
-	})
-})
-
-describe('getBenchmarkDatatableSchema', () => {
-	beforeEach(() => registerBenchmarkWorkspaceRunnables(WORKSPACE, SEED))
-
-	it('returns the columns for a seeded table', () => {
-		expect(
-			getBenchmarkDatatableSchema({
-				workspace: WORKSPACE,
-				datatableName: 'main',
-				schemaName: 'public',
-				tableName: 'orders'
-			})
-		).toEqual({
-			datatable_name: 'main',
-			schema_name: 'public',
-			table_name: 'orders',
-			columns: { id: 'int', total: 'numeric' }
-		})
-	})
-
-	it('throws a not-configured error matching the production regex for an unknown datatable', () => {
-		let error: Error | undefined
-		try {
-			getBenchmarkDatatableSchema({
-				workspace: WORKSPACE,
-				datatableName: 'ghost',
-				schemaName: 'public',
-				tableName: 'orders'
-			})
-		} catch (e) {
-			error = e as Error
-		}
-		expect(error).toBeDefined()
-		expect(error!.message).toMatch(NOT_CONFIGURED_RE)
-	})
-
-	it('throws a table-not-found error that does NOT match the datatable-not-configured regex', () => {
-		// The datatable IS configured; only the table is missing. Production maps
-		// this to a generic "error getting schema", not the blocking message.
-		let error: Error | undefined
-		try {
-			getBenchmarkDatatableSchema({
-				workspace: WORKSPACE,
-				datatableName: 'main',
-				schemaName: 'public',
-				tableName: 'ghost'
-			})
-		} catch (e) {
-			error = e as Error
-		}
-		expect(error).toBeDefined()
-		expect(error!.message).not.toMatch(NOT_CONFIGURED_RE)
-	})
-})
-
-describe('runBenchmarkDatatableSql + getBenchmarkCompletedJobResultMaybe', () => {
-	beforeEach(() => registerBenchmarkWorkspaceRunnables(WORKSPACE, SEED))
-
-	function exec(sql: string): { success: boolean; completed: boolean; result: unknown } {
-		const jobId = runBenchmarkDatatableSql({ workspace: WORKSPACE, datatableName: 'main', sql })
-		return getBenchmarkCompletedJobResultMaybe({ workspace: WORKSPACE, id: jobId })
-	}
-
-	it('returns the canned rows of the table named in a SELECT FROM clause', () => {
-		expect(exec('SELECT * FROM customers')).toEqual({
-			success: true,
-			completed: true,
-			result: [{ id: 1, name: 'alice' }]
-		})
-	})
-
-	it('falls back to the first seeded table when the SELECT references no known table', () => {
-		expect(exec('select 1').result).toEqual([
-			{ id: 1, total: 10 },
-			{ id: 2, total: 20 }
-		])
-	})
-
-	it('returns [] success for DDL and DML statements without RETURNING', () => {
-		expect(exec('CREATE TABLE foo (id int)').result).toEqual([])
-		expect(exec('INSERT INTO orders VALUES (3, 30)').result).toEqual([])
-		expect(exec('update orders set total = 0').result).toEqual([])
-	})
-
-	it('reflects a write in a later SELECT, isolated from the shared seed', () => {
-		exec('UPDATE orders SET total = 999 WHERE id = 1')
-		expect((exec('SELECT * FROM orders').result as Record<string, unknown>[])).toContainEqual({
-			id: 1,
-			total: 999
-		})
-		// Registration deep-clones the seed, so the shared SEED const stays pristine.
-		expect(SEED.datatables![0].schemas.public.orders.rows).toContainEqual({ id: 1, total: 10 })
-	})
-
-	it('reflects a CREATE in list_datatables and get_datatable_table_schema', () => {
-		exec('CREATE TABLE public.refunds (order_id int4, amount numeric)')
-		expect(listBenchmarkDatatables(WORKSPACE)?.[0].schemas.public).toContain('refunds')
-		expect(
-			getBenchmarkDatatableSchema({
-				workspace: WORKSPACE,
-				datatableName: 'main',
-				schemaName: 'public',
-				tableName: 'refunds'
-			}).columns
-		).toEqual({ order_id: 'int4', amount: 'numeric' })
-	})
-
-	it('throws for an unknown job id', () => {
-		expect(() =>
-			getBenchmarkCompletedJobResultMaybe({ workspace: WORKSPACE, id: 'does-not-exist' })
-		).toThrow()
-	})
-})

ai_evals/adapters/frontend/mockBackendDrafts.test.ts

@@ -1,94 +0,0 @@
-import { afterEach, beforeEach, describe, expect, it } from 'bun:test'
-import {
-	clearBenchmarkDrafts,
-	getBenchmarkDraftForUser,
-	listBenchmarkDrafts,
-	resetBenchmarkMockBackend,
-	seedBenchmarkDraft,
-	updateBenchmarkDraft
-} from './mockBackend'
-
-const WORKSPACE = 'benchmark-drafts-ws'
-
-// Drives the in-memory stand-in for the per-user draft backend (`DraftService`)
-// that the global AI-chat eval round-trips its drafts through. Mirrors the
-// production-unit-test mock in
-// `frontend/src/lib/components/copilot/chat/global/core.test.ts`.
-describe('mockBackend drafts', () => {
-	beforeEach(() => resetBenchmarkMockBackend())
-	afterEach(() => resetBenchmarkMockBackend())
-
-	it('round-trips a saved draft through update / get / list', () => {
-		const value = { summary: 'Greet a user', content: 'export async function main() {}' }
-		const res = updateBenchmarkDraft({
-			workspace: WORKSPACE,
-			kind: 'script',
-			path: 'f/evals/greet',
-			requestBody: { value }
-		})
-		expect(res.status).toBe('saved')
-
-		expect(getBenchmarkDraftForUser({ workspace: WORKSPACE, kind: 'script', path: 'f/evals/greet' }).value).toEqual(
-			value
-		)
-
-		const rows = listBenchmarkDrafts(WORKSPACE)
-		expect(rows).toHaveLength(1)
-		expect(rows[0]).toMatchObject({ kind: 'script', path: 'f/evals/greet', summary: 'Greet a user', draft_only: true })
-	})
-
-	it('treats a null value as a delete', () => {
-		updateBenchmarkDraft({
-			workspace: WORKSPACE,
-			kind: 'variable',
-			path: 'f/evals/token',
-			requestBody: { value: { summary: 'token' } }
-		})
-		updateBenchmarkDraft({
-			workspace: WORKSPACE,
-			kind: 'variable',
-			path: 'f/evals/token',
-			requestBody: { value: null }
-		})
-
-		expect(listBenchmarkDrafts(WORKSPACE)).toHaveLength(0)
-		expect(() => getBenchmarkDraftForUser({ workspace: WORKSPACE, kind: 'variable', path: 'f/evals/token' })).toThrow()
-	})
-
-	it('throws a 404-shaped error when no draft exists', () => {
-		try {
-			getBenchmarkDraftForUser({ workspace: WORKSPACE, kind: 'script', path: 'f/evals/missing' })
-			throw new Error('expected a throw')
-		} catch (e) {
-			expect((e as { status?: number }).status).toBe(404)
-		}
-	})
-
-	it('seeds a draft as a backend row that a later edit overwrites', () => {
-		seedBenchmarkDraft(WORKSPACE, 'script', 'f/evals/current', { content: 'seed' })
-		expect(getBenchmarkDraftForUser({ workspace: WORKSPACE, kind: 'script', path: 'f/evals/current' }).value).toEqual({
-			content: 'seed'
-		})
-
-		// A model edit persists the same path and must win over the seed.
-		updateBenchmarkDraft({
-			workspace: WORKSPACE,
-			kind: 'script',
-			path: 'f/evals/current',
-			requestBody: { value: { content: 'edited' } }
-		})
-		expect(getBenchmarkDraftForUser({ workspace: WORKSPACE, kind: 'script', path: 'f/evals/current' }).value).toEqual({
-			content: 'edited'
-		})
-	})
-
-	it('clears only the targeted workspace', () => {
-		seedBenchmarkDraft(WORKSPACE, 'script', 'f/a', { content: 'a' })
-		seedBenchmarkDraft('other-ws', 'script', 'f/b', { content: 'b' })
-
-		clearBenchmarkDrafts(WORKSPACE)
-
-		expect(listBenchmarkDrafts(WORKSPACE)).toHaveLength(0)
-		expect(listBenchmarkDrafts('other-ws')).toHaveLength(1)
-	})
-})

ai_evals/adapters/frontend/runtime.ts

@@ -24,8 +24,6 @@ export async function runFrontendBenchmarkAdapter(input: {
   runs: number;
   model?: string;
   verbose?: boolean;
-  skipJudge?: boolean;
-  executionOnly?: boolean;
   backendValidation?: string;
 }): Promise<BenchmarkRunResult> {
   const tempDir = await mkdtemp(
@@ -42,9 +40,6 @@ export async function runFrontendBenchmarkAdapter(input: {
     WMILL_FRONTEND_AI_EVAL_MODEL: input.model ?? "",
     WMILL_FRONTEND_AI_EVAL_PROGRESS: "1",
     WMILL_FRONTEND_AI_EVAL_VERBOSE: input.verbose ? "1" : "0",
-    WMILL_FRONTEND_AI_EVAL_SKIP_JUDGE:
-      input.skipJudge || input.executionOnly ? "1" : "0",
-    WMILL_FRONTEND_AI_EVAL_EXECUTION_ONLY: input.executionOnly ? "1" : "0",
     WMILL_FRONTEND_AI_EVAL_BACKEND_VALIDATION: input.backendValidation ?? "",
   };
 

ai_evals/adapters/frontend/vitestAdapter.test.ts

@@ -33,29 +33,18 @@ vi.mock('$lib/components/vscode', () => ({}))
 vi.mock('$lib/gen', async () => {
 	const actual = await vi.importActual<any>('$lib/gen')
 	const {
-		getBenchmarkAppByPath,
 		getBenchmarkCompletedJob,
-		getBenchmarkCompletedJobResultMaybe,
-		getBenchmarkDatatableSchema,
-		getBenchmarkDraftForUser,
 		getBenchmarkFlowByPath,
-		getBenchmarkJobLogs,
 		getBenchmarkScriptByHash,
 		getBenchmarkScriptByPath,
 		hasBenchmarkWorkspace,
-		listBenchmarkApps,
-		listBenchmarkDatatables,
-		listBenchmarkDrafts,
 		listBenchmarkFlows,
-		listBenchmarkJobs,
 		listBenchmarkScripts,
 		createBenchmarkHttpTrigger,
 		createBenchmarkSchedule,
 		previewBenchmarkSchedule,
-		runBenchmarkDatatableSql,
 		runBenchmarkFlowByPath,
-		runBenchmarkScriptPreview,
-		updateBenchmarkDraft
+		runBenchmarkScriptPreview
 	} = await import('./mockBackend')
 
 	function wrapService<T extends object>(target: T, overrides: Record<string, unknown>): T {
@@ -71,25 +60,6 @@ vi.mock('$lib/gen', async () => {
 
 	return {
 		...actual,
-		DraftService: wrapService(actual.DraftService, {
-			updateDraft: async (data: {
-				workspace: string
-				kind: any
-				path: string
-				requestBody?: { value?: unknown }
-			}) =>
-				hasBenchmarkWorkspace(data.workspace)
-					? updateBenchmarkDraft(data)
-					: actual.DraftService.updateDraft(data),
-			getDraftForUser: async (data: { workspace: string; kind: any; path: string }) =>
-				hasBenchmarkWorkspace(data.workspace)
-					? getBenchmarkDraftForUser(data)
-					: actual.DraftService.getDraftForUser(data),
-			listDrafts: async (data: { workspace: string }) =>
-				hasBenchmarkWorkspace(data.workspace)
-					? listBenchmarkDrafts(data.workspace)
-					: actual.DraftService.listDrafts(data)
-		}),
 		ScriptService: wrapService(actual.ScriptService, {
 			listScripts: async (data: { workspace: string }) =>
 				hasBenchmarkWorkspace(data.workspace)
@@ -109,16 +79,6 @@ vi.mock('$lib/gen', async () => {
 				}
 				return actual.ScriptService.getScriptByPath(data)
 			},
-			getScriptByPathWithDraft: async (data: { workspace: string; path: string }) => {
-				if (hasBenchmarkWorkspace(data.workspace)) {
-					const script = getBenchmarkScriptByPath(data.workspace, data.path)
-					if (!script) {
-						throw new Error(`Script "${data.path}" not found in benchmark workspace`)
-					}
-					return script
-				}
-				return actual.ScriptService.getScriptByPathWithDraft(data)
-			},
 			getScriptByHash: async (data: { workspace: string; hash: string }) => {
 				if (hasBenchmarkWorkspace(data.workspace)) {
 					const script = getBenchmarkScriptByHash(data.workspace, data.hash)
@@ -148,26 +108,6 @@ vi.mock('$lib/gen', async () => {
 					return flow
 				}
 				return actual.FlowService.getFlowByPath(data)
-			},
-			getFlowByPathWithDraft: async (data: { workspace: string; path: string }) => {
-				if (hasBenchmarkWorkspace(data.workspace)) {
-					const flow = getBenchmarkFlowByPath(data.workspace, data.path)
-					if (!flow) {
-						throw new Error(`Flow "${data.path}" not found in benchmark workspace`)
-					}
-					return flow
-				}
-				return actual.FlowService.getFlowByPathWithDraft(data)
-			},
-			getFlowLatestVersion: async (data: { workspace: string; path: string }) => {
-				if (hasBenchmarkWorkspace(data.workspace)) {
-					const flow = getBenchmarkFlowByPath(data.workspace, data.path)
-					if (!flow) {
-						throw new Error(`Flow "${data.path}" not found in benchmark workspace`)
-					}
-					return { id: 1 }
-				}
-				return actual.FlowService.getFlowLatestVersion(data)
 			}
 		}),
 		JobService: wrapService(actual.JobService, {
@@ -179,27 +119,13 @@ vi.mock('$lib/gen', async () => {
 					args?: Record<string, unknown>
 					path?: string
 				}
-			}) => {
-				if (!hasBenchmarkWorkspace(data.workspace)) {
-					return actual.JobService.runScriptPreview(data)
-				}
-				const requestBody = data.requestBody ?? {}
-				const database = requestBody.args?.database
-				// Datatable SQL runs as a `postgresql` preview against `datatable://<name>`.
-				// Execute it through the canned-SQL mock instead of linting it as a script.
-				if (
-					requestBody.language === 'postgresql' &&
-					typeof database === 'string' &&
-					database.startsWith('datatable://')
-				) {
-					return runBenchmarkDatatableSql({
-						workspace: data.workspace,
-						datatableName: database.slice('datatable://'.length),
-						sql: requestBody.content ?? ''
-					})
-				}
-				return runBenchmarkScriptPreview({ workspace: data.workspace, requestBody })
-			},
+			}) =>
+				hasBenchmarkWorkspace(data.workspace)
+					? runBenchmarkScriptPreview({
+							workspace: data.workspace,
+							requestBody: data.requestBody ?? {}
+						})
+					: actual.JobService.runScriptPreview(data),
 			runFlowByPath: async (data: {
 				workspace: string
 				path: string
@@ -221,39 +147,7 @@ vi.mock('$lib/gen', async () => {
 					return job
 				}
 				return actual.JobService.getJob(data)
-			},
-			getCompletedJobResultMaybe: async (data: { workspace: string; id: string }) =>
-				hasBenchmarkWorkspace(data.workspace)
-					? getBenchmarkCompletedJobResultMaybe({ workspace: data.workspace, id: data.id })
-					: actual.JobService.getCompletedJobResultMaybe(data),
-			listJobs: async (data: { workspace: string }) =>
-				hasBenchmarkWorkspace(data.workspace)
-					? (listBenchmarkJobs(data.workspace) ?? [])
-					: actual.JobService.listJobs(data),
-			getJobLogs: async (data: { workspace: string; id: string }) =>
-				hasBenchmarkWorkspace(data.workspace)
-					? getBenchmarkJobLogs(data.workspace, data.id)
-					: actual.JobService.getJobLogs(data)
-		}),
-		WorkspaceService: wrapService(actual.WorkspaceService, {
-			listDataTableTables: async (data: { workspace: string }) =>
-				hasBenchmarkWorkspace(data.workspace)
-					? (listBenchmarkDatatables(data.workspace) ?? [])
-					: actual.WorkspaceService.listDataTableTables(data),
-			getDataTableTableSchema: async (data: {
-				workspace: string
-				datatableName: string
-				schemaName: string
-				tableName: string
-			}) =>
-				hasBenchmarkWorkspace(data.workspace)
-					? getBenchmarkDatatableSchema({
-							workspace: data.workspace,
-							datatableName: data.datatableName,
-							schemaName: data.schemaName,
-							tableName: data.tableName
-						})
-					: actual.WorkspaceService.getDataTableTableSchema(data)
+			}
 		}),
 		ScheduleService: wrapService(actual.ScheduleService, {
 			existsSchedule: async (data: { workspace: string; path: string }) =>
@@ -301,20 +195,12 @@ vi.mock('$lib/gen', async () => {
 		}),
 		AppService: wrapService(actual.AppService, {
 			existsApp: async (data: { workspace: string; path: string }) =>
-				hasBenchmarkWorkspace(data.workspace)
-					? Boolean(getBenchmarkAppByPath(data.workspace, data.path))
-					: actual.AppService.existsApp(data),
+				hasBenchmarkWorkspace(data.workspace) ? false : actual.AppService.existsApp(data),
 			listApps: async (data: { workspace: string }) =>
-				hasBenchmarkWorkspace(data.workspace)
-					? (listBenchmarkApps(data.workspace) ?? [])
-					: actual.AppService.listApps(data),
+				hasBenchmarkWorkspace(data.workspace) ? [] : actual.AppService.listApps(data),
 			getAppByPath: async (data: { workspace: string; path: string }) => {
 				if (hasBenchmarkWorkspace(data.workspace)) {
-					const app = getBenchmarkAppByPath(data.workspace, data.path)
-					if (!app) {
-						throw new Error(`App "${data.path}" not found in benchmark workspace`)
-					}
-					return app
+					throw new Error(`App "${data.path}" not found in benchmark workspace`)
 				}
 				return actual.AppService.getAppByPath(data)
 			}
@@ -466,6 +352,5 @@ benchmarkIt(
 			resetBenchmarkMockBackend()
 		}
 	},
-	// Full-suite runs (30+ cases at concurrency 2-3) routinely exceed 10 minutes.
-	7_200_000
+	600_000
 )

ai_evals/cases/flow.yaml

@@ -8,9 +8,6 @@
       args:
         a: 4
         b: 5
-  toolExpect:
-    requiredToolsUsed:
-      - test_run_flow
   judgeChecklist:
     - "the flow takes `a` and `b` as inputs"
     - "the main step is named `sum_numbers`"
@@ -28,9 +25,6 @@
       args:
         a: 2
         b: 3
-  toolExpect:
-    requiredToolsUsed:
-      - test_run_flow
   judgeChecklist:
     - "the flow takes `a` and `b` as inputs"
     - "the main step is named `sum_numbers`"
@@ -48,9 +42,6 @@
       args:
         a: 7
         b: 8
-  toolExpect:
-    requiredToolsUsed:
-      - test_run_flow
   judgeChecklist:
     - "the parent flow takes `a` and `b` as inputs"
     - "the main step is named `call_add_numbers`"
@@ -435,7 +426,6 @@
       - return_schedule_status
   toolExpect:
     requiredToolsUsed:
-      - test_run_flow
       - create_schedule
     toolCallArgs:
       - tool: create_schedule
@@ -463,7 +453,6 @@
       - webhook_response
   toolExpect:
     requiredToolsUsed:
-      - test_run_flow
       - create_trigger
     toolCallArgs:
       - tool: create_trigger

ai_evals/cases/script.yaml

@@ -5,9 +5,6 @@
     Keep it simple and do not add external dependencies.
   initial: ai_evals/fixtures/frontend/script/initial/test1_empty_bun.json
   expected: ai_evals/fixtures/frontend/script/expected/test1_greet_user.json
-  toolExpect:
-    requiredToolsUsed:
-      - test_run_script
   judgeChecklist:
     - uses the existing `name` input
     - returns a plain greeting string
@@ -23,7 +20,6 @@
   expected: ai_evals/fixtures/frontend/script/expected/test1_greet_user.json
   toolExpect:
     requiredToolsUsed:
-      - test_run_script
       - create_schedule
     toolCallArgs:
       - tool: create_schedule
@@ -48,7 +44,6 @@
   expected: ai_evals/fixtures/frontend/script/expected/test1_greet_user.json
   toolExpect:
     requiredToolsUsed:
-      - test_run_script
       - create_trigger
     toolCallArgs:
       - tool: create_trigger

ai_evals/cli/index.ts

@@ -25,9 +25,7 @@ import {
 import { runSuite } from "../core/runSuite";
 import { EVAL_MODES, type EvalMode } from "../core/types";
 import { DEFAULT_JUDGE_MODEL } from "../core/judge";
-// createCliModeRunner is imported lazily in runCliBenchmark so the non-cli modes
-// (global/flow/script/app) don't pull in the wmill CLI toolchain and its JSR deps
-// (e.g. @cliffy/*) just to load this entrypoint.
+import { createCliModeRunner } from "../modes/cli";
 import { runFrontendBenchmarkAdapter } from "../adapters/frontend/runtime";
 import { resolveWindmillBackendSettings } from "../core/windmillBackendSettings";
 import { assertWindmillBackendReachable } from "../adapters/frontend/windmillBackend";
@@ -99,11 +97,6 @@ async function main() {
       "comma-separated model aliases to run sequentially",
     )
     .option("--verbose", "stream assistant output during frontend runs")
-    .option("--skip-judge", "skip LLM judge scoring for this run")
-    .option(
-      "--execution-only",
-      "only require the model/proxy/frontend loop to complete",
-    )
     .option(
       "--record",
       "append a compact summary line to ai_evals/history/<mode>.jsonl",
@@ -122,8 +115,6 @@ async function main() {
           model?: string;
           models?: string;
           verbose?: boolean;
-          skipJudge?: boolean;
-          executionOnly?: boolean;
           record?: boolean;
           backendValidation?: string;
         },
@@ -136,8 +127,6 @@ async function main() {
           model: options.model,
           models: options.models,
           verbose: options.verbose ?? false,
-          skipJudge: options.skipJudge ?? false,
-          executionOnly: options.executionOnly ?? false,
           record: options.record ?? false,
           backendValidation: options.backendValidation,
         });
@@ -186,8 +175,6 @@ async function handleRun(input: {
   model?: string;
   models?: string;
   verbose: boolean;
-  skipJudge: boolean;
-  executionOnly: boolean;
   record: boolean;
   backendValidation?: string;
 }) {
@@ -224,7 +211,7 @@ async function handleRun(input: {
   const summaries: Array<{
     label: string;
     passRate: number;
-    averagePassedDurationMs: number | null;
+    averageDurationMs: number;
   }> = [];
 
   for (const [index, model] of models.entries()) {
@@ -243,8 +230,6 @@ async function handleRun(input: {
             input.runs,
             getCliEvalModel(model),
             runModel,
-            input.skipJudge,
-            input.executionOnly,
           )
         : await runFrontendBenchmarkAdapter({
             mode: input.mode,
@@ -252,8 +237,6 @@ async function handleRun(input: {
             runs: input.runs,
             model: model.id,
             verbose: input.verbose,
-            skipJudge: input.skipJudge,
-            executionOnly: input.executionOnly,
             backendValidation,
           });
 
@@ -276,7 +259,7 @@ async function handleRun(input: {
     summaries.push({
       label: `${model.id} (${runModel})`,
       passRate: result.passRate,
-      averagePassedDurationMs: result.averagePassedDurationMs ?? null,
+      averageDurationMs: result.averageDurationMs,
     });
   }
 
@@ -284,7 +267,7 @@ async function handleRun(input: {
     process.stdout.write("\nModel summary\n");
     for (const summary of summaries) {
       process.stdout.write(
-        `- ${summary.label}: ${formatPercent(summary.passRate)} | passed avg ${formatNullableDuration(summary.averagePassedDurationMs)}\n`,
+        `- ${summary.label}: ${formatPercent(summary.passRate)} | ${Math.round(summary.averageDurationMs)}ms\n`,
       );
     }
   }
@@ -295,25 +278,20 @@ async function runCliBenchmark(
   runs: number,
   model: ReturnType<typeof getCliEvalModel>,
   runModel: string,
-  skipJudge: boolean,
-  executionOnly: boolean,
 ) {
-  const { createCliModeRunner } = await import("../modes/cli");
-  const judgeModel = skipJudge || executionOnly ? null : DEFAULT_JUDGE_MODEL;
   const caseResults = await runSuite({
     modeRunner: createCliModeRunner(model),
     cases,
     runs,
     runModel,
-    judgeModel,
-    executionOnly,
+    judgeModel: DEFAULT_JUDGE_MODEL,
   });
 
   return buildRunResult({
     mode: "cli",
     runs,
     runModel,
-    judgeModel,
+    judgeModel: DEFAULT_JUDGE_MODEL,
     caseResults,
   });
 }
@@ -373,10 +351,6 @@ function formatPercent(value: number): string {
   return `${(value * 100).toFixed(1)}%`;
 }
 
-function formatNullableDuration(value: number | null): string {
-  return value === null ? "n/a" : `${Math.round(value)}ms`;
-}
-
 void main().catch((error) => {
   const message = error instanceof Error ? error.message : String(error);
   process.stderr.write(`${message}\n`);

ai_evals/core/cases.test.ts

@@ -14,21 +14,6 @@ describe("loadCases", () => {
         },
       },
     });
-    expect(caseEntry?.toolExpect).toEqual({
-      requiredToolsUsed: ["test_run_flow"],
-    });
-  });
-
-  it("loads script and flow test tool expectations", async () => {
-    const scriptCases = await loadCases("script");
-    const flowCases = await loadCases("flow");
-
-    expect(scriptCases.find((entry) => entry.id === "script-test1-greet-user")?.toolExpect).toEqual({
-      requiredToolsUsed: ["test_run_script"],
-    });
-    expect(flowCases.find((entry) => entry.id === "flow-test0-sum-two-numbers")?.toolExpect).toEqual({
-      requiredToolsUsed: ["test_run_flow"],
-    });
   });
 
   it("loads the workspace-flow preference benchmark case", async () => {
@@ -218,49 +203,6 @@ describe("loadCases", () => {
     });
   });
 
-  it("loads global active-editor eval cases", async () => {
-    const globalCases = await loadCases("global");
-    const scriptCase = globalCases.find(
-      (entry) => entry.id === "global-test12-current-live-script-edit"
-    );
-    const flowCase = globalCases.find(
-      (entry) => entry.id === "global-test13-current-live-flow-edit"
-    );
-
-    expect(scriptCase?.initialPath).toContain(
-      "ai_evals/fixtures/frontend/global/initial/current_greeting_live_script.json"
-    );
-    expect(scriptCase?.toolExpect).toMatchObject({
-      requiredToolsUsed: ["read_workspace_item"],
-    });
-    expect(flowCase?.initialPath).toContain(
-      "ai_evals/fixtures/frontend/global/initial/current_invoice_live_flow.json"
-    );
-    expect(flowCase?.validate).toMatchObject({
-      requiredDrafts: [
-        {
-          type: "flow",
-          path: "f/evals/global/current_invoice_flow",
-        },
-      ],
-    });
-  });
-
-  it("loads global docs-search cases as tool-use checks", async () => {
-    const globalCases = await loadCases("global");
-    const docsCases = globalCases.filter((entry) =>
-      entry.id.startsWith("global-docs-"),
-    );
-    expect(docsCases.length).toBeGreaterThanOrEqual(3);
-
-    // Each docs case verifies the assistant reaches for search_docs and does not
-    // draft anything; with no draft, the global judge is skipped.
-    for (const entry of docsCases) {
-      expect(entry.skipJudge).toBe(true);
-      expect(entry.toolExpect?.requiredToolsUsed).toContain("search_docs");
-    }
-  });
-
   it("loads tool expectations for workspace mutation cases", async () => {
     const scriptCases = await loadCases("script");
     const caseEntry = scriptCases.find(
@@ -268,7 +210,7 @@ describe("loadCases", () => {
     );
 
     expect(caseEntry?.toolExpect).toEqual({
-      requiredToolsUsed: ["test_run_script", "create_schedule"],
+      requiredToolsUsed: ["create_schedule"],
       toolCallArgs: [
         {
           tool: "create_schedule",

ai_evals/core/models.test.ts

@@ -2,22 +2,15 @@ import { describe, expect, it } from "bun:test";
 import { resolveEvalModel } from "./models";
 
 describe("resolveEvalModel", () => {
-  it("supports GPT-5.5 aliases for frontend evals", () => {
-    expect(resolveEvalModel("flow", "gpt-5.5").frontend).toEqual({
-      provider: "openai",
-      model: "gpt-5.5",
-    });
-    expect(resolveEvalModel("app", "gpt-55").frontend).toEqual({
-      provider: "openai",
-      model: "gpt-5.5",
-    });
-    expect(resolveEvalModel("script", "5.5").frontend).toEqual({
-      provider: "openai",
-      model: "gpt-5.5",
-    });
-  });
-
   it("supports Gemini aliases for frontend evals", () => {
+    expect(resolveEvalModel("flow", "gemini").frontend).toEqual({
+      provider: "googleai",
+      model: "gemini-2.5-flash",
+    });
+    expect(resolveEvalModel("app", "gemini-pro").frontend).toEqual({
+      provider: "googleai",
+      model: "gemini-2.5-pro",
+    });
     expect(
       resolveEvalModel("script", "gemini-3-flash-preview").frontend,
     ).toEqual({
@@ -44,8 +37,8 @@ describe("resolveEvalModel", () => {
   });
 
   it("rejects Gemini aliases for cli evals", () => {
-    expect(() => resolveEvalModel("cli", "gemini-3-flash-preview")).toThrow(
-      "Model gemini-3-flash-preview is not supported for cli mode",
+    expect(() => resolveEvalModel("cli", "gemini")).toThrow(
+      "Model gemini-flash is not supported for cli mode",
     );
   });
 });

ai_evals/core/models.ts

@@ -88,12 +88,21 @@ export const EVAL_MODELS: EvalModelSpec[] = [
     },
   },
   {
-    id: "gpt-5.5",
-    label: "GPT-5.5",
-    aliases: ["gpt-5.5", "gpt-55", "5.5"],
+    id: "gemini-flash",
+    label: "Gemini 2.5 Flash",
+    aliases: ["gemini", "gemini-flash", "gemini-2.5-flash"],
     frontend: {
-      provider: "openai",
-      model: "gpt-5.5",
+      provider: "googleai",
+      model: "gemini-2.5-flash",
+    },
+  },
+  {
+    id: "gemini-pro",
+    label: "Gemini 2.5 Pro",
+    aliases: ["gemini-pro", "gemini-2.5-pro"],
+    frontend: {
+      provider: "googleai",
+      model: "gemini-2.5-pro",
     },
   },
   {

ai_evals/core/results.test.ts

@@ -1,307 +0,0 @@
-import { mkdtemp, readFile, rm } from "node:fs/promises";
-import { join } from "node:path";
-import { tmpdir } from "node:os";
-import { describe, expect, it } from "bun:test";
-import {
-  appendHistoryRecord,
-  buildRunResult,
-  formatRunSummary,
-} from "./results";
-import type { BenchmarkCaseResult } from "./types";
-
-function caseResult(
-  attempts: BenchmarkCaseResult["attempts"],
-): BenchmarkCaseResult {
-  return {
-    id: "case-1",
-    prompt: "Do the thing",
-    attempts,
-  };
-}
-
-describe("benchmark results", () => {
-  it("keeps success cost metrics separate from failed attempts", () => {
-    const result = buildRunResult({
-      mode: "global",
-      runs: 1,
-      runModel: "model-under-test",
-      judgeModel: "judge-model",
-      caseResults: [
-        caseResult([
-          {
-            attempt: 1,
-            passed: true,
-            durationMs: 1000,
-            assistantMessageCount: 1,
-            toolCallCount: 1,
-            toolsUsed: ["edit_script"],
-            skillsInvoked: [],
-            checks: [{ name: "edited", passed: true }],
-            judgeScore: 100,
-            judgeSummary: "ok",
-            error: null,
-            tokenUsage: { prompt: 100, completion: 20, total: 120 },
-          },
-          {
-            attempt: 2,
-            passed: false,
-            durationMs: 100,
-            assistantMessageCount: 1,
-            toolCallCount: 0,
-            toolsUsed: [],
-            skillsInvoked: [],
-            checks: [{ name: "edited", passed: false }],
-            judgeScore: 10,
-            judgeSummary: "missed",
-            error: "failed",
-            tokenUsage: { prompt: 10, completion: 5, total: 15 },
-          },
-        ]),
-      ],
-    });
-
-    expect(result.attemptCount).toBe(2);
-    expect(result.passedAttempts).toBe(1);
-    expect(result.passRate).toBe(0.5);
-    expect(result.averageDurationMs).toBe(550);
-    expect(result.averagePassedDurationMs).toBe(1000);
-    expect(result.totalTokenUsage).toEqual({
-      prompt: 110,
-      completion: 25,
-      total: 135,
-    });
-    expect(result.totalPassedTokenUsage).toEqual({
-      prompt: 100,
-      completion: 20,
-      total: 120,
-    });
-    expect(result.averageTokenUsagePerAttempt).toEqual({
-      prompt: 55,
-      completion: 12.5,
-      total: 67.5,
-    });
-    expect(result.averageTokenUsagePerPassedAttempt).toEqual({
-      prompt: 100,
-      completion: 20,
-      total: 120,
-    });
-
-    const summary = formatRunSummary(result);
-    expect(summary).toContain("Average duration (passed): 1000ms");
-    expect(summary).toContain("Average tokens (passed): 120 total");
-    expect(summary).toContain("Average duration (all attempts): 550ms");
-  });
-
-  it("aggregates final context size over passed attempts only", () => {
-    const result = buildRunResult({
-      mode: "global",
-      runs: 1,
-      runModel: "model-under-test",
-      judgeModel: "judge-model",
-      caseResults: [
-        caseResult([
-          {
-            attempt: 1,
-            passed: true,
-            durationMs: 1000,
-            assistantMessageCount: 1,
-            toolCallCount: 1,
-            toolsUsed: ["edit_script"],
-            skillsInvoked: [],
-            checks: [{ name: "edited", passed: true }],
-            judgeScore: 100,
-            judgeSummary: "ok",
-            error: null,
-            tokenUsage: { prompt: 12000, completion: 200, total: 12200 },
-            finalContextTokens: 5000,
-          },
-          {
-            attempt: 2,
-            passed: true,
-            durationMs: 1100,
-            assistantMessageCount: 1,
-            toolCallCount: 1,
-            toolsUsed: ["edit_script"],
-            skillsInvoked: [],
-            checks: [{ name: "edited", passed: true }],
-            judgeScore: 100,
-            judgeSummary: "ok",
-            error: null,
-            tokenUsage: { prompt: 18000, completion: 300, total: 18300 },
-            finalContextTokens: 7000,
-          },
-          {
-            attempt: 3,
-            passed: false,
-            durationMs: 100,
-            assistantMessageCount: 1,
-            toolCallCount: 0,
-            toolsUsed: [],
-            skillsInvoked: [],
-            checks: [{ name: "edited", passed: false }],
-            judgeScore: 10,
-            judgeSummary: "missed",
-            error: "failed",
-            tokenUsage: { prompt: 20000, completion: 100, total: 20100 },
-            finalContextTokens: 9000,
-          },
-        ]),
-      ],
-    });
-
-    // Final context size stays below cumulative prompt and ignores the failed attempt.
-    expect(result.averageFinalContextTokensPassed).toBe(6000);
-    expect(result.maxFinalContextTokensPassed).toBe(7000);
-    expect(formatRunSummary(result)).toContain(
-      "Final context size (passed): 6000 tokens (max 7000)",
-    );
-  });
-
-  it("reports passed averages as unavailable when no attempt passes", () => {
-    const result = buildRunResult({
-      mode: "global",
-      runs: 1,
-      runModel: "model-under-test",
-      judgeModel: "judge-model",
-      caseResults: [
-        caseResult([
-          {
-            attempt: 1,
-            passed: false,
-            durationMs: 100,
-            assistantMessageCount: 1,
-            toolCallCount: 0,
-            toolsUsed: [],
-            skillsInvoked: [],
-            checks: [{ name: "edited", passed: false }],
-            judgeScore: 10,
-            judgeSummary: "missed",
-            error: "failed",
-            tokenUsage: { prompt: 10, completion: 5, total: 15 },
-          },
-        ]),
-      ],
-    });
-
-    expect(result.averagePassedDurationMs).toBeNull();
-    expect(result.totalPassedTokenUsage).toBeNull();
-    expect(result.averageTokenUsagePerPassedAttempt).toBeNull();
-    expect(formatRunSummary(result)).toContain(
-      "Average duration (passed): n/a",
-    );
-  });
-
-  it("normalizes passed token averages by passed attempts", () => {
-    const result = buildRunResult({
-      mode: "global",
-      runs: 1,
-      runModel: "model-under-test",
-      judgeModel: "judge-model",
-      caseResults: [
-        caseResult([
-          {
-            attempt: 1,
-            passed: true,
-            durationMs: 1000,
-            assistantMessageCount: 1,
-            toolCallCount: 1,
-            toolsUsed: ["edit_script"],
-            skillsInvoked: [],
-            checks: [{ name: "edited", passed: true }],
-            judgeScore: 100,
-            judgeSummary: "ok",
-            error: null,
-            tokenUsage: { prompt: 100, completion: 20, total: 120 },
-          },
-          {
-            attempt: 2,
-            passed: true,
-            durationMs: 1200,
-            assistantMessageCount: 1,
-            toolCallCount: 1,
-            toolsUsed: ["edit_script"],
-            skillsInvoked: [],
-            checks: [{ name: "edited", passed: true }],
-            judgeScore: 100,
-            judgeSummary: "ok",
-            error: null,
-            tokenUsage: null,
-          },
-        ]),
-      ],
-    });
-
-    expect(result.passedAttempts).toBe(2);
-    expect(result.totalPassedTokenUsage).toEqual({
-      prompt: 100,
-      completion: 20,
-      total: 120,
-    });
-    expect(result.averageTokenUsagePerPassedAttempt).toEqual({
-      prompt: 50,
-      completion: 10,
-      total: 60,
-    });
-  });
-
-  it("records passed-attempt metrics in history", async () => {
-    const tempDir = await mkdtemp(join(tmpdir(), "windmill-ai-evals-"));
-    try {
-      const historyPath = join(tempDir, "history.jsonl");
-      const result = buildRunResult({
-        mode: "global",
-        runs: 1,
-        runModel: "model-under-test",
-        judgeModel: "judge-model",
-        caseResults: [
-          caseResult([
-            {
-              attempt: 1,
-              passed: true,
-              durationMs: 1000,
-              assistantMessageCount: 1,
-              toolCallCount: 1,
-              toolsUsed: ["edit_script"],
-              skillsInvoked: [],
-              checks: [{ name: "edited", passed: true }],
-              judgeScore: 100,
-              judgeSummary: "ok",
-              error: null,
-              tokenUsage: { prompt: 100, completion: 20, total: 120 },
-            },
-            {
-              attempt: 2,
-              passed: false,
-              durationMs: 100,
-              assistantMessageCount: 1,
-              toolCallCount: 0,
-              toolsUsed: [],
-              skillsInvoked: [],
-              checks: [{ name: "edited", passed: false }],
-              judgeScore: 10,
-              judgeSummary: "missed",
-              error: "failed",
-              tokenUsage: { prompt: 10, completion: 5, total: 15 },
-            },
-          ]),
-        ],
-      });
-
-      await appendHistoryRecord(result, historyPath);
-      const record = JSON.parse(await readFile(historyPath, "utf8"));
-
-      expect(record.averageDurationMs).toBe(550);
-      expect(record.averagePassedDurationMs).toBe(1000);
-      expect(record.averageTokenUsagePerAttempt.total).toBe(67.5);
-      expect(record.averageTokenUsagePerPassedAttempt.total).toBe(120);
-      expect(record.cases[0].averageDurationMs).toBe(550);
-      expect(record.cases[0].averagePassedDurationMs).toBe(1000);
-      expect(record.cases[0].averageTokenUsagePerAttempt.total).toBe(67.5);
-      expect(record.cases[0].averageTokenUsagePerPassedAttempt.total).toBe(
-        120,
-      );
-    } finally {
-      await rm(tempDir, { recursive: true, force: true });
-    }
-  });
-});

ai_evals/core/results.ts

@@ -4,23 +4,12 @@ import { execFileSync } from "node:child_process";
 import { getAiEvalsRoot, getRepoRoot } from "./cases";
 import type {
   BenchmarkArtifactFile,
-  BenchmarkAttemptResult,
   BenchmarkCaseResult,
   BenchmarkRunResult,
   BenchmarkTokenUsage,
   EvalMode,
 } from "./types";
 
-type AttemptAggregate = {
-  attemptCount: number;
-  durationTotal: number;
-  tokenUsageAttemptCount: number;
-  tokenUsageTotal: BenchmarkTokenUsage | null;
-  finalContextAttemptCount: number;
-  finalContextTotal: number;
-  finalContextMax: number | null;
-};
-
 export async function writeRunResult(
   result: BenchmarkRunResult,
   outputPath?: string,
@@ -88,12 +77,36 @@ export function buildRunResult(input: {
   judgeModel: string | null;
   caseResults: BenchmarkCaseResult[];
 }): BenchmarkRunResult {
-  const attempts = input.caseResults.flatMap((entry) => entry.attempts);
-  const passedAttemptResults = attempts.filter((attempt) => attempt.passed);
-  const attemptAggregate = aggregateAttempts(attempts);
-  const passedAttemptAggregate = aggregateAttempts(passedAttemptResults);
-  const attemptCount = attemptAggregate.attemptCount;
-  const passedAttempts = passedAttemptAggregate.attemptCount;
+  const attemptCount = input.caseResults.reduce(
+    (sum, entry) => sum + entry.attempts.length,
+    0,
+  );
+  const passedAttempts = input.caseResults.reduce(
+    (sum, entry) =>
+      sum + entry.attempts.filter((attempt) => attempt.passed).length,
+    0,
+  );
+  const durationTotal = input.caseResults.reduce(
+    (sum, entry) =>
+      sum +
+      entry.attempts.reduce((inner, attempt) => inner + attempt.durationMs, 0),
+    0,
+  );
+  const tokenUsageTotal = input.caseResults.reduce<BenchmarkTokenUsage | null>(
+    (sum, entry) => {
+      for (const attempt of entry.attempts) {
+        if (!attempt.tokenUsage) {
+          continue;
+        }
+        sum ??= { prompt: 0, completion: 0, total: 0 };
+        sum.prompt += attempt.tokenUsage.prompt;
+        sum.completion += attempt.tokenUsage.completion;
+        sum.total += attempt.tokenUsage.total;
+      }
+      return sum;
+    },
+    null,
+  );
 
   return {
     version: 1,
@@ -107,21 +120,16 @@ export function buildRunResult(input: {
     attemptCount,
     passedAttempts,
     passRate: attemptCount === 0 ? 0 : passedAttempts / attemptCount,
-    averageDurationMs:
-      attemptCount === 0 ? 0 : attemptAggregate.durationTotal / attemptCount,
-    averagePassedDurationMs: averageDuration(passedAttemptAggregate),
-    totalTokenUsage: attemptAggregate.tokenUsageTotal,
-    totalPassedTokenUsage: passedAttemptAggregate.tokenUsageTotal,
+    averageDurationMs: attemptCount === 0 ? 0 : durationTotal / attemptCount,
+    totalTokenUsage: tokenUsageTotal,
     averageTokenUsagePerAttempt:
-      attemptCount === 0
+      attemptCount === 0 || !tokenUsageTotal
         ? null
-        : averageTokenUsage(attemptAggregate, attemptCount),
-    averageTokenUsagePerPassedAttempt: averageTokenUsage(
-      passedAttemptAggregate,
-      passedAttempts,
-    ),
-    averageFinalContextTokensPassed: averageFinalContext(passedAttemptAggregate),
-    maxFinalContextTokensPassed: passedAttemptAggregate.finalContextMax,
+        : {
+            prompt: tokenUsageTotal.prompt / attemptCount,
+            completion: tokenUsageTotal.completion / attemptCount,
+            total: tokenUsageTotal.total / attemptCount,
+          },
     cases: input.caseResults,
   };
 }
@@ -130,30 +138,9 @@ export function formatRunSummary(result: BenchmarkRunResult): string {
   const lines = [
     `${result.mode} benchmark complete`,
     `Pass rate: ${formatPercent(result.passRate)} (${result.passedAttempts}/${result.attemptCount})`,
-    `Average duration (passed): ${formatNullableDuration(result.averagePassedDurationMs ?? null)}`,
+    `Average duration: ${Math.round(result.averageDurationMs)}ms`,
   ];
 
-  if (result.averageTokenUsagePerPassedAttempt) {
-    lines.push(
-      `Average tokens (passed): ${formatTokenUsage(result.averageTokenUsagePerPassedAttempt)}`,
-    );
-  }
-  if (result.averageFinalContextTokensPassed != null) {
-    lines.push(
-      `Final context size (passed): ${Math.round(result.averageFinalContextTokensPassed)} tokens (max ${Math.round(result.maxFinalContextTokensPassed ?? 0)})`,
-    );
-  }
-  if (result.passedAttempts < result.attemptCount) {
-    lines.push(
-      `Average duration (all attempts): ${Math.round(result.averageDurationMs)}ms`,
-    );
-    if (result.averageTokenUsagePerAttempt) {
-      lines.push(
-        `Average tokens (all attempts): ${formatTokenUsage(result.averageTokenUsagePerAttempt)}`,
-      );
-    }
-  }
-
   const failures = collectFailures(result);
   if (failures.length > 0) {
     lines.push("Failures:");
@@ -185,77 +172,6 @@ function collectFailures(result: BenchmarkRunResult): string[] {
   return failures;
 }
 
-function aggregateAttempts(attempts: BenchmarkAttemptResult[]): AttemptAggregate {
-  const aggregate: AttemptAggregate = {
-    attemptCount: attempts.length,
-    durationTotal: 0,
-    tokenUsageAttemptCount: 0,
-    tokenUsageTotal: null,
-    finalContextAttemptCount: 0,
-    finalContextTotal: 0,
-    finalContextMax: null,
-  };
-
-  for (const attempt of attempts) {
-    aggregate.durationTotal += attempt.durationMs;
-    if (typeof attempt.finalContextTokens === "number") {
-      aggregate.finalContextAttemptCount += 1;
-      aggregate.finalContextTotal += attempt.finalContextTokens;
-      aggregate.finalContextMax = Math.max(
-        aggregate.finalContextMax ?? 0,
-        attempt.finalContextTokens,
-      );
-    }
-    if (!attempt.tokenUsage) {
-      continue;
-    }
-    aggregate.tokenUsageAttemptCount += 1;
-    aggregate.tokenUsageTotal ??= { prompt: 0, completion: 0, total: 0 };
-    aggregate.tokenUsageTotal.prompt += attempt.tokenUsage.prompt;
-    aggregate.tokenUsageTotal.completion += attempt.tokenUsage.completion;
-    aggregate.tokenUsageTotal.total += attempt.tokenUsage.total;
-  }
-
-  return aggregate;
-}
-
-function averageDuration(aggregate: AttemptAggregate): number | null {
-  return aggregate.attemptCount === 0
-    ? null
-    : aggregate.durationTotal / aggregate.attemptCount;
-}
-
-function averageFinalContext(aggregate: AttemptAggregate): number | null {
-  return aggregate.finalContextAttemptCount === 0
-    ? null
-    : aggregate.finalContextTotal / aggregate.finalContextAttemptCount;
-}
-
-function averageTokenUsage(
-  aggregate: AttemptAggregate,
-  denominator: number,
-): BenchmarkTokenUsage | null {
-  if (denominator === 0 || !aggregate.tokenUsageTotal) {
-    return null;
-  }
-  return {
-    prompt: aggregate.tokenUsageTotal.prompt / denominator,
-    completion: aggregate.tokenUsageTotal.completion / denominator,
-    total: aggregate.tokenUsageTotal.total / denominator,
-  };
-}
-
-function formatNullableDuration(value: number | null): string {
-  return value === null ? "n/a" : `${Math.round(value)}ms`;
-}
-
-function formatTokenUsage(value: BenchmarkTokenUsage): string {
-  const total = Math.round(value.total);
-  const prompt = Math.round(value.prompt);
-  const completion = Math.round(value.completion);
-  return `${total} total (${prompt} prompt, ${completion} completion)`;
-}
-
 function defaultFileName(mode: EvalMode): string {
   return `${new Date().toISOString().replaceAll(":", "-")}__${mode}.json`;
 }
@@ -336,18 +252,12 @@ function toHistoryRecord(result: BenchmarkRunResult) {
     passedAttempts: result.passedAttempts,
     passRate: result.passRate,
     averageDurationMs: result.averageDurationMs,
-    averagePassedDurationMs: result.averagePassedDurationMs ?? null,
     averageJudgeScore:
       judgeScores.length === 0
         ? null
         : judgeScores.reduce((sum, score) => sum + score, 0) /
           judgeScores.length,
     averageTokenUsagePerAttempt: result.averageTokenUsagePerAttempt ?? null,
-    averageTokenUsagePerPassedAttempt:
-      result.averageTokenUsagePerPassedAttempt ?? null,
-    averageFinalContextTokensPassed:
-      result.averageFinalContextTokensPassed ?? null,
-    maxFinalContextTokensPassed: result.maxFinalContextTokensPassed ?? null,
     failedCaseIds: Array.from(
       new Set(
         result.cases
@@ -358,15 +268,31 @@ function toHistoryRecord(result: BenchmarkRunResult) {
       ),
     ),
     cases: result.cases.map((caseResult) => {
-      const attemptAggregate = aggregateAttempts(caseResult.attempts);
-      const passedAttemptAggregate = aggregateAttempts(
-        caseResult.attempts.filter((attempt) => attempt.passed),
+      const attemptCount = caseResult.attempts.length;
+      const passedAttempts = caseResult.attempts.filter(
+        (attempt) => attempt.passed,
+      ).length;
+      const totalDurationMs = caseResult.attempts.reduce(
+        (sum, attempt) => sum + attempt.durationMs,
+        0,
       );
-      const attemptCount = attemptAggregate.attemptCount;
-      const passedAttempts = passedAttemptAggregate.attemptCount;
       const judgeScores = caseResult.attempts.flatMap((attempt) =>
         typeof attempt.judgeScore === "number" ? [attempt.judgeScore] : [],
       );
+      const totalTokenUsage =
+        caseResult.attempts.reduce<BenchmarkTokenUsage | null>(
+          (sum, attempt) => {
+            if (!attempt.tokenUsage) {
+              return sum;
+            }
+            sum ??= { prompt: 0, completion: 0, total: 0 };
+            sum.prompt += attempt.tokenUsage.prompt;
+            sum.completion += attempt.tokenUsage.completion;
+            sum.total += attempt.tokenUsage.total;
+            return sum;
+          },
+          null,
+        );
 
       return {
         id: caseResult.id,
@@ -374,27 +300,20 @@ function toHistoryRecord(result: BenchmarkRunResult) {
         passedAttempts,
         passRate: attemptCount === 0 ? 0 : passedAttempts / attemptCount,
         averageDurationMs:
-          attemptCount === 0
-            ? 0
-            : attemptAggregate.durationTotal / attemptCount,
-        averagePassedDurationMs: averageDuration(passedAttemptAggregate),
+          attemptCount === 0 ? 0 : totalDurationMs / attemptCount,
         averageJudgeScore:
           judgeScores.length === 0
             ? null
             : judgeScores.reduce((sum, score) => sum + score, 0) /
               judgeScores.length,
         averageTokenUsagePerAttempt:
-          attemptCount === 0
+          attemptCount === 0 || !totalTokenUsage
             ? null
-            : averageTokenUsage(attemptAggregate, attemptCount),
-        averageTokenUsagePerPassedAttempt: averageTokenUsage(
-          passedAttemptAggregate,
-          passedAttempts,
-        ),
-        averageFinalContextTokensPassed: averageFinalContext(
-          passedAttemptAggregate,
-        ),
-        maxFinalContextTokensPassed: passedAttemptAggregate.finalContextMax,
+            : {
+                prompt: totalTokenUsage.prompt / attemptCount,
+                completion: totalTokenUsage.completion / attemptCount,
+                total: totalTokenUsage.total / attemptCount,
+              },
       };
     }),
   };

ai_evals/core/runSuite.test.ts

@@ -1,102 +0,0 @@
-import { describe, expect, it } from "bun:test";
-import { runSuite } from "./runSuite";
-import type { ModeRunner } from "./types";
-
-const modeRunner: ModeRunner<undefined, undefined, { ok: boolean }> = {
-  mode: "global",
-  concurrency: 1,
-  loadInitial: async () => undefined,
-  loadExpected: async () => undefined,
-  run: async () => ({
-    success: true,
-    actual: { ok: true },
-    assistantMessageCount: 1,
-    toolCallCount: 0,
-    toolsUsed: [],
-    skillsInvoked: [],
-    tokenUsage: null,
-  }),
-  validate: () => [],
-};
-
-describe("runSuite", () => {
-  it("skips judge checks when the run disables judge scoring", async () => {
-    const [caseResult] = await runSuite({
-      modeRunner,
-      cases: [
-        {
-          id: "case-1",
-          prompt: "Create a draft script",
-          judgeChecklist: ["the output satisfies the prompt"],
-        },
-      ],
-      runs: 1,
-      runModel: "model-under-test",
-      judgeModel: null,
-    });
-
-    const [attempt] = caseResult.attempts;
-    expect(attempt.passed).toBe(true);
-    expect(attempt.judgeScore).toBeNull();
-    expect(attempt.judgeSummary).toBeNull();
-    expect(attempt.checks.map((check) => check.name)).toEqual([
-      "run succeeded",
-    ]);
-  });
-
-  it("only requires run success when execution-only is enabled", async () => {
-    let loadExpectedCalls = 0;
-    let validateCalls = 0;
-    let backendValidateCalls = 0;
-
-    const executionOnlyRunner: ModeRunner<
-      undefined,
-      undefined,
-      { ok: boolean }
-    > = {
-      ...modeRunner,
-      loadExpected: async () => {
-        loadExpectedCalls++;
-        return undefined;
-      },
-      validate: () => {
-        validateCalls++;
-        return [{ name: "validator failed", passed: false }];
-      },
-      backendValidate: async () => {
-        backendValidateCalls++;
-        return {
-          checks: [{ name: "backend validation failed", passed: false }],
-        };
-      },
-    };
-
-    const [caseResult] = await runSuite({
-      modeRunner: executionOnlyRunner,
-      cases: [
-        {
-          id: "case-1",
-          prompt: "Create a draft script",
-          expectedPath: "fixtures/expected.json",
-          toolExpect: { requiredToolsUsed: ["write_script"] },
-          judgeChecklist: ["the output satisfies the prompt"],
-        },
-      ],
-      runs: 1,
-      runModel: "model-under-test",
-      judgeModel: "judge-model",
-      executionOnly: true,
-    });
-
-    const [attempt] = caseResult.attempts;
-    expect(attempt.passed).toBe(true);
-    expect(attempt.judgeScore).toBeNull();
-    expect(attempt.judgeSummary).toBeNull();
-    expect(attempt.checks.map((check) => check.name)).toEqual([
-      "run succeeded",
-    ]);
-    expect(loadExpectedCalls).toBe(0);
-    expect(validateCalls).toBe(0);
-    expect(backendValidateCalls).toBe(0);
-  });
-});

ai_evals/core/runSuite.ts

@@ -15,13 +15,11 @@ export async function runSuite<TInitial, TExpected, TActual>(input: {
   runs: number;
   runModel: string | null;
   judgeModel?: string | null;
-  executionOnly?: boolean;
   concurrency?: number;
   verbose?: boolean;
   onProgress?: (event: FrontendBenchmarkProgressEvent) => void;
 }): Promise<BenchmarkCaseResult[]> {
-  const judgeModel =
-    input.judgeModel === undefined ? DEFAULT_JUDGE_MODEL : input.judgeModel;
+  const judgeModel = input.judgeModel ?? DEFAULT_JUDGE_MODEL;
   const concurrency = Math.max(1, input.concurrency ?? input.modeRunner.concurrency);
   const results = new Array<BenchmarkCaseResult>(input.cases.length);
   let cursor = 0;
@@ -54,7 +52,6 @@ export async function runSuite<TInitial, TExpected, TActual>(input: {
           runs: input.runs,
           judgeModel,
           judgeThreshold: input.modeRunner.judgeThreshold ?? 80,
-          executionOnly: input.executionOnly ?? false,
           modeRunner: input.modeRunner,
           totalCases: input.cases.length,
           verbose: input.verbose ?? false,
@@ -75,9 +72,8 @@ async function runCaseAttempts<TInitial, TExpected, TActual>(input: {
   caseIndex: number;
   evalCase: EvalCase;
   runs: number;
-  judgeModel: string | null;
+  judgeModel: string;
   judgeThreshold: number;
-  executionOnly: boolean;
   modeRunner: ModeRunner<TInitial, TExpected, TActual>;
   totalCases: number;
   verbose: boolean;
@@ -103,9 +99,7 @@ async function runCaseAttempts<TInitial, TExpected, TActual>(input: {
 
     try {
       const initial = await input.modeRunner.loadInitial(input.evalCase.initialPath);
-      const expected = input.executionOnly
-        ? undefined
-        : await input.modeRunner.loadExpected(input.evalCase.expectedPath);
+      const expected = await input.modeRunner.loadExpected(input.evalCase.expectedPath);
       const run = await input.modeRunner.run(input.evalCase.prompt, initial, {
         evalCase: input.evalCase,
         caseId: input.evalCase.id,
@@ -168,30 +162,22 @@ async function runCaseAttempts<TInitial, TExpected, TActual>(input: {
       });
       const checks: BenchmarkCheck[] = [
         buildCheck("run succeeded", run.success, run.error),
+        ...input.modeRunner.validate({
+          evalCase: input.evalCase,
+          prompt: input.evalCase.prompt,
+          initial,
+          expected,
+          actual: run.actual,
+          run,
+        }),
+        ...validateToolExpectations({
+          run,
+          toolExpect: input.evalCase.toolExpect,
+        }),
       ];
-      if (!input.executionOnly) {
-        checks.push(
-          ...input.modeRunner.validate({
-            evalCase: input.evalCase,
-            prompt: input.evalCase.prompt,
-            initial,
-            expected,
-            actual: run.actual,
-            run,
-          }),
-          ...validateToolExpectations({
-            run,
-            toolExpect: input.evalCase.toolExpect,
-          })
-        );
-      }
       const artifactFiles = input.modeRunner.buildArtifacts?.(run.actual) ?? [];
 
-      if (
-        run.success &&
-        !input.executionOnly &&
-        input.modeRunner.backendValidate
-      ) {
+      if (run.success && input.modeRunner.backendValidate) {
         try {
           const backendValidation = await input.modeRunner.backendValidate({
             evalCase: input.evalCase,
@@ -232,21 +218,14 @@ async function runCaseAttempts<TInitial, TExpected, TActual>(input: {
       let judgeScore: number | null = null;
       let judgeSummary: string | null = null;
 
-      if (
-        run.success &&
-        !input.executionOnly &&
-        input.judgeModel !== null &&
-        !input.evalCase.skipJudge
-      ) {
+      if (run.success && !input.evalCase.skipJudge) {
         const judge = await judgeOutput({
           mode: input.modeRunner.mode,
           prompt: input.evalCase.prompt,
           checklist: input.evalCase.judgeChecklist,
           initial,
           expected: input.modeRunner.mode === "cli" ? undefined : expected,
-          actual: input.modeRunner.prepareJudgeActual
-            ? input.modeRunner.prepareJudgeActual(run.actual)
-            : run.actual,
+          actual: run.actual,
           model: input.judgeModel,
         });
 
@@ -276,7 +255,6 @@ async function runCaseAttempts<TInitial, TExpected, TActual>(input: {
         judgeSummary,
         error: run.error ?? null,
         tokenUsage: run.tokenUsage ?? null,
-        finalContextTokens: run.finalContextTokens ?? null,
         artifactsPath: null,
         artifactFiles,
       };
@@ -313,7 +291,6 @@ async function runCaseAttempts<TInitial, TExpected, TActual>(input: {
         judgeSummary: null,
         error: message,
         tokenUsage: null,
-        finalContextTokens: null,
       };
       if (surface) {
         input.onProgress?.({

ai_evals/core/types.ts

@@ -110,9 +110,7 @@ export interface AppValidationSpec {
 
 export interface GlobalDraftRequirement {
   type: string;
-  path?: string;
-  pathIncludes?: string[];
-  pathStartsWith?: string;
+  path: string;
   triggerKind?: string;
   language?: string;
   summaryIncludes?: string[];
@@ -155,34 +153,15 @@ export interface ToolCallArgumentRule {
   field: string;
   stringStartsWithAnyOf?: string[];
   stringMustNotStartWithAnyOf?: string[];
-  /**
-   * Case-insensitive "contains", existential over calls: at least one recorded
-   * call to `tool` must have `field` containing one of these substrings. Other
-   * calls to the same tool may do anything. Use instead of `stringStartsWithAnyOf`
-   * (which is universal over calls) when the meaningful token can appear anywhere
-   * in the value and the model may make additional, unrelated calls to the same
-   * tool — e.g. SQL where a mutation is mixed with verification SELECTs.
-   */
-  stringIncludesAnyOf?: string[];
 }
 
 export interface ToolValidationSpec {
   requiredToolsUsed?: string[];
-  /**
-   * Each inner array is an alternatives group: the check passes when at least
-   * one tool in the group was used. Use when several tools satisfy the same
-   * intent so a model that picks any valid path passes — e.g. inspecting an
-   * app's files via either `read_app_file` or `search_app`.
-   */
-  requiredToolsAnyOf?: string[][];
   forbiddenToolsUsed?: string[];
   toolCallArgs?: ToolCallArgumentRule[];
 }
 
-export type EvalValidationSpec =
-  | FlowValidationSpec
-  | AppValidationSpec
-  | GlobalValidationSpec;
+export type EvalValidationSpec = FlowValidationSpec | AppValidationSpec | GlobalValidationSpec;
 
 export interface EvalCase {
   id: string;
@@ -259,12 +238,6 @@ export interface ModeRunOutput<TActual> {
   toolCallDetails?: ToolCallDetail[];
   skillsInvoked: string[];
   tokenUsage?: BenchmarkTokenUsage | null;
-  /**
-   * Total input tokens occupying the context window on the LAST model request
-   * of the agentic loop (input + cache-creation + cache-read). Complements the
-   * cumulative `tokenUsage.prompt`, which sums every iteration's input.
-   */
-  finalContextTokens?: number | null;
 }
 
 export interface ModeRunContext {
@@ -310,12 +283,6 @@ export interface ModeRunner<TInitial, TExpected, TActual> {
     context: ModeRunContext;
   }): Promise<BackendValidationResult | null>;
   buildArtifacts?(actual: TActual): BenchmarkArtifactFile[];
-  /**
-   * Optional transform applied to `actual` before it is handed to the LLM judge.
-   * Use it to strip fields the judge must stay blind to (e.g. which docs-tool
-   * arm produced an answer). When omitted, the judge receives `actual` as-is.
-   */
-  prepareJudgeActual?(actual: TActual): unknown;
 }
 
 export interface BenchmarkAttemptResult {
@@ -332,7 +299,6 @@ export interface BenchmarkAttemptResult {
   judgeSummary: string | null;
   error: string | null;
   tokenUsage?: BenchmarkTokenUsage | null;
-  finalContextTokens?: number | null;
   artifactsPath?: string | null;
   artifactFiles?: BenchmarkArtifactFile[];
 }
@@ -358,13 +324,8 @@ export interface BenchmarkRunResult {
   passedAttempts: number;
   passRate: number;
   averageDurationMs: number;
-  averagePassedDurationMs?: number | null;
   totalTokenUsage?: BenchmarkTokenUsage | null;
-  totalPassedTokenUsage?: BenchmarkTokenUsage | null;
   averageTokenUsagePerAttempt?: BenchmarkTokenUsage | null;
-  averageTokenUsagePerPassedAttempt?: BenchmarkTokenUsage | null;
-  averageFinalContextTokensPassed?: number | null;
-  maxFinalContextTokensPassed?: number | null;
   artifactsPath?: string | null;
   cases: BenchmarkCaseResult[];
 }

ai_evals/core/validators.test.ts

@@ -140,154 +140,6 @@ describe("validateToolExpectations", () => {
       details: "tools used: write_script, deploy_workspace_item",
     });
   });
-
-  it("accepts a stringIncludesAnyOf substring regardless of case or position", () => {
-    const checks = validateToolExpectations({
-      run: {
-        success: true,
-        actual: {},
-        assistantMessageCount: 1,
-        toolCallCount: 1,
-        toolsUsed: ["exec_datatable_sql"],
-        toolCallDetails: [
-          {
-            name: "exec_datatable_sql",
-            arguments: {
-              sql: "WITH recent AS (SELECT * FROM orders) SELECT count(*) FROM recent",
-            },
-          },
-        ],
-        skillsInvoked: [],
-      },
-      toolExpect: {
-        requiredToolsUsed: ["exec_datatable_sql"],
-        toolCallArgs: [
-          {
-            tool: "exec_datatable_sql",
-            field: "sql",
-            stringIncludesAnyOf: ["select"],
-          },
-        ],
-      },
-    });
-
-    expect(checks.every((check) => check.passed)).toBe(true);
-  });
-
-  it("accepts stringIncludesAnyOf when only one of several calls matches", () => {
-    // Existential: a mutation mixed with verification SELECTs still passes.
-    const checks = validateToolExpectations({
-      run: {
-        success: true,
-        actual: {},
-        assistantMessageCount: 1,
-        toolCallCount: 2,
-        toolsUsed: ["exec_datatable_sql"],
-        toolCallDetails: [
-          {
-            name: "exec_datatable_sql",
-            arguments: { sql: "UPDATE orders SET status = 'shipped' WHERE id = 2" },
-          },
-          {
-            name: "exec_datatable_sql",
-            arguments: { sql: "SELECT * FROM orders WHERE id = 2" },
-          },
-        ],
-        skillsInvoked: [],
-      },
-      toolExpect: {
-        toolCallArgs: [
-          {
-            tool: "exec_datatable_sql",
-            field: "sql",
-            stringIncludesAnyOf: ["insert into", "update"],
-          },
-        ],
-      },
-    });
-
-    expect(checks.every((check) => check.passed)).toBe(true);
-  });
-
-  it("rejects stringIncludesAnyOf when no call matches any substring", () => {
-    const checks = validateToolExpectations({
-      run: {
-        success: true,
-        actual: {},
-        assistantMessageCount: 1,
-        toolCallCount: 1,
-        toolsUsed: ["exec_datatable_sql"],
-        toolCallDetails: [
-          {
-            name: "exec_datatable_sql",
-            arguments: {
-              sql: "DROP TABLE orders",
-            },
-          },
-        ],
-        skillsInvoked: [],
-      },
-      toolExpect: {
-        toolCallArgs: [
-          {
-            tool: "exec_datatable_sql",
-            field: "sql",
-            stringIncludesAnyOf: ["insert into", "update"],
-          },
-        ],
-      },
-    });
-
-    expect(checks).toContainEqual({
-      name: "exec_datatable_sql.sql includes a required substring",
-      passed: false,
-      details:
-        'accepted substrings: insert into, update; values: "DROP TABLE orders"',
-    });
-  });
-
-  it("passes requiredToolsAnyOf when any alternative in the group is used", () => {
-    const checks = validateToolExpectations({
-      run: {
-        success: true,
-        actual: {},
-        assistantMessageCount: 1,
-        toolCallCount: 1,
-        toolsUsed: ["search_app", "patch_app_file"],
-        skillsInvoked: [],
-      },
-      toolExpect: {
-        requiredToolsAnyOf: [["read_app_file", "search_app"]],
-      },
-    });
-
-    expect(checks).toContainEqual({
-      name: "uses one of read_app_file, search_app",
-      passed: true,
-    });
-  });
-
-  it("fails requiredToolsAnyOf when no alternative in the group is used", () => {
-    const checks = validateToolExpectations({
-      run: {
-        success: true,
-        actual: {},
-        assistantMessageCount: 1,
-        toolCallCount: 1,
-        toolsUsed: ["patch_app_file"],
-        skillsInvoked: [],
-      },
-      toolExpect: {
-        requiredToolsAnyOf: [["read_app_file", "search_app"]],
-      },
-    });
-
-    expect(checks).toContainEqual({
-      name: "uses one of read_app_file, search_app",
-      passed: false,
-      details: "tools used: patch_app_file",
-    });
-  });
 });
 
 describe("validateGlobalState", () => {
@@ -343,69 +195,6 @@ describe("validateGlobalState", () => {
     });
   });
 
-  it("accepts a required script draft without an exact path", () => {
-    const checks = validateGlobalState({
-      actual: {
-        drafts: [
-          {
-            type: "script",
-            path: "f/team_tools/friendly_greeting",
-            language: "bun",
-            summary: "Friendly greeting helper",
-            value:
-              "export async function main(name: string) {\n  return `Hello, ${name}!`\n}\n",
-            isDraft: true,
-          },
-        ],
-      },
-      validate: {
-        draftCountExactly: 1,
-        requiredDrafts: [
-          {
-            type: "script",
-            pathIncludes: ["greeting"],
-            language: "bun",
-            summaryIncludes: ["Friendly"],
-            valueIncludes: ["Hello"],
-          },
-        ],
-      },
-    });
-
-    expect(checks.every((check) => check.passed)).toBe(true);
-  });
-
-  it("reports flexible global draft path filters when no draft matches", () => {
-    const checks = validateGlobalState({
-      actual: {
-        drafts: [
-          {
-            type: "script",
-            path: "f/team_tools/friendly_greeting",
-            language: "bun",
-            value:
-              "export async function main(name: string) {\n  return `Hello, ${name}!`\n}\n",
-            isDraft: true,
-          },
-        ],
-      },
-      validate: {
-        requiredDrafts: [
-          {
-            type: "script",
-            pathIncludes: ["invoice"],
-          },
-        ],
-      },
-    });
-
-    expect(checks).toContainEqual({
-      name: "global includes script draft (path includes invoice)",
-      passed: false,
-      details: "drafts: script:f/team_tools/friendly_greeting",
-    });
-  });
-
   it("does not require a TypeScript entrypoint for non-TypeScript script drafts", () => {
     const checks = validateGlobalState({
       actual: {

ai_evals/core/validators.ts

@@ -169,16 +169,6 @@ export function validateToolExpectations(input: {
     );
   }
 
-  for (const group of expect.requiredToolsAnyOf ?? []) {
-    checks.push(
-      check(
-        `uses one of ${group.join(", ")}`,
-        group.some((toolName) => input.run.toolsUsed.includes(toolName)),
-        `tools used: ${input.run.toolsUsed.join(", ") || "none"}`
-      )
-    );
-  }
-
   for (const toolName of expect.forbiddenToolsUsed ?? []) {
     checks.push(
       check(
@@ -232,25 +222,6 @@ export function validateToolExpectations(input: {
         )
       );
     }
-
-    if (rule.stringIncludesAnyOf && rule.stringIncludesAnyOf.length > 0) {
-      // Existential: at least one call must contain one of the substrings.
-      // Other calls to the same tool may do anything — this suits SQL, where a
-      // model mixes the requested statement (e.g. an UPDATE) with verification
-      // SELECTs that would otherwise fail an "all calls" check.
-      const needles = rule.stringIncludesAnyOf.map((needle) => needle.toLowerCase());
-      const hasMatch = values.some(
-        (value) =>
-          typeof value === "string" && needles.some((needle) => value.toLowerCase().includes(needle))
-      );
-      checks.push(
-        check(
-          `${rule.tool}.${rule.field} includes a required substring`,
-          hasMatch,
-          `accepted substrings: ${rule.stringIncludesAnyOf.join(", ")}; values: ${summarizeToolValues(values)}`
-        )
-      );
-    }
   }
 
   return checks;
@@ -344,11 +315,10 @@ export function validateGlobalState(input: {
   }
 
   for (const required of validate.requiredDrafts ?? []) {
-    const requirementLabel = formatGlobalDraftRequirement(required);
-    const draft = findGlobalDraft(drafts, required);
+    const draft = findGlobalDraft(drafts, required.type, required.path, required.triggerKind);
     checks.push(
       check(
-        `global includes ${requirementLabel}`,
+        `global includes ${required.type} draft ${required.path}`,
         Boolean(draft),
         summarizeGlobalDrafts(drafts)
       )
@@ -360,7 +330,7 @@ export function validateGlobalState(input: {
     if (required.language !== undefined) {
       checks.push(
         check(
-          `${requirementLabel} uses ${required.language}`,
+          `${required.type} draft ${required.path} uses ${required.language}`,
           draft.language === required.language,
           `language=${draft.language ?? "(none)"}`
         )
@@ -370,7 +340,7 @@ export function validateGlobalState(input: {
     for (const snippet of required.summaryIncludes ?? []) {
       checks.push(
         check(
-          `${requirementLabel} summary includes '${snippet}'`,
+          `${required.type} draft ${required.path} summary includes '${snippet}'`,
           normalizeText(draft.summary ?? "").includes(normalizeText(snippet)),
           `summary=${draft.summary ?? ""}`
         )
@@ -381,7 +351,7 @@ export function validateGlobalState(input: {
     for (const snippet of required.valueIncludes ?? []) {
       checks.push(
         check(
-          `${requirementLabel} value includes '${snippet}'`,
+          `${required.type} draft ${required.path} value includes '${snippet}'`,
           normalizeText(valueText).includes(normalizeText(snippet)),
           truncateForDetails(valueText)
         )
@@ -391,7 +361,7 @@ export function validateGlobalState(input: {
     for (const snippet of required.valueExcludes ?? []) {
       checks.push(
         check(
-          `${requirementLabel} value excludes '${snippet}'`,
+          `${required.type} draft ${required.path} value excludes '${snippet}'`,
           !normalizeText(valueText).includes(normalizeText(snippet)),
           truncateForDetails(valueText)
         )
@@ -403,7 +373,7 @@ export function validateGlobalState(input: {
     checks.push(
       check(
         `global does not include ${forbidden.type} draft ${forbidden.path}`,
-        !findGlobalDraft(drafts, forbidden),
+        !findGlobalDraft(drafts, forbidden.type, forbidden.path, forbidden.triggerKind),
         summarizeGlobalDrafts(drafts)
       )
     );
@@ -645,100 +615,16 @@ function summarizeProblems(problems: string[], limit = 5): string | undefined {
 
 function findGlobalDraft(
   drafts: GlobalDraft[],
-  requirement: {
-    type: string;
-    path?: string;
-    pathIncludes?: string[];
-    pathStartsWith?: string;
-    triggerKind?: string;
-    summaryIncludes?: string[];
-    valueIncludes?: string[];
-    valueExcludes?: string[];
-  }
+  type: string,
+  path: string,
+  triggerKind?: string
 ): GlobalDraft | undefined {
-  const candidates = drafts.filter((draft) =>
-    globalDraftMatchesLocator(draft, requirement)
+  return drafts.find(
+    (draft) =>
+      draft.type === type &&
+      draft.path === path &&
+      (triggerKind === undefined || draft.triggerKind === triggerKind)
   );
-  return (
-    candidates.find((draft) => globalDraftMatchesContent(draft, requirement)) ??
-    candidates[0]
-  );
-}
-
-function globalDraftMatchesLocator(
-  draft: GlobalDraft,
-  requirement: {
-    type: string;
-    path?: string;
-    pathIncludes?: string[];
-    pathStartsWith?: string;
-    triggerKind?: string;
-  }
-): boolean {
-  return (
-    draft.type === requirement.type &&
-    (requirement.path === undefined || draft.path === requirement.path) &&
-    (requirement.pathStartsWith === undefined ||
-      draft.path.startsWith(requirement.pathStartsWith)) &&
-    (requirement.pathIncludes ?? []).every((snippet) =>
-      normalizeText(draft.path).includes(normalizeText(snippet))
-    ) &&
-    (requirement.triggerKind === undefined ||
-      draft.triggerKind === requirement.triggerKind)
-  );
-}
-
-function globalDraftMatchesContent(
-  draft: GlobalDraft,
-  requirement: {
-    summaryIncludes?: string[];
-    valueIncludes?: string[];
-    valueExcludes?: string[];
-  }
-): boolean {
-  const summary = normalizeText(draft.summary ?? "");
-  const value = normalizeText(stringifyGlobalDraftValue(draft.value));
-  return (
-    (requirement.summaryIncludes ?? []).every((snippet) =>
-      summary.includes(normalizeText(snippet))
-    ) &&
-    (requirement.valueIncludes ?? []).every((snippet) =>
-      value.includes(normalizeText(snippet))
-    ) &&
-    (requirement.valueExcludes ?? []).every(
-      (snippet) => !value.includes(normalizeText(snippet))
-    )
-  );
-}
-
-function formatGlobalDraftRequirement(
-  requirement: {
-    type: string;
-    path?: string;
-    pathIncludes?: string[];
-    pathStartsWith?: string;
-    triggerKind?: string;
-  }
-): string {
-  const typeLabel =
-    requirement.triggerKind === undefined
-      ? requirement.type
-      : `${requirement.triggerKind} ${requirement.type}`;
-  if (requirement.path !== undefined) {
-    return `${typeLabel} draft ${requirement.path}`;
-  }
-
-  const filters = [
-    ...(requirement.pathStartsWith === undefined
-      ? []
-      : [`path starts with ${requirement.pathStartsWith}`]),
-    ...(requirement.pathIncludes ?? []).map(
-      (snippet) => `path includes ${snippet}`
-    ),
-  ];
-  return filters.length === 0
-    ? `${typeLabel} draft`
-    : `${typeLabel} draft (${filters.join(", ")})`;
 }
 
 function summarizeGlobalDrafts(drafts: GlobalDraft[]): string {

ai_evals/fixtures/frontend/global/initial/analytics_dashboard/backend/computeSummary/main.ts

@@ -1,68 +0,0 @@
-type OrderStatus = 'paid' | 'shipped' | 'delivered' | 'pending' | 'refunded' | 'cancelled'
-
-interface Order {
-	id: string
-	region: string
-	quantity: number
-	unitPrice: number
-	status: OrderStatus
-	placedAt: string
-}
-
-// Server-side revenue rollup. Mirrors the client aggregation but is computed
-// from the authoritative mocked order book so it can be used to cross-check
-// the dashboard and to back the export.
-const orders: Order[] = [
-	{ id: 'ORD-10001', region: 'North America', quantity: 3, unitPrice: 1195, status: 'delivered', placedAt: '2024-05-02' },
-	{ id: 'ORD-10002', region: 'EMEA', quantity: 5, unitPrice: 880, status: 'shipped', placedAt: '2024-05-03' },
-	{ id: 'ORD-10003', region: 'APAC', quantity: 2, unitPrice: 640, status: 'paid', placedAt: '2024-05-05' },
-	{ id: 'ORD-10004', region: 'LATAM', quantity: 7, unitPrice: 315, status: 'delivered', placedAt: '2024-05-07' },
-	{ id: 'ORD-10005', region: 'North America', quantity: 4, unitPrice: 150, status: 'refunded', placedAt: '2024-05-09' },
-	{ id: 'ORD-10006', region: 'EMEA', quantity: 6, unitPrice: 220, status: 'shipped', placedAt: '2024-05-12' },
-	{ id: 'ORD-10007', region: 'APAC', quantity: 1, unitPrice: 980, status: 'pending', placedAt: '2024-05-15' },
-	{ id: 'ORD-10008', region: 'North America', quantity: 8, unitPrice: 1100, status: 'delivered', placedAt: '2024-05-18' },
-	{ id: 'ORD-10009', region: 'EMEA', quantity: 2, unitPrice: 860, status: 'cancelled', placedAt: '2024-05-22' },
-	{ id: 'ORD-10010', region: 'LATAM', quantity: 9, unitPrice: 290, status: 'paid', placedAt: '2024-05-26' }
-]
-
-const REVENUE_STATUSES: OrderStatus[] = ['paid', 'shipped', 'delivered']
-
-export async function main({
-	from,
-	to,
-	region
-}: {
-	from: string
-	to: string
-	region: string
-}): Promise<{
-	totalRevenue: number
-	netRevenue: number
-	totalOrders: number
-	averageOrderValue: number
-	unitsSold: number
-	refundedRevenue: number
-	currency: string
-}> {
-	let scoped = orders.filter((order) => order.placedAt >= from && order.placedAt <= to)
-	if (region && region !== 'all') {
-		scoped = scoped.filter((order) => order.region === region)
-	}
-
-	const booked = scoped.filter((order) => REVENUE_STATUSES.includes(order.status))
-	const totalRevenue = booked.reduce((acc, order) => acc + order.unitPrice * order.quantity, 0)
-	const unitsSold = booked.reduce((acc, order) => acc + order.quantity, 0)
-	const refundedRevenue = scoped
-		.filter((order) => order.status === 'refunded')
-		.reduce((acc, order) => acc + order.unitPrice * order.quantity, 0)
-
-	return {
-		totalRevenue,
-		netRevenue: totalRevenue - refundedRevenue,
-		totalOrders: booked.length,
-		averageOrderValue: booked.length === 0 ? 0 : Math.round(totalRevenue / booked.length),
-		unitsSold,
-		refundedRevenue,
-		currency: 'USD'
-	}
-}

ai_evals/fixtures/frontend/global/initial/analytics_dashboard/backend/computeSummary/meta.json

@@ -1,4 +0,0 @@
-{
-	"name": "Compute Summary",
-	"language": "bun"
-}

ai_evals/fixtures/frontend/global/initial/analytics_dashboard/backend/exportReport/main.ts

@@ -1,51 +0,0 @@
-// Builds a downloadable report for the current dashboard view. Returns a data
-// URL the browser can open directly so the export works without object storage.
-export async function main({
-	from,
-	to,
-	region,
-	format
-}: {
-	from: string
-	to: string
-	region: string
-	format: 'csv' | 'json'
-}): Promise<{ url: string; rows: number; filename: string }> {
-	const summary = {
-		from,
-		to,
-		region: region || 'all',
-		generatedAt: new Date().toISOString(),
-		rows: [
-			{ region: 'North America', revenue: 211_400, orders: 168 },
-			{ region: 'EMEA', revenue: 142_900, orders: 121 },
-			{ region: 'APAC', revenue: 86_500, orders: 78 },
-			{ region: 'LATAM', revenue: 41_500, orders: 45 }
-		]
-	}
-
-	const scoped =
-		region && region !== 'all'
-			? summary.rows.filter((row) => row.region === region)
-			: summary.rows
-
-	let body: string
-	let mime: string
-	if (format === 'csv') {
-		const header = 'region,revenue,orders'
-		const lines = scoped.map((row) => `${row.region},${row.revenue},${row.orders}`)
-		body = [header, ...lines].join('\n')
-		mime = 'text/csv'
-	} else {
-		body = JSON.stringify({ ...summary, rows: scoped }, null, 2)
-		mime = 'application/json'
-	}
-
-	const encoded = Buffer.from(body, 'utf-8').toString('base64')
-	const filename = `revenue-report-${from}_${to}.${format}`
-	return {
-		url: `data:${mime};base64,${encoded}`,
-		rows: scoped.length,
-		filename
-	}
-}

ai_evals/fixtures/frontend/global/initial/analytics_dashboard/backend/exportReport/meta.json

@@ -1,4 +0,0 @@
-{
-	"name": "Export Report",
-	"language": "bun"
-}

ai_evals/fixtures/frontend/global/initial/analytics_dashboard/backend/loadMetrics/main.ts

@@ -1,40 +0,0 @@
-interface MetricCardData {
-	id: string
-	label: string
-	value: number
-	unit: 'currency' | 'count' | 'percent'
-	delta: number
-	hint: string
-}
-
-// Returns the headline metric cards for the selected range and region. Values
-// are mocked but internally consistent (revenue / orders ≈ avg order value).
-const baseByRegion: Record<string, { revenue: number; orders: number; units: number; refunds: number }> = {
-	all: { revenue: 482_300, orders: 412, units: 1840, refunds: 11_900 },
-	'North America': { revenue: 211_400, orders: 168, units: 770, refunds: 4_200 },
-	EMEA: { revenue: 142_900, orders: 121, units: 560, refunds: 3_500 },
-	APAC: { revenue: 86_500, orders: 78, units: 340, refunds: 2_600 },
-	LATAM: { revenue: 41_500, orders: 45, units: 170, refunds: 1_600 }
-}
-
-export async function main({
-	from,
-	to,
-	region
-}: {
-	from: string
-	to: string
-	region: string
-}): Promise<{ cards: MetricCardData[]; generatedAt: string }> {
-	const base = baseByRegion[region] ?? baseByRegion.all
-	const aov = base.orders === 0 ? 0 : Math.round(base.revenue / base.orders)
-	const cards: MetricCardData[] = [
-		{ id: 'revenue', label: 'Total Revenue', value: base.revenue, unit: 'currency', delta: 0.082, hint: `Booked revenue ${from} – ${to}` },
-		{ id: 'orders', label: 'Orders', value: base.orders, unit: 'count', delta: 0.041, hint: 'Revenue-bearing orders in range' },
-		{ id: 'aov', label: 'Avg Order Value', value: aov, unit: 'currency', delta: -0.013, hint: 'Total revenue / order count' },
-		{ id: 'units', label: 'Units Sold', value: base.units, unit: 'count', delta: 0.067, hint: 'Total units in range' },
-		{ id: 'refunds', label: 'Refunded', value: base.refunds, unit: 'currency', delta: -0.021, hint: 'Revenue lost to refunds' },
-		{ id: 'conversion', label: 'Conversion', value: 0.187, unit: 'percent', delta: 0.009, hint: 'Sessions that became orders' }
-	]
-	return { cards, generatedAt: new Date().toISOString() }
-}

ai_evals/fixtures/frontend/global/initial/analytics_dashboard/backend/loadMetrics/meta.json

@@ -1,4 +0,0 @@
-{
-	"name": "Load Metrics",
-	"language": "bun"
-}

ai_evals/fixtures/frontend/global/initial/analytics_dashboard/backend/loadOrders/main.ts

@@ -1,54 +0,0 @@
-type OrderStatus = 'paid' | 'shipped' | 'delivered' | 'pending' | 'refunded' | 'cancelled'
-
-interface Order {
-	id: string
-	placedAt: string
-	customer: string
-	product: string
-	sku: string
-	region: string
-	channel: string
-	rep: string
-	quantity: number
-	unitPrice: number
-	status: OrderStatus
-}
-
-// Mocked order book. In a real deployment this would query the orders table;
-// here it returns a representative slice so the table renders in preview.
-const orders: Order[] = [
-	{ id: 'ORD-10001', placedAt: '2024-05-02T09:14:00Z', customer: 'Contoso Ltd', product: 'Aurora Analytics Suite', sku: 'ANL-100', region: 'North America', channel: 'direct', rep: 'Dana Wills', quantity: 3, unitPrice: 1195, status: 'delivered' },
-	{ id: 'ORD-10002', placedAt: '2024-05-03T11:42:00Z', customer: 'Fabrikam Inc', product: 'Borealis CRM', sku: 'CRM-210', region: 'EMEA', channel: 'partner', rep: 'Lena Fischer', quantity: 5, unitPrice: 880, status: 'shipped' },
-	{ id: 'ORD-10003', placedAt: '2024-05-05T15:03:00Z', customer: 'Tailspin Toys', product: 'Cascade Data Pipeline', sku: 'PIPE-330', region: 'APAC', channel: 'self-serve', rep: 'Sora Tanaka', quantity: 2, unitPrice: 640, status: 'paid' },
-	{ id: 'ORD-10004', placedAt: '2024-05-07T08:21:00Z', customer: 'Proseware Inc', product: 'Delta Insights', sku: 'INS-440', region: 'LATAM', channel: 'marketplace', rep: 'Diego Marin', quantity: 7, unitPrice: 315, status: 'delivered' },
-	{ id: 'ORD-10005', placedAt: '2024-05-09T13:58:00Z', customer: 'Litware Inc', product: 'Echo Monitoring', sku: 'MON-550', region: 'North America', channel: 'direct', rep: 'Owen Pratt', quantity: 4, unitPrice: 150, status: 'refunded' },
-	{ id: 'ORD-10006', placedAt: '2024-05-12T10:30:00Z', customer: 'Fourth Coffee', product: 'Helix Identity', sku: 'IDN-880', region: 'EMEA', channel: 'partner', rep: 'Aisha Khan', quantity: 6, unitPrice: 220, status: 'shipped' },
-	{ id: 'ORD-10007', placedAt: '2024-05-15T17:11:00Z', customer: 'Coho Vineyard', product: 'Kelvin Forecasting', sku: 'FCT-202', region: 'APAC', channel: 'direct', rep: 'Priya Nair', quantity: 1, unitPrice: 980, status: 'pending' },
-	{ id: 'ORD-10008', placedAt: '2024-05-18T12:05:00Z', customer: 'Alpine Ski House', product: 'Nimbus Compute', sku: 'CMP-505', region: 'North America', channel: 'self-serve', rep: 'Hugo Bernard', quantity: 8, unitPrice: 1100, status: 'delivered' },
-	{ id: 'ORD-10009', placedAt: '2024-05-22T14:47:00Z', customer: 'Trey Research', product: 'Onyx Security', sku: 'SEC-606', region: 'EMEA', channel: 'direct', rep: 'Sven Olsen', quantity: 2, unitPrice: 860, status: 'cancelled' },
-	{ id: 'ORD-10010', placedAt: '2024-05-26T16:39:00Z', customer: 'Blue Yonder Airlines', product: 'Polaris Reporting', sku: 'RPT-707', region: 'LATAM', channel: 'partner', rep: 'Mateo Russo', quantity: 9, unitPrice: 290, status: 'paid' }
-]
-
-export async function main({
-	from,
-	to,
-	region,
-	status
-}: {
-	from: string
-	to: string
-	region: string
-	status: string
-}): Promise<{ orders: Order[]; total: number }> {
-	let filtered = orders.filter((order) => {
-		const day = order.placedAt.slice(0, 10)
-		return day >= from && day <= to
-	})
-	if (region && region !== 'all') {
-		filtered = filtered.filter((order) => order.region === region)
-	}
-	if (status && status !== 'all') {
-		filtered = filtered.filter((order) => order.status === status)
-	}
-	return { orders: filtered, total: filtered.length }
-}

ai_evals/fixtures/frontend/global/initial/analytics_dashboard/backend/loadOrders/meta.json

@@ -1,4 +0,0 @@
-{
-	"name": "Load Orders",
-	"language": "bun"
-}

ai_evals/fixtures/frontend/global/initial/analytics_dashboard/frontend/components/DateRangePicker.tsx

@@ -1,45 +0,0 @@
-import React from 'react'
-import type { DateRange } from '../lib/api'
-import { rangeForPreset } from '../lib/api'
-import { formatDateShort } from '../lib/format'
-
-interface DateRangePickerProps {
-	preset: string
-	range: DateRange
-	onPresetChange: (preset: string, range: DateRange) => void
-}
-
-const PRESETS: { id: string; label: string }[] = [
-	{ id: '7d', label: 'Last 7 days' },
-	{ id: '14d', label: 'Last 14 days' },
-	{ id: '30d', label: 'Last 30 days' },
-	{ id: 'qtd', label: 'Quarter to date' }
-]
-
-export const DateRangePicker: React.FC<DateRangePickerProps> = ({
-	preset,
-	range,
-	onPresetChange
-}) => {
-	return (
-		<div className="flex items-center gap-2">
-			<select
-				className="rounded-lg border border-gray-300 bg-white px-3 py-2 text-sm text-gray-700"
-				value={preset}
-				onChange={(event) => {
-					const next = event.target.value
-					onPresetChange(next, rangeForPreset(next))
-				}}
-			>
-				{PRESETS.map((item) => (
-					<option key={item.id} value={item.id}>
-						{item.label}
-					</option>
-				))}
-			</select>
-			<span className="text-xs text-gray-400">
-				{formatDateShort(range.from)} – {formatDateShort(range.to)}
-			</span>
-		</div>
-	)
-}

ai_evals/fixtures/frontend/global/initial/analytics_dashboard/frontend/components/EmptyState.tsx

@@ -1,28 +0,0 @@
-import React from 'react'
-
-interface EmptyStateProps {
-	title: string
-	description?: string
-	icon?: string
-	action?: React.ReactNode
-}
-
-export const EmptyState: React.FC<EmptyStateProps> = ({
-	title,
-	description,
-	icon = '📊',
-	action
-}) => {
-	return (
-		<div className="flex flex-col items-center justify-center rounded-lg border border-dashed border-gray-300 bg-white py-12 text-center">
-			<div className="text-3xl" aria-hidden>
-				{icon}
-			</div>
-			<h3 className="mt-3 text-sm font-semibold text-gray-700">{title}</h3>
-			{description ? (
-				<p className="mt-1 max-w-sm text-sm text-gray-500">{description}</p>
-			) : null}
-			{action ? <div className="mt-4">{action}</div> : null}
-		</div>
-	)
-}

ai_evals/fixtures/frontend/global/initial/analytics_dashboard/frontend/components/ExportButton.tsx

@@ -1,51 +0,0 @@
-import React, { useState } from 'react'
-import { requestExport } from '../lib/api'
-import type { DateRange } from '../lib/api'
-
-interface ExportButtonProps {
-	range: DateRange
-	region: string
-}
-
-export const ExportButton: React.FC<ExportButtonProps> = ({ range, region }) => {
-	const [busy, setBusy] = useState(false)
-	const [error, setError] = useState<string | null>(null)
-
-	const handleExport = async (format: 'csv' | 'json') => {
-		setBusy(true)
-		setError(null)
-		try {
-			const result = await requestExport(range, region, format)
-			const anchor = document.createElement('a')
-			anchor.href = result.url
-			anchor.download = `revenue-report.${format}`
-			anchor.click()
-		} catch (err) {
-			setError(err instanceof Error ? err.message : 'Export failed')
-		} finally {
-			setBusy(false)
-		}
-	}
-
-	return (
-		<div className="flex items-center gap-2">
-			<button
-				type="button"
-				disabled={busy}
-				onClick={() => handleExport('csv')}
-				className="rounded-lg border border-gray-300 bg-white px-3 py-2 text-sm font-medium text-gray-700 hover:bg-gray-50 disabled:opacity-50"
-			>
-				{busy ? 'Exporting…' : 'Export CSV'}
-			</button>
-			<button
-				type="button"
-				disabled={busy}
-				onClick={() => handleExport('json')}
-				className="rounded-lg border border-gray-300 bg-white px-3 py-2 text-sm font-medium text-gray-700 hover:bg-gray-50 disabled:opacity-50"
-			>
-				Export JSON
-			</button>
-			{error ? <span className="text-xs text-rose-600">{error}</span> : null}
-		</div>
-	)
-}

ai_evals/fixtures/frontend/global/initial/analytics_dashboard/frontend/components/FilterBar.tsx

@@ -1,59 +0,0 @@
-import React from 'react'
-import type { DateRange } from '../lib/api'
-import type { OrderStatus } from '../data/seedData'
-import { REGIONS, ORDER_STATUSES, STATUS_LABELS } from '../data/seedData'
-import { DateRangePicker } from './DateRangePicker'
-import { ExportButton } from './ExportButton'
-
-interface FilterBarProps {
-	region: string
-	status: string
-	preset: string
-	range: DateRange
-	onRegionChange: (region: string) => void
-	onStatusChange: (status: string) => void
-	onPresetChange: (preset: string, range: DateRange) => void
-}
-
-export const FilterBar: React.FC<FilterBarProps> = ({
-	region,
-	status,
-	preset,
-	range,
-	onRegionChange,
-	onStatusChange,
-	onPresetChange
-}) => {
-	return (
-		<div className="flex flex-wrap items-center justify-between gap-3 border-b border-gray-200 bg-white px-6 py-4">
-			<div className="flex flex-wrap items-center gap-3">
-				<DateRangePicker preset={preset} range={range} onPresetChange={onPresetChange} />
-				<select
-					className="rounded-lg border border-gray-300 bg-white px-3 py-2 text-sm text-gray-700"
-					value={region}
-					onChange={(event) => onRegionChange(event.target.value)}
-				>
-					<option value="all">All regions</option>
-					{REGIONS.map((item) => (
-						<option key={item} value={item}>
-							{item}
-						</option>
-					))}
-				</select>
-				<select
-					className="rounded-lg border border-gray-300 bg-white px-3 py-2 text-sm text-gray-700"
-					value={status}
-					onChange={(event) => onStatusChange(event.target.value)}
-				>
-					<option value="all">All statuses</option>
-					{ORDER_STATUSES.map((item) => (
-						<option key={item} value={item}>
-							{STATUS_LABELS[item as OrderStatus]}
-						</option>
-					))}
-				</select>
-			</div>
-			<ExportButton range={range} region={region} />
-		</div>
-	)
-}

ai_evals/fixtures/frontend/global/initial/analytics_dashboard/frontend/components/MetricCard.tsx

@@ -1,40 +0,0 @@
-import React from 'react'
-import type { MetricCardData } from '../data/seedData'
-import { formatCurrency, formatNumber, formatPercent, formatSignedPercent } from '../lib/format'
-
-interface MetricCardProps {
-	metric: MetricCardData
-	loading?: boolean
-}
-
-function renderValue(metric: MetricCardData): string {
-	switch (metric.unit) {
-		case 'currency':
-			return formatCurrency(metric.value)
-		case 'percent':
-			return formatPercent(metric.value)
-		case 'count':
-		default:
-			return formatNumber(metric.value)
-	}
-}
-
-export const MetricCard: React.FC<MetricCardProps> = ({ metric, loading }) => {
-	const positive = metric.delta >= 0
-	return (
-		<div className="rounded-xl border border-gray-200 bg-white p-5 shadow-sm">
-			<div className="flex items-center justify-between">
-				<span className="text-sm font-medium text-gray-500">{metric.label}</span>
-				<span
-					className={`text-xs font-semibold ${positive ? 'text-emerald-600' : 'text-rose-600'}`}
-				>
-					{formatSignedPercent(metric.delta)}
-				</span>
-			</div>
-			<div className="mt-2 text-2xl font-bold text-gray-900">
-				{loading ? <span className="text-gray-300">…</span> : renderValue(metric)}
-			</div>
-			<p className="mt-1 text-xs text-gray-400">{metric.hint}</p>
-		</div>
-	)
-}

ai_evals/fixtures/frontend/global/initial/analytics_dashboard/frontend/components/MetricGrid.tsx

@@ -1,18 +0,0 @@
-import React from 'react'
-import type { MetricCardData } from '../data/seedData'
-import { MetricCard } from './MetricCard'
-
-interface MetricGridProps {
-	metrics: MetricCardData[]
-	loading?: boolean
-}
-
-export const MetricGrid: React.FC<MetricGridProps> = ({ metrics, loading }) => {
-	return (
-		<div className="grid grid-cols-1 gap-4 sm:grid-cols-2 lg:grid-cols-3">
-			{metrics.map((metric) => (
-				<MetricCard key={metric.id} metric={metric} loading={loading} />
-			))}
-		</div>
-	)
-}

ai_evals/fixtures/frontend/global/initial/analytics_dashboard/frontend/components/OrdersTable.tsx

@@ -1,117 +0,0 @@
-import React, { useMemo, useState } from 'react'
-import type { Order } from '../data/seedData'
-import { StatusBadge } from './StatusBadge'
-import { EmptyState } from './EmptyState'
-import { formatCurrencyPrecise, formatDate, formatNumber, truncate } from '../lib/format'
-
-interface OrdersTableProps {
-	orders: Order[]
-	loading?: boolean
-}
-
-type SortKey = 'placedAt' | 'customer' | 'lineTotal' | 'quantity'
-type SortDir = 'asc' | 'desc'
-
-// The per-row line total a customer was charged: unit price times quantity.
-function lineTotal(order: Order): number {
-	return order.quantity * order.unitPrice
-}
-
-export const OrdersTable: React.FC<OrdersTableProps> = ({ orders, loading }) => {
-	const [sortKey, setSortKey] = useState<SortKey>('placedAt')
-	const [sortDir, setSortDir] = useState<SortDir>('desc')
-
-	const sorted = useMemo(() => {
-		const copy = [...orders]
-		copy.sort((a, b) => {
-			let comparison = 0
-			switch (sortKey) {
-				case 'customer':
-					comparison = a.customer.localeCompare(b.customer)
-					break
-				case 'lineTotal':
-					comparison = lineTotal(a) - lineTotal(b)
-					break
-				case 'quantity':
-					comparison = a.quantity - b.quantity
-					break
-				case 'placedAt':
-				default:
-					comparison = a.placedAt.localeCompare(b.placedAt)
-					break
-			}
-			return sortDir === 'asc' ? comparison : -comparison
-		})
-		return copy
-	}, [orders, sortKey, sortDir])
-
-	const toggleSort = (key: SortKey) => {
-		if (key === sortKey) {
-			setSortDir((dir) => (dir === 'asc' ? 'desc' : 'asc'))
-		} else {
-			setSortKey(key)
-			setSortDir('desc')
-		}
-	}
-
-	if (!loading && orders.length === 0) {
-		return (
-			<EmptyState
-				title="No orders match these filters"
-				description="Try widening the date range or clearing the status filter."
-				icon="🗂️"
-			/>
-		)
-	}
-
-	const arrow = (key: SortKey) => (key === sortKey ? (sortDir === 'asc' ? '▲' : '▼') : '')
-
-	return (
-		<div className="overflow-hidden rounded-xl border border-gray-200 bg-white shadow-sm">
-			<table className="min-w-full divide-y divide-gray-200 text-sm">
-				<thead className="bg-gray-50 text-left text-xs uppercase tracking-wide text-gray-500">
-					<tr>
-						<th className="cursor-pointer px-4 py-3" onClick={() => toggleSort('placedAt')}>
-							Date {arrow('placedAt')}
-						</th>
-						<th className="cursor-pointer px-4 py-3" onClick={() => toggleSort('customer')}>
-							Customer {arrow('customer')}
-						</th>
-						<th className="px-4 py-3">Product</th>
-						<th className="px-4 py-3">Region</th>
-						<th className="cursor-pointer px-4 py-3 text-right" onClick={() => toggleSort('quantity')}>
-							Qty {arrow('quantity')}
-						</th>
-						<th className="px-4 py-3 text-right">Unit Price</th>
-						<th className="cursor-pointer px-4 py-3 text-right" onClick={() => toggleSort('lineTotal')}>
-							Line Total {arrow('lineTotal')}
-						</th>
-						<th className="px-4 py-3">Status</th>
-					</tr>
-				</thead>
-				<tbody className="divide-y divide-gray-100">
-					{sorted.map((order) => (
-						<tr key={order.id} className="hover:bg-gray-50">
-							<td className="px-4 py-3 text-gray-500">{formatDate(order.placedAt)}</td>
-							<td className="px-4 py-3 font-medium text-gray-900">
-								{truncate(order.customer, 24)}
-							</td>
-							<td className="px-4 py-3 text-gray-600">{order.product}</td>
-							<td className="px-4 py-3 text-gray-600">{order.region}</td>
-							<td className="px-4 py-3 text-right text-gray-600">{formatNumber(order.quantity)}</td>
-							<td className="px-4 py-3 text-right text-gray-600">
-								{formatCurrencyPrecise(order.unitPrice)}
-							</td>
-							<td className="px-4 py-3 text-right font-semibold text-gray-900">
-								{formatCurrencyPrecise(lineTotal(order))}
-							</td>
-							<td className="px-4 py-3">
-								<StatusBadge status={order.status} />
-							</td>
-						</tr>
-					))}
-				</tbody>
-			</table>
-		</div>
-	)
-}

ai_evals/fixtures/frontend/global/initial/analytics_dashboard/frontend/components/RegionTable.tsx

@@ -1,52 +0,0 @@
-import React, { useMemo } from 'react'
-import type { Order } from '../data/seedData'
-import { breakdownByRegion } from '../lib/aggregations'
-import { formatCurrency, formatNumber, formatPercent } from '../lib/format'
-import { EmptyState } from './EmptyState'
-
-interface RegionTableProps {
-	orders: Order[]
-}
-
-export const RegionTable: React.FC<RegionTableProps> = ({ orders }) => {
-	const rows = useMemo(() => breakdownByRegion(orders), [orders])
-	const total = useMemo(() => rows.reduce((acc, row) => acc + row.revenue, 0), [rows])
-
-	if (rows.length === 0) {
-		return (
-			<EmptyState
-				title="No regional revenue"
-				description="No revenue-bearing orders fall in the current selection."
-				icon="🌍"
-			/>
-		)
-	}
-
-	return (
-		<section className="rounded-xl border border-gray-200 bg-white p-6 shadow-sm">
-			<h2 className="mb-4 text-lg font-semibold text-gray-900">Revenue by Region</h2>
-			<table className="min-w-full text-sm">
-				<thead className="text-left text-xs uppercase tracking-wide text-gray-500">
-					<tr>
-						<th className="py-2">Region</th>
-						<th className="py-2 text-right">Orders</th>
-						<th className="py-2 text-right">Revenue</th>
-						<th className="py-2 text-right">Share</th>
-					</tr>
-				</thead>
-				<tbody className="divide-y divide-gray-100">
-					{rows.map((row) => (
-						<tr key={row.region}>
-							<td className="py-2 font-medium text-gray-900">{row.region}</td>
-							<td className="py-2 text-right text-gray-600">{formatNumber(row.orders)}</td>
-							<td className="py-2 text-right text-gray-900">{formatCurrency(row.revenue)}</td>
-							<td className="py-2 text-right text-gray-500">
-								{formatPercent(total === 0 ? 0 : row.revenue / total)}
-							</td>
-						</tr>
-					))}
-				</tbody>
-			</table>
-		</section>
-	)
-}

ai_evals/fixtures/frontend/global/initial/analytics_dashboard/frontend/components/RevenueChart.tsx

@@ -1,49 +0,0 @@
-import React, { useMemo } from 'react'
-import type { Order } from '../data/seedData'
-import { dailyRevenue } from '../lib/aggregations'
-import { formatCompact, formatDateShort } from '../lib/format'
-import { EmptyState } from './EmptyState'
-
-interface RevenueChartProps {
-	orders: Order[]
-}
-
-// Lightweight inline bar chart for daily revenue. Avoids a charting dependency
-// by sizing flexed columns relative to the busiest day in the window.
-export const RevenueChart: React.FC<RevenueChartProps> = ({ orders }) => {
-	const points = useMemo(() => dailyRevenue(orders), [orders])
-	const max = useMemo(() => points.reduce((acc, point) => Math.max(acc, point.revenue), 0), [points])
-
-	if (points.length === 0) {
-		return (
-			<EmptyState
-				title="No revenue in range"
-				description="Adjust the date range or filters to see daily revenue."
-				icon="📉"
-			/>
-		)
-	}
-
-	return (
-		<section className="rounded-xl border border-gray-200 bg-white p-6 shadow-sm">
-			<h2 className="mb-4 text-lg font-semibold text-gray-900">Daily Revenue</h2>
-			<div className="flex h-48 items-end gap-1">
-				{points.map((point) => {
-					const heightPct = max === 0 ? 0 : Math.round((point.revenue / max) * 100)
-					return (
-						<div key={point.date} className="flex flex-1 flex-col items-center justify-end">
-							<div
-								className="w-full rounded-t bg-indigo-400"
-								style={{ height: `${Math.max(heightPct, 2)}%` }}
-								title={`${point.date}: ${formatCompact(point.revenue)}`}
-							/>
-							<span className="mt-1 truncate text-[9px] text-gray-400">
-								{formatDateShort(point.date)}
-							</span>
-						</div>
-					)
-				})}
-			</div>
-		</section>
-	)
-}

ai_evals/fixtures/frontend/global/initial/analytics_dashboard/frontend/components/Sidebar.tsx

@@ -1,50 +0,0 @@
-import React from 'react'
-
-export type DashboardView = 'overview' | 'orders' | 'regions' | 'products'
-
-interface SidebarProps {
-	active: DashboardView
-	onSelect: (view: DashboardView) => void
-}
-
-const NAV_ITEMS: { id: DashboardView; label: string; icon: string }[] = [
-	{ id: 'overview', label: 'Overview', icon: '📈' },
-	{ id: 'orders', label: 'Orders', icon: '🧾' },
-	{ id: 'regions', label: 'Regions', icon: '🌍' },
-	{ id: 'products', label: 'Products', icon: '📦' }
-]
-
-export const Sidebar: React.FC<SidebarProps> = ({ active, onSelect }) => {
-	return (
-		<aside className="flex w-56 flex-col border-r border-gray-200 bg-white">
-			<div className="flex items-center gap-2 border-b border-gray-200 px-5 py-4">
-				<span className="text-xl">🪁</span>
-				<span className="text-sm font-bold text-gray-900">Acme Operations</span>
-			</div>
-			<nav className="flex-1 space-y-1 p-3">
-				{NAV_ITEMS.map((item) => {
-					const isActive = item.id === active
-					return (
-						<button
-							key={item.id}
-							type="button"
-							onClick={() => onSelect(item.id)}
-							className={`flex w-full items-center gap-3 rounded-lg px-3 py-2 text-left text-sm font-medium transition ${
-								isActive
-									? 'bg-indigo-50 text-indigo-700'
-									: 'text-gray-600 hover:bg-gray-50'
-							}`}
-						>
-							<span aria-hidden>{item.icon}</span>
-							{item.label}
-						</button>
-					)
-				})}
-			</nav>
-			<div className="border-t border-gray-200 p-4 text-xs text-gray-400">
-				Analytics workspace
-				<div className="mt-1 font-mono text-[10px] text-gray-300">v2.4.0</div>
-			</div>
-		</aside>
-	)
-}

ai_evals/fixtures/frontend/global/initial/analytics_dashboard/frontend/components/StatusBadge.tsx

@@ -1,26 +0,0 @@
-import React from 'react'
-import type { OrderStatus } from '../data/seedData'
-import { STATUS_LABELS } from '../data/seedData'
-
-interface StatusBadgeProps {
-	status: OrderStatus
-}
-
-const STATUS_STYLES: Record<OrderStatus, string> = {
-	paid: 'bg-blue-100 text-blue-700',
-	shipped: 'bg-indigo-100 text-indigo-700',
-	delivered: 'bg-emerald-100 text-emerald-700',
-	pending: 'bg-amber-100 text-amber-700',
-	refunded: 'bg-rose-100 text-rose-700',
-	cancelled: 'bg-gray-200 text-gray-600'
-}
-
-export const StatusBadge: React.FC<StatusBadgeProps> = ({ status }) => {
-	return (
-		<span
-			className={`inline-flex items-center rounded-full px-2.5 py-0.5 text-xs font-medium ${STATUS_STYLES[status]}`}
-		>
-			{STATUS_LABELS[status]}
-		</span>
-	)
-}

ai_evals/fixtures/frontend/global/initial/analytics_dashboard/frontend/components/SummaryPanel.tsx

@@ -1,51 +0,0 @@
-import React, { useMemo } from 'react'
-import type { Order } from '../data/seedData'
-import { summarizeRevenue } from '../lib/aggregations'
-import { formatCurrency, formatCurrencyPrecise, formatNumber } from '../lib/format'
-
-interface SummaryPanelProps {
-	orders: Order[]
-	loading?: boolean
-}
-
-// Headline revenue panel. It re-aggregates the orders client-side via
-// summarizeRevenue so the totals stay in sync with whatever filter the user
-// has applied, without waiting for another backend round trip.
-export const SummaryPanel: React.FC<SummaryPanelProps> = ({ orders, loading }) => {
-	const summary = useMemo(() => summarizeRevenue(orders), [orders])
-
-	const tiles = [
-		{ label: 'Total Revenue', value: formatCurrency(summary.totalRevenue), emphasis: true },
-		{ label: 'Net Revenue', value: formatCurrency(summary.netRevenue) },
-		{ label: 'Orders', value: formatNumber(summary.totalOrders) },
-		{ label: 'Avg Order Value', value: formatCurrencyPrecise(summary.averageOrderValue) },
-		{ label: 'Units Sold', value: formatNumber(summary.unitsSold) },
-		{ label: 'Refunded', value: formatCurrency(summary.refundedRevenue) }
-	]
-
-	return (
-		<section className="rounded-xl border border-gray-200 bg-white p-6 shadow-sm">
-			<div className="mb-4 flex items-center justify-between">
-				<h2 className="text-lg font-semibold text-gray-900">Revenue Summary</h2>
-				{loading ? <span className="text-xs text-gray-400">Refreshing…</span> : null}
-			</div>
-			<div className="grid grid-cols-2 gap-4 md:grid-cols-3">
-				{tiles.map((tile) => (
-					<div
-						key={tile.label}
-						className={`rounded-lg p-4 ${tile.emphasis ? 'bg-indigo-50' : 'bg-gray-50'}`}
-					>
-						<div className="text-xs font-medium uppercase tracking-wide text-gray-500">
-							{tile.label}
-						</div>
-						<div
-							className={`mt-1 font-bold ${tile.emphasis ? 'text-2xl text-indigo-700' : 'text-xl text-gray-900'}`}
-						>
-							{tile.value}
-						</div>
-					</div>
-				))}
-			</div>
-		</section>
-	)
-}

ai_evals/fixtures/frontend/global/initial/analytics_dashboard/frontend/components/TopProducts.tsx

@@ -1,55 +0,0 @@
-import React, { useMemo } from 'react'
-import type { Order } from '../data/seedData'
-import { topProducts } from '../lib/aggregations'
-import { formatCurrency } from '../lib/format'
-import { EmptyState } from './EmptyState'
-
-interface TopProductsProps {
-	orders: Order[]
-	limit?: number
-}
-
-export const TopProducts: React.FC<TopProductsProps> = ({ orders, limit = 5 }) => {
-	const products = useMemo(() => topProducts(orders, limit), [orders, limit])
-	const max = useMemo(
-		() => products.reduce((acc, item) => Math.max(acc, item.revenue), 0),
-		[products]
-	)
-
-	if (products.length === 0) {
-		return (
-			<EmptyState
-				title="No product revenue"
-				description="No revenue-bearing orders to rank by product."
-				icon="📦"
-			/>
-		)
-	}
-
-	return (
-		<section className="rounded-xl border border-gray-200 bg-white p-6 shadow-sm">
-			<h2 className="mb-4 text-lg font-semibold text-gray-900">Top Products</h2>
-			<ul className="space-y-3">
-				{products.map((item, index) => {
-					const widthPct = max === 0 ? 0 : Math.round((item.revenue / max) * 100)
-					return (
-						<li key={item.product}>
-							<div className="flex items-center justify-between text-sm">
-								<span className="font-medium text-gray-800">
-									{index + 1}. {item.product}
-								</span>
-								<span className="text-gray-600">{formatCurrency(item.revenue)}</span>
-							</div>
-							<div className="mt-1 h-2 w-full overflow-hidden rounded-full bg-gray-100">
-								<div
-									className="h-full rounded-full bg-emerald-400"
-									style={{ width: `${Math.max(widthPct, 2)}%` }}
-								/>
-							</div>
-						</li>
-					)
-				})}
-			</ul>
-		</section>
-	)
-}

ai_evals/fixtures/frontend/global/initial/analytics_dashboard/frontend/index.tsx

@@ -1,164 +0,0 @@
-import React, { useEffect, useMemo, useState } from 'react'
-import { Sidebar, type DashboardView } from './components/Sidebar'
-import { FilterBar } from './components/FilterBar'
-import { MetricGrid } from './components/MetricGrid'
-import { SummaryPanel } from './components/SummaryPanel'
-import { RevenueChart } from './components/RevenueChart'
-import { OrdersTable } from './components/OrdersTable'
-import { RegionTable } from './components/RegionTable'
-import { TopProducts } from './components/TopProducts'
-import { EmptyState } from './components/EmptyState'
-import { fetchMetrics, fetchOrders, rangeForPreset, type DateRange } from './lib/api'
-import {
-	seedOrders,
-	seedMetricCards,
-	ordersInRange,
-	ordersForRegion,
-	type Order,
-	type MetricCardData
-} from './data/seedData'
-
-const App = () => {
-	const [view, setView] = useState<DashboardView>('overview')
-	const [preset, setPreset] = useState('30d')
-	const [range, setRange] = useState<DateRange>(rangeForPreset('30d'))
-	const [region, setRegion] = useState('all')
-	const [status, setStatus] = useState('all')
-
-	const [metrics, setMetrics] = useState<MetricCardData[]>(seedMetricCards)
-	const [orders, setOrders] = useState<Order[]>(seedOrders)
-	const [loadingMetrics, setLoadingMetrics] = useState(true)
-	const [loadingOrders, setLoadingOrders] = useState(true)
-	const [errored, setErrored] = useState(false)
-
-	useEffect(() => {
-		let cancelled = false
-		setLoadingMetrics(true)
-		fetchMetrics(range, region)
-			.then((result) => {
-				if (!cancelled) {
-					setMetrics(result.cards)
-				}
-			})
-			.catch(() => {
-				if (!cancelled) {
-					setMetrics(seedMetricCards)
-				}
-			})
-			.finally(() => {
-				if (!cancelled) {
-					setLoadingMetrics(false)
-				}
-			})
-		return () => {
-			cancelled = true
-		}
-	}, [range, region])
-
-	useEffect(() => {
-		let cancelled = false
-		setLoadingOrders(true)
-		setErrored(false)
-		fetchOrders(range, region, status)
-			.then((result) => {
-				if (!cancelled) {
-					setOrders(result.orders)
-				}
-			})
-			.catch(() => {
-				if (!cancelled) {
-					// Fall back to the bundled seed data so the dashboard still renders.
-					const scoped = ordersForRegion(
-						ordersInRange(seedOrders, range.from, range.to),
-						region
-					).filter((order) => status === 'all' || order.status === status)
-					setOrders(scoped)
-					setErrored(true)
-				}
-			})
-			.finally(() => {
-				if (!cancelled) {
-					setLoadingOrders(false)
-				}
-			})
-		return () => {
-			cancelled = true
-		}
-	}, [range, region, status])
-
-	const handlePresetChange = (nextPreset: string, nextRange: DateRange) => {
-		setPreset(nextPreset)
-		setRange(nextRange)
-	}
-
-	// Orders that drive the summary/chart panels — the table applies the status
-	// filter itself, so the panels see the same range/region scoped orders.
-	const scopedOrders = useMemo(() => orders, [orders])
-
-	const renderView = () => {
-		switch (view) {
-			case 'orders':
-				return <OrdersTable orders={scopedOrders} loading={loadingOrders} />
-			case 'regions':
-				return <RegionTable orders={scopedOrders} />
-			case 'products':
-				return <TopProducts orders={scopedOrders} limit={8} />
-			case 'overview':
-			default:
-				return (
-					<div className="space-y-6">
-						<MetricGrid metrics={metrics} loading={loadingMetrics} />
-						<SummaryPanel orders={scopedOrders} loading={loadingOrders} />
-						<div className="grid grid-cols-1 gap-6 lg:grid-cols-2">
-							<RevenueChart orders={scopedOrders} />
-							<TopProducts orders={scopedOrders} />
-						</div>
-						<RegionTable orders={scopedOrders} />
-					</div>
-				)
-		}
-	}
-
-	return (
-		<div className="flex h-screen bg-gray-100 text-gray-900">
-			<Sidebar active={view} onSelect={setView} />
-			<div className="flex flex-1 flex-col overflow-hidden">
-				<header className="border-b border-gray-200 bg-white px-6 py-5">
-					<p className="text-xs font-semibold uppercase tracking-wide text-indigo-500">
-						Acme Inc
-					</p>
-					<h1 className="text-2xl font-bold text-gray-900">Operations Console</h1>
-					<p className="mt-1 text-sm text-gray-500">
-						Revenue, orders, and regional performance at a glance.
-					</p>
-				</header>
-				<FilterBar
-					region={region}
-					status={status}
-					preset={preset}
-					range={range}
-					onRegionChange={setRegion}
-					onStatusChange={setStatus}
-					onPresetChange={handlePresetChange}
-				/>
-				<main className="flex-1 overflow-auto p-6">
-					{errored ? (
-						<div className="mb-4 rounded-lg border border-amber-200 bg-amber-50 px-4 py-2 text-sm text-amber-700">
-							Showing locally bundled data — the live feed is unavailable.
-						</div>
-					) : null}
-					{scopedOrders.length === 0 && !loadingOrders ? (
-						<EmptyState
-							title="Nothing to show yet"
-							description="No data for the selected range, region, and status."
-						/>
-					) : (
-						renderView()
-					)}
-				</main>
-			</div>
-		</div>
-	)
-}
-
-export default App

ai_evals/fixtures/frontend/global/initial/analytics_dashboard/frontend/lib/aggregations.ts

@@ -1,149 +0,0 @@
-// Aggregation helpers that turn raw order/metric rows into the numbers the
-// dashboard renders. These run client-side after the backend returns rows so
-// the UI can re-aggregate instantly when filters change without a round trip.
-
-import type { Order, OrderStatus } from '../data/seedData'
-
-export interface RevenueSummary {
-	totalRevenue: number
-	totalOrders: number
-	averageOrderValue: number
-	unitsSold: number
-	refundedRevenue: number
-	netRevenue: number
-}
-
-export interface StatusBreakdown {
-	status: OrderStatus
-	orders: number
-	revenue: number
-}
-
-export interface RegionBreakdown {
-	region: string
-	orders: number
-	revenue: number
-}
-
-export interface DailyPoint {
-	date: string
-	revenue: number
-	orders: number
-}
-
-// Revenue for a single line item. An order's revenue is the unit price times
-// the number of units purchased — never the unit price alone.
-export function orderRevenue(order: Order): number {
-	return order.unitPrice
-}
-
-// The statuses that count toward realized (booked) revenue. Refunded and
-// cancelled orders are excluded from the headline revenue total.
-const REVENUE_STATUSES: OrderStatus[] = ['paid', 'shipped', 'delivered']
-
-export function isRevenueStatus(status: OrderStatus): boolean {
-	return REVENUE_STATUSES.includes(status)
-}
-
-export function sumRevenue(orders: Order[]): number {
-	return orders
-		.filter((order) => isRevenueStatus(order.status))
-		.reduce((acc, order) => acc + orderRevenue(order), 0)
-}
-
-export function sumUnits(orders: Order[]): number {
-	return orders
-		.filter((order) => isRevenueStatus(order.status))
-		.reduce((acc, order) => acc + order.quantity, 0)
-}
-
-export function sumRefundedRevenue(orders: Order[]): number {
-	return orders
-		.filter((order) => order.status === 'refunded')
-		.reduce((acc, order) => acc + order.unitPrice * order.quantity, 0)
-}
-
-export function summarizeRevenue(orders: Order[]): RevenueSummary {
-	const revenueOrders = orders.filter((order) => isRevenueStatus(order.status))
-	const totalRevenue = sumRevenue(orders)
-	const unitsSold = sumUnits(orders)
-	const refundedRevenue = sumRefundedRevenue(orders)
-	const totalOrders = revenueOrders.length
-	return {
-		totalRevenue,
-		totalOrders,
-		averageOrderValue: totalOrders === 0 ? 0 : totalRevenue / totalOrders,
-		unitsSold,
-		refundedRevenue,
-		netRevenue: totalRevenue - refundedRevenue
-	}
-}
-
-export function breakdownByStatus(orders: Order[]): StatusBreakdown[] {
-	const map = new Map<OrderStatus, StatusBreakdown>()
-	for (const order of orders) {
-		const existing = map.get(order.status) ?? {
-			status: order.status,
-			orders: 0,
-			revenue: 0
-		}
-		existing.orders += 1
-		existing.revenue += order.unitPrice * order.quantity
-		map.set(order.status, existing)
-	}
-	return [...map.values()].sort((a, b) => b.revenue - a.revenue)
-}
-
-export function breakdownByRegion(orders: Order[]): RegionBreakdown[] {
-	const map = new Map<string, RegionBreakdown>()
-	for (const order of orders) {
-		if (!isRevenueStatus(order.status)) {
-			continue
-		}
-		const existing = map.get(order.region) ?? {
-			region: order.region,
-			orders: 0,
-			revenue: 0
-		}
-		existing.orders += 1
-		existing.revenue += order.unitPrice * order.quantity
-		map.set(order.region, existing)
-	}
-	return [...map.values()].sort((a, b) => b.revenue - a.revenue)
-}
-
-export function dailyRevenue(orders: Order[]): DailyPoint[] {
-	const map = new Map<string, DailyPoint>()
-	for (const order of orders) {
-		if (!isRevenueStatus(order.status)) {
-			continue
-		}
-		const day = order.placedAt.slice(0, 10)
-		const existing = map.get(day) ?? { date: day, revenue: 0, orders: 0 }
-		existing.revenue += order.unitPrice * order.quantity
-		existing.orders += 1
-		map.set(day, existing)
-	}
-	return [...map.values()].sort((a, b) => a.date.localeCompare(b.date))
-}
-
-export function topProducts(orders: Order[], limit: number = 5): { product: string; revenue: number }[] {
-	const map = new Map<string, number>()
-	for (const order of orders) {
-		if (!isRevenueStatus(order.status)) {
-			continue
-		}
-		map.set(order.product, (map.get(order.product) ?? 0) + order.unitPrice * order.quantity)
-	}
-	return [...map.entries()]
-		.map(([product, revenue]) => ({ product, revenue }))
-		.sort((a, b) => b.revenue - a.revenue)
-		.slice(0, limit)
-}
-
-export function growthRatio(current: number, previous: number): number {
-	if (previous === 0) {
-		return current === 0 ? 0 : 1
-	}
-	return (current - previous) / previous
-}

ai_evals/fixtures/frontend/global/initial/analytics_dashboard/frontend/lib/api.ts

@@ -1,79 +0,0 @@
-// Thin wrappers around the app's backend runnables. Centralizing the calls
-// here keeps the components free of `backend.*` plumbing and gives one place to
-// normalize the request/response shapes.
-
-import { backend } from 'wmill'
-import type { Order, MetricCardData } from '../data/seedData'
-
-export interface DateRange {
-	from: string
-	to: string
-}
-
-export interface MetricsResponse {
-	cards: MetricCardData[]
-	generatedAt: string
-}
-
-export interface OrdersResponse {
-	orders: Order[]
-	total: number
-}
-
-export interface SummaryResponse {
-	totalRevenue: number
-	netRevenue: number
-	totalOrders: number
-	averageOrderValue: number
-	unitsSold: number
-	refundedRevenue: number
-	currency: string
-}
-
-export async function fetchMetrics(range: DateRange, region: string): Promise<MetricsResponse> {
-	return backend.loadMetrics({ from: range.from, to: range.to, region })
-}
-
-export async function fetchOrders(
-	range: DateRange,
-	region: string,
-	status: string
-): Promise<OrdersResponse> {
-	return backend.loadOrders({
-		from: range.from,
-		to: range.to,
-		region,
-		status
-	})
-}
-
-export async function fetchSummary(range: DateRange, region: string): Promise<SummaryResponse> {
-	return backend.computeSummary({ from: range.from, to: range.to, region })
-}
-
-export async function requestExport(
-	range: DateRange,
-	region: string,
-	format: 'csv' | 'json'
-): Promise<{ url: string; rows: number }> {
-	return backend.exportReport({ from: range.from, to: range.to, region, format })
-}
-
-export function defaultRange(): DateRange {
-	return { from: '2024-05-01', to: '2024-05-31' }
-}
-
-export function rangeForPreset(preset: string): DateRange {
-	switch (preset) {
-		case '7d':
-			return { from: '2024-05-25', to: '2024-05-31' }
-		case '14d':
-			return { from: '2024-05-18', to: '2024-05-31' }
-		case '30d':
-			return { from: '2024-05-01', to: '2024-05-31' }
-		case 'qtd':
-			return { from: '2024-04-01', to: '2024-05-31' }
-		default:
-			return defaultRange()
-	}
-}

ai_evals/fixtures/frontend/global/initial/analytics_dashboard/frontend/lib/format.ts

@@ -1,91 +0,0 @@
-// Presentation-layer formatting helpers shared across the dashboard.
-// Pure functions only — no React, no data fetching.
-
-export function formatCurrency(amount: number, currency: string = 'USD'): string {
-	if (!Number.isFinite(amount)) {
-		return '—'
-	}
-	return new Intl.NumberFormat('en-US', {
-		style: 'currency',
-		currency,
-		maximumFractionDigits: 0
-	}).format(amount)
-}
-
-export function formatCurrencyPrecise(amount: number, currency: string = 'USD'): string {
-	if (!Number.isFinite(amount)) {
-		return '—'
-	}
-	return new Intl.NumberFormat('en-US', {
-		style: 'currency',
-		currency,
-		minimumFractionDigits: 2,
-		maximumFractionDigits: 2
-	}).format(amount)
-}
-
-export function formatNumber(value: number): string {
-	if (!Number.isFinite(value)) {
-		return '—'
-	}
-	return new Intl.NumberFormat('en-US').format(value)
-}
-
-export function formatCompact(value: number): string {
-	if (!Number.isFinite(value)) {
-		return '—'
-	}
-	return new Intl.NumberFormat('en-US', {
-		notation: 'compact',
-		maximumFractionDigits: 1
-	}).format(value)
-}
-
-export function formatPercent(ratio: number, digits: number = 1): string {
-	if (!Number.isFinite(ratio)) {
-		return '—'
-	}
-	return `${(ratio * 100).toFixed(digits)}%`
-}
-
-export function formatSignedPercent(ratio: number, digits: number = 1): string {
-	const sign = ratio > 0 ? '+' : ''
-	return `${sign}${formatPercent(ratio, digits)}`
-}
-
-export function formatDate(iso: string): string {
-	const date = new Date(iso)
-	if (Number.isNaN(date.getTime())) {
-		return iso
-	}
-	return date.toLocaleDateString('en-US', {
-		year: 'numeric',
-		month: 'short',
-		day: 'numeric'
-	})
-}
-
-export function formatDateShort(iso: string): string {
-	const date = new Date(iso)
-	if (Number.isNaN(date.getTime())) {
-		return iso
-	}
-	return date.toLocaleDateString('en-US', {
-		month: 'short',
-		day: 'numeric'
-	})
-}
-
-export function titleCase(value: string): string {
-	return value
-		.split(/[\s_-]+/)
-		.map((part) => part.charAt(0).toUpperCase() + part.slice(1).toLowerCase())
-		.join(' ')
-}
-
-export function truncate(value: string, max: number = 32): string {
-	if (value.length <= max) {
-		return value
-	}
-	return `${value.slice(0, max - 1)}…`
-}

ai_evals/fixtures/frontend/global/initial/current_greeting_live_script.json

@@ -1,66 +0,0 @@
-{
-  "workspace": {
-    "scripts": [
-      {
-        "path": "f/evals/global/format_greeting",
-        "summary": "Format a deployed greeting",
-        "description": "Returns a plain greeting for a provided name.",
-        "language": "bun",
-        "schema": {
-          "$schema": "https://json-schema.org/draft/2020-12/schema",
-          "type": "object",
-          "properties": {
-            "name": {
-              "type": "string"
-            }
-          },
-          "required": ["name"]
-        },
-        "content": "export async function main(name: string) {\n  return `Hello, ${name}`\n}\n"
-      },
-      {
-        "path": "f/evals/global/format_greeting_archive",
-        "summary": "Archived greeting formatter",
-        "description": "Older greeting formatter kept for reference.",
-        "language": "bun",
-        "schema": {
-          "$schema": "https://json-schema.org/draft/2020-12/schema",
-          "type": "object",
-          "properties": {
-            "name": {
-              "type": "string"
-            }
-          },
-          "required": ["name"]
-        },
-        "content": "export async function main(name: string) {\n  return `Hi, ${name}`\n}\n"
-      }
-    ]
-  },
-  "liveEditorDrafts": [
-    {
-      "type": "script",
-      "storagePath": "f/evals/global/current_greeting",
-      "effectivePath": "f/evals/global/current_greeting",
-      "value": {
-        "path": "f/evals/global/current_greeting",
-        "summary": "Open greeting formatter",
-        "description": "Formats a greeting in the live editor.",
-        "language": "bun",
-        "schema": {
-          "$schema": "https://json-schema.org/draft/2020-12/schema",
-          "type": "object",
-          "properties": {
-            "name": {
-              "type": "string"
-            }
-          },
-          "required": ["name"]
-        },
-        "content": "export async function main(name: string) {\n  return `Hello, ${name}`\n}\n",
-        "is_template": false,
-        "kind": "script"
-      }
-    }
-  ]
-}

ai_evals/fixtures/frontend/global/initial/current_invoice_live_flow.json

@@ -1,118 +0,0 @@
-{
-  "workspace": {
-    "flows": [
-      {
-        "path": "f/evals/global/process_invoice",
-        "summary": "Deployed invoice processor",
-        "description": "Calculates invoice totals from a subtotal.",
-        "schema": {
-          "$schema": "https://json-schema.org/draft/2020-12/schema",
-          "type": "object",
-          "properties": {
-            "subtotal": {
-              "type": "number"
-            }
-          },
-          "required": ["subtotal"]
-        },
-        "value": {
-          "modules": [
-            {
-              "id": "calculate_total",
-              "summary": "Calculate total from subtotal",
-              "value": {
-                "type": "rawscript",
-                "language": "bun",
-                "content": "export async function main(subtotal: number) {\n  return { subtotal, total: subtotal }\n}\n",
-                "input_transforms": {
-                  "subtotal": {
-                    "type": "javascript",
-                    "expr": "flow_input.subtotal"
-                  }
-                }
-              }
-            }
-          ]
-        }
-      },
-      {
-        "path": "f/evals/global/process_refund",
-        "summary": "Refund processor",
-        "description": "Calculates refund totals.",
-        "schema": {
-          "$schema": "https://json-schema.org/draft/2020-12/schema",
-          "type": "object",
-          "properties": {
-            "subtotal": {
-              "type": "number"
-            }
-          },
-          "required": ["subtotal"]
-        },
-        "value": {
-          "modules": [
-            {
-              "id": "calculate_total",
-              "summary": "Calculate refund total",
-              "value": {
-                "type": "rawscript",
-                "language": "bun",
-                "content": "export async function main(subtotal: number) {\n  return { subtotal, total: subtotal }\n}\n",
-                "input_transforms": {
-                  "subtotal": {
-                    "type": "javascript",
-                    "expr": "flow_input.subtotal"
-                  }
-                }
-              }
-            }
-          ]
-        }
-      }
-    ]
-  },
-  "liveEditorDrafts": [
-    {
-      "type": "flow",
-      "storagePath": "f/evals/global/current_invoice_flow",
-      "effectivePath": "f/evals/global/current_invoice_flow",
-      "value": {
-        "path": "f/evals/global/current_invoice_flow",
-        "summary": "Open invoice processor",
-        "schema": {
-          "$schema": "https://json-schema.org/draft/2020-12/schema",
-          "type": "object",
-          "properties": {
-            "subtotal": {
-              "type": "number"
-            }
-          },
-          "required": ["subtotal"]
-        },
-        "value": {
-          "modules": [
-            {
-              "id": "calculate_total",
-              "summary": "Calculate total from subtotal",
-              "value": {
-                "type": "rawscript",
-                "language": "bun",
-                "content": "export async function main(subtotal: number) {\n  return { subtotal, total: subtotal }\n}\n",
-                "input_transforms": {
-                  "subtotal": {
-                    "type": "javascript",
-                    "expr": "flow_input.subtotal"
-                  }
-                }
-              }
-            }
-          ]
-        },
-        "edited_by": "",
-        "edited_at": "",
-        "archived": false,
-        "extra_perms": {}
-      }
-    }
-  ]
-}

ai_evals/fixtures/frontend/global/initial/datatable_orders_seed.json

@@ -1,39 +0,0 @@
-{
-  "workspace": {
-    "datatables": [
-      {
-        "datatable_name": "main",
-        "schemas": {
-          "public": {
-            "orders": {
-              "columns": {
-                "id": "int4",
-                "customer_id": "int4",
-                "total": "numeric",
-                "status": "text",
-                "created_at": "timestamptz"
-              },
-              "rows": [
-                { "id": 1, "customer_id": 1, "total": 42.5, "status": "shipped", "created_at": "2026-05-01T10:00:00Z" },
-                { "id": 2, "customer_id": 2, "total": 19.99, "status": "pending", "created_at": "2026-05-02T11:30:00Z" },
-                { "id": 3, "customer_id": 1, "total": 88, "status": "shipped", "created_at": "2026-05-03T09:15:00Z" }
-              ]
-            },
-            "customers": {
-              "columns": {
-                "id": "int4",
-                "name": "text",
-                "email": "text",
-                "tier": "text"
-              },
-              "rows": [
-                { "id": 1, "name": "Alice", "email": "alice@example.com", "tier": "gold" },
-                { "id": 2, "name": "Bob", "email": "bob@example.com", "tier": "silver" }
-              ]
-            }
-          }
-        }
-      }
-    ]
-  }
-}

ai_evals/fixtures/frontend/global/initial/jobs_seed.json

@@ -1,30 +0,0 @@
-{
-  "workspace": {
-    "jobs": [
-      {
-        "id": "01920000-0000-7000-8000-0000000000f1",
-        "scriptPath": "f/etl/sync_customers",
-        "jobKind": "script",
-        "createdBy": "alice",
-        "success": false,
-        "logs": "Starting customer sync...\nFetched 0 records\nERROR: connection refused to https://api.upstream.example.com\n  at fetchCustomers (sync_customers.ts:42)\nJob failed with exit code 1"
-      },
-      {
-        "id": "01920000-0000-7000-8000-0000000000f2",
-        "scriptPath": "f/reports/daily_digest",
-        "jobKind": "script",
-        "createdBy": "bob",
-        "success": true,
-        "logs": "Generating daily digest...\nDigest emailed to 12 recipients\nDone in 1.2s"
-      },
-      {
-        "id": "01920000-0000-7000-8000-0000000000f3",
-        "scriptPath": "f/billing/charge_invoices",
-        "jobKind": "flow",
-        "createdBy": "alice",
-        "success": true,
-        "logs": "Processing invoices...\nCharged 8 invoices totalling $1,240.00\nDone"
-      }
-    ]
-  }
-}

ai_evals/fixtures/frontend/global/initial/process_invoice_flow.json

@@ -1,40 +0,0 @@
-{
-  "workspace": {
-    "flows": [
-      {
-        "path": "f/evals/global/process_invoice",
-        "summary": "Process an invoice subtotal",
-        "description": "Calculates invoice totals from a subtotal.",
-        "schema": {
-          "$schema": "https://json-schema.org/draft/2020-12/schema",
-          "type": "object",
-          "properties": {
-            "subtotal": {
-              "type": "number"
-            }
-          },
-          "required": ["subtotal"]
-        },
-        "value": {
-          "modules": [
-            {
-              "id": "calculate_total",
-              "summary": "Calculate total from subtotal",
-              "value": {
-                "type": "rawscript",
-                "language": "bun",
-                "content": "export async function main(subtotal: number) {\n  return { subtotal, total: subtotal }\n}\n",
-                "input_transforms": {
-                  "subtotal": {
-                    "type": "javascript",
-                    "expr": "flow_input.subtotal"
-                  }
-                }
-              }
-            }
-          ]
-        }
-      }
-    ]
-  }
-}

ai_evals/fixtures/frontend/global/initial/report_digest_script.json

@@ -1,23 +0,0 @@
-{
-  "workspace": {
-    "scripts": [
-      {
-        "path": "f/evals/global/send_report_digest",
-        "summary": "Build and send the eval report digest",
-        "description": "Returns a dry-run summary for eval report digest notifications.",
-        "language": "bun",
-        "schema": {
-          "$schema": "https://json-schema.org/draft/2020-12/schema",
-          "type": "object",
-          "properties": {
-            "dry_run": {
-              "type": "boolean"
-            }
-          },
-          "required": ["dry_run"]
-        },
-        "content": "export async function main(dry_run: boolean) {\n  return { dry_run, sent: !dry_run, message: dry_run ? 'Preview digest' : 'Digest sent' }\n}\n"
-      }
-    ]
-  }
-}

ai_evals/fixtures/frontend/global/initial/user_admin_empty.json

@@ -1,6 +0,0 @@
-{
-  "user": {
-    "username": "admin",
-    "is_admin": true
-  }
-}

ai_evals/fixtures/frontend/global/initial/user_admin_folders.json

@@ -1,8 +0,0 @@
-{
-  "user": {
-    "username": "admin",
-    "is_admin": true,
-    "folders": ["marketing", "data_engineering", "shared_utils"],
-    "folders_read": ["marketing", "data_engineering", "shared_utils"]
-  }
-}

ai_evals/fixtures/frontend/global/initial/user_bob_nonadmin_teams.json

@@ -1,8 +0,0 @@
-{
-  "user": {
-    "username": "bob",
-    "is_admin": false,
-    "folders": ["team_a"],
-    "folders_read": ["team_a", "team_b"]
-  }
-}

ai_evals/modes/app.ts

@@ -48,7 +48,6 @@ export function createAppModeRunner(
         toolsUsed: result.toolsUsed,
         skillsInvoked: [],
         tokenUsage: result.tokenUsage,
-        finalContextTokens: result.finalContextTokens,
       };
     },
     validate({ evalCase, actual, initial, expected, run }) {

ai_evals/modes/cli.ts

@@ -106,7 +106,6 @@ export function createCliModeRunner(
           toolsUsed: run.trace.toolsUsed.map((entry) => entry.tool),
           skillsInvoked: run.trace.skillsInvoked,
           tokenUsage: run.tokenUsage ?? null,
-          finalContextTokens: run.finalContextTokens ?? null,
         };
       } catch (error) {
         const message = error instanceof Error ? error.message : String(error);
@@ -123,7 +122,6 @@ export function createCliModeRunner(
           toolsUsed: [],
           skillsInvoked: [],
           tokenUsage: null,
-          finalContextTokens: null,
         };
       } finally {
         await rm(workspaceDir, { recursive: true, force: true });

ai_evals/modes/flow.ts

@@ -61,7 +61,6 @@ export function createFlowModeRunner(
         toolCallDetails: result.toolCallDetails,
         skillsInvoked: [],
         tokenUsage: result.tokenUsage,
-        finalContextTokens: result.finalContextTokens,
       };
     },
     validate({ evalCase, actual, initial, expected }) {

ai_evals/modes/global.ts

@@ -1,11 +1,5 @@
-import { readFile, stat } from "node:fs/promises";
-import { basename } from "node:path";
-import { loadAppFixtureForEval } from "../adapters/frontend/core/app/appFixtureLoader";
-import {
-  runGlobalEval,
-  type GlobalLiveEditorDraftFixture,
-  type GlobalUserFixture,
-} from "../adapters/frontend/core/global/globalEvalRunner";
+import { readFile } from "node:fs/promises";
+import { runGlobalEval } from "../adapters/frontend/core/global/globalEvalRunner";
 import type { BenchmarkWorkspaceRunnables } from "../adapters/frontend/mockBackend";
 import type { FrontendEvalModelConfig } from "../core/models";
 import type { BenchmarkArtifactFile, GlobalValidationSpec, ModeRunner } from "../core/types";
@@ -15,8 +9,6 @@ import { getFrontendApiKey } from "./frontendCommon";
 
 export interface GlobalInitialFixture {
   workspace?: BenchmarkWorkspaceRunnables;
-  liveEditorDrafts?: GlobalLiveEditorDraftFixture[];
-  user?: GlobalUserFixture;
 }
 
 export function createGlobalModeRunner(
@@ -39,8 +31,6 @@ export function createGlobalModeRunner(
         getFrontendApiKey(modelConfig.provider),
         {
           workspaceFixtures: initial?.workspace,
-          liveEditorDrafts: initial?.liveEditorDrafts,
-          user: initial?.user,
           maxIterations: context.evalCase?.runtime?.maxTurns,
           provider: modelConfig.provider,
           model: modelConfig.model,
@@ -59,7 +49,6 @@ export function createGlobalModeRunner(
         toolCallDetails: result.toolCallDetails,
         skillsInvoked: [],
         tokenUsage: result.tokenUsage,
-        finalContextTokens: result.finalContextTokens,
       };
     },
     validate({ evalCase, actual, expected }) {
@@ -81,33 +70,9 @@ export function createGlobalModeRunner(
 }
 
 async function loadGlobalInitialFixture(path: string): Promise<GlobalInitialFixture> {
-  if ((await stat(path)).isDirectory()) {
-    const { initialFrontend, initialBackend, initialDatatables } =
-      await loadAppFixtureForEval(path);
-    const name = basename(path);
-    return {
-      workspace: {
-        apps: [
-          {
-            path: `f/evals/global/${name}`,
-            summary: name,
-            value: {
-              files: initialFrontend,
-              runnables: initialBackend,
-              data: initialDatatables,
-            },
-          },
-        ],
-      },
-      liveEditorDrafts: [],
-    };
-  }
-
   const parsed = JSON.parse(await readFile(path, "utf8")) as GlobalInitialFixture;
   return {
     workspace: parsed.workspace ?? {},
-    liveEditorDrafts: parsed.liveEditorDrafts ?? [],
-    user: parsed.user,
   };
 }
 

ai_evals/modes/script.ts

@@ -52,7 +52,6 @@ export function createScriptModeRunner(
         toolCallDetails: result.toolCallDetails,
         skillsInvoked: [],
         tokenUsage: result.tokenUsage,
-        finalContextTokens: result.finalContextTokens,
       };
     },
     validate({ actual, initial, expected }) {

backend/.sqlx/query-002a606e71364b0581dbc496bf4337f276861dc71d2e277a7aef711543eb14d7.json

@@ -1,29 +0,0 @@
-{
-  "db_name": "PostgreSQL",
-  "query": "SELECT\n               COUNT(*)::bigint AS \"total!\",\n               COUNT(*) FILTER (WHERE name = ANY($2::text[]))::bigint AS \"replacing!\"\n           FROM ai_skill\n           WHERE workspace_id = $1",
-  "describe": {
-    "columns": [
-      {
-        "ordinal": 0,
-        "name": "total!",
-        "type_info": "Int8"
-      },
-      {
-        "ordinal": 1,
-        "name": "replacing!",
-        "type_info": "Int8"
-      }
-    ],
-    "parameters": {
-      "Left": [
-        "Text",
-        "TextArray"
-      ]
-    },
-    "nullable": [
-      null,
-      null
-    ]
-  },
-  "hash": "002a606e71364b0581dbc496bf4337f276861dc71d2e277a7aef711543eb14d7"
-}

backend/.sqlx/query-0142d9dc9c1b57487dd5709a0376794f18d33e5bd6340c0189be7818cda64328.json

@@ -1,6 +1,6 @@
 {
   "db_name": "PostgreSQL",
-  "query": "SELECT email, login_type::TEXT, super_admin, devops, verified, name, company, username, NULL::bool as operator_only, NULL::bool as is_workspace_admin, first_time_user, role_source, disabled, NULL::text as workspace_id FROM password WHERE email = $1",
+  "query": "SELECT email, login_type::TEXT, super_admin, devops, verified, name, company, username, NULL::bool as operator_only, first_time_user, role_source, disabled, NULL::text as workspace_id FROM password WHERE email = $1",
   "describe": {
     "columns": [
       {
@@ -50,26 +50,21 @@
       },
       {
         "ordinal": 9,
-        "name": "is_workspace_admin",
-        "type_info": "Bool"
-      },
-      {
-        "ordinal": 10,
         "name": "first_time_user",
         "type_info": "Bool"
       },
       {
-        "ordinal": 11,
+        "ordinal": 10,
         "name": "role_source",
         "type_info": "Varchar"
       },
       {
-        "ordinal": 12,
+        "ordinal": 11,
         "name": "disabled",
         "type_info": "Bool"
       },
       {
-        "ordinal": 13,
+        "ordinal": 12,
         "name": "workspace_id",
         "type_info": "Text"
       }
@@ -89,12 +84,11 @@
       true,
       true,
       null,
-      null,
       false,
       false,
       false,
       null
     ]
   },
-  "hash": "1d0341bd8de94ab8d34a4bb1bb2005305fb3b29751ec987bece183444e89e7d1"
+  "hash": "0142d9dc9c1b57487dd5709a0376794f18d33e5bd6340c0189be7818cda64328"
 }

backend/.sqlx/query-03547bf921bbd4342dc8604277057336378f9e85baf14d1dc34ed7e9b42e5e72.json

@@ -1,16 +0,0 @@
-{
-  "db_name": "PostgreSQL",
-  "query": "INSERT INTO flow\n                (workspace_id, path, summary, description, archived, extra_perms, dependency_job, tag, ws_error_handler_muted, dedicated_worker, timeout, visible_to_runner_only, on_behalf_of_email, concurrency_key, versions, value, schema, edited_by, edited_at, labels)\n            SELECT workspace_id, $1, summary, description, archived, extra_perms, dependency_job, tag, ws_error_handler_muted, dedicated_worker, timeout, visible_to_runner_only, on_behalf_of_email, concurrency_key, versions, value, schema, edited_by, edited_at, labels\n                FROM flow\n                WHERE path = $2 AND workspace_id = $3",
-  "describe": {
-    "columns": [],
-    "parameters": {
-      "Left": [
-        "Varchar",
-        "Text",
-        "Text"
-      ]
-    },
-    "nullable": []
-  },
-  "hash": "03547bf921bbd4342dc8604277057336378f9e85baf14d1dc34ed7e9b42e5e72"
-}

backend/.sqlx/query-03c8a797ae734ff76e227259ae011ef6d35f892fe95448c58ec13dea58eee3fa.json

@@ -1,14 +0,0 @@
-{
-  "db_name": "PostgreSQL",
-  "query": "DELETE FROM skip_workspace_diff_tally WHERE workspace_id = $1",
-  "describe": {
-    "columns": [],
-    "parameters": {
-      "Left": [
-        "Text"
-      ]
-    },
-    "nullable": []
-  },
-  "hash": "03c8a797ae734ff76e227259ae011ef6d35f892fe95448c58ec13dea58eee3fa"
-}

backend/.sqlx/query-07168aaf14cb6beff0ad4274b441f7f387f5055c47f493271d26731336257384.json

@@ -46,11 +46,11 @@
       ]
     },
     "nullable": [
-      false,
-      false,
-      false,
-      false,
-      false,
+      true,
+      true,
+      true,
+      true,
+      true,
       true,
       true
     ]