chore: point ee-repo-ref at the draft-only-removal EE companion commit

Companion PR: windmill-labs/windmill-ee-private#601

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

.agents/skills/pr/SKILL.md

@@ -55,55 +55,18 @@ The body MUST be explicit about what changed. Structure:
 
 The harness/tooling that invoked the skill may add its own attribution trailer; the skill itself does not prescribe one.
 
-## Screenshots (required for frontend changes)
-
-If `git diff main...HEAD --name-only` matches `^frontend/`, the PR body **must** include
-screenshots of the affected UI. Skip only when there is no visible UI effect (types,
-tests, build config) — and say so in the body.
-
-1. Verify the change in the browser (AGENTS.md → "Verifying Frontend Changes").
-2. Screenshot each affected page with `mcp__playwright__browser_take_screenshot` (save to a file).
-3. Host each image and get its Markdown embed by pushing to the public
-   `windmill-labs/agent-screenshots-internal` repo. **Pipe base64 through stdin** —
-   passing it as `-f content=…` fails with `argument list too long` on real images:
-
-   ```bash
-   REPO=windmill-labs/agent-screenshots-internal
-   IMG=screenshot.png                                          # repeat per page
-   DEST="shots/$(git branch --show-current)/$(date +%s)-$(basename "$IMG")"
-   base64 -w0 "$IMG" | jq -Rs --arg m "add $DEST" '{message:$m, content:.}' \
-     | gh api -X PUT "repos/$REPO/contents/$DEST" --input - >/dev/null
-   echo "![$(basename "$IMG" .png)](https://raw.githubusercontent.com/$REPO/main/$DEST)"
-   ```
-   Derive `$DEST` from the file name (as above) so distinct pages never collide — a
-   fixed name would make same-second uploads reuse one path, and the second `PUT`
-   then 422s (the Contents API needs the existing file's `sha` to overwrite).
-4. Put the printed `![…](…)` lines under a `## Screenshots` heading in the PR body.
-
-Requires `gh` (`repo` scope), `jq`, `base64` — all in the devShell. The host repo is
-public (so the raw URLs render for reviewers without a token) and its history is
-permanent — **never screenshot pages that show secrets or sensitive values** (workspace
-variables, resource values, instance settings, OAuth/SMTP config); deleting the file
-can't undo an accidental capture. (GitHub's drag-and-drop uploader needs a browser
-session and can't be driven from a token.)
-
-If `gh` can't push to the host repo (e.g. a CI token scoped only to `windmill`), do
-**not** fail the PR or skip silently — hand the upload to the user, who has push access,
-and continue once they confirm it's done.
-
 ## Execution Steps
 
 1. Run `git status` to check for uncommitted changes
 2. Run `git log main..HEAD --oneline` to see all commits in this branch
 3. Run `git diff main...HEAD` to see the full diff against main
 4. **Invoke the `local-review` skill** before creating the PR (`/local-review` in Claude Code, `$local-review` in Codex, `pi --skill local-review` / `/skill:local-review` in Pi). If issues are found, fix them and commit before proceeding. Do not skip this step.
-5. **Screenshots for frontend changes**: if `git diff main...HEAD --name-only` matches `^frontend/`, capture and embed screenshots of the affected UI per "Screenshots" above before writing the PR body (skip only if there is no visible UI effect).
-6. Check if remote branch exists and is up to date:
+5. Check if remote branch exists and is up to date:
    ```bash
    git rev-parse --abbrev-ref --symbolic-full-name @{u} 2>/dev/null || echo "no upstream"
    ```
-7. Push to remote if needed: `git push -u origin HEAD`
-8. Create draft PR using gh CLI:
+6. Push to remote if needed: `git push -u origin HEAD`
+7. Create draft PR using gh CLI:
    ```bash
    gh pr create --draft --title "<type>: <description>" --body "$(cat <<'EOF'
    ## Summary
@@ -119,7 +82,7 @@ and continue once they confirm it's done.
    EOF
    )"
    ```
-9. Return the PR URL to the user
+8. Return the PR URL to the user
 
 ## EE Companion PR (when `*_ee.rs` files were modified)
 

.github/DockerfileBackendTests

@@ -28,7 +28,7 @@ ENV PATH="${PATH}:/usr/local/go/bin"
 ENV GO_PATH=/usr/local/go/bin/go
 
 # UV
-RUN curl --proto '=https' --tlsv1.2 -LsSf https://github.com/astral-sh/uv/releases/download/0.9.25/uv-installer.sh | sh && mv /usr/local/cargo/bin/uv /usr/local/bin/uv
+RUN curl --proto '=https' --tlsv1.2 -LsSf https://github.com/astral-sh/uv/releases/download/0.9.24/uv-installer.sh | sh && mv /usr/local/cargo/bin/uv /usr/local/bin/uv
 
 ENV TZ=Etc/UTC
 

.github/change-versions-mac.sh

@@ -7,7 +7,7 @@ VERSION=$1
 echo "Updating versions to: $VERSION"
 
 sed -i '' -e "/^version =/s/= .*/= \"$VERSION\"/" ${root_dirpath}/backend/Cargo.toml
-sed -i '' -e "/^export const VERSION =/s/= .*/= \"v$VERSION\";/"  ${root_dirpath}/cli/src/core/constants.ts
+sed -i '' -e "/^export const VERSION =/s/= .*/= \"v$VERSION\";/"  ${root_dirpath}/cli/src/main.ts
 sed -i '' -e "/^export const VERSION =/s/= .*/= \"v$VERSION\";/" ${root_dirpath}/benchmarks/lib.ts
 sed -i '' -e "/version: /s/: .*/: $VERSION/" ${root_dirpath}/backend/windmill-api/openapi.yaml
 sed -i '' -e "/version: /s/: .*/: $VERSION/" ${root_dirpath}/openflow.openapi.yaml

.github/change-versions.sh

@@ -7,7 +7,7 @@ VERSION=$1
 echo "Updating versions to: $VERSION"
 
 sed -i -e "/^version =/s/= .*/= \"$VERSION\"/" ${root_dirpath}/backend/Cargo.toml
-sed -i -e "/^export const VERSION =/s/= .*/= \"$VERSION\";/" ${root_dirpath}/cli/src/core/constants.ts
+sed -i -e "/^export const VERSION =/s/= .*/= \"$VERSION\";/" ${root_dirpath}/cli/src/main.ts
 sed -i -e "/^export const VERSION =/s/= .*/= \"v$VERSION\";/" ${root_dirpath}/benchmarks/lib.ts
 sed -i -e "/version: /s/: .*/: $VERSION/" ${root_dirpath}/backend/windmill-api/openapi.yaml
 sed -i -e "/version: /s/: .*/: $VERSION/" ${root_dirpath}/openflow.openapi.yaml

.github/workflows/ai-agent-tests.yml

@@ -1,132 +0,0 @@
-name: AI Agent Integration Tests
-
-# Exercises the AI agent flow path (preview_flow with `aiagent` modules) against
-# real LLM providers. Runs only when AI-agent backend code or the tests change,
-# because each run makes real (paid) LLM calls. To avoid spending on every commit,
-# the PR side triggers only when a PR is marked ready for review (out of draft) —
-# not on `synchronize` — plus push to main and manual dispatch.
-on:
-  workflow_dispatch:
-  push:
-    branches: [main]
-    paths:
-      - "integration_tests/ai_agent_tests/**"
-      - "backend/windmill-ai/**"
-      - "backend/windmill-api/src/ai.rs"
-      - "backend/windmill-worker/src/ai_executor.rs"
-      - "backend/windmill-worker/src/ai/**"
-      - "backend/windmill-worker/src/memory_common.rs"
-      - "backend/windmill-common/src/flow_conversations.rs"
-      - ".github/workflows/ai-agent-tests.yml"
-  pull_request:
-    types: [opened, reopened, ready_for_review]
-    paths:
-      - "integration_tests/ai_agent_tests/**"
-      - "backend/windmill-ai/**"
-      - "backend/windmill-api/src/ai.rs"
-      - "backend/windmill-worker/src/ai_executor.rs"
-      - "backend/windmill-worker/src/ai/**"
-      - "backend/windmill-worker/src/memory_common.rs"
-      - "backend/windmill-common/src/flow_conversations.rs"
-      - ".github/workflows/ai-agent-tests.yml"
-
-concurrency:
-  group: ai-agent-tests-${{ github.ref }}
-  cancel-in-progress: true
-
-jobs:
-  ai_agent_e2e:
-    # Skip draft PRs; the `opened`/`reopened` types would otherwise fire while
-    # still a draft. `ready_for_review` always arrives non-draft.
-    if: github.event_name != 'pull_request' || github.event.pull_request.draft == false
-    runs-on: ubicloud-standard-16
-    services:
-      postgres:
-        image: postgres:16
-        ports:
-          - 5432:5432
-        env:
-          POSTGRES_DB: windmill
-          POSTGRES_PASSWORD: changeme
-        options: >-
-          --health-cmd pg_isready --health-interval 10s --health-timeout 5s
-          --health-retries 5
-    steps:
-      - uses: actions/checkout@v4
-
-      - uses: actions-rust-lang/setup-rust-toolchain@v1
-        with:
-          cache-workspaces: backend
-          toolchain: 1.93.0
-
-      - uses: oven-sh/setup-bun@v2
-        with:
-          bun-version: 1.3.10
-
-      - uses: actions/setup-node@v4
-        with:
-          node-version: "20"
-
-      - uses: actions/setup-python@v5
-        with:
-          python-version: "3.11"
-
-      # CE build (no enterprise/license needed for AI agents). `quickjs` powers
-      # flow input-transform JS eval; `mcp` is required by the deepwiki MCP tool
-      # test. Bun tool scripts run via the always-on worker (BUN_PATH).
-      - name: Build Windmill
-        working-directory: ./backend
-        env:
-          SQLX_OFFLINE: true
-          CARGO_BUILD_JOBS: 12
-          RUSTFLAGS: ""
-        run: cargo build --features quickjs,mcp
-
-      - name: Start Windmill
-        working-directory: ./backend
-        env:
-          DATABASE_URL: postgres://postgres:changeme@localhost:5432/windmill
-          BUN_PATH: bun
-          NODE_BIN_PATH: node
-          RUST_LOG: info
-        run: |
-          mkdir -p ../integration_tests/logs
-          ./target/debug/windmill > ../integration_tests/logs/windmill.log 2>&1 &
-          echo "Waiting for Windmill to be ready..."
-          for i in $(seq 1 60); do
-            if curl -sf http://localhost:8000/api/version > /dev/null 2>&1; then
-              echo "Windmill is ready"
-              break
-            fi
-            sleep 2
-          done
-          curl -sf http://localhost:8000/api/version > /dev/null || { echo "Windmill failed to start"; tail -50 ../integration_tests/logs/windmill.log; exit 1; }
-
-      - name: Run AI agent integration tests
-        timeout-minutes: 20
-        working-directory: ./integration_tests/ai_agent_tests
-        env:
-          WINDMILL_URL: http://localhost:8000
-          # Only the providers we have org secrets for. Other providers
-          # (Azure, Bedrock, OpenRouter) are skipped by conftest when their
-          # keys are absent — see skip_provider_without_credentials.
-          ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }}
-          OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
-          GOOGLE_AI_API_KEY: ${{ secrets.GOOGLE_API_KEY }}
-        run: |
-          python -m venv .venv
-          .venv/bin/pip install -r requirements.txt
-          # The S3/vision-attachment tests need MinIO large-file storage and
-          # image-capable provider setup; out of scope for this cost-controlled
-          # smoke. Add MinIO secrets + a storage service to enable them.
-          .venv/bin/python -m pytest -v \
-            --ignore=test_user_attachments.py \
-            --ignore=test_user_images.py \
-            --ignore=test_image_output.py
-
-      - name: Archive Windmill logs
-        uses: actions/upload-artifact@v4
-        if: always()
-        with:
-          name: ai-agent-tests-windmill-logs
-          path: integration_tests/logs

.github/workflows/ai-evals-test.yml

@@ -1,166 +0,0 @@
-name: AI Evals (global mode)
-
-# Smoke-tests the production global AI chat proxy/frontend execution path via
-# the ai_evals harness, one case across one cheap model per provider. Runs only
-# when the eval harness or the global chat code change, since each run makes real
-# (paid) LLM calls. The backend is built from source purely as the AI proxy the
-# harness routes model calls through; the global tools/drafts run in-process in
-# the Vitest bridge against production frontend code. To avoid spending on every
-# commit, the PR side triggers only when a PR is marked ready for review (out of
-# draft) — not on `synchronize` — plus push to main and manual dispatch.
-on:
-  workflow_dispatch:
-  push:
-    branches: [main]
-    paths:
-      - "ai_evals/**"
-      - "backend/windmill-api/src/ai.rs"
-      - "backend/windmill-ai/**"
-      - "frontend/src/lib/components/copilot/**"
-      # The eval harness runs production frontend code in-process; these are the
-      # AI/draft-specific deps outside copilot/ that the global smoke exercises.
-      - "frontend/src/lib/userDraft.svelte.ts"
-      - "frontend/src/lib/userDraftDbSyncer.svelte.ts"
-      - "frontend/src/lib/infer.ts"
-      - ".github/workflows/ai-evals-test.yml"
-  pull_request:
-    types: [opened, reopened, ready_for_review]
-    paths:
-      - "ai_evals/**"
-      - "backend/windmill-api/src/ai.rs"
-      - "backend/windmill-ai/**"
-      - "frontend/src/lib/components/copilot/**"
-      # The eval harness runs production frontend code in-process; these are the
-      # AI/draft-specific deps outside copilot/ that the global smoke exercises.
-      - "frontend/src/lib/userDraft.svelte.ts"
-      - "frontend/src/lib/userDraftDbSyncer.svelte.ts"
-      - "frontend/src/lib/infer.ts"
-      - ".github/workflows/ai-evals-test.yml"
-
-concurrency:
-  group: ai-evals-test-${{ github.ref }}
-  cancel-in-progress: true
-
-jobs:
-  ai_evals_global:
-    # Provider secrets are unavailable to forked and Dependabot PRs.
-    if: >-
-      github.event_name != 'pull_request' ||
-      (
-        github.event.pull_request.draft == false &&
-        github.event.pull_request.head.repo.full_name == github.repository &&
-        github.event.pull_request.user.login != 'dependabot[bot]'
-      )
-    runs-on: ubicloud-standard-16
-    services:
-      postgres:
-        image: postgres:16
-        ports:
-          - 5432:5432
-        env:
-          POSTGRES_DB: windmill
-          POSTGRES_PASSWORD: changeme
-        options: >-
-          --health-cmd pg_isready --health-interval 10s --health-timeout 5s
-          --health-retries 5
-    steps:
-      - uses: actions/checkout@v4
-
-      - uses: actions-rust-lang/setup-rust-toolchain@v1
-        with:
-          cache-workspaces: backend
-          toolchain: 1.93.0
-
-      - uses: oven-sh/setup-bun@v2
-        with:
-          bun-version: 1.3.10
-
-      - uses: actions/setup-node@v4
-        with:
-          # Node 22.19+ is required by the frontend's undici 8.x, which the
-          # Vitest bridge loads; Node 20 fails with markAsUncloneable.
-          node-version: "22"
-
-      # CE build used only as the AI proxy (login, workspace, provider resource,
-      # /ai/proxy). No worker execution or MCP needed — global tools/drafts run
-      # in the Vitest bridge. quickjs matches the standard CE feature set.
-      - name: Build Windmill (AI proxy)
-        working-directory: ./backend
-        env:
-          SQLX_OFFLINE: true
-          CARGO_BUILD_JOBS: 12
-          RUSTFLAGS: ""
-        run: cargo build --features quickjs
-
-      - name: Start Windmill
-        working-directory: ./backend
-        env:
-          DATABASE_URL: postgres://postgres:changeme@localhost:5432/windmill
-          RUST_LOG: info
-        run: |
-          mkdir -p ../ai_evals/logs
-          ./target/debug/windmill > ../ai_evals/logs/windmill.log 2>&1 &
-          echo "Waiting for Windmill to be ready..."
-          for i in $(seq 1 60); do
-            if curl -sf http://localhost:8000/api/version > /dev/null 2>&1; then
-              echo "Windmill is ready"
-              break
-            fi
-            sleep 2
-          done
-          curl -sf http://localhost:8000/api/version > /dev/null || { echo "Windmill failed to start"; tail -50 ../ai_evals/logs/windmill.log; exit 1; }
-
-      - name: Install frontend deps + generate client
-        working-directory: ./frontend
-        run: |
-          npm ci
-          npm run generate-backend-client
-
-      - name: Run global AI evals
-        timeout-minutes: 20
-        working-directory: ./ai_evals
-        env:
-          WMILL_AI_EVAL_BACKEND_URL: http://localhost:8000
-          WMILL_AI_EVAL_BACKEND_WORKSPACE: integration-tests
-          # Anthropic backs the haiku model. Google AI uses GEMINI_API_KEY.
-          ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }}
-          OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
-          GEMINI_API_KEY: ${{ secrets.GOOGLE_API_KEY }}
-          DEEPSEEK_API_KEY: ${{ secrets.DEEPSEEK_API_KEY }}
-        run: |
-          bun install
-          mkdir -p results
-          # One cheap model per provider (anthropic/openai/googleai/deepseek).
-          fail=0
-          for m in haiku 4o gemini-3-flash-preview deepseek-v4-flash; do
-            echo "::group::global-test1-script-create ($m)"
-            if ! bun run cli -- run global global-test1-script-create \
-                  --model "$m" --execution-only --output "$PWD/results/ci-$m.json"; then
-              echo "$m: harness/proxy errored"
-              fail=1
-              echo "::endgroup::"
-              continue
-            fi
-            # The CLI exits 0 when the harness records failed attempts, so gate
-            # on execution-only pass counts while ignoring model output quality.
-            if jq -e \
-              '.attemptCount > 0 and .passedAttempts == .attemptCount' \
-              "results/ci-$m.json" > /dev/null; then
-              echo "$m: OK — proxy/frontend execution completed"
-            else
-              echo "$m: FAILED proxy/frontend execution"
-              jq -c '.cases[0].attempts[0].checks' "results/ci-$m.json" || true
-              fail=1
-            fi
-            echo "::endgroup::"
-          done
-          [ "$fail" = 0 ] || { echo "ai_evals global smoke failed"; exit 1; }
-
-      - name: Archive logs and results
-        uses: actions/upload-artifact@v4
-        if: always()
-        with:
-          name: ai-evals-global-logs
-          path: |
-            ai_evals/logs
-            ai_evals/results

.github/workflows/backend-test-windows.yml

@@ -74,7 +74,7 @@ jobs:
 
       - uses: astral-sh/setup-uv@v6.2.1
         with:
-          version: "0.9.25"
+          version: "0.9.24"
 
       - uses: shivammathur/setup-php@v2
         with:

.github/workflows/backend-test.yml

@@ -62,7 +62,7 @@ jobs:
           node-version: "20"
       - uses: astral-sh/setup-uv@v6.2.1
         with:
-          version: "0.9.25"
+          version: "0.9.24"
       - uses: shivammathur/setup-php@v2
         with:
           php-version: "8.3"

.github/workflows/claude-plan.yml

@@ -45,7 +45,7 @@ jobs:
           allowed_bots: 'windmill-internal-app[bot]'
           trigger_phrase: '/plan'
           claude_args: |
-            --model claude-opus-4-8
+            --model opus
             --system-prompt "# Claude Planning Mode
 
             You are operating in PLANNING MODE ONLY. Your role is to create detailed, structured plans without making any code changes.

.github/workflows/claude.yml

@@ -51,4 +51,4 @@ jobs:
             }
           claude_args: |
             --allowedTools "Bash,WebFetch,WebSearch"
-            --model claude-opus-4-8
+            --model opus

.github/workflows/pr-ready-review.yml

@@ -160,4 +160,4 @@ jobs:
             ${{ env.REVIEW_PROMPT }}
           claude_args: |
             --allowedTools "mcp__github_inline_comment__create_inline_comment,Bash(gh pr comment:*),Bash(gh pr diff:*),Bash(gh pr view:*)"
-            --model claude-opus-4-8
+            --model opus

.gitignore

@@ -33,5 +33,4 @@ backend/chrome_profiler.json
 .fast-check/
 __pycache__/
 .playwright-mcp/
-.codex
-.claude/scheduled_tasks.lock
+.codex

AGENTS.md

@@ -15,7 +15,6 @@ Open-source platform for internal tools, workflows, API integrations, background
 - **Enterprise**: `docs/enterprise.md` — EE file conventions and PR workflow
 - **Backend patterns**: use the `rust-backend` skill when writing Rust code
 - **Frontend patterns**: use the `svelte-frontend` skill when writing Svelte code. Do NOT edit svelte files unless you have read that skill.
-- **Frontend UUIDs**: do not call `crypto.randomUUID()` in frontend code. Import `randomUUID` from `$lib/utils/uuid` instead.
 - **Code review**: review the current PR or branch against the shared review policy in `REVIEW.md` (severity triage, public-surface checklist, AGENTS.md compliance, test-coverage assessment). The skill at `.agents/skills/local-review/SKILL.md` orchestrates it. All three CLIs auto-discover the same SKILL — Claude reads `.claude/skills/` (symlinked to the canonical `.agents/skills/` file), Codex and Pi read `.agents/skills/` directly. Invoke with `/local-review` in Claude Code, `$local-review` (or `/skills` selector) in Codex, or `pi --skill local-review` / `/skill:local-review` in Pi.
 - **Domain guides**: `.claude/skills/native-trigger/` and `frontend/tutorial-system-guide.mdc`
 - **Brand/UI guidelines**: `frontend/brand-guidelines.md`
@@ -48,8 +47,6 @@ Typical flow:
 5. `mcp__playwright__browser_take_screenshot` for visual confirmation
 6. `mcp__playwright__browser_console_messages` / `browser_network_requests` to surface errors
 
-**Attach the screenshots to the PR.** For any change under `frontend/`, embed screenshots of the affected UI in the PR body — the `pr` skill requires this and carries the upload recipe.
-
 If you cannot exercise a UI change (no dev server, etc.), say so explicitly rather than claiming success.
 
 ## Banned Patterns
@@ -109,5 +106,3 @@ $NAV --root backend callees "X"                           # what does X call?
 - Search for existing code to reuse before writing new code
 - Follow established patterns in the codebase
 - Keep changes focused — don't refactor beyond what's asked
-- **Comments record constraints, not narration.** Write a comment only for what the code can't show: why a non-obvious approach is required, what breaks if it's "simplified" away. State each invariant once, at the place where someone would break it, in ≤4 lines. Don't describe what the next line does, don't repeat the same rationale at multiple sites, and don't address the PR reviewer (justifying a change belongs in the PR description, not the code). Describe the code as it is, never its drafting history: "we no longer do X", "unchanged behavior", "instead of the previous approach" are meaningless to a reader who never saw the earlier iteration — before finishing, reread your comments as if the current state is the only state that ever existed.
-- **Never attribute work to a specific customer, account, or "requested by a customer" in repo-tracked content** (PR descriptions, commit messages, code comments, docs). Describe changes by their technical motivation instead.

CHANGELOG.md

@@ -1,446 +1,5 @@
 # Changelog
 
-## [1.737.0](https://github.com/windmill-labs/windmill/compare/v1.736.0...v1.737.0) (2026-06-23)
-
-
-### Features
-
-* **apps:** opt-in sandbox isolation for published & raw apps (alpha) ([#9420](https://github.com/windmill-labs/windmill/issues/9420)) ([2879cbb](https://github.com/windmill-labs/windmill/commit/2879cbb65a4122c86b4a472206d74d9009b07904))
-
-
-### Bug Fixes
-
-* allow SQL args in managed // materialize scripts ([#9733](https://github.com/windmill-labs/windmill/issues/9733)) ([fa35968](https://github.com/windmill-labs/windmill/commit/fa3596885bf2d7ee8859f295e820ee362c756911))
-* bound orphan-cleanup drain rate with capped multi-batch loop ([#9730](https://github.com/windmill-labs/windmill/issues/9730)) ([31d9215](https://github.com/windmill-labs/windmill/commit/31d9215e5a61f19662cc87be8147007e1d47ebb6))
-* **ext-jwt:** reject external JWT auth for non-existent workspaces ([#9723](https://github.com/windmill-labs/windmill/issues/9723)) ([c644311](https://github.com/windmill-labs/windmill/commit/c644311eca4bcaf4b68058cf1d5d79d4078aee1a))
-* optimize cleanup_job_perms_orphaned and job_result_stream cleanup queries ([#9727](https://github.com/windmill-labs/windmill/issues/9727)) ([6d94865](https://github.com/windmill-labs/windmill/commit/6d9486510933af9109f52011d93b13847dbdbb39))
-* prevent silent audit-partition outage via monitor watchdog + alert ([#9729](https://github.com/windmill-labs/windmill/issues/9729)) ([8dea383](https://github.com/windmill-labs/windmill/commit/8dea38383f884f59b2956c39f1424005a21265bd))
-
-
-### Performance Improvements
-
-* **monitor:** hash active-root exclusion in retention delete (WIN-2088) ([#9732](https://github.com/windmill-labs/windmill/issues/9732)) ([75bafab](https://github.com/windmill-labs/windmill/commit/75bafabeeec76cf6da33eef41f588e37071df011))
-
-## [1.736.0](https://github.com/windmill-labs/windmill/compare/v1.735.0...v1.736.0) (2026-06-23)
-
-
-### Features
-
-* **ai-chat:** workspace AI chat skills (SKILL.md upload + read_skill tool) ([#9648](https://github.com/windmill-labs/windmill/issues/9648)) ([6f4017d](https://github.com/windmill-labs/windmill/commit/6f4017d694494a158ebd0579c93336c378cba0fd))
-
-
-### Bug Fixes
-
-* **drafts:** stop mis-filing workspace-blind legacy drafts on migration ([#9725](https://github.com/windmill-labs/windmill/issues/9725)) ([3bf5b72](https://github.com/windmill-labs/windmill/commit/3bf5b72afab3241ea41a261a40c2434764bdaf72))
-* **frontend:** destroy old WebsocketProvider on workspace switch in MultiplayerMenu ([#9719](https://github.com/windmill-labs/windmill/issues/9719)) ([6e96f90](https://github.com/windmill-labs/windmill/commit/6e96f90065dfe2f6ccc5eb4f85f4facd3515c70c))
-* **frontend:** ensure type:object in test_run_flow tool schema for Anthropic ([#9721](https://github.com/windmill-labs/windmill/issues/9721)) ([d5cb944](https://github.com/windmill-labs/windmill/commit/d5cb944cf92f074b2ee42c876595eacdfa2f4d76))
-* **health:** detect read-only replica via pg_is_in_recovery() ([#9722](https://github.com/windmill-labs/windmill/issues/9722)) ([e16061d](https://github.com/windmill-labs/windmill/commit/e16061df06babeae935a9396de5bdcd46e8119a9))
-* re-enforce scoped API token boundaries across handlers ([#9712](https://github.com/windmill-labs/windmill/issues/9712)) ([e19594d](https://github.com/windmill-labs/windmill/commit/e19594df2ad015a0336ade95e04562f5562ec3f6))
-
-## [1.735.0](https://github.com/windmill-labs/windmill/compare/v1.734.0...v1.735.0) (2026-06-22)
-
-
-### Features
-
-* clarify session draft bar tracks all workspace draft changes ([#9714](https://github.com/windmill-labs/windmill/issues/9714)) ([ed016a5](https://github.com/windmill-labs/windmill/commit/ed016a5edb4527877bf7e6bf92feafc2710669c2))
-* **copilot:** improve global-mode path selection + add path-selection evals ([#9698](https://github.com/windmill-labs/windmill/issues/9698)) ([74a2329](https://github.com/windmill-labs/windmill/commit/74a2329d2e8395141807c43acef07ef132490039))
-* link files & folders to the global AI chat ([#9520](https://github.com/windmill-labs/windmill/issues/9520)) ([84cc043](https://github.com/windmill-labs/windmill/commit/84cc043406d63a1e1472165cf24ce8c09905fc5f))
-* scope default instance db name to workspace (dt_/dl_) ([#9699](https://github.com/windmill-labs/windmill/issues/9699)) ([4a8a724](https://github.com/windmill-labs/windmill/commit/4a8a724895dcecb835e1eb1e4fd7d1bbc8b3e0fb))
-
-
-### Bug Fixes
-
-* enforce job_dir containment when writing module files ([#9703](https://github.com/windmill-labs/windmill/issues/9703)) ([e403f92](https://github.com/windmill-labs/windmill/commit/e403f92d7e84cebc78709dce1a0928048ba2506d))
-* **frontend:** deploy full script/flow draft from AI chat via shared module ([#9642](https://github.com/windmill-labs/windmill/issues/9642)) ([23bf6bf](https://github.com/windmill-labs/windmill/commit/23bf6bf3da01d552d2dfab6dbcfd30f758ed34d2))
-* **frontend:** strip raw-app post-deploy diff noise (raw_app/lock/data) ([#9706](https://github.com/windmill-labs/windmill/issues/9706)) ([e20a277](https://github.com/windmill-labs/windmill/commit/e20a27745a08d552e6d2c5a8bbaf08ccfe89c68f))
-* ignore NotFound errors when deleting log files from object store ([#9707](https://github.com/windmill-labs/windmill/issues/9707)) ([8a0b0ab](https://github.com/windmill-labs/windmill/commit/8a0b0abead71320c4f69eb3007739a19f76d4126))
-* **oauth:** restore bring-your-own CC token URL override ([#9711](https://github.com/windmill-labs/windmill/issues/9711)) ([ef4962e](https://github.com/windmill-labs/windmill/commit/ef4962e52aba0bc79bf72523de9101853c660654))
-* sanitize git credentials from ansible executor errors and logs ([#9697](https://github.com/windmill-labs/windmill/issues/9697)) ([ace7b68](https://github.com/windmill-labs/windmill/commit/ace7b68a28b00d715298ffcb6ae907c1974a74b8))
-
-## [1.734.0](https://github.com/windmill-labs/windmill/compare/v1.733.1...v1.734.0) (2026-06-20)
-
-
-### Features
-
-* ducklake materialization for data pipelines ([#9689](https://github.com/windmill-labs/windmill/issues/9689)) ([3ebf243](https://github.com/windmill-labs/windmill/commit/3ebf24359d66048d6361ce65cd879cdc04b737ed))
-
-
-### Bug Fixes
-
-* **frontend:** clear branch step state when switching outer loop iterations ([#9650](https://github.com/windmill-labs/windmill/issues/9650)) ([09a8004](https://github.com/windmill-labs/windmill/commit/09a80040ca268a4379d5302e3401435ba93247e0))
-
-## [1.733.1](https://github.com/windmill-labs/windmill/compare/v1.733.0...v1.733.1) (2026-06-19)
-
-
-### Bug Fixes
-
-* **backend:** validate ansible vault_id entries before config generation ([#9681](https://github.com/windmill-labs/windmill/issues/9681)) ([c1f31c0](https://github.com/windmill-labs/windmill/commit/c1f31c0e4777bf0cfed0dd7f03249e9a61cd8cb9))
-* **frontend:** group live pipeline runs in the activity panel ([#9684](https://github.com/windmill-labs/windmill/issues/9684)) ([1be4df9](https://github.com/windmill-labs/windmill/commit/1be4df9acb935250d4cc12e83cf67e366d870d5a))
-* require super admin for object storage config test endpoint ([#9683](https://github.com/windmill-labs/windmill/issues/9683)) ([fb44fe7](https://github.com/windmill-labs/windmill/commit/fb44fe7af2b8ebe8ef64ffb0e5acce8580bf4200))
-* validate websocket trigger urls and gate trigger test route ([#9682](https://github.com/windmill-labs/windmill/issues/9682)) ([c39ee07](https://github.com/windmill-labs/windmill/commit/c39ee07c0bcd2249dd19ffa5cd988125eefc6c9f))
-
-## [1.733.0](https://github.com/windmill-labs/windmill/compare/v1.732.0...v1.733.0) (2026-06-19)
-
-
-### Features
-
-* **ai-chat:** cap read_app_file + search_app grep tool to bound context in large raw apps ([#9653](https://github.com/windmill-labs/windmill/issues/9653)) ([4296a6a](https://github.com/windmill-labs/windmill/commit/4296a6ae1f73564de4df54fe1df0a03c1df05dfd))
-* **python, windows:** enable S3 to cache wheels ([#5199](https://github.com/windmill-labs/windmill/issues/5199)) ([ab3bc97](https://github.com/windmill-labs/windmill/commit/ab3bc97cd92b6480327029bcf018280442462af7))
-
-
-### Bug Fixes
-
-* allow users to always discard their own drafts without write permission ([#9659](https://github.com/windmill-labs/windmill/issues/9659)) ([6833a55](https://github.com/windmill-labs/windmill/commit/6833a554aeddb3e63173d3c3140b490c0bf2822b))
-* **backend:** clean up unique_ext_jwt_token on workspace deletion ([#9676](https://github.com/windmill-labs/windmill/issues/9676)) ([9add719](https://github.com/windmill-labs/windmill/commit/9add719d936cdcfb2c4062629e3e1f792694dafe))
-* **backend:** strip NUL bytes from draft values on write ([#9673](https://github.com/windmill-labs/windmill/issues/9673)) ([924f9c7](https://github.com/windmill-labs/windmill/commit/924f9c7e8d8863d9af40aee246a519b4be0e1ea2))
-* **python:** split PIP_TRUSTED_HOST by whitespace to support multiple hosts ([#9675](https://github.com/windmill-labs/windmill/issues/9675)) ([cafb473](https://github.com/windmill-labs/windmill/commit/cafb473494d9cff3a8b2aeaf9f18b015f966e7b3))
-
-## [1.732.0](https://github.com/windmill-labs/windmill/compare/v1.731.0...v1.732.0) (2026-06-19)
-
-
-### Features
-
-* **ansible:** add AI chat and editor bar buttons for ansible ([#9671](https://github.com/windmill-labs/windmill/issues/9671)) ([017c3d3](https://github.com/windmill-labs/windmill/commit/017c3d3343c2577501103be4b2dd8dac9727d80d))
-
-
-### Bug Fixes
-
-* **ai:** emit token usage in gemini proxy streaming translation ([#9669](https://github.com/windmill-labs/windmill/issues/9669)) ([0cc2257](https://github.com/windmill-labs/windmill/commit/0cc2257596a3965cf6db21a0090edcce6e1b8419))
-* **backend:** grant script_trigger access to windmill roles ([#9674](https://github.com/windmill-labs/windmill/issues/9674)) ([3361736](https://github.com/windmill-labs/windmill/commit/33617367d09537667d2ab3f91135c736194b9e7e))
-* **frontend:** ignore hash/assets in script diffs and drafts (WIN-2071) ([#9664](https://github.com/windmill-labs/windmill/issues/9664)) ([3371265](https://github.com/windmill-labs/windmill/commit/33712653821e83f2562dd5f271dbec0188d5d2f8))
-
-## [1.731.0](https://github.com/windmill-labs/windmill/compare/v1.730.0...v1.731.0) (2026-06-19)
-
-
-### Features
-
-* **backend:** auto-reconnect postgres trigger listener with backoff (WIN-2073) ([#9666](https://github.com/windmill-labs/windmill/issues/9666)) ([a425431](https://github.com/windmill-labs/windmill/commit/a425431e9067bcf85474fdc7b7ef7f73e41b9071))
-
-
-### Bug Fixes
-
-* **backend:** grant notify_event access to windmill roles ([#9665](https://github.com/windmill-labs/windmill/issues/9665)) ([a682d02](https://github.com/windmill-labs/windmill/commit/a682d02311a2110bfc0d5e0a5b52e96147fe0dd7))
-* **mcp:** repair invalid type keywords in tool JSON schemas ([#9667](https://github.com/windmill-labs/windmill/issues/9667)) ([c30bdec](https://github.com/windmill-labs/windmill/commit/c30bdecea77ff9b4d74d52961f3101201099b683))
-* trigger flow error handler on unrecoverable (OOM/zombie) step failures ([#9662](https://github.com/windmill-labs/windmill/issues/9662)) ([7e4df02](https://github.com/windmill-labs/windmill/commit/7e4df02bd60c4d6ee8c92d3dfd19f4e587ff9632))
-
-## [1.730.0](https://github.com/windmill-labs/windmill/compare/v1.729.0...v1.730.0) (2026-06-18)
-
-
-### Features
-
-* **ai-chat:** summary-based conversation compaction ([#9645](https://github.com/windmill-labs/windmill/issues/9645)) ([5d553b8](https://github.com/windmill-labs/windmill/commit/5d553b81c06664aab61131a93b198575c088d12d))
-* Data Pipelines alpha ([#9193](https://github.com/windmill-labs/windmill/issues/9193)) ([7155a0b](https://github.com/windmill-labs/windmill/commit/7155a0bb96cf30bd878272a0f4c3c3b02341b261))
-
-
-### Bug Fixes
-
-* **ai-chat:** stop echoing app draft value in global chat write tool results ([#9658](https://github.com/windmill-labs/windmill/issues/9658)) ([2fed808](https://github.com/windmill-labs/windmill/commit/2fed808b9e716d9a44b34c7a073ec0d37374be05))
-* **backend:** include raw_app drafts in list_apps draft_users ([#9647](https://github.com/windmill-labs/windmill/issues/9647)) ([19bc005](https://github.com/windmill-labs/windmill/commit/19bc0052f1069d732231950a0ec958f675d57417))
-* **frontend:** keep ?new_draft flag until first save is confirmed ([#9656](https://github.com/windmill-labs/windmill/issues/9656)) ([9b6b7c3](https://github.com/windmill-labs/windmill/commit/9b6b7c3862d9988e5e91eaab2b967a23f41cdc0d))
-* **frontend:** re-key raw-app autosave on post-deploy navigation ([#9646](https://github.com/windmill-labs/windmill/issues/9646)) ([1058bde](https://github.com/windmill-labs/windmill/commit/1058bdeccdc4c403ef4599db0ee74a65a66c715f))
-* gate agent-worker global setting reads with a blocklist ([#9623](https://github.com/windmill-labs/windmill/issues/9623)) ([fdd82f0](https://github.com/windmill-labs/windmill/commit/fdd82f0c48f29805cd9e219649f27fba45c7fd92))
-* **workspaces:** add instance setting to disable workspace invite/add emails ([#9643](https://github.com/windmill-labs/windmill/issues/9643)) ([796230d](https://github.com/windmill-labs/windmill/commit/796230d90a7e6d1debc15e139ab708881e527862))
-
-## [1.729.0](https://github.com/windmill-labs/windmill/compare/v1.728.1...v1.729.0) (2026-06-18)
-
-
-### Features
-
-* add ducklake schema support to the database manager ([#9633](https://github.com/windmill-labs/windmill/issues/9633)) ([3eeccaf](https://github.com/windmill-labs/windmill/commit/3eeccaf9682b7803fdf5be8dcbc4d243e0ba2e49))
-* **ai-chat:** self-hosted docs tools via windmill.dev llms.txt + ask benchmark ([#9578](https://github.com/windmill-labs/windmill/issues/9578)) ([f4425fc](https://github.com/windmill-labs/windmill/commit/f4425fca9fb0d02b845bd72888ade54905c5a30b))
-* **frontend:** View Diff and in-place Load for other users' drafts ([#9621](https://github.com/windmill-labs/windmill/issues/9621)) ([5508f1d](https://github.com/windmill-labs/windmill/commit/5508f1da9cd04c2583eb3f7ee6bce19d067f2227))
-* per-user draft review & deploy page (gating, badges, rename, raw-app deploy fixes) ([#9625](https://github.com/windmill-labs/windmill/issues/9625)) ([e09cd58](https://github.com/windmill-labs/windmill/commit/e09cd5862cb636e143027fe8d9a5be9c7097b031))
-* queue messages typed while ai chat is streaming ([#9525](https://github.com/windmill-labs/windmill/issues/9525)) ([51bd869](https://github.com/windmill-labs/windmill/commit/51bd8692a482850f7ac8b04dd16db5876336b5b9))
-* zero-setup oauth client credentials for registry providers ([#9559](https://github.com/windmill-labs/windmill/issues/9559)) ([e26a923](https://github.com/windmill-labs/windmill/commit/e26a9239a62a25abf90ef06ade4dde7f36e791bb))
-
-
-### Bug Fixes
-
-* **ai_evals:** adapt global eval harness to DB-backed user drafts ([#9641](https://github.com/windmill-labs/windmill/issues/9641)) ([e87ff79](https://github.com/windmill-labs/windmill/commit/e87ff79ecf6a6e0958916ed1b3756fb3addf719f))
-* **drafts:** preserve original timestamp when migrating localStorage drafts ([#9638](https://github.com/windmill-labs/windmill/issues/9638)) ([8021775](https://github.com/windmill-labs/windmill/commit/8021775f5f961ef6fd01b022639b85855326a1da))
-* **frontend:** don't save drafts on leave when auto-save is off, warn instead ([#9630](https://github.com/windmill-labs/windmill/issues/9630)) ([2523465](https://github.com/windmill-labs/windmill/commit/252346500945a9571af744c839ac0c7d6870504f))
-* **frontend:** render Modal2 dialogs above the AI chat panel ([#9636](https://github.com/windmill-labs/windmill/issues/9636)) ([b67c8cf](https://github.com/windmill-labs/windmill/commit/b67c8cf42b477575fc1bc448058ec0d3b7e54fee))
-* **frontend:** show AI sessions when AI unconfigured, with disabled chat ([#9644](https://github.com/windmill-labs/windmill/issues/9644)) ([ba69d81](https://github.com/windmill-labs/windmill/commit/ba69d8147b615e160cf3d2885fc65a0777b78b71))
-* **git-sync:** bump default sync script to hub/28719 (windmill-cli 1.728.1) for WAC modules ([#9649](https://github.com/windmill-labs/windmill/issues/9649)) ([3c0e38b](https://github.com/windmill-labs/windmill/commit/3c0e38b5890d77983cb5cf5f422a62a73e7a4f22))
-
-## [1.728.1](https://github.com/windmill-labs/windmill/compare/v1.728.0...v1.728.1) (2026-06-17)
-
-
-### Bug Fixes
-
-* **backend:** purge workspace_diff cache on workspace delete ([#9627](https://github.com/windmill-labs/windmill/issues/9627)) ([8a3f69d](https://github.com/windmill-labs/windmill/commit/8a3f69dda8f2088fb859ed8ed6e54458940423d0))
-* **cli:** fall back to esbuild-wasm on native host/binary mismatch ([#9629](https://github.com/windmill-labs/windmill/issues/9629)) ([86d1d16](https://github.com/windmill-labs/windmill/commit/86d1d160f0d3bd9faabdafada07e2956dd98445d))
-* **frontend:** persist session-editor draft path/summary edits + per-line diff tooltips ([#9622](https://github.com/windmill-labs/windmill/issues/9622)) ([e4bfeb2](https://github.com/windmill-labs/windmill/commit/e4bfeb29bc4e89669863b5f6396904a331167658))
-
-## [1.728.0](https://github.com/windmill-labs/windmill/compare/v1.727.0...v1.728.0) (2026-06-16)
-
-
-### Features
-
-* **frontend:** adapt AI-chat/sessions drafts to DB-backed model ([#9601](https://github.com/windmill-labs/windmill/issues/9601)) ([611c70a](https://github.com/windmill-labs/windmill/commit/611c70acd211cf4b8f8308da4a264c670a2f5f43))
-* **frontend:** consolidate draft-migration errors into a single toast + modal ([#9612](https://github.com/windmill-labs/windmill/issues/9612)) ([bc0d5bf](https://github.com/windmill-labs/windmill/commit/bc0d5bf241df3633921bd9d43d171e91034fbfcf))
-* **frontend:** dedup user drafts against the deployed baseline ([#9618](https://github.com/windmill-labs/windmill/issues/9618)) ([a2ce446](https://github.com/windmill-labs/windmill/commit/a2ce44645fdbfa98bf250fac2d15d2b5b26c4b47))
-
-
-### Bug Fixes
-
-* **frontend:** reset deleteWorkspaceForkModal on confirm in SidebarContent ([#9619](https://github.com/windmill-labs/windmill/issues/9619)) ([7cb5c6e](https://github.com/windmill-labs/windmill/commit/7cb5c6e749b2020dee5ee1499f0dc69c5109a6d8))
-* **frontend:** session Drafts drawer uses raw_app kind for the raw-app diff ([#9617](https://github.com/windmill-labs/windmill/issues/9617)) ([46288b6](https://github.com/windmill-labs/windmill/commit/46288b6143efae4dfdf6fe068b97a1e8831fce6a))
-* **nativets:** respect custom CA certs in in-process fetch runtime ([#9615](https://github.com/windmill-labs/windmill/issues/9615)) ([41562c7](https://github.com/windmill-labs/windmill/commit/41562c7d7c708d7d056d9b3d0c39b994a6f4a016))
-* **ResourceForm:** initialize JSON editor when resource type schema is unavailable ([#9611](https://github.com/windmill-labs/windmill/issues/9611)) ([5a24057](https://github.com/windmill-labs/windmill/commit/5a2405743b4622fc1021109114d007057abd5dfd))
-* show folder labels in the folder list table ([#9620](https://github.com/windmill-labs/windmill/issues/9620)) ([651fa13](https://github.com/windmill-labs/windmill/commit/651fa13ee80ff76e5a53ef1ed545b03ce6792294))
-* show last updated date per user in other-users-drafts modal ([#9614](https://github.com/windmill-labs/windmill/issues/9614)) ([f6104ce](https://github.com/windmill-labs/windmill/commit/f6104ce05c4005ffb9fe8112782d1ef6d3065300))
-
-## [1.727.0](https://github.com/windmill-labs/windmill/compare/v1.726.1...v1.727.0) (2026-06-16)
-
-
-### Features
-
-* support temp_script_refs in wmill dev for local relative imports ([#9554](https://github.com/windmill-labs/windmill/issues/9554)) ([33ac287](https://github.com/windmill-labs/windmill/commit/33ac287065742df53f363a5fe09f54f5584a85a6))
-
-
-### Bug Fixes
-
-* **cli:** harden legacy flow lock migration ordering and collision guard ([#9557](https://github.com/windmill-labs/windmill/issues/9557)) ([cd09870](https://github.com/windmill-labs/windmill/commit/cd098700c2cd8d7e9150f760938f4eaf34d188ec))
-* **cli:** include __mod/ folder in gitSyncIncludePattern for scripts ([#9606](https://github.com/windmill-labs/windmill/issues/9606)) ([252c1b3](https://github.com/windmill-labs/windmill/commit/252c1b35fc716c3486109d89615127c588bbe90a))
-* **frontend:** allow same-origin redirects in isValidLogoutRedirect ([#9568](https://github.com/windmill-labs/windmill/issues/9568)) ([8500435](https://github.com/windmill-labs/windmill/commit/8500435e82231e13a1b8a874fd0545f0a0a73fee))
-* **frontend:** make UserDraft read-after-write work without live entry ([#9609](https://github.com/windmill-labs/windmill/issues/9609)) ([51e82d7](https://github.com/windmill-labs/windmill/commit/51e82d7c6d30c66c84236feb743c09929934e564))
-* **frontend:** seed detached user-draft handles so new-item drawers render ([#9608](https://github.com/windmill-labs/windmill/issues/9608)) ([9e3c0de](https://github.com/windmill-labs/windmill/commit/9e3c0decf95378c66055d82215c15cd3bf4a69cb))
-* **frontend:** strip server-managed fields from value diffs ([#9599](https://github.com/windmill-labs/windmill/issues/9599)) ([c213801](https://github.com/windmill-labs/windmill/commit/c213801b5aee54d801c14b9eb31422f2a312ef7e))
-
-## [1.726.1](https://github.com/windmill-labs/windmill/compare/v1.726.0...v1.726.1) (2026-06-15)
-
-
-### Bug Fixes
-
-* **apps:** prevent decision tree graph editor crash on missing graph context ([#9602](https://github.com/windmill-labs/windmill/issues/9602))  ([24f3259](https://github.com/windmill-labs/windmill/commit/24f32596e9ca39c963c19af4a8b14fbcd04e3a78))
-* db-backed draft fixes — review-page UX, legacy drafts, session restore ([#9600](https://github.com/windmill-labs/windmill/issues/9600)) ([4e4b224](https://github.com/windmill-labs/windmill/commit/4e4b2247ef471dada1b8c894974fe921d14b3947))
-
-## [1.726.0](https://github.com/windmill-labs/windmill/compare/v1.725.1...v1.726.0) (2026-06-15)
-
-
-### Features
-
-* **audit:** record workspace archive/unarchive/delete in instance audit log ([#9596](https://github.com/windmill-labs/windmill/issues/9596)) ([9de5708](https://github.com/windmill-labs/windmill/commit/9de57086086bb5626d175c7f926915d1d6ac67ca))
-* **frontend:** add user-level toggle to disable Windmill AI ([#9585](https://github.com/windmill-labs/windmill/issues/9585)) ([5709a56](https://github.com/windmill-labs/windmill/commit/5709a564fbafd9aa91943572ecd8c3e0c45c20b1))
-
-
-### Bug Fixes
-
-* **embeddings:** retry HuggingFace model downloads with backoff ([#9597](https://github.com/windmill-labs/windmill/issues/9597)) ([6a62959](https://github.com/windmill-labs/windmill/commit/6a6295921d681359155d814507908792be405679))
-* resolve release CI failures (pypi bundle, flow serde test, cli windows) ([#9595](https://github.com/windmill-labs/windmill/issues/9595)) ([5ccaae8](https://github.com/windmill-labs/windmill/commit/5ccaae8ab36f2be18b67863ea069763455908029))
-
-## [1.725.1](https://github.com/windmill-labs/windmill/compare/v1.725.0...v1.725.1) (2026-06-15)
-
-
-### Bug Fixes
-
-* **apps:** apply scope-path predicate to app list/search endpoints ([#9581](https://github.com/windmill-labs/windmill/issues/9581)) ([3bf6e10](https://github.com/windmill-labs/windmill/commit/3bf6e102afbdad41e558617bc812012eaaaecd9b))
-* **auth:** add scope checks to scripts/flows list_tokens endpoints ([#9582](https://github.com/windmill-labs/windmill/issues/9582)) ([36c9f86](https://github.com/windmill-labs/windmill/commit/36c9f8612b5778aa2c981454729590b71671ce8d))
-* **cli:** preserve committed script.lock on transient NULL lock during git-sync deploy ([#9593](https://github.com/windmill-labs/windmill/issues/9593)) ([6b916ac](https://github.com/windmill-labs/windmill/commit/6b916ac688e0305284e6cf819bf28803bcca0118))
-* expose parent_hash in MCP createScript tool for updates ([#9586](https://github.com/windmill-labs/windmill/issues/9586)) ([a69505d](https://github.com/windmill-labs/windmill/commit/a69505df9bf25d7c4f11d0528a7450c08dbb422c))
-* **flows:** stop serializing default retry/stop_after_if fields ([#9583](https://github.com/windmill-labs/windmill/issues/9583)) ([e1e2a24](https://github.com/windmill-labs/windmill/commit/e1e2a24b6a6752b3ac779cb0db38061cbc54425e))
-* **security:** sanitize dependency names & connection strings against command/SQL injection ([#9590](https://github.com/windmill-labs/windmill/issues/9590)) ([aff0a4e](https://github.com/windmill-labs/windmill/commit/aff0a4ec189cd8e315282e878bb858ef00635b90))
-
-## [1.725.0](https://github.com/windmill-labs/windmill/compare/v1.724.0...v1.725.0) (2026-06-15)
-
-
-### Features
-
-* Db-backed user drafts ([#9351](https://github.com/windmill-labs/windmill/issues/9351)) ([1fc3557](https://github.com/windmill-labs/windmill/commit/1fc355709c025fd256c5a4035356e15a5a05b23d))
-* scope AI session storage per user, session list in IndexedDB ([#9518](https://github.com/windmill-labs/windmill/issues/9518)) ([aa26c4d](https://github.com/windmill-labs/windmill/commit/aa26c4d9b22b3a353a6c0605eb9a4193e34aa18c))
-
-
-### Bug Fixes
-
-* **powershell:** sanitize module names to prevent command injection (CWE-78) ([#9587](https://github.com/windmill-labs/windmill/issues/9587)) ([6acce7a](https://github.com/windmill-labs/windmill/commit/6acce7a88733683153db534cb18752b31d93af82))
-
-## [1.724.0](https://github.com/windmill-labs/windmill/compare/v1.723.0...v1.724.0) (2026-06-15)
-
-
-### Features
-
-* **cli:** add --yes, --secret/--no-secret and --description to variable add ([#9548](https://github.com/windmill-labs/windmill/issues/9548)) ([4e9e0c0](https://github.com/windmill-labs/windmill/commit/4e9e0c024b4b95f9676b1646591d8f0c662e84ab))
-* **frontend:** improve AI chat cancel and interrupted-turn handling ([#9539](https://github.com/windmill-labs/windmill/issues/9539)) ([114c412](https://github.com/windmill-labs/windmill/commit/114c41251a8c738b58a1a3dd9434d09d33feb6f1))
-* **frontend:** precise AI chat context usage tracking + indicator ([#9551](https://github.com/windmill-labs/windmill/issues/9551)) ([2b47180](https://github.com/windmill-labs/windmill/commit/2b471805bf1c92bb210cfacda217a4341e1f989c))
-* wire chat reasoning effort through gemini and bedrock proxies ([#9545](https://github.com/windmill-labs/windmill/issues/9545)) ([aaf0563](https://github.com/windmill-labs/windmill/commit/aaf05635cedadc73455dc522474b673719f9fd5c))
-
-
-### Bug Fixes
-
-* actually isolate windows job children from CTRL_BREAK_EVENT + reap on worker death ([#9563](https://github.com/windmill-labs/windmill/issues/9563)) ([61f3291](https://github.com/windmill-labs/windmill/commit/61f3291b240bdb5c26bee8947351a9590bc3bd45))
-* **ai:** enforce resource authz when loading MCP tools in agent worker ([#9571](https://github.com/windmill-labs/windmill/issues/9571)) ([317a862](https://github.com/windmill-labs/windmill/commit/317a8629d1c8436d2a6f3443bd25b81d606ce283))
-* append system CA bundle to tracing proxy cert file ([#9549](https://github.com/windmill-labs/windmill/issues/9549)) ([3cf4083](https://github.com/windmill-labs/windmill/commit/3cf40839602e5c3d1df51f0a29b01736bade09da))
-* **cli:** consistent flow inline lock filenames for compound extensions ([#9555](https://github.com/windmill-labs/windmill/issues/9555)) ([f0659a7](https://github.com/windmill-labs/windmill/commit/f0659a755a161420833e3bfdbe04befc6ebeb977))
-* **flows:** skip_if evaluates wrong previous_result during retry ([#9547](https://github.com/windmill-labs/windmill/issues/9547)) ([2aab352](https://github.com/windmill-labs/windmill/commit/2aab35245c362c2f911c60ea435f29bfb1369ebf))
-* **folders:** allow hyphens in folder names ([#9566](https://github.com/windmill-labs/windmill/issues/9566)) ([84df111](https://github.com/windmill-labs/windmill/commit/84df11177f2009bff007e9b722b55a9a5a63c06a)), closes [#8474](https://github.com/windmill-labs/windmill/issues/8474)
-* **frontend:** load resource value in JSON editor when resource type is missing ([#9574](https://github.com/windmill-labs/windmill/issues/9574)) ([251266c](https://github.com/windmill-labs/windmill/commit/251266cd8119dbef314314daed43aa43ab92f1c9))
-* isolate windows job children from worker CTRL_BREAK_EVENT ([#9562](https://github.com/windmill-labs/windmill/issues/9562)) ([1d6191e](https://github.com/windmill-labs/windmill/commit/1d6191ebb75843917eec6c76a4be347f9ac4cb72))
-* stop sending temperature for AI chat across all providers ([#9553](https://github.com/windmill-labs/windmill/issues/9553)) ([3585716](https://github.com/windmill-labs/windmill/commit/358571687296fbe5c378533b3c1662707955c64a))
-
-## [1.723.0](https://github.com/windmill-labs/windmill/compare/v1.722.0...v1.723.0) (2026-06-11)
-
-
-### Features
-
-* add get_app_runtime_logs tool to global chat ([#9502](https://github.com/windmill-labs/windmill/issues/9502)) ([f86d0d7](https://github.com/windmill-labs/windmill/commit/f86d0d79fc6aa23119fd59761330ab79592d0a2a))
-* **cli:** improve agent prompts/skills and workspace fork workflow ([#9531](https://github.com/windmill-labs/windmill/issues/9531)) ([5bdc4f8](https://github.com/windmill-labs/windmill/commit/5bdc4f83ce37302a2c375d0ff73763acfee2aadb))
-* enable native web search in copilot ([#9522](https://github.com/windmill-labs/windmill/issues/9522)) ([d3f5fe1](https://github.com/windmill-labs/windmill/commit/d3f5fe1c8c39ff07d05f0922b8f40aa95756a707))
-
-
-### Bug Fixes
-
-* **frontend:** stop live activity flickering when user has multiple tabs ([#9543](https://github.com/windmill-labs/windmill/issues/9543)) ([57e627e](https://github.com/windmill-labs/windmill/commit/57e627eabf7c4144ce1c07214ad44d026b82f0b4))
-* omit temperature for claude fable 5 ([#9540](https://github.com/windmill-labs/windmill/issues/9540)) ([bd00bee](https://github.com/windmill-labs/windmill/commit/bd00beeac54dbcfa9ab86fd336fca1a8fa289341))
-* refetch license key from settings when in-memory key is invalid ([#9534](https://github.com/windmill-labs/windmill/issues/9534)) ([38c0ccd](https://github.com/windmill-labs/windmill/commit/38c0ccdf563d3655a4cba390ce9372bc9f9c4a9b))
-
-## [1.722.0](https://github.com/windmill-labs/windmill/compare/v1.721.0...v1.722.0) (2026-06-11)
-
-
-### Features
-
-* add reasoning effort control and thinking display to AI chat ([#9511](https://github.com/windmill-labs/windmill/issues/9511)) ([7f987e8](https://github.com/windmill-labs/windmill/commit/7f987e8c9807b72d9cc3901b6e4d02a24c423f50))
-* **ai-chat:** collapse big pastes, cap input height, escape HTML ([#9487](https://github.com/windmill-labs/windmill/issues/9487)) ([365e204](https://github.com/windmill-labs/windmill/commit/365e20410ed528d5b4e967b64fb282bcc1e03ccd))
-* **ai-chat:** quick access to AI prompt settings from chat ([#9508](https://github.com/windmill-labs/windmill/issues/9508)) ([b894f78](https://github.com/windmill-labs/windmill/commit/b894f783f183ab795eab6f57da8654275d9ad82d))
-* **ai:** add list_runs and get_job_logs tools to global chat mode ([#9488](https://github.com/windmill-labs/windmill/issues/9488)) ([cfe5119](https://github.com/windmill-labs/windmill/commit/cfe51190356a9e922f6398dd875abc368accbd15))
-* clear conflict error + force delete when reusing a fork workspace id ([#9499](https://github.com/windmill-labs/windmill/issues/9499)) ([fddabe9](https://github.com/windmill-labs/windmill/commit/fddabe9c5c6f178b4b09854dc11e50adafd44c87))
-* **flow:** support worker tag override on AI agent steps ([#9513](https://github.com/windmill-labs/windmill/issues/9513)) ([a6a5600](https://github.com/windmill-labs/windmill/commit/a6a5600833063e36b35aac7f90dcbdcd3db7c627))
-* folder-level label inheritance for scripts, flows and jobs ([#9524](https://github.com/windmill-labs/windmill/issues/9524)) ([765f50c](https://github.com/windmill-labs/windmill/commit/765f50c474f8abf76550664ed15418ddb3c0b221))
-* **frontend:** show AI sessions in narrow-screen burger menu ([#9523](https://github.com/windmill-labs/windmill/issues/9523)) ([a8f1062](https://github.com/windmill-labs/windmill/commit/a8f1062f37228e7a8257aac1cd3295bd5084ff90))
-* prefer idle worker pods on k8s autoscaling scale-in via pod-deletion-cost ([#9515](https://github.com/windmill-labs/windmill/issues/9515)) ([3c3f157](https://github.com/windmill-labs/windmill/commit/3c3f15722fd22713cc8baa744d9a81207943a33c))
-* prompt browser confirmation on page exit with unsaved changes ([#9503](https://github.com/windmill-labs/windmill/issues/9503)) ([3119e16](https://github.com/windmill-labs/windmill/commit/3119e16ed8df0daec0e019ae2efb2176d7e9e953))
-* **worker:** #ssh directive to run a bash script on a remote SSH host ([#9479](https://github.com/windmill-labs/windmill/issues/9479)) ([afddfe8](https://github.com/windmill-labs/windmill/commit/afddfe84452357b06f4fb815566d4c8269ceafbf))
-* workspace protection rule to restrict anonymous app deployment ([#9509](https://github.com/windmill-labs/windmill/issues/9509)) ([cf9ad54](https://github.com/windmill-labs/windmill/commit/cf9ad54181d38c5d3c3aa05208e17d8d97eae4ef))
-
-
-### Bug Fixes
-
-* **cli:** include lock-relevant script content in lock cache key ([#9528](https://github.com/windmill-labs/windmill/issues/9528)) ([dc60e1a](https://github.com/windmill-labs/windmill/commit/dc60e1aa174f2a7616d2d4a37117a463ef25a9d6))
-* **frontend:** allow copy/paste shortcuts inside ConfirmationModal ([#9505](https://github.com/windmill-labs/windmill/issues/9505)) ([7fc5340](https://github.com/windmill-labs/windmill/commit/7fc5340da39c049d059aa0ae15ceb6a98ca58053))
-* **frontend:** clarify trigger filters match the message parsed as JSON ([#9516](https://github.com/windmill-labs/windmill/issues/9516)) ([0b17843](https://github.com/windmill-labs/windmill/commit/0b178437ce8cd8295be21c0c6077c39e957a3337))
-* **frontend:** enable Apply button when env vars change in worker group config ([#9501](https://github.com/windmill-labs/windmill/issues/9501)) ([c80c6d8](https://github.com/windmill-labs/windmill/commit/c80c6d8fcdfc1de8fe841e9699452f67a25cad23))
-* **frontend:** improve AI chat markdown and typing dots in dark mode ([#9497](https://github.com/windmill-labs/windmill/issues/9497)) ([4e86806](https://github.com/windmill-labs/windmill/commit/4e868062d4c22c5574149d35235b7ad1abe805f3))
-* **frontend:** stop echoing draft values in global chat write tool results ([#9530](https://github.com/windmill-labs/windmill/issues/9530)) ([ce6e2f7](https://github.com/windmill-labs/windmill/commit/ce6e2f7ade25ca91c375f0de5f1be3f1c82ccb47))
-* inherit container NO_PROXY into MITM tracing proxy job exclusions ([#9492](https://github.com/windmill-labs/windmill/issues/9492)) ([4c22e3b](https://github.com/windmill-labs/windmill/commit/4c22e3b712a74828cf654ea7d89aeab5b50cfbd7))
-* make default chat model optional in AI settings ([#9514](https://github.com/windmill-labs/windmill/issues/9514)) ([1d43288](https://github.com/windmill-labs/windmill/commit/1d4328877fcc87352fb56de63f0570739d5a9dc4))
-* **nsjail:** make ansible collections mount non-mandatory ([#9510](https://github.com/windmill-labs/windmill/issues/9510)) ([08da7a1](https://github.com/windmill-labs/windmill/commit/08da7a121b4b835500dfc2bd943c4bdce912c63e))
-* **nsjail:** make ansible uv tools mount non-mandatory ([#9507](https://github.com/windmill-labs/windmill/issues/9507)) ([dc368a9](https://github.com/windmill-labs/windmill/commit/dc368a9669e812c593ad848266f645b6223501e6))
-
-## [1.721.0](https://github.com/windmill-labs/windmill/compare/v1.720.0...v1.721.0) (2026-06-09)
-
-
-### Features
-
-* deployed↔draft compare + AI-session draft bar ([#9435](https://github.com/windmill-labs/windmill/issues/9435)) ([b0b330c](https://github.com/windmill-labs/windmill/commit/b0b330c7864d0159af4b0f17dbb3c09bd015145b))
-
-
-### Bug Fixes
-
-* **cli:** reconcile case-only path drift during sync on case-insensitive filesystems (WIN-2020) ([#9485](https://github.com/windmill-labs/windmill/issues/9485)) ([c258928](https://github.com/windmill-labs/windmill/commit/c258928ab62adc1327913c21556520dfd1e5c24c))
-* drop archived items from fork compare (spurious 'not visible' warning) ([#9481](https://github.com/windmill-labs/windmill/issues/9481)) ([92c21bb](https://github.com/windmill-labs/windmill/commit/92c21bbe6586f3c285796a98692e976515a629d5))
-* require auth to view approval details when user_auth_required ([#9482](https://github.com/windmill-labs/windmill/issues/9482)) ([5f41ddd](https://github.com/windmill-labs/windmill/commit/5f41ddd3a592bcd504f94fc99060ca5d79c36190))
-
-## [1.720.0](https://github.com/windmill-labs/windmill/compare/v1.719.0...v1.720.0) (2026-06-08)
-
-
-### Features
-
-* allow private MCP server URLs ([#9470](https://github.com/windmill-labs/windmill/issues/9470)) ([3bc5800](https://github.com/windmill-labs/windmill/commit/3bc5800197db383ae6f708415701a4bdbe2e3345))
-* **api:** add endpoint to update token label ([#9474](https://github.com/windmill-labs/windmill/issues/9474)) ([e8e0701](https://github.com/windmill-labs/windmill/commit/e8e0701a360d0614c4c5a74f6410ba6ac0638caa))
-* **frontend:** use unified drill picker for AI chat @-mention dropdown ([#9159](https://github.com/windmill-labs/windmill/issues/9159)) ([64b089c](https://github.com/windmill-labs/windmill/commit/64b089cd23cca4601abb09f092a32becb80d9394))
-
-
-### Bug Fixes
-
-* center auth0/okta icons and respect currentColor ([#9457](https://github.com/windmill-labs/windmill/issues/9457)) ([5d0ef7d](https://github.com/windmill-labs/windmill/commit/5d0ef7dfd91b3021d125a1b34f81f0788f173786))
-* **forks:** keep trigger/schedule operational state owned by the parent - WIN-2019 ([#9476](https://github.com/windmill-labs/windmill/issues/9476)) ([192574a](https://github.com/windmill-labs/windmill/commit/192574ab8f98d9521a232fc8a4935d407b00cb3a))
-* **frontend:** respect forced column order for numeric column names ([#9463](https://github.com/windmill-labs/windmill/issues/9463)) ([44f5dd6](https://github.com/windmill-labs/windmill/commit/44f5dd6636d4b23aa55383b8b8abe4c2f73bc88d))
-* **frontend:** use ban icon for canceled jobs instead of hourglass ([#9478](https://github.com/windmill-labs/windmill/issues/9478)) ([fa86c62](https://github.com/windmill-labs/windmill/commit/fa86c62b6600e7d47dadf4706d7002706333d919))
-* gate native integration pickers behind non-operator check ([#9465](https://github.com/windmill-labs/windmill/issues/9465)) ([6156e23](https://github.com/windmill-labs/windmill/commit/6156e2372a785ccd0c6f29cb74e90bee69e76483))
-* **oauth:** persist refreshed token through configured secret backend ([#9471](https://github.com/windmill-labs/windmill/issues/9471)) ([76c0d97](https://github.com/windmill-labs/windmill/commit/76c0d970a18bf28ddc48dd746570e73486542606))
-* refresh session editor preview on breadcrumb target switch ([#9475](https://github.com/windmill-labs/windmill/issues/9475)) ([6d522b3](https://github.com/windmill-labs/windmill/commit/6d522b3989ace1f214bd049d910bc0d2a2a6893e))
-
-## [1.719.0](https://github.com/windmill-labs/windmill/compare/v1.718.0...v1.719.0) (2026-06-06)
-
-
-### Features
-
-* **otel:** connect jobs to the inbound distributed trace ([#9456](https://github.com/windmill-labs/windmill/issues/9456)) ([fad1a54](https://github.com/windmill-labs/windmill/commit/fad1a549d95c00d0746a48163c4f95fc69733e1a))
-
-
-### Bug Fixes
-
-* authenticate slack callback payload with per-workspace hmac ([#9461](https://github.com/windmill-labs/windmill/issues/9461)) ([fbdf81b](https://github.com/windmill-labs/windmill/commit/fbdf81ba5f77d282c025360ecee14138dd4cb4a2))
-* prevent token label collision bypassing job read access control ([#9462](https://github.com/windmill-labs/windmill/issues/9462)) ([e1e7af6](https://github.com/windmill-labs/windmill/commit/e1e7af6a25a44eb06b67332ce1efeae2a21e0c6d))
-* **python:** escape reserved-keyword step ids in wrapper codegen ([#9460](https://github.com/windmill-labs/windmill/issues/9460)) ([6a15a9b](https://github.com/windmill-labs/windmill/commit/6a15a9b152ad20be4b5c3de6000516da231e41e0)), closes [#8893](https://github.com/windmill-labs/windmill/issues/8893)
-
-## [1.718.0](https://github.com/windmill-labs/windmill/compare/v1.717.1...v1.718.0) (2026-06-05)
-
-
-### Features
-
-* **flows:** opt-in to include the stopping step's result in early-stop errors ([#9446](https://github.com/windmill-labs/windmill/issues/9446)) ([f2f0812](https://github.com/windmill-labs/windmill/commit/f2f0812a04c9256cfc8eba5e0dcf38d71d971410))
-* make C# dotnet target framework configurable via DOTNET_TARGET_FRAMEWORK ([#9454](https://github.com/windmill-labs/windmill/issues/9454)) ([9a609bf](https://github.com/windmill-labs/windmill/commit/9a609bf08ac1b6157dbdfb827fc01e771d71262e))
-* sandboxed daemonless container runtime via '# sandbox <image>' ([#9453](https://github.com/windmill-labs/windmill/issues/9453)) ([1727271](https://github.com/windmill-labs/windmill/commit/1727271e197b34026efeaf1b6561bb404a440baa))
-* **sandbox:** pull/extract images with crane instead of podman ([#9455](https://github.com/windmill-labs/windmill/issues/9455)) ([7590b28](https://github.com/windmill-labs/windmill/commit/7590b281085afd1fc2774e8fb37a4c0af3aedbad))
-
-
-### Bug Fixes
-
-* distinguish canceled jobs in runs ([#9452](https://github.com/windmill-labs/windmill/issues/9452)) ([9067787](https://github.com/windmill-labs/windmill/commit/90677872f6185eb0c81e0e84a426a54653818457))
-
-## [1.717.1](https://github.com/windmill-labs/windmill/compare/v1.717.0...v1.717.1) (2026-06-04)
-
-
-### Bug Fixes
-
-* invalidate relative-import cache when imported script changes ([#9443](https://github.com/windmill-labs/windmill/issues/9443)) ([f595787](https://github.com/windmill-labs/windmill/commit/f595787409a3fcda9278bbcf2cfcc80092f16460))
-
-## [1.717.0](https://github.com/windmill-labs/windmill/compare/v1.716.0...v1.717.0) (2026-06-04)
-
-
-### Features
-
-* let flow AI chat create and edit sticky notes ([#9412](https://github.com/windmill-labs/windmill/issues/9412)) ([e4e0984](https://github.com/windmill-labs/windmill/commit/e4e0984e55afd3c73f1c365cd0608493a9fd87ed))
-
-
-### Bug Fixes
-
-* **cli:** push whole raw app instead of treating frontend files as scripts ([#9442](https://github.com/windmill-labs/windmill/issues/9442)) ([b5a6a1e](https://github.com/windmill-labs/windmill/commit/b5a6a1eeab663c2d6aaec2c89eab7a550cb0bb6b))
-* read latest db draft for scripts/flows in global mode read tool ([#9441](https://github.com/windmill-labs/windmill/issues/9441)) ([819ba5e](https://github.com/windmill-labs/windmill/commit/819ba5e150ec9f5199919fbea50874fc156d0189))
-
-## [1.716.0](https://github.com/windmill-labs/windmill/compare/v1.715.0...v1.716.0) (2026-06-03)
-
-
-### Features
-
-* add metadata generation model setting ([#9418](https://github.com/windmill-labs/windmill/issues/9418)) ([cf5fefb](https://github.com/windmill-labs/windmill/commit/cf5fefb521479170b9dc64b884630c4dac789931))
-* auto-generate AI session names ([#9399](https://github.com/windmill-labs/windmill/issues/9399)) ([26b7270](https://github.com/windmill-labs/windmill/commit/26b727041830c9b741668a9ab73e2eb90c7cec74))
-* support $f/ and $u/ import path aliases for scripts ([#9378](https://github.com/windmill-labs/windmill/issues/9378)) ([220cd35](https://github.com/windmill-labs/windmill/commit/220cd35cf799c42ebf588bc97a6d8e6f4e97c2e3))
-* use metadata model for small AI tasks ([#9431](https://github.com/windmill-labs/windmill/issues/9431)) ([79178f6](https://github.com/windmill-labs/windmill/commit/79178f6f5a7c606a2e05677c6efcbdd84c608325))
-
-
-### Bug Fixes
-
-* **apps:** relock no longer reverts raw app to a stale version ([#9432](https://github.com/windmill-labs/windmill/issues/9432)) ([073857a](https://github.com/windmill-labs/windmill/commit/073857ac0a9ed54bdeac8f373f7c855fe34eb0ac))
-* **security:** scope variable and resource value caches by caller identity ([#9427](https://github.com/windmill-labs/windmill/issues/9427)) ([0ba128a](https://github.com/windmill-labs/windmill/commit/0ba128afe797bd016da60563949ac3abbbfe1978))
-
-## [1.715.0](https://github.com/windmill-labs/windmill/compare/v1.714.1...v1.715.0) (2026-06-03)
-
-
-### Features
-
-* **frontend:** add rebuild dependency map button to workspace settings ([#9424](https://github.com/windmill-labs/windmill/issues/9424)) ([3b2e748](https://github.com/windmill-labs/windmill/commit/3b2e748daf0a8ec4447c30423068df803f3f9ca2))
-
-
-### Bug Fixes
-
-* **auth:** filter script/flow listings by token scope (GHSA-2ppx-66jv-wpw5) ([#9426](https://github.com/windmill-labs/windmill/issues/9426)) ([7edf3f0](https://github.com/windmill-labs/windmill/commit/7edf3f02122e20fde1e95e0252e7bda641075326))
-* **backend:** authorize single-job read endpoints by job/flow visibility ([#9416](https://github.com/windmill-labs/windmill/issues/9416)) ([89a7a37](https://github.com/windmill-labs/windmill/commit/89a7a377764086911db18252f2478f42f0e1e3ea))
-* **mcp:** resolve MCP resource token via caller RLS + SSRF-guard url ([#9428](https://github.com/windmill-labs/windmill/issues/9428)) ([8053266](https://github.com/windmill-labs/windmill/commit/8053266f88bd4c94fc86278412df5a0beeed5e77))
-* **nsjail:** precompile python stdlib + raise download rlimit_as ([#9429](https://github.com/windmill-labs/windmill/issues/9429)) ([7031744](https://github.com/windmill-labs/windmill/commit/7031744a199f0bf8b8e35043afa959977e5ecdbd))
-* omit temperature for gpt-5+ and o-series models on all providers ([#9422](https://github.com/windmill-labs/windmill/issues/9422)) ([11d1ad9](https://github.com/windmill-labs/windmill/commit/11d1ad9a872d2ec2f14cde35708c84a0c7bdc172))
-
 ## [1.714.1](https://github.com/windmill-labs/windmill/compare/v1.714.0...v1.714.1) (2026-06-02)
 
 

Dockerfile

@@ -233,14 +233,11 @@ ENV PATH="${PATH}:/usr/local/go/bin"
 ENV GO_PATH=/usr/local/go/bin/go
 
 # Install UV
-RUN curl --proto '=https' --tlsv1.2 -LsSf https://github.com/astral-sh/uv/releases/download/0.9.25/uv-installer.sh | sh && mv /root/.local/bin/uv /usr/local/bin/uv
+RUN curl --proto '=https' --tlsv1.2 -LsSf https://github.com/astral-sh/uv/releases/download/0.9.24/uv-installer.sh | sh && mv /root/.local/bin/uv /usr/local/bin/uv
 
 # Preinstall python runtimes to temp build location (will copy with world-writable perms later)
-# --compile-bytecode precompiles the stdlib to .pyc so jobs don't recompile it on every run
-# under the read-only nsjail runtime mount (uv >= 0.9.25). The copy below MUST preserve
-# timestamps or Python's mtime-based .pyc invalidation discards these compiled files.
-RUN UV_CACHE_DIR=/tmp/build_cache/uv UV_PYTHON_INSTALL_DIR=/tmp/build_cache/py_runtime uv python install 3.11 --compile-bytecode
-RUN UV_CACHE_DIR=/tmp/build_cache/uv UV_PYTHON_INSTALL_DIR=/tmp/build_cache/py_runtime uv python install $LATEST_STABLE_PY --compile-bytecode
+RUN UV_CACHE_DIR=/tmp/build_cache/uv UV_PYTHON_INSTALL_DIR=/tmp/build_cache/py_runtime uv python install 3.11
+RUN UV_CACHE_DIR=/tmp/build_cache/uv UV_PYTHON_INSTALL_DIR=/tmp/build_cache/py_runtime uv python install $LATEST_STABLE_PY
 
 
 RUN curl -sL https://deb.nodesource.com/setup_20.x | bash -
@@ -262,7 +259,7 @@ RUN export GOCACHE=/tmp/build_cache/go && \
 # chmod a+rw adds read+write WITHOUT removing execute bits (755->777, 644->666)
 # Note: uv python install only creates py_runtime, not uv cache - we create uv/go dirs for runtime
 RUN mkdir -p /tmp/windmill/cache && \
-    cp -r --preserve=timestamps /tmp/build_cache/* /tmp/windmill/cache/ && \
+    cp -r /tmp/build_cache/* /tmp/windmill/cache/ && \
     chmod -R a+rw /tmp/windmill/cache && \
     rm -rf /tmp/build_cache && \
     mkdir -p -m 777 /tmp/windmill/cache/uv /tmp/windmill/cache/go /tmp/windmill/cache/rustup /tmp/windmill/cache/cargo
@@ -307,16 +304,6 @@ RUN apt-get update && apt-get install -y --no-install-recommends libprotobuf32 l
     && apt-get clean && rm -rf /var/lib/apt/lists/*
 COPY --from=nsjail /nsjail/nsjail /bin/nsjail
 
-# crane: pulls + flattens images for the sandboxed container runtime (`# sandbox <image>`).
-# Single static binary — no daemon/store/root needed. See docs/docker-v2-runtime.md.
-ARG CRANE_VERSION=v0.20.6
-RUN arch="$(dpkg --print-architecture)"; \
-    case "$arch" in amd64) crane_arch=x86_64 ;; arm64) crane_arch=arm64 ;; *) echo >&2 "error: unsupported arch '$arch' for crane"; exit 1 ;; esac; \
-    wget -O /tmp/crane.tgz "https://github.com/google/go-containerregistry/releases/download/${CRANE_VERSION}/go-containerregistry_Linux_${crane_arch}.tar.gz" \
-    && tar -xzf /tmp/crane.tgz -C /usr/local/bin crane \
-    && rm /tmp/crane.tgz \
-    && chmod +x /usr/local/bin/crane
-
 WORKDIR ${APP}
 
 RUN ln -s ${APP}/windmill /usr/local/bin/windmill

ai_evals/AGENTS.md

@@ -86,31 +86,6 @@ Global prompts should exercise workspace-level drafting behavior:
 
 Keep deterministic validation focused on the draft contract: required draft type/path, required content snippets, forbidden draft paths, and forbidden mutating tools such as deploy/delete unless the case explicitly asks for them.
 
-Datatable cases should set `skipJudge: true` and validate through tool-use
-(`requiredToolsUsed` / `forbiddenToolsUsed`) and SQL-argument assertions
-(`toolCallArgs` with `stringIncludesAnyOf`, e.g. `['select']`, `['create table']`,
-`['update', 'insert into']`). Two reasons the judge is unreliable here:
-
-- `list_datatables`, `get_datatable_table_schema`, and `exec_datatable_sql`
-  produce no drafts, and the global judge only sees the drafts artifact — it
-  scores a no-draft conversational answer as empty (same as the
-  `askUserQuestion` cases).
-- Even a case that *does* produce a draft (a script reading the data table via
-  `wmill.datatable()` at runtime) is mis-judged: the judge has no datatable SDK
-  reference and penalizes correct `wmill.datatable()` usage as wrong. Verify the
-  SDK call deterministically instead — `requiredDrafts.valueIncludes: ['wmill.datatable(']`
-  plus forbidding `exec_datatable_sql` (keeping chat-time SQL distinct from
-  runtime SDK use).
-
-`stringIncludesAnyOf` is existential over calls (at least one matching call), so a
-mutation case still passes when the model mixes its UPDATE/INSERT with
-verification SELECTs. The in-memory engine (`datatableSqlEngine.ts`) is stateful
-within a case — writes persist, so a model that re-queries to verify its
-CREATE/UPDATE sees the change and does not loop. But the engine is best-effort
-(SELECT returns all rows of the referenced/first table with no WHERE/projection),
-so still never assert specific returned row values. Seed data via
-`workspace.datatables` in the `initial` fixture (see README).
-
 ## Deterministic validation
 
 Use deterministic validation only for hard failures such as:

ai_evals/README.md

@@ -75,8 +75,6 @@ Public CLI surface:
 - `--model <alias>`: choose the model under test
 - `--models <a,b,c>`: run the same cases sequentially against several model aliases
 - `--verbose`: stream assistant output for frontend runs
-- `--skip-judge`: skip LLM judge scoring for the run
-- `--execution-only`: only require the model/proxy/frontend loop to complete; skip validators, tool expectations, backend artifact validation, and judge scoring
 - `--record`: append a compact tracked summary line to `ai_evals/history/<mode>.jsonl` for full-suite runs only
 - `--backend-validation <mode>`: optional backend smoke validation (`off` or `preview`) for `script` and `flow` evals
 
@@ -101,7 +99,7 @@ Notes:
 - the command also prints accepted alias spellings such as `gpt-4o`, `gpt-55`, `claude-opus-4.6`, and `claude-haiku-4.5`
 - frontend modes (`flow`, `script`, `app`, `global`) can use Anthropic, OpenAI, Gemini, and DeepSeek-backed aliases
 - `cli` mode always uses the Anthropic agent SDK, so only Anthropic aliases are valid there
-- the judge model is separate and currently defaults to `claude-sonnet-4-6`; use `--skip-judge` for deterministic-only runs
+- the judge model is separate and currently defaults to `claude-sonnet-4-6`
 
 ## Case Format
 
@@ -150,23 +148,6 @@ Global initial fixtures can also seed `liveEditorDrafts` with `type`,
 currently open script, flow, or raw app editor so cases can test prompts that
 refer to "this" or the "current" item.
 
-Global (and flow) initial fixtures can seed `workspace.datatables` so the
-`list_datatables`, `get_datatable_table_schema`, and `exec_datatable_sql` tools
-return seeded data during evals. Each entry is
-`{ datatable_name, schemas: { <schema>: { <table>: { columns, rows? } } } }`.
-SQL runs through a small in-memory engine (`datatableSqlEngine.ts`), not a real
-database. Writes are **stateful within a case**: `CREATE`/`DROP`/`INSERT`/`UPDATE`/
-`DELETE` mutate the seeded datatable in place, so a later `list_datatables`,
-`get_datatable_table_schema`, `SELECT`, or `information_schema` query reflects them
-— this is what stops a model from looping when it re-queries to verify a write.
-The engine is best-effort: `SELECT` returns all rows of the referenced (or first)
-table with no WHERE filtering/projection/joins, `WHERE` on UPDATE/DELETE supports
-`col = value` predicates joined by `AND`, and anything unparseable is a no-op
-success. So validate datatable cases through tool-use and SQL-argument assertions
-(`requiredToolsUsed`, `stringIncludesAnyOf`) — not through exact returned row
-values. An empty/absent `datatables` seed makes `list_datatables` return `[]`,
-which is what the "no datatable configured" blocking cases rely on.
-
 Set `WMILL_AI_EVAL_DISABLE_ACTIVE_EDITOR_CONTEXT=1` to run those cases with
 the old behavior where the live editor is only discoverable through
 `list_workspace_items`.

ai_evals/adapters/cli/runtime.ts

@@ -16,9 +16,6 @@ export interface PromptRunResult {
   output: string;
   durationMs: number;
   tokenUsage: BenchmarkTokenUsage | null;
-  // Input tokens on the last assistant turn. The SDK `result` message reports
-  // usage cumulatively, so the final context size comes from per-turn usage.
-  finalContextTokens: number | null;
   trace: CliTrace;
 }
 
@@ -147,7 +144,6 @@ export async function runPromptAndCapture(
   let output = "";
   let assistantMessageCount = 0;
   let tokenUsage: BenchmarkTokenUsage | null = null;
-  let finalContextTokens: number | null = null;
   const startedAt = Date.now();
   const stubBinDir = join(cwd, WMILL_STUB_DIR_NAME);
   const wmillLogPath = join(cwd, WMILL_LOG_FILE_NAME);
@@ -170,12 +166,6 @@ export async function runPromptAndCapture(
   for await (const message of query({ prompt, options })) {
     if (message.type === "assistant") {
       assistantMessageCount += 1;
-      const turnContext = anthropicUsageToBenchmarkTokenUsage(
-        message.message?.usage
-      )?.prompt;
-      if (turnContext && turnContext > 0) {
-        finalContextTokens = turnContext;
-      }
       const content = message.message?.content;
       if (Array.isArray(content)) {
         for (const block of content) {
@@ -220,7 +210,6 @@ export async function runPromptAndCapture(
     output,
     durationMs: Date.now() - startedAt,
     tokenUsage,
-    finalContextTokens,
     trace: {
       toolsUsed,
       skillsInvoked,

ai_evals/adapters/frontend/benchmarkRunner.ts

@@ -25,12 +25,6 @@ export async function runFrontendBenchmarkFromEnv(): Promise<BenchmarkRunResult>
   );
   const emitProgress = process.env.WMILL_FRONTEND_AI_EVAL_PROGRESS === "1";
   const verbose = process.env.WMILL_FRONTEND_AI_EVAL_VERBOSE === "1";
-  const executionOnly =
-    process.env.WMILL_FRONTEND_AI_EVAL_EXECUTION_ONLY === "1";
-  const judgeModel =
-    process.env.WMILL_FRONTEND_AI_EVAL_SKIP_JUDGE === "1" || executionOnly
-      ? null
-      : DEFAULT_JUDGE_MODEL;
   const model = resolveEvalModel(
     mode,
     process.env.WMILL_FRONTEND_AI_EVAL_MODEL,
@@ -54,8 +48,7 @@ export async function runFrontendBenchmarkFromEnv(): Promise<BenchmarkRunResult>
     cases: selectedCases,
     runs,
     runModel,
-    judgeModel,
-    executionOnly,
+    judgeModel: DEFAULT_JUDGE_MODEL,
     concurrency: verbose ? 1 : undefined,
     verbose,
     onProgress: emitProgress
@@ -67,7 +60,7 @@ export async function runFrontendBenchmarkFromEnv(): Promise<BenchmarkRunResult>
     mode,
     runs,
     runModel,
-    judgeModel,
+    judgeModel: DEFAULT_JUDGE_MODEL,
     caseResults,
   });
 }
@@ -103,12 +96,7 @@ async function getModeRunner(
 }
 
 function parseMode(value: string | undefined): FrontendBenchmarkMode {
-  if (
-    value === "flow" ||
-    value === "app" ||
-    value === "script" ||
-    value === "global"
-  ) {
+  if (value === "flow" || value === "app" || value === "script" || value === "global") {
     return value;
   }
   throw new Error(`Unsupported frontend benchmark mode: ${String(value)}`);

ai_evals/adapters/frontend/core/app/appEvalRunner.ts

@@ -38,7 +38,6 @@ export interface AppEvalResult {
   toolCallCount: number;
   toolsUsed: string[];
   tokenUsage: TokenUsage;
-  finalContextTokens: number | null;
 }
 
 export interface AppEvalOptions {
@@ -114,7 +113,6 @@ export async function runAppEval(
       toolCallCount: rawResult.toolCallsCount,
       toolsUsed: rawResult.toolsCalled,
       tokenUsage: rawResult.tokenUsage,
-      finalContextTokens: rawResult.finalContextTokens,
     };
   } finally {
     await cleanup();

ai_evals/adapters/frontend/core/flow/flowEvalRunner.ts

@@ -39,7 +39,6 @@ export interface FlowEvalResult {
   toolsUsed: string[];
   toolCallDetails: ToolCallDetail[];
   tokenUsage: TokenUsage;
-  finalContextTokens: number | null;
 }
 
 export interface FlowEvalOptions {
@@ -114,7 +113,6 @@ export async function runFlowEval(
       toolsUsed: rawResult.toolsCalled,
       toolCallDetails: rawResult.toolCallDetails,
       tokenUsage: rawResult.tokenUsage,
-      finalContextTokens: rawResult.finalContextTokens,
     };
   } finally {
     await cleanup();

ai_evals/adapters/frontend/core/global/globalEvalRunner.ts

@@ -9,7 +9,6 @@ import {
 } from "../../../../../frontend/src/lib/components/copilot/chat/global/core";
 import {
   clearGlobalDrafts,
-  getGlobalDraft,
   listGlobalDrafts,
 } from "../../../../../frontend/src/lib/components/copilot/chat/global/userDraftAdapter";
 import type { Tool as ProductionTool } from "../../../../../frontend/src/lib/components/copilot/chat/shared";
@@ -19,7 +18,6 @@ import type { GlobalDraftState } from "../../../../core/validators";
 import type { WindmillBackendSettings } from "../../../../core/windmillBackendSettings";
 import {
   registerBenchmarkWorkspaceRunnables,
-  seedBenchmarkDraft,
   unregisterBenchmarkWorkspaceRunnables,
   type BenchmarkWorkspaceRunnables,
 } from "../../mockBackend";
@@ -32,10 +30,6 @@ const MUTATING_GLOBAL_TOOLS = new Set([
 ]);
 const DISABLE_ACTIVE_EDITOR_CONTEXT_ENV =
   "WMILL_AI_EVAL_DISABLE_ACTIVE_EDITOR_CONTEXT";
-// A/B gate for the search_app read tool: set to "1" to run the baseline arm
-// (toolset without search_app) so its token cost can be compared against the arm
-// that offers it.
-const DISABLE_SEARCH_APP_ENV = "WMILL_AI_EVAL_DISABLE_SEARCH_APP";
 
 const LIVE_EDITOR_ITEM_KINDS = {
   script: "script",
@@ -50,20 +44,6 @@ export interface GlobalLiveEditorDraftFixture {
   value?: unknown;
 }
 
-// Identity the global system prompt builds paths from. Production reads
-// `userStore` (whoami) to fill `u/{username}/...`; the eval harness never logs
-// in, so without this the prompt sees an empty username (`u//...`) and no
-// path-selection case is meaningful. Seeded per-case via the initial fixture and
-// passed straight to `prepareGlobalSystemMessage` (no global-store mutation).
-export interface GlobalUserFixture {
-  username: string;
-  is_admin?: boolean;
-  /** Folders the user can write to (the writable set whoami returns). */
-  folders?: string[];
-  /** Folders the user can read; read-only folders = folders_read \ folders. */
-  folders_read?: string[];
-}
-
 export interface GlobalEvalResult {
   success: boolean;
   state: GlobalDraftState;
@@ -73,13 +53,11 @@ export interface GlobalEvalResult {
   toolsUsed: string[];
   toolCallDetails: ToolCallDetail[];
   tokenUsage: TokenUsage;
-  finalContextTokens: number | null;
 }
 
 export interface GlobalEvalOptions {
   workspaceFixtures?: BenchmarkWorkspaceRunnables;
   liveEditorDrafts?: GlobalLiveEditorDraftFixture[];
-  user?: GlobalUserFixture;
   model?: string;
   maxIterations?: number;
   provider?: AIProvider;
@@ -105,11 +83,9 @@ export async function runGlobalEval(
     const model = options.model ?? "claude-haiku-4-5-20251001";
     const injectActiveEditorContext =
       process.env[DISABLE_ACTIVE_EDITOR_CONTEXT_ENV] !== "1";
-    // Pass the seeded identity straight to the prompt builder rather than mutating
-    // the process-global `userStore`, so concurrent cases never race on it.
     const rawResult = await runEval({
       userPrompt,
-      systemMessage: prepareGlobalSystemMessage(undefined, { user: options.user }),
+      systemMessage: prepareGlobalSystemMessage(),
       userMessage: prepareGlobalUserMessage(
         userPrompt,
         [],
@@ -118,7 +94,7 @@ export async function runGlobalEval(
       tools: getGlobalEvalTools(),
       helpers: {},
       apiKey,
-      getOutput: () => collectGlobalDraftState(workspaceRoot),
+      getOutput: () => ({ drafts: listGlobalDrafts(workspaceRoot) }),
       onAssistantMessageStart: options.runContext?.onAssistantMessageStart,
       onAssistantToken: options.runContext?.onAssistantChunk,
       onAssistantMessageEnd: options.runContext?.onAssistantMessageEnd,
@@ -143,7 +119,6 @@ export async function runGlobalEval(
       toolsUsed: rawResult.toolsCalled,
       toolCallDetails: rawResult.toolCallDetails,
       tokenUsage: rawResult.tokenUsage,
-      finalContextTokens: rawResult.finalContextTokens,
     };
   } finally {
     clearGlobalDrafts(workspaceRoot);
@@ -155,32 +130,6 @@ export async function runGlobalEval(
   }
 }
 
-// Build the harness output from the DB-backed drafts. `listGlobalDrafts` returns
-// metadata-only rows for backend drafts (the model's `write_script` etc. persist
-// straight to the backend with no in-tab editor cell), so re-read each such row
-// with `getGlobalDraft` to attach the full value the validators assert on. A row
-// that already carries a value (the production in-tab cell overlay) is kept as-is.
-async function collectGlobalDraftState(
-  workspace: string,
-): Promise<GlobalDraftState> {
-  const items = await listGlobalDrafts(workspace);
-  const drafts = await Promise.all(
-    items.map(async (item) => {
-      if (item.value !== undefined) {
-        return item;
-      }
-      const full = await getGlobalDraft(
-        workspace,
-        item.type,
-        item.path,
-        item.triggerKind,
-      );
-      return full ?? item;
-    }),
-  );
-  return { drafts: drafts as GlobalDraftState["drafts"] };
-}
-
 function seedLiveEditorDrafts(
   workspace: string,
   fixtures: GlobalLiveEditorDraftFixture[],
@@ -189,9 +138,7 @@ function seedLiveEditorDrafts(
     const itemKind = LIVE_EDITOR_ITEM_KINDS[fixture.type];
     const storagePath = fixture.storagePath ?? fixture.effectivePath ?? "";
     if (fixture.value !== undefined) {
-      // Seed as a backend draft row, not an in-tab cell: a cell would shadow the
-      // model's DB-backed edit when the output is read back via listGlobalDrafts.
-      seedBenchmarkDraft(workspace, itemKind, storagePath, fixture.value);
+      UserDraft.save(itemKind, storagePath, fixture.value, { workspace });
     }
     UserDraft.setLiveEditorDraft({
       workspace,
@@ -214,28 +161,25 @@ function clearLiveEditorDrafts(
 }
 
 function getGlobalEvalTools(): ProductionTool<{}>[] {
-  const disableSearchApp = process.env[DISABLE_SEARCH_APP_ENV] === "1";
-  return (globalTools as ProductionTool<{}>[])
-    .filter((tool) => !(disableSearchApp && tool.def.function.name === "search_app"))
-    .map((tool) => {
-      if (!MUTATING_GLOBAL_TOOLS.has(tool.def.function.name)) {
-        return tool;
-      }
+  return (globalTools as ProductionTool<{}>[]).map((tool) => {
+    if (!MUTATING_GLOBAL_TOOLS.has(tool.def.function.name)) {
+      return tool;
+    }
 
-      return {
-        ...tool,
-        requiresConfirmation: false,
-        validateBeforeConfirmation: undefined,
-        fn: async () =>
-          JSON.stringify(
-            {
-              success: false,
-              error:
-                "This mutating workspace tool is disabled during ai_evals global mode.",
-            },
-            null,
-            2,
-          ),
-      };
-    });
+    return {
+      ...tool,
+      requiresConfirmation: false,
+      validateBeforeConfirmation: undefined,
+      fn: async () =>
+        JSON.stringify(
+          {
+            success: false,
+            error:
+              "This mutating workspace tool is disabled during ai_evals global mode.",
+          },
+          null,
+          2,
+        ),
+    };
+  });
 }

ai_evals/adapters/frontend/core/script/scriptEvalRunner.ts

@@ -25,7 +25,6 @@ export interface ScriptEvalResult {
   toolsUsed: string[];
   toolCallDetails: ToolCallDetail[];
   tokenUsage: TokenUsage;
-  finalContextTokens: number | null;
 }
 
 export interface ScriptEvalOptions {
@@ -112,7 +111,6 @@ export async function runScriptEval(
       toolsUsed: rawResult.toolsCalled,
       toolCallDetails: rawResult.toolCallDetails,
       tokenUsage: rawResult.tokenUsage,
-      finalContextTokens: rawResult.finalContextTokens,
     };
   } finally {
     await cleanup();

ai_evals/adapters/frontend/core/shared/baseEvalRunner.ts

@@ -38,9 +38,8 @@ export interface RunEvalParams<THelpers, TOutput> {
   helpers: THelpers;
   /** API key for the provider */
   apiKey: string;
-  /** Function to get the current output state. May be async — global mode reads
-   * DB-backed drafts back through the (mocked) backend to build its output. */
-  getOutput: () => TOutput | Promise<TOutput>;
+  /** Function to get the current output state */
+  getOutput: () => TOutput;
   /** Model and Windmill backend configuration */
   options: EvalRunnerOptions;
   onAssistantMessageStart?: () => void;
@@ -155,10 +154,9 @@ export async function runEval<THelpers, TOutput>(
       if (result.hitMaxIterations) {
         return {
           success: false,
-          output: (await getOutput()) as TOutput,
+          output: getOutput(),
           error: `Reached max turns (${maxIterations})`,
           tokenUsage: result.tokenUsage,
-          finalContextTokens: result.lastIterationUsage?.prompt ?? null,
           toolCallsCount,
           toolsCalled,
           toolCallDetails,
@@ -172,9 +170,8 @@ export async function runEval<THelpers, TOutput>(
 
       return {
         success: true,
-        output: (await getOutput()) as TOutput,
+        output: getOutput(),
         tokenUsage: result.tokenUsage,
-        finalContextTokens: result.lastIterationUsage?.prompt ?? null,
         toolCallsCount,
         toolsCalled,
         toolCallDetails,
@@ -194,10 +191,9 @@ export async function runEval<THelpers, TOutput>(
 
       return {
         success: false,
-        output: (await getOutput()) as TOutput,
+        output: getOutput(),
         error: errorMessage,
         tokenUsage: { prompt: 0, completion: 0, total: 0 },
-        finalContextTokens: null,
         toolCallsCount,
         toolsCalled,
         toolCallDetails,

ai_evals/adapters/frontend/core/shared/types.ts

@@ -28,8 +28,6 @@ export interface RawEvalResult<TOutput> {
   output: TOutput;
   error?: string;
   tokenUsage: TokenUsage;
-  /** Input tokens on the last model request of the loop (see BenchmarkAttemptResult.finalContextTokens). */
-  finalContextTokens: number | null;
   toolCallsCount: number;
   toolsCalled: string[];
   toolCallDetails: ToolCallDetail[];

ai_evals/adapters/frontend/datatableSqlEngine.test.ts

@@ -1,262 +0,0 @@
-import { describe, expect, it } from 'bun:test'
-import { applyDatatableSql, type BenchmarkDatatableSeed } from './datatableSqlEngine'
-
-function makeDatatable(): BenchmarkDatatableSeed {
-	return {
-		datatable_name: 'main',
-		schemas: {
-			public: {
-				orders: {
-					columns: { id: 'int4', customer_id: 'int4', total: 'numeric', status: 'text' },
-					rows: [
-						{ id: 1, customer_id: 1, total: 42.5, status: 'shipped' },
-						{ id: 2, customer_id: 2, total: 19.99, status: 'pending' },
-						{ id: 3, customer_id: 1, total: 88, status: 'shipped' }
-					]
-				},
-				customers: {
-					columns: { id: 'int4', name: 'text' },
-					rows: [{ id: 1, name: 'Alice' }]
-				}
-			}
-		}
-	}
-}
-
-describe('SELECT', () => {
-	it('returns the referenced table rows', () => {
-		const dt = makeDatatable()
-		expect(applyDatatableSql(dt, 'SELECT id, name FROM customers').rows).toEqual([
-			{ id: 1, name: 'Alice' }
-		])
-	})
-
-	it('falls back to the first table when no known table is referenced', () => {
-		const dt = makeDatatable()
-		expect(applyDatatableSql(dt, 'select 1').rows).toHaveLength(3)
-	})
-
-	it('resolves a schema-qualified table', () => {
-		const dt = makeDatatable()
-		expect(applyDatatableSql(dt, 'SELECT * FROM public.customers').rows).toEqual([
-			{ id: 1, name: 'Alice' }
-		])
-	})
-})
-
-describe('CREATE TABLE', () => {
-	it('adds a table with parsed columns, skipping table constraints and FK clauses', () => {
-		const dt = makeDatatable()
-		const result = applyDatatableSql(
-			dt,
-			'CREATE TABLE public.refunds (\n  order_id int4 NOT NULL REFERENCES public.orders(id),\n  amount numeric(10,2),\n  PRIMARY KEY (order_id)\n)'
-		)
-		expect(result.rows).toEqual([])
-		expect(dt.schemas.public.refunds).toEqual({
-			columns: { order_id: 'int4', amount: 'numeric(10,2)' },
-			rows: []
-		})
-	})
-
-	it('defaults an unqualified table to the public schema', () => {
-		const dt = makeDatatable()
-		applyDatatableSql(dt, 'CREATE TABLE notes (id int4, body text)')
-		expect(dt.schemas.public.notes.columns).toEqual({ id: 'int4', body: 'text' })
-	})
-
-	it('is a no-op for an existing table with IF NOT EXISTS', () => {
-		const dt = makeDatatable()
-		applyDatatableSql(dt, 'CREATE TABLE IF NOT EXISTS public.orders (x int4)')
-		expect(Object.keys(dt.schemas.public.orders.columns)).toContain('status')
-	})
-})
-
-describe('DROP TABLE', () => {
-	it('removes the table', () => {
-		const dt = makeDatatable()
-		applyDatatableSql(dt, 'DROP TABLE IF EXISTS public.customers')
-		expect(dt.schemas.public.customers).toBeUndefined()
-	})
-})
-
-describe('INSERT', () => {
-	it('appends a row using an explicit column list', () => {
-		const dt = makeDatatable()
-		applyDatatableSql(dt, "INSERT INTO customers (id, name) VALUES (2, 'Bob')")
-		expect(dt.schemas.public.customers.rows).toContainEqual({ id: 2, name: 'Bob' })
-	})
-
-	it('infers columns from the table when none are given, and appends multiple tuples', () => {
-		const dt = makeDatatable()
-		applyDatatableSql(dt, "INSERT INTO customers VALUES (2, 'Bob'), (3, 'Carol')")
-		expect(dt.schemas.public.customers.rows).toHaveLength(3)
-	})
-
-	it('returns the inserted rows when RETURNING is present', () => {
-		const dt = makeDatatable()
-		const result = applyDatatableSql(
-			dt,
-			"INSERT INTO customers (id, name) VALUES (2, 'Bob') RETURNING *"
-		)
-		expect(result.rows).toEqual([{ id: 2, name: 'Bob' }])
-	})
-})
-
-describe('UPDATE', () => {
-	it('updates only the rows matching an equality WHERE', () => {
-		const dt = makeDatatable()
-		const result = applyDatatableSql(
-			dt,
-			"UPDATE public.orders SET status = 'shipped' WHERE id = 2"
-		)
-		expect(result.rows).toEqual([])
-		expect(dt.schemas.public.orders.rows?.find((r) => r.id === 2)?.status).toBe('shipped')
-		expect(dt.schemas.public.orders.rows?.find((r) => r.id === 1)?.status).toBe('shipped')
-	})
-
-	it('strips a Postgres cast in the WHERE value', () => {
-		const dt = makeDatatable()
-		applyDatatableSql(dt, "UPDATE orders SET status = 'done' WHERE id = 2::int4")
-		expect(dt.schemas.public.orders.rows?.find((r) => r.id === 2)?.status).toBe('done')
-	})
-
-	it('matches multiple AND predicates including a numeric literal', () => {
-		const dt = makeDatatable()
-		applyDatatableSql(
-			dt,
-			"UPDATE orders SET status = 'done' WHERE customer_id = 2 AND total = 19.99"
-		)
-		expect(dt.schemas.public.orders.rows?.find((r) => r.id === 2)?.status).toBe('done')
-		expect(dt.schemas.public.orders.rows?.find((r) => r.id === 1)?.status).toBe('shipped')
-	})
-
-	it('updates every row when there is no WHERE', () => {
-		const dt = makeDatatable()
-		applyDatatableSql(dt, "UPDATE orders SET status = 'archived'")
-		expect(dt.schemas.public.orders.rows?.every((r) => r.status === 'archived')).toBe(true)
-	})
-
-	it('returns the affected rows when RETURNING is present', () => {
-		const dt = makeDatatable()
-		const result = applyDatatableSql(
-			dt,
-			"UPDATE orders SET status = 'shipped' WHERE id = 2 RETURNING *"
-		)
-		expect(result.rows).toHaveLength(1)
-		expect(result.rows[0]).toMatchObject({ id: 2, status: 'shipped' })
-	})
-
-	it('affects no rows when the WHERE clause cannot be parsed', () => {
-		const dt = makeDatatable()
-		applyDatatableSql(dt, "UPDATE orders SET status = 'x' WHERE total > 20")
-		expect(dt.schemas.public.orders.rows?.some((r) => r.status === 'x')).toBe(false)
-	})
-})
-
-describe('DELETE', () => {
-	it('removes only the matching rows', () => {
-		const dt = makeDatatable()
-		applyDatatableSql(dt, 'DELETE FROM orders WHERE id = 2')
-		expect(dt.schemas.public.orders.rows?.map((r) => r.id)).toEqual([1, 3])
-	})
-
-	it('returns the removed rows when RETURNING is present', () => {
-		const dt = makeDatatable()
-		const result = applyDatatableSql(dt, 'DELETE FROM orders WHERE id = 2 RETURNING *')
-		expect(result.rows).toEqual([{ id: 2, customer_id: 2, total: 19.99, status: 'pending' }])
-	})
-})
-
-describe('writes are reflected by later reads', () => {
-	it('UPDATE then SELECT sees the new value (the verify-loop fix)', () => {
-		const dt = makeDatatable()
-		applyDatatableSql(dt, "UPDATE orders SET status = 'shipped' WHERE id = 2")
-		const seen = applyDatatableSql(dt, 'SELECT * FROM orders').rows
-		expect(seen.find((r) => r.id === 2)?.status).toBe('shipped')
-	})
-
-	it('INSERT then SELECT sees the new row', () => {
-		const dt = makeDatatable()
-		applyDatatableSql(dt, "INSERT INTO customers (id, name) VALUES (9, 'Zed')")
-		const seen = applyDatatableSql(dt, 'SELECT * FROM customers').rows
-		expect(seen).toContainEqual({ id: 9, name: 'Zed' })
-	})
-
-	it('CREATE then SELECT on the new table returns its (empty) rows', () => {
-		const dt = makeDatatable()
-		applyDatatableSql(dt, 'CREATE TABLE public.refunds (order_id int4, amount numeric)')
-		expect(applyDatatableSql(dt, 'SELECT * FROM refunds').rows).toEqual([])
-	})
-})
-
-describe('system-catalog queries reflect the current tables/columns', () => {
-	it('lists current tables (including a freshly created one) via information_schema.tables', () => {
-		const dt = makeDatatable()
-		applyDatatableSql(dt, 'CREATE TABLE public.refunds (order_id int4)')
-		const rows = applyDatatableSql(
-			dt,
-			"SELECT table_name FROM information_schema.tables WHERE table_name = 'refunds'"
-		).rows
-		expect(rows.map((r) => r.table_name)).toContain('refunds')
-	})
-
-	it('does not list a dropped table', () => {
-		const dt = makeDatatable()
-		applyDatatableSql(dt, 'DROP TABLE public.customers')
-		const rows = applyDatatableSql(dt, 'SELECT table_name FROM information_schema.tables').rows
-		expect(rows.map((r) => r.table_name)).not.toContain('customers')
-	})
-
-	it('reports columns via information_schema.columns', () => {
-		const dt = makeDatatable()
-		const rows = applyDatatableSql(
-			dt,
-			"SELECT column_name FROM information_schema.columns WHERE table_name = 'orders'"
-		).rows
-		expect(rows.map((r) => r.column_name)).toContain('status')
-	})
-})
-
-describe('parser robustness (string/paren-aware splitting)', () => {
-	it('does not treat the word "returning" inside a string value as a RETURNING clause', () => {
-		const dt = makeDatatable()
-		const result = applyDatatableSql(
-			dt,
-			"INSERT INTO customers (id, name) VALUES (5, 'is returning soon')"
-		)
-		expect(result.rows).toEqual([])
-		expect(dt.schemas.public.customers.rows).toContainEqual({ id: 5, name: 'is returning soon' })
-	})
-
-	it('does not split on the word "where" inside a SET string value', () => {
-		const dt = makeDatatable()
-		applyDatatableSql(dt, "UPDATE orders SET status = 'ship where ordered' WHERE id = 2")
-		expect(dt.schemas.public.orders.rows?.find((r) => r.id === 2)?.status).toBe('ship where ordered')
-		expect(dt.schemas.public.orders.rows?.find((r) => r.id === 1)?.status).toBe('shipped')
-	})
-
-	it('keeps INSERT tuples intact when a value contains a function call', () => {
-		const dt = makeDatatable()
-		applyDatatableSql(dt, "INSERT INTO customers (id, name) VALUES (6, coalesce(NULL, 'x'))")
-		expect(dt.schemas.public.customers.rows).toHaveLength(2)
-		expect(dt.schemas.public.customers.rows?.[1]).toMatchObject({ id: 6 })
-	})
-
-	it('CREATE TABLE ignores a trailing semicolon-separated statement', () => {
-		const dt = makeDatatable()
-		applyDatatableSql(
-			dt,
-			'CREATE TABLE public.refunds (id int4, amount numeric); INSERT INTO refunds VALUES (1, 5)'
-		)
-		expect(dt.schemas.public.refunds.columns).toEqual({ id: 'int4', amount: 'numeric' })
-		expect(dt.schemas.public.refunds.rows).toEqual([])
-	})
-})
-
-describe('unparseable statements are a safe no-op', () => {
-	it('returns [] and does not throw', () => {
-		const dt = makeDatatable()
-		expect(applyDatatableSql(dt, 'VACUUM ANALYZE').rows).toEqual([])
-		expect(applyDatatableSql(dt, 'GRANT SELECT ON orders TO someone').rows).toEqual([])
-	})
-})

ai_evals/adapters/frontend/datatableSqlEngine.ts

@@ -1,541 +0,0 @@
-/**
- * A deliberately small, best-effort SQL engine for the benchmark datatable mock.
- *
- * This is NOT a real SQL implementation — it exists only so that writes a model
- * issues during an eval (`CREATE TABLE`, `INSERT`, `UPDATE`, `DELETE`, `DROP`)
- * become visible to its later reads (`list_datatables`, `get_datatable_table_schema`,
- * `SELECT`). Without that, a model that re-queries to verify a write sees stale
- * seed data, concludes the write failed, and loops until it exhausts its turns.
- *
- * It parses only the common statement shapes models produce. Anything it cannot
- * parse is a no-op success (it never throws) — behavioral evals assert that the
- * right statement was issued, not its exact data effects. Notable limits:
- *  - `SELECT` returns all rows of the referenced (or first) table — no WHERE
- *    filtering, projection, joins, or aggregation.
- *  - `WHERE` supports `col = value` predicates joined by `AND` only; an
- *    unparseable WHERE on UPDATE/DELETE affects zero rows (never the whole table).
- */
-
-/** One seeded datatable table: its columns (col -> compact_type) and optional rows. */
-export interface BenchmarkDatatableTableSeed {
-	columns: Record<string, string>
-	rows?: Record<string, unknown>[]
-}
-
-/** A seeded datatable: `datatable_name` plus a `schema -> table -> seed` map. */
-export interface BenchmarkDatatableSeed {
-	datatable_name: string
-	schemas: {
-		[schema: string]: {
-			[table: string]: BenchmarkDatatableTableSeed
-		}
-	}
-}
-
-export interface DatatableSqlResult {
-	rows: Record<string, unknown>[]
-}
-
-const DEFAULT_SCHEMA = 'public'
-
-type ParsedRef = { schema: string; table: string }
-type Predicate = { column: string; value: unknown }
-
-/**
- * Apply one SQL statement to `datatable` IN PLACE and return the result rows.
- * SELECT returns the referenced/first table's rows; a mutation returns its
- * affected rows when it has a RETURNING clause, otherwise `[]`.
- */
-export function applyDatatableSql(
-	datatable: BenchmarkDatatableSeed,
-	sql: string
-): DatatableSqlResult {
-	const statement = stripTrailingSemicolon(sql.trim())
-	if (/^\s*(with|select)\b/i.test(statement)) {
-		return { rows: selectRows(datatable, statement) }
-	}
-	if (/^\s*create\s+table\b/i.test(statement)) {
-		return { rows: applyCreateTable(datatable, statement) }
-	}
-	if (/^\s*drop\s+table\b/i.test(statement)) {
-		return { rows: applyDropTable(datatable, statement) }
-	}
-	if (/^\s*insert\s+into\b/i.test(statement)) {
-		return { rows: applyInsert(datatable, statement) }
-	}
-	if (/^\s*update\b/i.test(statement)) {
-		return { rows: applyUpdate(datatable, statement) }
-	}
-	if (/^\s*delete\s+from\b/i.test(statement)) {
-		return { rows: applyDelete(datatable, statement) }
-	}
-	return { rows: [] }
-}
-
-// ============= Reads =============
-
-function selectRows(
-	datatable: BenchmarkDatatableSeed,
-	sql: string
-): Record<string, unknown>[] {
-	const fromRef = sql.match(/\bfrom\s+([a-zA-Z_"][\w."]*)/i)?.[1]
-	if (fromRef) {
-		const catalog = catalogRows(datatable, fromRef)
-		if (catalog) {
-			return catalog
-		}
-	}
-	const table = fromRef ? resolveTable(datatable, fromRef) : undefined
-	const seed = table ?? firstTable(datatable)
-	return seed?.rows ?? []
-}
-
-/**
- * Synthesize rows for a system-catalog query so a model verifying a `CREATE`/`DROP`
- * via `information_schema.tables` / `.columns` (or `pg_tables`) sees the current
- * tables/columns instead of fallback data. WHERE is not applied, so the model gets
- * the full set and finds (or no longer finds) the table it just changed.
- * Returns `undefined` for non-catalog refs so normal table resolution proceeds.
- */
-function catalogRows(
-	datatable: BenchmarkDatatableSeed,
-	ref: string
-): Record<string, unknown>[] | undefined {
-	const normalized = ref.toLowerCase().replace(/"/g, '')
-	const name = normalized.split('.').pop()
-	const isCatalog = normalized.includes('information_schema.') || normalized.startsWith('pg_')
-	if (!isCatalog) {
-		return undefined
-	}
-	const tables = allTables(datatable)
-	if (name === 'tables' || name === 'pg_tables') {
-		return tables.map(({ schema, table }) => ({
-			table_schema: schema,
-			table_name: table,
-			schemaname: schema,
-			tablename: table
-		}))
-	}
-	if (name === 'columns') {
-		return tables.flatMap(({ schema, table, seed }) =>
-			Object.entries(seed.columns).map(([column, type]) => ({
-				table_schema: schema,
-				table_name: table,
-				column_name: column,
-				data_type: type
-			}))
-		)
-	}
-	return undefined
-}
-
-function allTables(
-	datatable: BenchmarkDatatableSeed
-): { schema: string; table: string; seed: BenchmarkDatatableTableSeed }[] {
-	return Object.entries(datatable.schemas).flatMap(([schema, tables]) =>
-		Object.entries(tables).map(([table, seed]) => ({ schema, table, seed }))
-	)
-}
-
-// ============= DDL =============
-
-function applyCreateTable(
-	datatable: BenchmarkDatatableSeed,
-	sql: string
-): Record<string, unknown>[] {
-	const head = sql.match(
-		/^\s*create\s+table\s+(?:if\s+not\s+exists\s+)?([a-zA-Z_"][\w."]*)/i
-	)
-	// The first top-level paren group is the column-definition list; using it (rather
-	// than a greedy `(...)` capture) ignores any trailing `;`-separated statement.
-	const columnText = extractParenGroups(sql)[0]
-	if (!head || columnText === undefined) {
-		return []
-	}
-	const { schema, table } = parseRef(head[1])
-	const existing = datatable.schemas[schema]?.[table]
-	if (existing) {
-		return []
-	}
-	const columns: Record<string, string> = {}
-	for (const rawDef of splitTopLevel(columnText)) {
-		const def = rawDef.trim()
-		if (!def || isTableConstraint(def)) {
-			continue
-		}
-		const tokens = def.split(/\s+/)
-		const column = unquoteIdentifier(tokens[0])
-		if (!column) {
-			continue
-		}
-		columns[column] = tokens[1] ?? 'text'
-	}
-	if (!datatable.schemas[schema]) {
-		datatable.schemas[schema] = {}
-	}
-	datatable.schemas[schema][table] = { columns, rows: [] }
-	return []
-}
-
-function applyDropTable(
-	datatable: BenchmarkDatatableSeed,
-	sql: string
-): Record<string, unknown>[] {
-	const match = sql.match(
-		/^\s*drop\s+table\s+(?:if\s+exists\s+)?([a-zA-Z_"][\w."]*)/i
-	)
-	if (!match) {
-		return []
-	}
-	const { schema, table } = parseRef(match[1])
-	if (datatable.schemas[schema]?.[table]) {
-		delete datatable.schemas[schema][table]
-	}
-	return []
-}
-
-// ============= DML =============
-
-function applyInsert(
-	datatable: BenchmarkDatatableSeed,
-	sql: string
-): Record<string, unknown>[] {
-	const { body, returning } = splitOffReturning(sql)
-	const match = body.match(
-		/^\s*insert\s+into\s+([a-zA-Z_"][\w."]*)\s*(?:\(([^)]*)\))?\s*values\s*([\s\S]+)$/i
-	)
-	if (!match) {
-		return []
-	}
-	const table = resolveTable(datatable, match[1])
-	if (!table) {
-		return []
-	}
-	const columns = match[2]
-		? splitTopLevel(match[2]).map((entry) => unquoteIdentifier(entry.trim()))
-		: Object.keys(table.columns)
-	const inserted: Record<string, unknown>[] = []
-	for (const tuple of extractParenGroups(match[3])) {
-		const values = splitTopLevel(tuple).map((entry) => parseValue(entry))
-		const row: Record<string, unknown> = {}
-		columns.forEach((column, index) => {
-			row[column] = values[index]
-		})
-		inserted.push(row)
-	}
-	table.rows ??= []
-	table.rows.push(...inserted)
-	return returning ? inserted : []
-}
-
-function applyUpdate(
-	datatable: BenchmarkDatatableSeed,
-	sql: string
-): Record<string, unknown>[] {
-	const { body, returning } = splitOffReturning(sql)
-	const match = body.match(/^\s*update\s+([a-zA-Z_"][\w."]*)\s+set\s+([\s\S]+)$/i)
-	if (!match) {
-		return []
-	}
-	const table = resolveTable(datatable, match[1])
-	if (!table) {
-		return []
-	}
-	let assignmentText = match[2]
-	let whereText: string | undefined
-	const whereMatch = maskForClauseScan(assignmentText).match(/\swhere\s/i)
-	if (whereMatch && whereMatch.index !== undefined) {
-		whereText = assignmentText.slice(whereMatch.index + whereMatch[0].length)
-		assignmentText = assignmentText.slice(0, whereMatch.index)
-	}
-	const predicates = parsePredicates(whereText)
-	if (predicates === null) {
-		return []
-	}
-	const assignments: Record<string, unknown> = {}
-	for (const entry of splitTopLevel(assignmentText)) {
-		const pair = entry.match(/^\s*([a-zA-Z_"][\w."]*)\s*=\s*([\s\S]+?)\s*$/)
-		if (pair) {
-			assignments[lastIdentifier(pair[1])] = parseValue(pair[2])
-		}
-	}
-	const affected = (table.rows ?? []).filter((row) => rowMatches(row, predicates))
-	for (const row of affected) {
-		Object.assign(row, assignments)
-	}
-	return returning ? affected : []
-}
-
-function applyDelete(
-	datatable: BenchmarkDatatableSeed,
-	sql: string
-): Record<string, unknown>[] {
-	const { body, returning } = splitOffReturning(sql)
-	const match = body.match(/^\s*delete\s+from\s+([a-zA-Z_"][\w."]*)\s*([\s\S]*)$/i)
-	if (!match) {
-		return []
-	}
-	const table = resolveTable(datatable, match[1])
-	if (!table) {
-		return []
-	}
-	const whereText = match[2].replace(/^\s*where\s+/i, '').trim() || undefined
-	const predicates = parsePredicates(whereText)
-	if (predicates === null) {
-		return []
-	}
-	const rows = table.rows ?? []
-	const removed = rows.filter((row) => rowMatches(row, predicates))
-	table.rows = rows.filter((row) => !rowMatches(row, predicates))
-	return returning ? removed : []
-}
-
-// ============= Parsing helpers =============
-
-function resolveTable(
-	datatable: BenchmarkDatatableSeed,
-	ref: string
-): BenchmarkDatatableTableSeed | undefined {
-	const { schema, table } = parseRef(ref)
-	const direct = datatable.schemas[schema]?.[table]
-	if (direct) {
-		return direct
-	}
-	// Bare table name: fall back to searching every schema for a matching table.
-	if (!ref.includes('.')) {
-		for (const tables of Object.values(datatable.schemas)) {
-			if (tables[table]) {
-				return tables[table]
-			}
-		}
-	}
-	return undefined
-}
-
-function firstTable(
-	datatable: BenchmarkDatatableSeed
-): BenchmarkDatatableTableSeed | undefined {
-	for (const tables of Object.values(datatable.schemas)) {
-		for (const seed of Object.values(tables)) {
-			return seed
-		}
-	}
-	return undefined
-}
-
-function parseRef(ref: string): ParsedRef {
-	const parts = ref.split('.').map(unquoteIdentifier)
-	if (parts.length >= 2) {
-		return { schema: parts[parts.length - 2], table: parts[parts.length - 1] }
-	}
-	return { schema: DEFAULT_SCHEMA, table: parts[0] }
-}
-
-/** A WHERE clause with no parseable form returns `null`; absent WHERE returns `[]` (match all). */
-function parsePredicates(whereText: string | undefined): Predicate[] | null {
-	if (whereText === undefined || whereText.trim() === '') {
-		return []
-	}
-	const predicates: Predicate[] = []
-	for (const part of whereText.split(/\s+and\s+/i)) {
-		const match = part.match(/^\s*([a-zA-Z_"][\w."]*)\s*=\s*([\s\S]+?)\s*$/)
-		if (!match) {
-			return null
-		}
-		predicates.push({ column: lastIdentifier(match[1]), value: parseValue(match[2]) })
-	}
-	return predicates
-}
-
-function rowMatches(row: Record<string, unknown>, predicates: Predicate[]): boolean {
-	return predicates.every((predicate) => looseEquals(row[predicate.column], predicate.value))
-}
-
-function looseEquals(left: unknown, right: unknown): boolean {
-	if (left === null || left === undefined) {
-		return right === null || right === undefined
-	}
-	if (typeof left === 'number' && typeof right === 'number') {
-		return left === right
-	}
-	return String(left) === String(right)
-}
-
-function parseValue(raw: string): unknown {
-	// Drop a trailing Postgres cast (e.g. `2::int4`) before interpreting the literal.
-	const token = raw.trim().replace(/::\s*[a-zA-Z_][\w]*(\([^)]*\))?\s*$/, '').trim()
-	const stringMatch = token.match(/^'([\s\S]*)'$/)
-	if (stringMatch) {
-		return stringMatch[1].replace(/''/g, "'")
-	}
-	if (/^-?\d+(\.\d+)?$/.test(token)) {
-		return Number(token)
-	}
-	if (/^true$/i.test(token)) {
-		return true
-	}
-	if (/^false$/i.test(token)) {
-		return false
-	}
-	if (/^null$/i.test(token)) {
-		return null
-	}
-	return token
-}
-
-function splitOffReturning(sql: string): { body: string; returning: boolean } {
-	const match = maskForClauseScan(sql).match(/\sreturning\s/i)
-	if (!match || match.index === undefined) {
-		return { body: sql, returning: false }
-	}
-	return { body: sql.slice(0, match.index), returning: true }
-}
-
-/**
- * A same-length copy of `sql` with the contents of single-quoted strings and
- * parenthesized groups blanked to spaces, so a top-level keyword scan
- * (WHERE / RETURNING) cannot match inside a string literal or a subquery. Index
- * positions in the result map 1:1 back onto the original.
- */
-function maskForClauseScan(sql: string): string {
-	let masked = ''
-	let depth = 0
-	let inString = false
-	for (let i = 0; i < sql.length; i++) {
-		const char = sql[i]
-		if (inString) {
-			if (char === "'") {
-				if (sql[i + 1] === "'") {
-					masked += '  '
-					i++
-					continue
-				}
-				inString = false
-			}
-			masked += ' '
-			continue
-		}
-		if (char === "'") {
-			inString = true
-			masked += ' '
-		} else if (char === '(') {
-			depth++
-			masked += ' '
-		} else if (char === ')') {
-			depth = Math.max(0, depth - 1)
-			masked += ' '
-		} else {
-			masked += depth > 0 ? ' ' : char
-		}
-	}
-	return masked
-}
-
-/**
- * Inner text of each top-level `( ... )` group in `input`, honoring nested parens
- * (e.g. `now()`, `numeric(10,2)`) and single-quoted strings. Used for the CREATE
- * column-definition group and INSERT value tuples.
- */
-function extractParenGroups(input: string): string[] {
-	const groups: string[] = []
-	let depth = 0
-	let inString = false
-	let current = ''
-	for (let i = 0; i < input.length; i++) {
-		const char = input[i]
-		if (inString) {
-			current += char
-			if (char === "'") {
-				if (input[i + 1] === "'") {
-					current += input[++i]
-				} else {
-					inString = false
-				}
-			}
-			continue
-		}
-		if (char === "'") {
-			inString = true
-			current += char
-		} else if (char === '(') {
-			depth++
-			if (depth === 1) {
-				current = ''
-			} else {
-				current += char
-			}
-		} else if (char === ')') {
-			depth = Math.max(0, depth - 1)
-			if (depth === 0) {
-				groups.push(current)
-				current = ''
-			} else {
-				current += char
-			}
-		} else if (depth > 0) {
-			current += char
-		}
-	}
-	return groups
-}
-
-/** Split on commas that are not inside parentheses or single-quoted strings. */
-function splitTopLevel(input: string): string[] {
-	const parts: string[] = []
-	let depth = 0
-	let inString = false
-	let current = ''
-	for (let i = 0; i < input.length; i++) {
-		const char = input[i]
-		if (inString) {
-			current += char
-			if (char === "'") {
-				if (input[i + 1] === "'") {
-					current += input[++i]
-				} else {
-					inString = false
-				}
-			}
-			continue
-		}
-		if (char === "'") {
-			inString = true
-			current += char
-		} else if (char === '(') {
-			depth++
-			current += char
-		} else if (char === ')') {
-			depth = Math.max(0, depth - 1)
-			current += char
-		} else if (char === ',' && depth === 0) {
-			parts.push(current)
-			current = ''
-		} else {
-			current += char
-		}
-	}
-	if (current.trim() !== '') {
-		parts.push(current)
-	}
-	return parts
-}
-
-function isTableConstraint(def: string): boolean {
-	return /^(primary\s+key|foreign\s+key|constraint|unique|check|exclude|like)\b/i.test(def)
-}
-
-function unquoteIdentifier(identifier: string): string {
-	const trimmed = identifier.trim()
-	const quoted = trimmed.match(/^"([\s\S]*)"$/)
-	return quoted ? quoted[1] : trimmed
-}
-
-/** For a qualified reference like `orders.id`, keep only the final identifier. */
-function lastIdentifier(reference: string): string {
-	const parts = reference.split('.')
-	return unquoteIdentifier(parts[parts.length - 1])
-}
-
-function stripTrailingSemicolon(sql: string): string {
-	return sql.replace(/;\s*$/, '')
-}

ai_evals/adapters/frontend/mockBackend.ts

@@ -1,25 +1,7 @@
 import { randomUUID } from 'node:crypto'
-import type {
-	AppWithLastVersion,
-	CompletedJob,
-	Flow,
-	Job,
-	ListableApp,
-	Script
-} from '../../../frontend/src/lib/gen'
-import type {
-	DataTableTables,
-	DataTableTableSchema,
-	GetDraftForUserResponse,
-	ListDraftsResponse,
-	ScriptLang,
-	UpdateDraftResponse,
-	UserDraftItemKind
-} from '../../../frontend/src/lib/gen/types.gen'
+import type { CompletedJob, Flow, Script } from '../../../frontend/src/lib/gen'
+import type { ScriptLang } from '../../../frontend/src/lib/gen/types.gen'
 import { buildScriptLintResult } from './core/script/preview'
-import { applyDatatableSql, type BenchmarkDatatableSeed } from './datatableSqlEngine'
-
-export type { BenchmarkDatatableSeed, BenchmarkDatatableTableSeed } from './datatableSqlEngine'
 
 const BENCHMARK_TIMESTAMP = '1970-01-01T00:00:00.000Z'
 
@@ -40,54 +22,21 @@ export interface BenchmarkWorkspaceFlow {
 	value: Flow['value']
 }
 
-export interface BenchmarkWorkspaceApp {
-	path: string
-	summary: string
-	value: {
-		files: Record<string, string>
-		runnables: Record<string, unknown>
-		data?: unknown
-		policy?: unknown
-		custom_path?: unknown
-	}
-}
-
-export interface BenchmarkWorkspaceJob {
-	/** Stable id so a case prompt can reference a specific run (e.g. for get_job_logs). */
-	id?: string
-	jobKind?: CompletedJob['job_kind']
-	scriptPath?: string
-	createdBy?: string
-	label?: string
-	success?: boolean
-	logs?: string
-}
-
 export interface BenchmarkWorkspaceRunnables {
 	scripts?: BenchmarkWorkspaceScript[]
 	flows?: BenchmarkWorkspaceFlow[]
-	apps?: BenchmarkWorkspaceApp[]
-	datatables?: BenchmarkDatatableSeed[]
-	jobs?: BenchmarkWorkspaceJob[]
 }
 
 type BenchmarkCompletedJob = CompletedJob & { type: 'CompletedJob' }
 
 const benchmarkWorkspaces = new Set<string>()
 const benchmarkWorkspaceRunnables = new Map<string, BenchmarkWorkspaceRunnables>()
-// Keyed by `${workspace}::${jobId}` so concurrent attempts (or distinct cases)
-// can seed the same fixed job id without clobbering each other's entry.
 const benchmarkJobs = new Map<string, { workspace: string; job: BenchmarkCompletedJob }>()
 
-function benchmarkJobKey(workspace: string, jobId: string): string {
-	return `${workspace}::${jobId}`
-}
-
 export function resetBenchmarkMockBackend(): void {
 	benchmarkWorkspaces.clear()
 	benchmarkWorkspaceRunnables.clear()
 	benchmarkJobs.clear()
-	benchmarkDrafts.clear()
 }
 
 export function registerBenchmarkWorkspace(workspace: string): void {
@@ -99,33 +48,12 @@ export function registerBenchmarkWorkspaceRunnables(
 	runnables: BenchmarkWorkspaceRunnables
 ): void {
 	benchmarkWorkspaces.add(workspace)
-	// Fresh case: drop any drafts left from a prior run on this workspace id.
-	clearBenchmarkDrafts(workspace)
-	// Datatables are mutated in place by exec_datatable_sql (a write must be visible
-	// to later reads), so store an isolated deep copy — never mutate the caller's seed.
-	benchmarkWorkspaceRunnables.set(workspace, {
-		...runnables,
-		datatables: runnables.datatables ? structuredClone(runnables.datatables) : undefined
-	})
-	// Seed any fixture jobs so list_runs / get_job_logs have data to return.
-	for (const seed of runnables.jobs ?? []) {
-		createBenchmarkCompletedJob({
-			workspace,
-			id: seed.id,
-			jobKind: seed.jobKind ?? 'script',
-			success: seed.success,
-			scriptPath: seed.scriptPath,
-			createdBy: seed.createdBy,
-			label: seed.label,
-			logs: seed.logs
-		})
-	}
+	benchmarkWorkspaceRunnables.set(workspace, runnables)
 }
 
 export function unregisterBenchmarkWorkspace(workspace: string): void {
 	benchmarkWorkspaces.delete(workspace)
 	benchmarkWorkspaceRunnables.delete(workspace)
-	clearBenchmarkDrafts(workspace)
 	for (const [jobId, entry] of benchmarkJobs.entries()) {
 		if (entry.workspace === workspace) {
 			benchmarkJobs.delete(jobId)
@@ -181,22 +109,6 @@ export function getBenchmarkFlowByPath(workspace: string, path: string): Flow |
 	return flow ? buildBenchmarkFlow(flow) : null
 }
 
-export function listBenchmarkApps(workspace: string): ListableApp[] | null {
-	const runnables = benchmarkWorkspaceRunnables.get(workspace)
-	if (!runnables) {
-		return null
-	}
-	return (runnables.apps ?? []).map(buildBenchmarkListableApp)
-}
-
-export function getBenchmarkAppByPath(workspace: string, path: string): AppWithLastVersion | null {
-	const app = benchmarkWorkspaceRunnables
-		.get(workspace)
-		?.apps?.find((entry) => entry.path === path)
-
-	return app ? buildBenchmarkApp(app) : null
-}
-
 export function createBenchmarkCompletedJob(input: {
 	workspace: string
 	jobKind: CompletedJob['job_kind']
@@ -206,17 +118,14 @@ export function createBenchmarkCompletedJob(input: {
 	scriptPath?: string
 	scriptHash?: string
 	args?: Record<string, unknown>
-	id?: string
-	createdBy?: string
-	label?: string
 }): string {
-	const jobId = input.id ?? `benchmark-job-${randomUUID()}`
+	const jobId = `benchmark-job-${randomUUID()}`
 	const now = new Date().toISOString()
 	const job: BenchmarkCompletedJob = {
 		type: 'CompletedJob',
 		id: jobId,
 		workspace_id: input.workspace,
-		created_by: input.createdBy ?? 'ai-evals',
+		created_by: 'ai-evals',
 		created_at: now,
 		started_at: now,
 		completed_at: now,
@@ -234,11 +143,10 @@ export function createBenchmarkCompletedJob(input: {
 		is_skipped: false,
 		email: 'ai-evals@local',
 		visible_to_owner: true,
-		tag: 'benchmark',
-		labels: input.label ? [input.label] : undefined
+		tag: 'benchmark'
 	}
 
-	benchmarkJobs.set(benchmarkJobKey(input.workspace, jobId), { workspace: input.workspace, job })
+	benchmarkJobs.set(jobId, { workspace: input.workspace, job })
 	return jobId
 }
 
@@ -246,239 +154,13 @@ export function getBenchmarkCompletedJob(
 	workspace: string,
 	jobId: string
 ): BenchmarkCompletedJob | null {
-	const entry = benchmarkJobs.get(benchmarkJobKey(workspace, jobId))
-	if (!entry) {
+	const entry = benchmarkJobs.get(jobId)
+	if (!entry || entry.workspace !== workspace) {
 		return null
 	}
 	return structuredClone(entry.job)
 }
 
-/**
- * List seeded/recorded jobs for a benchmark workspace, most recent first —
- * the shape `JobService.listJobs` returns. Returns `null` for a non-benchmark
- * workspace so the caller can fall through to the real backend. Server-side
- * filters (path/creator/status/limit) are intentionally not applied: global
- * eval cases assert on the recorded `list_runs` tool call, not on filtering.
- */
-export function listBenchmarkJobs(workspace: string): Job[] | null {
-	if (!hasBenchmarkWorkspace(workspace)) {
-		return null
-	}
-	return [...benchmarkJobs.values()]
-		.filter((entry) => entry.workspace === workspace)
-		.map((entry) => structuredClone(entry.job) as Job)
-		.sort((a, b) => (b.created_at ?? '').localeCompare(a.created_at ?? ''))
-}
-
-/**
- * Mirror `JobService.getJobLogs` (response is the raw log string). Throws a
- * "not found" error for an unknown id, matching the backend 404.
- */
-export function getBenchmarkJobLogs(workspace: string, jobId: string): string {
-	const job = getBenchmarkCompletedJob(workspace, jobId)
-	if (!job) {
-		throw new Error(`Job Logs not found for "${jobId}"`)
-	}
-	return job.logs ?? ''
-}
-
-// ============= Drafts (per-user, DB-backed in production) =============
-
-/**
- * In-memory stand-in for the per-user draft backend (`DraftService`). The global
- * AI chat now persists and reads drafts through the backend DB instead of an
- * in-tab `UserDraft` cell, so the eval mocks the three draft endpoints it
- * exercises (`updateDraft` / `getDraftForUser` / `listDrafts`) and keeps the
- * saved values here, keyed by workspace + draft kind + storage path. Mirrors the
- * semantics of the production unit test's mock in
- * `frontend/src/lib/components/copilot/chat/global/core.test.ts`.
- */
-const benchmarkDrafts = new Map<
-	string,
-	{ workspace: string; kind: UserDraftItemKind; path: string; value: unknown }
->()
-
-// Fixed timestamp so artifacts stay deterministic. No eval simulates a
-// concurrent writer, so every save is accepted and the conflict branch is
-// never taken — the syncer just records this as its `last_sync` baseline.
-const BENCHMARK_DRAFT_TIMESTAMP = '1970-01-01T00:00:00.000Z'
-
-function benchmarkDraftKey(workspace: string, kind: string, path: string): string {
-	return `${workspace}::${kind}::${path}`
-}
-
-export function clearBenchmarkDrafts(workspace: string): void {
-	for (const [key, entry] of benchmarkDrafts.entries()) {
-		if (entry.workspace === workspace) {
-			benchmarkDrafts.delete(key)
-		}
-	}
-}
-
-/**
- * Seed a draft straight into the store — used by the eval's live-editor draft
- * fixtures, which model "the user already has this draft open/saved". Writing it
- * here (instead of through `UserDraft.save`) keeps it a backend draft row with no
- * shadowing in-tab cell, so a model edit that persists to the backend is what the
- * output read-back captures — not the stale seed.
- */
-export function seedBenchmarkDraft(
-	workspace: string,
-	kind: UserDraftItemKind,
-	path: string,
-	value: unknown
-): void {
-	benchmarkDrafts.set(benchmarkDraftKey(workspace, kind, path), {
-		workspace,
-		kind,
-		path,
-		value
-	})
-}
-
-/** Mirror `DraftService.updateDraft`: a `null`/omitted value deletes the row. */
-export function updateBenchmarkDraft(input: {
-	workspace: string
-	kind: UserDraftItemKind
-	path: string
-	requestBody?: { value?: unknown }
-}): UpdateDraftResponse {
-	const key = benchmarkDraftKey(input.workspace, input.kind, input.path)
-	const value = input.requestBody?.value
-	if (value == null) {
-		benchmarkDrafts.delete(key)
-	} else {
-		benchmarkDrafts.set(key, {
-			workspace: input.workspace,
-			kind: input.kind,
-			path: input.path,
-			value
-		})
-	}
-	return { status: 'saved', current_timestamp: BENCHMARK_DRAFT_TIMESTAMP }
-}
-
-/** Mirror `DraftService.getDraftForUser`: 404-shaped throw when absent so the
- * adapter's narrowed catch treats it as "no draft" instead of re-throwing. */
-export function getBenchmarkDraftForUser(input: {
-	workspace: string
-	kind: UserDraftItemKind
-	path: string
-}): GetDraftForUserResponse {
-	const entry = benchmarkDrafts.get(benchmarkDraftKey(input.workspace, input.kind, input.path))
-	if (!entry) {
-		throw Object.assign(new Error(`no draft for "${input.path}"`), { status: 404 })
-	}
-	return { value: entry.value, created_at: BENCHMARK_DRAFT_TIMESTAMP }
-}
-
-/** Mirror `DraftService.listDrafts`: metadata rows (no value) for a workspace. */
-export function listBenchmarkDrafts(workspace: string): ListDraftsResponse {
-	return [...benchmarkDrafts.values()]
-		.filter((entry) => entry.workspace === workspace)
-		.map((entry) => ({
-			kind: entry.kind,
-			path: entry.path,
-			summary: (entry.value as { summary?: string } | null)?.summary,
-			draft_only: true,
-			legacy_draft: false,
-			created_at: BENCHMARK_DRAFT_TIMESTAMP
-		}))
-}
-
-// ============= Datatables (best-effort in-memory SQL) =============
-
-/**
- * Project the seeded datatables down to the `list_datatable_tables` response:
- * `datatable_name` + `schema -> table_names`, with no column detail.
- * Returns `null` for a non-benchmark workspace so callers can fall through to
- * the real backend; an empty seed yields `[]`.
- */
-export function listBenchmarkDatatables(workspace: string): DataTableTables[] | null {
-	const runnables = benchmarkWorkspaceRunnables.get(workspace)
-	if (!runnables) {
-		return null
-	}
-	return (runnables.datatables ?? []).map((datatable) => ({
-		datatable_name: datatable.datatable_name,
-		schemas: Object.fromEntries(
-			Object.entries(datatable.schemas).map(([schema, tables]) => [schema, Object.keys(tables)])
-		)
-	}))
-}
-
-export function getBenchmarkDatatableSchema(input: {
-	workspace: string
-	datatableName: string
-	schemaName: string
-	tableName: string
-}): DataTableTableSchema {
-	const runnables = benchmarkWorkspaceRunnables.get(input.workspace)
-	const datatable = (runnables?.datatables ?? []).find(
-		(entry) => entry.datatable_name === input.datatableName
-	)
-	if (!datatable) {
-		// Message MUST match the production `isDatatableNotConfiguredError` regex
-		// (/datatable\s+\S+\s+not found/i in datatableTools.ts) so the
-		// get_datatable_table_schema not-configured mapping is actually exercised.
-		throw new Error(`datatable "${input.datatableName}" not found`)
-	}
-	const table = datatable.schemas?.[input.schemaName]?.[input.tableName]
-	if (!table) {
-		throw new Error(
-			`table "${input.schemaName}.${input.tableName}" not found in datatable "${input.datatableName}"`
-		)
-	}
-	return {
-		datatable_name: input.datatableName,
-		schema_name: input.schemaName,
-		table_name: input.tableName,
-		columns: table.columns
-	}
-}
-
-/**
- * Execute SQL against a seeded datatable through the best-effort in-memory engine
- * (`applyDatatableSql`). Writes (CREATE/INSERT/UPDATE/DELETE/DROP) mutate the
- * stored datatable in place so a later list/schema/SELECT reflects them; SELECT
- * (and RETURNING) yield rows, other statements yield `[]`. Creates a benchmark
- * completed job and returns its id, like `runBenchmarkScriptPreview`.
- */
-export function runBenchmarkDatatableSql(input: {
-	workspace: string
-	datatableName: string
-	sql: string
-}): string {
-	const runnables = benchmarkWorkspaceRunnables.get(input.workspace)
-	const datatable = (runnables?.datatables ?? []).find(
-		(entry) => entry.datatable_name === input.datatableName
-	)
-	const rows = datatable ? applyDatatableSql(datatable, input.sql).rows : []
-	return createBenchmarkCompletedJob({
-		workspace: input.workspace,
-		jobKind: 'preview',
-		success: true,
-		args: { database: `datatable://${input.datatableName}` },
-		result: rows
-	})
-}
-
-/**
- * Mirror `JobService.getCompletedJobResultMaybe` for benchmark workspaces — the
- * shape `pollJobResult` consumes. The job is created synchronously before
- * polling, so it is always present and completed.
- */
-export function getBenchmarkCompletedJobResultMaybe(input: {
-	workspace: string
-	id: string
-}): { success: boolean; completed: boolean; result: unknown } {
-	const job = getBenchmarkCompletedJob(input.workspace, input.id)
-	if (!job) {
-		throw new Error(`Job "${input.id}" not found in benchmark workspace`)
-	}
-	return { success: job.success, completed: true, result: job.result }
-}
-
 export function runBenchmarkScriptPreview(input: {
 	workspace: string
 	requestBody: {
@@ -640,35 +322,3 @@ function buildBenchmarkFlow(flow: BenchmarkWorkspaceFlow): Flow {
 		extra_perms: {}
 	} as Flow
 }
-
-function buildBenchmarkListableApp(app: BenchmarkWorkspaceApp): ListableApp {
-	return {
-		id: 0,
-		workspace_id: 'benchmark',
-		path: app.path,
-		summary: app.summary,
-		version: 1,
-		extra_perms: {},
-		edited_at: BENCHMARK_TIMESTAMP,
-		execution_mode: 'viewer',
-		raw_app: true
-	}
-}
-
-function buildBenchmarkApp(app: BenchmarkWorkspaceApp): AppWithLastVersion {
-	return {
-		id: 0,
-		workspace_id: 'benchmark',
-		path: app.path,
-		summary: app.summary,
-		versions: [1],
-		created_by: 'benchmark',
-		created_at: BENCHMARK_TIMESTAMP,
-		value: app.value,
-		policy: (app.value.policy ?? {}) as AppWithLastVersion['policy'],
-		execution_mode: 'viewer',
-		extra_perms: {},
-		custom_path: app.value.custom_path as string | undefined,
-		raw_app: true
-	}
-}

ai_evals/adapters/frontend/mockBackendDatatables.test.ts

@@ -1,175 +0,0 @@
-import { afterEach, beforeEach, describe, expect, it } from 'bun:test'
-import {
-	getBenchmarkCompletedJobResultMaybe,
-	getBenchmarkDatatableSchema,
-	listBenchmarkDatatables,
-	registerBenchmarkWorkspaceRunnables,
-	resetBenchmarkMockBackend,
-	runBenchmarkDatatableSql,
-	type BenchmarkWorkspaceRunnables
-} from './mockBackend'
-
-const WORKSPACE = 'benchmark-datatable-ws'
-
-// Mirrors the production `isDatatableNotConfiguredError` regex in
-// datatableTools.ts. The schema mock's "not configured" message MUST match it,
-// otherwise the not-configured mapping in get_datatable_table_schema is silently
-// untested.
-const NOT_CONFIGURED_RE = /datatable\s+\S+\s+not found/i
-
-const SEED: BenchmarkWorkspaceRunnables = {
-	datatables: [
-		{
-			datatable_name: 'main',
-			schemas: {
-				public: {
-					orders: {
-						columns: { id: 'int', total: 'numeric' },
-						rows: [
-							{ id: 1, total: 10 },
-							{ id: 2, total: 20 }
-						]
-					},
-					customers: {
-						columns: { id: 'int', name: 'text' },
-						rows: [{ id: 1, name: 'alice' }]
-					}
-				}
-			}
-		}
-	]
-}
-
-beforeEach(() => resetBenchmarkMockBackend())
-afterEach(() => resetBenchmarkMockBackend())
-
-describe('listBenchmarkDatatables', () => {
-	it('returns null for a non-benchmark workspace (caller falls through to real backend)', () => {
-		expect(listBenchmarkDatatables('unregistered')).toBeNull()
-	})
-
-	it('returns [] for a registered workspace with no datatables seed', () => {
-		registerBenchmarkWorkspaceRunnables(WORKSPACE, {})
-		expect(listBenchmarkDatatables(WORKSPACE)).toEqual([])
-	})
-
-	it('projects seeded datatables to schema -> table names only (no columns)', () => {
-		registerBenchmarkWorkspaceRunnables(WORKSPACE, SEED)
-		expect(listBenchmarkDatatables(WORKSPACE)).toEqual([
-			{ datatable_name: 'main', schemas: { public: ['orders', 'customers'] } }
-		])
-	})
-})
-
-describe('getBenchmarkDatatableSchema', () => {
-	beforeEach(() => registerBenchmarkWorkspaceRunnables(WORKSPACE, SEED))
-
-	it('returns the columns for a seeded table', () => {
-		expect(
-			getBenchmarkDatatableSchema({
-				workspace: WORKSPACE,
-				datatableName: 'main',
-				schemaName: 'public',
-				tableName: 'orders'
-			})
-		).toEqual({
-			datatable_name: 'main',
-			schema_name: 'public',
-			table_name: 'orders',
-			columns: { id: 'int', total: 'numeric' }
-		})
-	})
-
-	it('throws a not-configured error matching the production regex for an unknown datatable', () => {
-		let error: Error | undefined
-		try {
-			getBenchmarkDatatableSchema({
-				workspace: WORKSPACE,
-				datatableName: 'ghost',
-				schemaName: 'public',
-				tableName: 'orders'
-			})
-		} catch (e) {
-			error = e as Error
-		}
-		expect(error).toBeDefined()
-		expect(error!.message).toMatch(NOT_CONFIGURED_RE)
-	})
-
-	it('throws a table-not-found error that does NOT match the datatable-not-configured regex', () => {
-		// The datatable IS configured; only the table is missing. Production maps
-		// this to a generic "error getting schema", not the blocking message.
-		let error: Error | undefined
-		try {
-			getBenchmarkDatatableSchema({
-				workspace: WORKSPACE,
-				datatableName: 'main',
-				schemaName: 'public',
-				tableName: 'ghost'
-			})
-		} catch (e) {
-			error = e as Error
-		}
-		expect(error).toBeDefined()
-		expect(error!.message).not.toMatch(NOT_CONFIGURED_RE)
-	})
-})
-
-describe('runBenchmarkDatatableSql + getBenchmarkCompletedJobResultMaybe', () => {
-	beforeEach(() => registerBenchmarkWorkspaceRunnables(WORKSPACE, SEED))
-
-	function exec(sql: string): { success: boolean; completed: boolean; result: unknown } {
-		const jobId = runBenchmarkDatatableSql({ workspace: WORKSPACE, datatableName: 'main', sql })
-		return getBenchmarkCompletedJobResultMaybe({ workspace: WORKSPACE, id: jobId })
-	}
-
-	it('returns the canned rows of the table named in a SELECT FROM clause', () => {
-		expect(exec('SELECT * FROM customers')).toEqual({
-			success: true,
-			completed: true,
-			result: [{ id: 1, name: 'alice' }]
-		})
-	})
-
-	it('falls back to the first seeded table when the SELECT references no known table', () => {
-		expect(exec('select 1').result).toEqual([
-			{ id: 1, total: 10 },
-			{ id: 2, total: 20 }
-		])
-	})
-
-	it('returns [] success for DDL and DML statements without RETURNING', () => {
-		expect(exec('CREATE TABLE foo (id int)').result).toEqual([])
-		expect(exec('INSERT INTO orders VALUES (3, 30)').result).toEqual([])
-		expect(exec('update orders set total = 0').result).toEqual([])
-	})
-
-	it('reflects a write in a later SELECT, isolated from the shared seed', () => {
-		exec('UPDATE orders SET total = 999 WHERE id = 1')
-		expect((exec('SELECT * FROM orders').result as Record<string, unknown>[])).toContainEqual({
-			id: 1,
-			total: 999
-		})
-		// Registration deep-clones the seed, so the shared SEED const stays pristine.
-		expect(SEED.datatables![0].schemas.public.orders.rows).toContainEqual({ id: 1, total: 10 })
-	})
-
-	it('reflects a CREATE in list_datatables and get_datatable_table_schema', () => {
-		exec('CREATE TABLE public.refunds (order_id int4, amount numeric)')
-		expect(listBenchmarkDatatables(WORKSPACE)?.[0].schemas.public).toContain('refunds')
-		expect(
-			getBenchmarkDatatableSchema({
-				workspace: WORKSPACE,
-				datatableName: 'main',
-				schemaName: 'public',
-				tableName: 'refunds'
-			}).columns
-		).toEqual({ order_id: 'int4', amount: 'numeric' })
-	})
-
-	it('throws for an unknown job id', () => {
-		expect(() =>
-			getBenchmarkCompletedJobResultMaybe({ workspace: WORKSPACE, id: 'does-not-exist' })
-		).toThrow()
-	})
-})

ai_evals/adapters/frontend/mockBackendDrafts.test.ts

@@ -1,94 +0,0 @@
-import { afterEach, beforeEach, describe, expect, it } from 'bun:test'
-import {
-	clearBenchmarkDrafts,
-	getBenchmarkDraftForUser,
-	listBenchmarkDrafts,
-	resetBenchmarkMockBackend,
-	seedBenchmarkDraft,
-	updateBenchmarkDraft
-} from './mockBackend'
-
-const WORKSPACE = 'benchmark-drafts-ws'
-
-// Drives the in-memory stand-in for the per-user draft backend (`DraftService`)
-// that the global AI-chat eval round-trips its drafts through. Mirrors the
-// production-unit-test mock in
-// `frontend/src/lib/components/copilot/chat/global/core.test.ts`.
-describe('mockBackend drafts', () => {
-	beforeEach(() => resetBenchmarkMockBackend())
-	afterEach(() => resetBenchmarkMockBackend())
-
-	it('round-trips a saved draft through update / get / list', () => {
-		const value = { summary: 'Greet a user', content: 'export async function main() {}' }
-		const res = updateBenchmarkDraft({
-			workspace: WORKSPACE,
-			kind: 'script',
-			path: 'f/evals/greet',
-			requestBody: { value }
-		})
-		expect(res.status).toBe('saved')
-
-		expect(getBenchmarkDraftForUser({ workspace: WORKSPACE, kind: 'script', path: 'f/evals/greet' }).value).toEqual(
-			value
-		)
-
-		const rows = listBenchmarkDrafts(WORKSPACE)
-		expect(rows).toHaveLength(1)
-		expect(rows[0]).toMatchObject({ kind: 'script', path: 'f/evals/greet', summary: 'Greet a user', draft_only: true })
-	})
-
-	it('treats a null value as a delete', () => {
-		updateBenchmarkDraft({
-			workspace: WORKSPACE,
-			kind: 'variable',
-			path: 'f/evals/token',
-			requestBody: { value: { summary: 'token' } }
-		})
-		updateBenchmarkDraft({
-			workspace: WORKSPACE,
-			kind: 'variable',
-			path: 'f/evals/token',
-			requestBody: { value: null }
-		})
-
-		expect(listBenchmarkDrafts(WORKSPACE)).toHaveLength(0)
-		expect(() => getBenchmarkDraftForUser({ workspace: WORKSPACE, kind: 'variable', path: 'f/evals/token' })).toThrow()
-	})
-
-	it('throws a 404-shaped error when no draft exists', () => {
-		try {
-			getBenchmarkDraftForUser({ workspace: WORKSPACE, kind: 'script', path: 'f/evals/missing' })
-			throw new Error('expected a throw')
-		} catch (e) {
-			expect((e as { status?: number }).status).toBe(404)
-		}
-	})
-
-	it('seeds a draft as a backend row that a later edit overwrites', () => {
-		seedBenchmarkDraft(WORKSPACE, 'script', 'f/evals/current', { content: 'seed' })
-		expect(getBenchmarkDraftForUser({ workspace: WORKSPACE, kind: 'script', path: 'f/evals/current' }).value).toEqual({
-			content: 'seed'
-		})
-
-		// A model edit persists the same path and must win over the seed.
-		updateBenchmarkDraft({
-			workspace: WORKSPACE,
-			kind: 'script',
-			path: 'f/evals/current',
-			requestBody: { value: { content: 'edited' } }
-		})
-		expect(getBenchmarkDraftForUser({ workspace: WORKSPACE, kind: 'script', path: 'f/evals/current' }).value).toEqual({
-			content: 'edited'
-		})
-	})
-
-	it('clears only the targeted workspace', () => {
-		seedBenchmarkDraft(WORKSPACE, 'script', 'f/a', { content: 'a' })
-		seedBenchmarkDraft('other-ws', 'script', 'f/b', { content: 'b' })
-
-		clearBenchmarkDrafts(WORKSPACE)
-
-		expect(listBenchmarkDrafts(WORKSPACE)).toHaveLength(0)
-		expect(listBenchmarkDrafts('other-ws')).toHaveLength(1)
-	})
-})

ai_evals/adapters/frontend/runtime.ts

@@ -24,8 +24,6 @@ export async function runFrontendBenchmarkAdapter(input: {
   runs: number;
   model?: string;
   verbose?: boolean;
-  skipJudge?: boolean;
-  executionOnly?: boolean;
   backendValidation?: string;
 }): Promise<BenchmarkRunResult> {
   const tempDir = await mkdtemp(
@@ -42,9 +40,6 @@ export async function runFrontendBenchmarkAdapter(input: {
     WMILL_FRONTEND_AI_EVAL_MODEL: input.model ?? "",
     WMILL_FRONTEND_AI_EVAL_PROGRESS: "1",
     WMILL_FRONTEND_AI_EVAL_VERBOSE: input.verbose ? "1" : "0",
-    WMILL_FRONTEND_AI_EVAL_SKIP_JUDGE:
-      input.skipJudge || input.executionOnly ? "1" : "0",
-    WMILL_FRONTEND_AI_EVAL_EXECUTION_ONLY: input.executionOnly ? "1" : "0",
     WMILL_FRONTEND_AI_EVAL_BACKEND_VALIDATION: input.backendValidation ?? "",
   };
 

ai_evals/adapters/frontend/vitestAdapter.test.ts

@@ -33,29 +33,18 @@ vi.mock('$lib/components/vscode', () => ({}))
 vi.mock('$lib/gen', async () => {
 	const actual = await vi.importActual<any>('$lib/gen')
 	const {
-		getBenchmarkAppByPath,
 		getBenchmarkCompletedJob,
-		getBenchmarkCompletedJobResultMaybe,
-		getBenchmarkDatatableSchema,
-		getBenchmarkDraftForUser,
 		getBenchmarkFlowByPath,
-		getBenchmarkJobLogs,
 		getBenchmarkScriptByHash,
 		getBenchmarkScriptByPath,
 		hasBenchmarkWorkspace,
-		listBenchmarkApps,
-		listBenchmarkDatatables,
-		listBenchmarkDrafts,
 		listBenchmarkFlows,
-		listBenchmarkJobs,
 		listBenchmarkScripts,
 		createBenchmarkHttpTrigger,
 		createBenchmarkSchedule,
 		previewBenchmarkSchedule,
-		runBenchmarkDatatableSql,
 		runBenchmarkFlowByPath,
-		runBenchmarkScriptPreview,
-		updateBenchmarkDraft
+		runBenchmarkScriptPreview
 	} = await import('./mockBackend')
 
 	function wrapService<T extends object>(target: T, overrides: Record<string, unknown>): T {
@@ -71,25 +60,6 @@ vi.mock('$lib/gen', async () => {
 
 	return {
 		...actual,
-		DraftService: wrapService(actual.DraftService, {
-			updateDraft: async (data: {
-				workspace: string
-				kind: any
-				path: string
-				requestBody?: { value?: unknown }
-			}) =>
-				hasBenchmarkWorkspace(data.workspace)
-					? updateBenchmarkDraft(data)
-					: actual.DraftService.updateDraft(data),
-			getDraftForUser: async (data: { workspace: string; kind: any; path: string }) =>
-				hasBenchmarkWorkspace(data.workspace)
-					? getBenchmarkDraftForUser(data)
-					: actual.DraftService.getDraftForUser(data),
-			listDrafts: async (data: { workspace: string }) =>
-				hasBenchmarkWorkspace(data.workspace)
-					? listBenchmarkDrafts(data.workspace)
-					: actual.DraftService.listDrafts(data)
-		}),
 		ScriptService: wrapService(actual.ScriptService, {
 			listScripts: async (data: { workspace: string }) =>
 				hasBenchmarkWorkspace(data.workspace)
@@ -179,27 +149,13 @@ vi.mock('$lib/gen', async () => {
 					args?: Record<string, unknown>
 					path?: string
 				}
-			}) => {
-				if (!hasBenchmarkWorkspace(data.workspace)) {
-					return actual.JobService.runScriptPreview(data)
-				}
-				const requestBody = data.requestBody ?? {}
-				const database = requestBody.args?.database
-				// Datatable SQL runs as a `postgresql` preview against `datatable://<name>`.
-				// Execute it through the canned-SQL mock instead of linting it as a script.
-				if (
-					requestBody.language === 'postgresql' &&
-					typeof database === 'string' &&
-					database.startsWith('datatable://')
-				) {
-					return runBenchmarkDatatableSql({
-						workspace: data.workspace,
-						datatableName: database.slice('datatable://'.length),
-						sql: requestBody.content ?? ''
-					})
-				}
-				return runBenchmarkScriptPreview({ workspace: data.workspace, requestBody })
-			},
+			}) =>
+				hasBenchmarkWorkspace(data.workspace)
+					? runBenchmarkScriptPreview({
+							workspace: data.workspace,
+							requestBody: data.requestBody ?? {}
+						})
+					: actual.JobService.runScriptPreview(data),
 			runFlowByPath: async (data: {
 				workspace: string
 				path: string
@@ -221,39 +177,7 @@ vi.mock('$lib/gen', async () => {
 					return job
 				}
 				return actual.JobService.getJob(data)
-			},
-			getCompletedJobResultMaybe: async (data: { workspace: string; id: string }) =>
-				hasBenchmarkWorkspace(data.workspace)
-					? getBenchmarkCompletedJobResultMaybe({ workspace: data.workspace, id: data.id })
-					: actual.JobService.getCompletedJobResultMaybe(data),
-			listJobs: async (data: { workspace: string }) =>
-				hasBenchmarkWorkspace(data.workspace)
-					? (listBenchmarkJobs(data.workspace) ?? [])
-					: actual.JobService.listJobs(data),
-			getJobLogs: async (data: { workspace: string; id: string }) =>
-				hasBenchmarkWorkspace(data.workspace)
-					? getBenchmarkJobLogs(data.workspace, data.id)
-					: actual.JobService.getJobLogs(data)
-		}),
-		WorkspaceService: wrapService(actual.WorkspaceService, {
-			listDataTableTables: async (data: { workspace: string }) =>
-				hasBenchmarkWorkspace(data.workspace)
-					? (listBenchmarkDatatables(data.workspace) ?? [])
-					: actual.WorkspaceService.listDataTableTables(data),
-			getDataTableTableSchema: async (data: {
-				workspace: string
-				datatableName: string
-				schemaName: string
-				tableName: string
-			}) =>
-				hasBenchmarkWorkspace(data.workspace)
-					? getBenchmarkDatatableSchema({
-							workspace: data.workspace,
-							datatableName: data.datatableName,
-							schemaName: data.schemaName,
-							tableName: data.tableName
-						})
-					: actual.WorkspaceService.getDataTableTableSchema(data)
+			}
 		}),
 		ScheduleService: wrapService(actual.ScheduleService, {
 			existsSchedule: async (data: { workspace: string; path: string }) =>
@@ -301,20 +225,12 @@ vi.mock('$lib/gen', async () => {
 		}),
 		AppService: wrapService(actual.AppService, {
 			existsApp: async (data: { workspace: string; path: string }) =>
-				hasBenchmarkWorkspace(data.workspace)
-					? Boolean(getBenchmarkAppByPath(data.workspace, data.path))
-					: actual.AppService.existsApp(data),
+				hasBenchmarkWorkspace(data.workspace) ? false : actual.AppService.existsApp(data),
 			listApps: async (data: { workspace: string }) =>
-				hasBenchmarkWorkspace(data.workspace)
-					? (listBenchmarkApps(data.workspace) ?? [])
-					: actual.AppService.listApps(data),
+				hasBenchmarkWorkspace(data.workspace) ? [] : actual.AppService.listApps(data),
 			getAppByPath: async (data: { workspace: string; path: string }) => {
 				if (hasBenchmarkWorkspace(data.workspace)) {
-					const app = getBenchmarkAppByPath(data.workspace, data.path)
-					if (!app) {
-						throw new Error(`App "${data.path}" not found in benchmark workspace`)
-					}
-					return app
+					throw new Error(`App "${data.path}" not found in benchmark workspace`)
 				}
 				return actual.AppService.getAppByPath(data)
 			}
@@ -466,6 +382,5 @@ benchmarkIt(
 			resetBenchmarkMockBackend()
 		}
 	},
-	// Full-suite runs (30+ cases at concurrency 2-3) routinely exceed 10 minutes.
-	7_200_000
+	600_000
 )

ai_evals/cases/global.yaml

@@ -4,7 +4,7 @@
     It should take a string `name` input and return `Hello, ${name}!`.
     Leave it as an AI draft only; do not deploy or save it.
   runtime:
-    maxTurns: 10
+    maxTurns: 8
   validate:
     draftCountExactly: 1
     requiredDrafts:
@@ -618,606 +618,3 @@
     - creates a Slack resource draft that references the secret variable instead of embedding the token
     - keeps both drafts under a sensible eval/global Slack-related path
     - does not create schedules, triggers, or deployed workspace changes
-
-- id: global-test19-datatable-not-configured-asks-to-set-up
-  prompt: |-
-    Here are two newsletter signups: alice@example.com and bob@example.com.
-    Save them into a workspace data table for me.
-  runtime:
-    maxTurns: 6
-  validate:
-    draftCountExactly: 0
-  toolExpect:
-    requiredToolsUsed:
-      - list_datatables
-    forbiddenToolsUsed:
-      - exec_datatable_sql
-      - write_script
-      - deploy_workspace_item
-      - delete_workspace_item
-  skipJudge: true
-  judgeChecklist:
-    - checks which data tables exist in the workspace before acting
-    - recognizes that no data table is configured in this workspace
-    - explains that a data table must first be set up by the user in the workspace settings (Data Tables) and is not created via SQL
-    - does not run SQL, write a script, or invent a data table to work around the missing configuration
-    - tells the user to configure a data table and then try again
-
-- id: global-test20-datatable-no-hallucinated-main
-  prompt: |-
-    Pull the latest rows from the orders table in our data table so I can see recent orders.
-  runtime:
-    maxTurns: 6
-  validate:
-    draftCountExactly: 0
-  toolExpect:
-    requiredToolsUsed:
-      - list_datatables
-    forbiddenToolsUsed:
-      - exec_datatable_sql
-      - write_script
-      - deploy_workspace_item
-      - delete_workspace_item
-  skipJudge: true
-  judgeChecklist:
-    - checks which data tables exist in the workspace before querying
-    - recognizes that no data table is configured in this workspace
-    - does not assume a data table named "main" (or any other name) exists
-    - does not run SQL against a guessed data table or fabricate order rows
-    - tells the user they need to set up a data table in the workspace settings first
-
-- id: global-test21-datatable-list-summarize
-  prompt: |-
-    What tables do we have in our workspace data table? Just give me the list.
-  initial: ai_evals/fixtures/frontend/global/initial/datatable_orders_seed.json
-  runtime:
-    maxTurns: 6
-  validate:
-    draftCountExactly: 0
-  toolExpect:
-    requiredToolsUsed:
-      - list_datatables
-    forbiddenToolsUsed:
-      - get_datatable_table_schema
-      - exec_datatable_sql
-      - deploy_workspace_item
-      - delete_workspace_item
-  skipJudge: true
-  judgeChecklist:
-    - lists the tables available in the workspace data table (orders and customers)
-    - answers from the data table listing rather than fabricating table names
-    - does not fetch column details or run SQL just to produce a table list
-
-- id: global-test22-datatable-inspect-columns
-  prompt: |-
-    What columns does the orders table have in our workspace data table?
-  initial: ai_evals/fixtures/frontend/global/initial/datatable_orders_seed.json
-  runtime:
-    maxTurns: 6
-  validate:
-    draftCountExactly: 0
-  toolExpect:
-    requiredToolsUsed:
-      - get_datatable_table_schema
-    forbiddenToolsUsed:
-      - deploy_workspace_item
-      - delete_workspace_item
-  skipJudge: true
-  judgeChecklist:
-    - inspects the orders table schema in the workspace data table
-    - reports the orders columns (such as id, customer_id, total, status, created_at)
-    - answers from the retrieved schema rather than guessing the columns
-
-- id: global-test23-datatable-query-select
-  prompt: |-
-    Show me the orders in our workspace data table, including their status and total.
-  initial: ai_evals/fixtures/frontend/global/initial/datatable_orders_seed.json
-  runtime:
-    maxTurns: 6
-  validate:
-    draftCountExactly: 0
-  toolExpect:
-    requiredToolsUsed:
-      - exec_datatable_sql
-    forbiddenToolsUsed:
-      - write_script
-      - deploy_workspace_item
-      - delete_workspace_item
-    toolCallArgs:
-      - tool: exec_datatable_sql
-        field: sql
-        stringIncludesAnyOf:
-          - select
-  skipJudge: true
-  judgeChecklist:
-    - runs a SELECT query against the orders table in the workspace data table
-    - reports the orders returned by the query back to the user instead of fabricating data
-    - does not tell the user to set up a data table, since one already exists
-
-- id: global-test24-datatable-create-table
-  prompt: |-
-    Add a new table called refunds to our workspace data table, with an order id and a refund amount.
-  initial: ai_evals/fixtures/frontend/global/initial/datatable_orders_seed.json
-  runtime:
-    maxTurns: 6
-  validate:
-    draftCountExactly: 0
-  toolExpect:
-    requiredToolsUsed:
-      - exec_datatable_sql
-    forbiddenToolsUsed:
-      - write_script
-      - deploy_workspace_item
-      - delete_workspace_item
-    toolCallArgs:
-      - tool: exec_datatable_sql
-        field: sql
-        stringIncludesAnyOf:
-          - create table
-  skipJudge: true
-  judgeChecklist:
-    - creates the refunds table with a plain CREATE TABLE statement on the data table
-    - includes an order id and a refund amount column
-    - treats creating the table as a normal SQL statement and does not claim a separate registration step is needed
-    - does not write a script to create the table
-
-- id: global-test25-datatable-mutate-rows
-  prompt: |-
-    Mark order number 2 as shipped in our workspace data table.
-  initial: ai_evals/fixtures/frontend/global/initial/datatable_orders_seed.json
-  runtime:
-    # Headroom for inspect-schema -> UPDATE -> verify; the in-memory engine now
-    # persists the write, so verification confirms on the first try (no loop).
-    maxTurns: 12
-  validate:
-    draftCountExactly: 0
-  toolExpect:
-    requiredToolsUsed:
-      - exec_datatable_sql
-    forbiddenToolsUsed:
-      - write_script
-      - deploy_workspace_item
-      - delete_workspace_item
-    toolCallArgs:
-      - tool: exec_datatable_sql
-        field: sql
-        stringIncludesAnyOf:
-          - update
-          - insert into
-  skipJudge: true
-  judgeChecklist:
-    - runs an UPDATE on the orders table setting the status of order id 2 to shipped
-    - targets only order number 2 rather than rewriting the whole table
-    - confirms the change back to the user
-
-- id: global-test26-datatable-script-sdk
-  prompt: |-
-    Write a script that reads our workspace data table and returns the total revenue across all orders.
-    Leave it as an AI draft only; do not deploy or save it.
-  initial: ai_evals/fixtures/frontend/global/initial/datatable_orders_seed.json
-  runtime:
-    maxTurns: 10
-  validate:
-    draftCountExactly: 1
-    requiredDrafts:
-      - type: script
-        valueIncludes:
-          - wmill.datatable(
-  toolExpect:
-    requiredToolsUsed:
-      - get_instructions
-      - write_script
-    forbiddenToolsUsed:
-      - exec_datatable_sql
-      - deploy_workspace_item
-      - delete_workspace_item
-  # The judge has no datatable SDK reference and wrongly penalizes correct
-  # wmill.datatable() tagged-template usage, so rely on the deterministic checks:
-  # required get_instructions + write_script, forbidden exec_datatable_sql, and a
-  # draft that contains wmill.datatable(.
-  skipJudge: true
-  judgeChecklist:
-    - writes a script (not a chat-time SQL execution) that reads the workspace data table at runtime
-    - uses the wmill.datatable() SDK to query the orders table and sum the order totals
-    - returns the total revenue from the script
-    - leaves the result as an AI draft and does not deploy or save it
-
-- id: global-test27-list-recent-runs
-  prompt: |-
-    What are the most recent runs in this workspace? Give me a quick rundown of what ran and whether it succeeded.
-  initial: ai_evals/fixtures/frontend/global/initial/jobs_seed.json
-  runtime:
-    maxTurns: 6
-  validate:
-    draftCountExactly: 0
-  toolExpect:
-    requiredToolsUsed:
-      - list_runs
-    forbiddenToolsUsed:
-      - deploy_workspace_item
-      - delete_workspace_item
-      - write_script
-  # Read-only job inspection produces no draft, so the global judge (which only
-  # sees the drafts artifact) would score it empty — validate via tool use.
-  skipJudge: true
-  judgeChecklist:
-    - lists the recent runs from the workspace rather than fabricating them
-    - summarizes each run's path and success/failure from the returned data
-
-- id: global-test28-fetch-failed-run-logs
-  prompt: |-
-    The run with id 01920000-0000-7000-8000-0000000000f1 failed. Pull its logs and tell me what went wrong.
-  initial: ai_evals/fixtures/frontend/global/initial/jobs_seed.json
-  runtime:
-    maxTurns: 6
-  validate:
-    draftCountExactly: 0
-  toolExpect:
-    requiredToolsUsed:
-      - get_job_logs
-    forbiddenToolsUsed:
-      - deploy_workspace_item
-      - delete_workspace_item
-      - write_script
-    toolCallArgs:
-      - tool: get_job_logs
-        field: id
-        stringIncludesAnyOf:
-          - 01920000-0000-7000-8000-0000000000f1
-  # Same as above — no draft is produced, so rely on the deterministic tool-use
-  # and argument checks rather than the judge.
-  skipJudge: true
-  judgeChecklist:
-    - fetches the logs for the requested job id
-    - explains the failure from the returned logs (connection refused to the upstream API)
-
-# --- Documentation search (search_docs) ---
-# Pure product-knowledge questions: the assistant should consult the docs via
-# search_docs and answer conversationally, not draft or mutate anything. No
-# draft is produced, so the global judge is skipped and we validate tool use.
-
-- id: global-docs-ai-agent-step
-  prompt: |-
-    Does Windmill support a flow step where an LLM decides which of my scripts to call based on the input?
-  runtime:
-    maxTurns: 6
-  validate:
-    draftCountExactly: 0
-  toolExpect:
-    requiredToolsUsed:
-      - search_docs
-    forbiddenToolsUsed:
-      - write_script
-      - write_flow
-      - deploy_workspace_item
-      - delete_workspace_item
-  skipJudge: true
-
-- id: global-docs-retry-step
-  prompt: |-
-    How does automatic retry work for a flow step that calls a flaky API?
-  runtime:
-    maxTurns: 6
-  validate:
-    draftCountExactly: 0
-  toolExpect:
-    requiredToolsUsed:
-      - search_docs
-    forbiddenToolsUsed:
-      - write_script
-      - write_flow
-      - deploy_workspace_item
-      - delete_workspace_item
-  skipJudge: true
-
-- id: global-docs-key-value-store
-  prompt: |-
-    Can I use a Redis-style key-value store from my Windmill scripts, and how?
-  runtime:
-    maxTurns: 6
-  validate:
-    draftCountExactly: 0
-  toolExpect:
-    requiredToolsUsed:
-      - search_docs
-    forbiddenToolsUsed:
-      - write_script
-      - write_flow
-      - deploy_workspace_item
-      - delete_workspace_item
-  skipJudge: true
-
-- id: global-docs-cron-schedule-format
-  prompt: |-
-    How do Windmill's cron schedules work, and what format does the schedule expression use?
-  runtime:
-    maxTurns: 6
-  validate:
-    draftCountExactly: 0
-  toolExpect:
-    requiredToolsUsed:
-      - search_docs
-    forbiddenToolsUsed:
-      - write_script
-      - write_flow
-      - deploy_workspace_item
-      - delete_workspace_item
-  skipJudge: true
-
-# --- Raw app on a large project (context-usage benchmark) ---
-# These cases run against the deliberately large `analytics_dashboard` raw-app
-# fixture (~20 frontend files incl. a 5k-line data module, plus backend runnables).
-# They exist to measure how much context the global chat consumes when working in a
-# big raw app: test29 is a read-heavy debugging hunt, test30 is a small edit baseline.
-# tokenUsage is recorded per run, so the same cases re-run after a read-tool change
-# (the read_app_file cap + offset/limit paging) quantify the optimization. skipJudge:
-# the judge only sees the drafts artifact and cannot run the app, so we validate
-# deterministically.
-
-- id: global-test29-raw-app-debug-large
-  prompt: |-
-    The analytics dashboard app at `f/evals/global/analytics_dashboard` has a bug:
-    the Revenue Summary tile shows a total that is lower than the per-order line
-    totals and the per-region breakdown. Track down what is computing revenue
-    incorrectly and fix it. Keep the change as an AI draft only; do not deploy or
-    save it.
-  initial: ai_evals/fixtures/frontend/global/initial/analytics_dashboard
-  runtime:
-    maxTurns: 20
-  validate:
-    draftCountExactly: 1
-    requiredDrafts:
-      - type: app
-        path: f/evals/global/analytics_dashboard
-        valueIncludes:
-          - "return order.unitPrice * order.quantity"
-  toolExpect:
-    requiredToolsAnyOf:
-      # Inspecting the app's files is satisfied by either reading them directly
-      # or grepping for the revenue calculation.
-      - [read_app_file, search_app]
-    forbiddenToolsUsed:
-      - deploy_workspace_item
-      - delete_workspace_item
-  skipJudge: true
-  judgeChecklist:
-    - inspects the dashboard app's files to locate the revenue calculation
-    - fixes the per-order revenue so it multiplies unit price by quantity
-    - leaves the result as an AI draft and does not deploy or save it
-
-- id: global-test30-raw-app-small-edit-large
-  prompt: |-
-    In the dashboard app at `f/evals/global/analytics_dashboard`, change the main
-    page heading from "Operations Console" to "Revenue Overview". Leave everything
-    else unchanged. Keep it as an AI draft only; do not deploy or save it.
-  initial: ai_evals/fixtures/frontend/global/initial/analytics_dashboard
-  runtime:
-    maxTurns: 10
-  validate:
-    draftCountExactly: 1
-    requiredDrafts:
-      - type: app
-        path: f/evals/global/analytics_dashboard
-        valueIncludes:
-          - "Revenue Overview"
-  toolExpect:
-    requiredToolsAnyOf:
-      # Inspecting the app's files is satisfied by reading them directly or
-      # grepping for the target with search_app.
-      - [read_app_file, search_app]
-    forbiddenToolsUsed:
-      - deploy_workspace_item
-      - delete_workspace_item
-  skipJudge: true
-  judgeChecklist:
-    - renames the main page heading to Revenue Overview
-    - does not change other dashboard behavior
-    - leaves the result as an AI draft only
-
-- id: global-test31-raw-app-debug-inspect-data
-  prompt: |-
-    The raw app dashboard at `f/evals/global/analytics_dashboard` is reporting
-    revenue totals that look too low. Inspect the app's files — both the sample
-    order data module and the revenue calculation — to work out whether the bug is
-    in the data or in the calculation, then fix the actual cause. Keep the change as
-    an AI draft only; do not deploy or save it.
-  initial: ai_evals/fixtures/frontend/global/initial/analytics_dashboard
-  runtime:
-    maxTurns: 22
-  validate:
-    draftCountExactly: 1
-    requiredDrafts:
-      - type: app
-        path: f/evals/global/analytics_dashboard
-        valueIncludes:
-          - "return order.unitPrice * order.quantity"
-  toolExpect:
-    requiredToolsAnyOf:
-      # Inspecting the app's files is satisfied by reading them directly or
-      # grepping for the target with search_app.
-      - [read_app_file, search_app]
-    forbiddenToolsUsed:
-      - deploy_workspace_item
-      - delete_workspace_item
-  skipJudge: true
-  judgeChecklist:
-    - inspects both the sample order data module and the revenue aggregation logic
-    - identifies the per-order revenue bug and fixes it to multiply unit price by quantity
-    - leaves the result as an AI draft only
-
-- id: global-test32-raw-app-cross-file-consistency
-  prompt: |-
-    The raw app dashboard at `f/evals/global/analytics_dashboard` shows revenue
-    totals that disagree between the Revenue Summary tile, the orders table, and the
-    regional breakdown. Investigate how each of those computes revenue, work out
-    which calculation is wrong, and fix it. Keep the change as an AI draft only; do
-    not deploy or save it.
-  # Cross-file investigation: forces the model through several overlapping files
-  # (the summary's aggregation helper, the orders table, the regional breakdown) —
-  # a realistic multi-file read load that exercises the read_app_file cap.
-  initial: ai_evals/fixtures/frontend/global/initial/analytics_dashboard
-  runtime:
-    maxTurns: 24
-  validate:
-    draftCountExactly: 1
-    requiredDrafts:
-      - type: app
-        path: f/evals/global/analytics_dashboard
-        valueIncludes:
-          - "return order.unitPrice * order.quantity"
-  toolExpect:
-    requiredToolsAnyOf:
-      # Inspecting the app's files is satisfied by reading them directly or
-      # grepping for the target with search_app.
-      - [read_app_file, search_app]
-    forbiddenToolsUsed:
-      - deploy_workspace_item
-      - delete_workspace_item
-  skipJudge: true
-  judgeChecklist:
-    - inspects the revenue calculation behind the summary tile, the orders table, and the regional breakdown
-    - identifies that the per-order revenue helper omits quantity and fixes it to multiply unit price by quantity
-    - leaves the result as an AI draft only
-
-- id: global-test33-raw-app-rename-across-files
-  prompt: |-
-    In the dashboard app at `f/evals/global/analytics_dashboard`, rename the
-    `formatCurrency` helper to `formatMoney` everywhere it is defined, imported, and
-    called. Leave the separate `formatCurrencyPrecise` helper exactly as it is. Keep
-    the change as an AI draft only; do not deploy or save it.
-  # Find-all-usages rename: formatCurrency is defined once and called in 6 places
-  # spread across 4 component files (and imported in 4). Locating every usage is the
-  # exact task search_app is meant to make cheap — one grep returns all file:line
-  # rows instead of reading each component whole. valueExcludes "formatCurrency("
-  # asserts the definition and all call sites were renamed while tolerating the
-  # preserved formatCurrencyPrecise (which is never followed by "(").
-  initial: ai_evals/fixtures/frontend/global/initial/analytics_dashboard
-  runtime:
-    maxTurns: 22
-  validate:
-    draftCountExactly: 1
-    requiredDrafts:
-      - type: app
-        path: f/evals/global/analytics_dashboard
-        valueIncludes:
-          - "export function formatMoney"
-          - "formatMoney("
-        valueExcludes:
-          - "formatCurrency("
-  toolExpect:
-    forbiddenToolsUsed:
-      - deploy_workspace_item
-      - delete_workspace_item
-  skipJudge: true
-  judgeChecklist:
-    - renames the formatCurrency definition, imports, and all call sites to formatMoney
-    - leaves the unrelated formatCurrencyPrecise helper unchanged
-    - leaves the result as an AI draft only
-
-# --- Path selection (u/<user> vs f/<folder>) ---
-# These cases assert how the assistant picks a workspace path when the user gives
-# none: a bare name defaults to the personal scope `u/<user>/`, an existing folder
-# whose purpose matches is used, a non-admin targets a writable folder and never a
-# read-only one, and shared intent with no matching folder asks rather than invents.
-# Each depends on the seeded `user` fixture (username / is_admin / folders /
-# folders_read) so the prompt's folder guidance and `u/{username}` are well-formed.
-
-- id: global-path1-bare-name-defaults-to-personal
-  prompt: |-
-    Stage a quick draft helper that takes a string and returns it trimmed of
-    leading and trailing whitespace. Just keep it as a draft.
-  initial: ai_evals/fixtures/frontend/global/initial/user_admin_empty.json
-  runtime:
-    maxTurns: 8
-  validate:
-    draftCountExactly: 1
-    requiredDrafts:
-      - type: script
-        pathStartsWith: u/admin/
-  toolExpect:
-    requiredToolsUsed:
-      - write_script
-    forbiddenToolsUsed:
-      - deploy_workspace_item
-      - delete_workspace_item
-  skipJudge: true
-  judgeChecklist:
-    - stages a single script draft for a trim helper
-    - defaults the path to the current user's personal scope (u/admin/...) since no path or folder was given
-    - does not invent an f/<folder> path
-    - leaves the result as a draft only
-
-- id: global-path2-match-existing-folder
-  prompt: |-
-    Draft a flow for the marketing team's weekly campaign report.
-    It should take a week number and return a short summary string.
-    Keep it as a draft only.
-  initial: ai_evals/fixtures/frontend/global/initial/user_admin_folders.json
-  runtime:
-    maxTurns: 10
-  validate:
-    draftCountExactly: 1
-    requiredDrafts:
-      - type: flow
-        pathStartsWith: f/marketing/
-  toolExpect:
-    requiredToolsUsed:
-      - write_flow
-    forbiddenToolsUsed:
-      - deploy_workspace_item
-      - delete_workspace_item
-  skipJudge: true
-  judgeChecklist:
-    - drafts a flow for the marketing campaign report
-    - places it in the existing marketing folder (f/marketing/...) rather than the personal scope or an invented folder
-    - leaves the result as a draft only
-
-- id: global-path3-shared-intent-unknown-folder-asks
-  prompt: |-
-    Put together a draft onboarding checklist flow for the People Ops team to use
-    when a new hire joins. Keep it as a draft.
-  initial: ai_evals/fixtures/frontend/global/initial/user_admin_folders.json
-  runtime:
-    maxTurns: 6
-  validate:
-    draftCountExactly: 0
-  toolExpect:
-    requiredToolsUsed:
-      - askUserQuestion
-    forbiddenToolsUsed:
-      - write_flow
-      - write_script
-      - deploy_workspace_item
-      - delete_workspace_item
-  skipJudge: true
-  judgeChecklist:
-    - recognizes the request implies shared/team work but names no existing folder (none of marketing/data_engineering/shared_utils fit People Ops)
-    - asks which folder to use instead of guessing or inventing one
-    - does not create a draft until the folder is known
-
-- id: global-path4-nonadmin-avoids-readonly-folder
-  prompt: |-
-    Draft a small flow that returns today's date as an ISO string, and stage it in
-    one of our shared team folders. Keep it as a draft.
-  initial: ai_evals/fixtures/frontend/global/initial/user_bob_nonadmin_teams.json
-  runtime:
-    maxTurns: 10
-  validate:
-    draftCountExactly: 1
-    requiredDrafts:
-      - type: flow
-        pathStartsWith: f/team_a/
-    forbiddenDrafts:
-      - type: flow
-        pathStartsWith: f/team_b/
-  toolExpect:
-    requiredToolsUsed:
-      - write_flow
-    forbiddenToolsUsed:
-      - deploy_workspace_item
-      - delete_workspace_item
-  skipJudge: true
-  judgeChecklist:
-    - drafts a flow that returns the current date as an ISO string
-    - places it in team_a (writable by this non-admin user) and not team_b (read-only)
-    - leaves the result as a draft only

ai_evals/cli/index.ts

@@ -25,9 +25,7 @@ import {
 import { runSuite } from "../core/runSuite";
 import { EVAL_MODES, type EvalMode } from "../core/types";
 import { DEFAULT_JUDGE_MODEL } from "../core/judge";
-// createCliModeRunner is imported lazily in runCliBenchmark so the non-cli modes
-// (global/flow/script/app) don't pull in the wmill CLI toolchain and its JSR deps
-// (e.g. @cliffy/*) just to load this entrypoint.
+import { createCliModeRunner } from "../modes/cli";
 import { runFrontendBenchmarkAdapter } from "../adapters/frontend/runtime";
 import { resolveWindmillBackendSettings } from "../core/windmillBackendSettings";
 import { assertWindmillBackendReachable } from "../adapters/frontend/windmillBackend";
@@ -99,11 +97,6 @@ async function main() {
       "comma-separated model aliases to run sequentially",
     )
     .option("--verbose", "stream assistant output during frontend runs")
-    .option("--skip-judge", "skip LLM judge scoring for this run")
-    .option(
-      "--execution-only",
-      "only require the model/proxy/frontend loop to complete",
-    )
     .option(
       "--record",
       "append a compact summary line to ai_evals/history/<mode>.jsonl",
@@ -122,8 +115,6 @@ async function main() {
           model?: string;
           models?: string;
           verbose?: boolean;
-          skipJudge?: boolean;
-          executionOnly?: boolean;
           record?: boolean;
           backendValidation?: string;
         },
@@ -136,8 +127,6 @@ async function main() {
           model: options.model,
           models: options.models,
           verbose: options.verbose ?? false,
-          skipJudge: options.skipJudge ?? false,
-          executionOnly: options.executionOnly ?? false,
           record: options.record ?? false,
           backendValidation: options.backendValidation,
         });
@@ -186,8 +175,6 @@ async function handleRun(input: {
   model?: string;
   models?: string;
   verbose: boolean;
-  skipJudge: boolean;
-  executionOnly: boolean;
   record: boolean;
   backendValidation?: string;
 }) {
@@ -243,8 +230,6 @@ async function handleRun(input: {
             input.runs,
             getCliEvalModel(model),
             runModel,
-            input.skipJudge,
-            input.executionOnly,
           )
         : await runFrontendBenchmarkAdapter({
             mode: input.mode,
@@ -252,8 +237,6 @@ async function handleRun(input: {
             runs: input.runs,
             model: model.id,
             verbose: input.verbose,
-            skipJudge: input.skipJudge,
-            executionOnly: input.executionOnly,
             backendValidation,
           });
 
@@ -295,25 +278,20 @@ async function runCliBenchmark(
   runs: number,
   model: ReturnType<typeof getCliEvalModel>,
   runModel: string,
-  skipJudge: boolean,
-  executionOnly: boolean,
 ) {
-  const { createCliModeRunner } = await import("../modes/cli");
-  const judgeModel = skipJudge || executionOnly ? null : DEFAULT_JUDGE_MODEL;
   const caseResults = await runSuite({
     modeRunner: createCliModeRunner(model),
     cases,
     runs,
     runModel,
-    judgeModel,
-    executionOnly,
+    judgeModel: DEFAULT_JUDGE_MODEL,
   });
 
   return buildRunResult({
     mode: "cli",
     runs,
     runModel,
-    judgeModel,
+    judgeModel: DEFAULT_JUDGE_MODEL,
     caseResults,
   });
 }

ai_evals/core/cases.test.ts

@@ -246,21 +246,6 @@ describe("loadCases", () => {
     });
   });
 
-  it("loads global docs-search cases as tool-use checks", async () => {
-    const globalCases = await loadCases("global");
-    const docsCases = globalCases.filter((entry) =>
-      entry.id.startsWith("global-docs-"),
-    );
-    expect(docsCases.length).toBeGreaterThanOrEqual(3);
-
-    // Each docs case verifies the assistant reaches for search_docs and does not
-    // draft anything; with no draft, the global judge is skipped.
-    for (const entry of docsCases) {
-      expect(entry.skipJudge).toBe(true);
-      expect(entry.toolExpect?.requiredToolsUsed).toContain("search_docs");
-    }
-  });
-
   it("loads tool expectations for workspace mutation cases", async () => {
     const scriptCases = await loadCases("script");
     const caseEntry = scriptCases.find(

ai_evals/core/results.test.ts

@@ -92,71 +92,6 @@ describe("benchmark results", () => {
     expect(summary).toContain("Average duration (all attempts): 550ms");
   });
 
-  it("aggregates final context size over passed attempts only", () => {
-    const result = buildRunResult({
-      mode: "global",
-      runs: 1,
-      runModel: "model-under-test",
-      judgeModel: "judge-model",
-      caseResults: [
-        caseResult([
-          {
-            attempt: 1,
-            passed: true,
-            durationMs: 1000,
-            assistantMessageCount: 1,
-            toolCallCount: 1,
-            toolsUsed: ["edit_script"],
-            skillsInvoked: [],
-            checks: [{ name: "edited", passed: true }],
-            judgeScore: 100,
-            judgeSummary: "ok",
-            error: null,
-            tokenUsage: { prompt: 12000, completion: 200, total: 12200 },
-            finalContextTokens: 5000,
-          },
-          {
-            attempt: 2,
-            passed: true,
-            durationMs: 1100,
-            assistantMessageCount: 1,
-            toolCallCount: 1,
-            toolsUsed: ["edit_script"],
-            skillsInvoked: [],
-            checks: [{ name: "edited", passed: true }],
-            judgeScore: 100,
-            judgeSummary: "ok",
-            error: null,
-            tokenUsage: { prompt: 18000, completion: 300, total: 18300 },
-            finalContextTokens: 7000,
-          },
-          {
-            attempt: 3,
-            passed: false,
-            durationMs: 100,
-            assistantMessageCount: 1,
-            toolCallCount: 0,
-            toolsUsed: [],
-            skillsInvoked: [],
-            checks: [{ name: "edited", passed: false }],
-            judgeScore: 10,
-            judgeSummary: "missed",
-            error: "failed",
-            tokenUsage: { prompt: 20000, completion: 100, total: 20100 },
-            finalContextTokens: 9000,
-          },
-        ]),
-      ],
-    });
-
-    // Final context size stays below cumulative prompt and ignores the failed attempt.
-    expect(result.averageFinalContextTokensPassed).toBe(6000);
-    expect(result.maxFinalContextTokensPassed).toBe(7000);
-    expect(formatRunSummary(result)).toContain(
-      "Final context size (passed): 6000 tokens (max 7000)",
-    );
-  });
-
   it("reports passed averages as unavailable when no attempt passes", () => {
     const result = buildRunResult({
       mode: "global",

ai_evals/core/results.ts

@@ -16,9 +16,6 @@ type AttemptAggregate = {
   durationTotal: number;
   tokenUsageAttemptCount: number;
   tokenUsageTotal: BenchmarkTokenUsage | null;
-  finalContextAttemptCount: number;
-  finalContextTotal: number;
-  finalContextMax: number | null;
 };
 
 export async function writeRunResult(
@@ -120,8 +117,6 @@ export function buildRunResult(input: {
       passedAttemptAggregate,
       passedAttempts,
     ),
-    averageFinalContextTokensPassed: averageFinalContext(passedAttemptAggregate),
-    maxFinalContextTokensPassed: passedAttemptAggregate.finalContextMax,
     cases: input.caseResults,
   };
 }
@@ -138,11 +133,6 @@ export function formatRunSummary(result: BenchmarkRunResult): string {
       `Average tokens (passed): ${formatTokenUsage(result.averageTokenUsagePerPassedAttempt)}`,
     );
   }
-  if (result.averageFinalContextTokensPassed != null) {
-    lines.push(
-      `Final context size (passed): ${Math.round(result.averageFinalContextTokensPassed)} tokens (max ${Math.round(result.maxFinalContextTokensPassed ?? 0)})`,
-    );
-  }
   if (result.passedAttempts < result.attemptCount) {
     lines.push(
       `Average duration (all attempts): ${Math.round(result.averageDurationMs)}ms`,
@@ -191,21 +181,10 @@ function aggregateAttempts(attempts: BenchmarkAttemptResult[]): AttemptAggregate
     durationTotal: 0,
     tokenUsageAttemptCount: 0,
     tokenUsageTotal: null,
-    finalContextAttemptCount: 0,
-    finalContextTotal: 0,
-    finalContextMax: null,
   };
 
   for (const attempt of attempts) {
     aggregate.durationTotal += attempt.durationMs;
-    if (typeof attempt.finalContextTokens === "number") {
-      aggregate.finalContextAttemptCount += 1;
-      aggregate.finalContextTotal += attempt.finalContextTokens;
-      aggregate.finalContextMax = Math.max(
-        aggregate.finalContextMax ?? 0,
-        attempt.finalContextTokens,
-      );
-    }
     if (!attempt.tokenUsage) {
       continue;
     }
@@ -225,12 +204,6 @@ function averageDuration(aggregate: AttemptAggregate): number | null {
     : aggregate.durationTotal / aggregate.attemptCount;
 }
 
-function averageFinalContext(aggregate: AttemptAggregate): number | null {
-  return aggregate.finalContextAttemptCount === 0
-    ? null
-    : aggregate.finalContextTotal / aggregate.finalContextAttemptCount;
-}
-
 function averageTokenUsage(
   aggregate: AttemptAggregate,
   denominator: number,
@@ -345,9 +318,6 @@ function toHistoryRecord(result: BenchmarkRunResult) {
     averageTokenUsagePerAttempt: result.averageTokenUsagePerAttempt ?? null,
     averageTokenUsagePerPassedAttempt:
       result.averageTokenUsagePerPassedAttempt ?? null,
-    averageFinalContextTokensPassed:
-      result.averageFinalContextTokensPassed ?? null,
-    maxFinalContextTokensPassed: result.maxFinalContextTokensPassed ?? null,
     failedCaseIds: Array.from(
       new Set(
         result.cases
@@ -391,10 +361,6 @@ function toHistoryRecord(result: BenchmarkRunResult) {
           passedAttemptAggregate,
           passedAttempts,
         ),
-        averageFinalContextTokensPassed: averageFinalContext(
-          passedAttemptAggregate,
-        ),
-        maxFinalContextTokensPassed: passedAttemptAggregate.finalContextMax,
       };
     }),
   };

ai_evals/core/runSuite.test.ts

@@ -1,102 +0,0 @@
-import { describe, expect, it } from "bun:test";
-import { runSuite } from "./runSuite";
-import type { ModeRunner } from "./types";
-
-const modeRunner: ModeRunner<undefined, undefined, { ok: boolean }> = {
-  mode: "global",
-  concurrency: 1,
-  loadInitial: async () => undefined,
-  loadExpected: async () => undefined,
-  run: async () => ({
-    success: true,
-    actual: { ok: true },
-    assistantMessageCount: 1,
-    toolCallCount: 0,
-    toolsUsed: [],
-    skillsInvoked: [],
-    tokenUsage: null,
-  }),
-  validate: () => [],
-};
-
-describe("runSuite", () => {
-  it("skips judge checks when the run disables judge scoring", async () => {
-    const [caseResult] = await runSuite({
-      modeRunner,
-      cases: [
-        {
-          id: "case-1",
-          prompt: "Create a draft script",
-          judgeChecklist: ["the output satisfies the prompt"],
-        },
-      ],
-      runs: 1,
-      runModel: "model-under-test",
-      judgeModel: null,
-    });
-
-    const [attempt] = caseResult.attempts;
-    expect(attempt.passed).toBe(true);
-    expect(attempt.judgeScore).toBeNull();
-    expect(attempt.judgeSummary).toBeNull();
-    expect(attempt.checks.map((check) => check.name)).toEqual([
-      "run succeeded",
-    ]);
-  });
-
-  it("only requires run success when execution-only is enabled", async () => {
-    let loadExpectedCalls = 0;
-    let validateCalls = 0;
-    let backendValidateCalls = 0;
-
-    const executionOnlyRunner: ModeRunner<
-      undefined,
-      undefined,
-      { ok: boolean }
-    > = {
-      ...modeRunner,
-      loadExpected: async () => {
-        loadExpectedCalls++;
-        return undefined;
-      },
-      validate: () => {
-        validateCalls++;
-        return [{ name: "validator failed", passed: false }];
-      },
-      backendValidate: async () => {
-        backendValidateCalls++;
-        return {
-          checks: [{ name: "backend validation failed", passed: false }],
-        };
-      },
-    };
-
-    const [caseResult] = await runSuite({
-      modeRunner: executionOnlyRunner,
-      cases: [
-        {
-          id: "case-1",
-          prompt: "Create a draft script",
-          expectedPath: "fixtures/expected.json",
-          toolExpect: { requiredToolsUsed: ["write_script"] },
-          judgeChecklist: ["the output satisfies the prompt"],
-        },
-      ],
-      runs: 1,
-      runModel: "model-under-test",
-      judgeModel: "judge-model",
-      executionOnly: true,
-    });
-
-    const [attempt] = caseResult.attempts;
-    expect(attempt.passed).toBe(true);
-    expect(attempt.judgeScore).toBeNull();
-    expect(attempt.judgeSummary).toBeNull();
-    expect(attempt.checks.map((check) => check.name)).toEqual([
-      "run succeeded",
-    ]);
-    expect(loadExpectedCalls).toBe(0);
-    expect(validateCalls).toBe(0);
-    expect(backendValidateCalls).toBe(0);
-  });
-});

ai_evals/core/runSuite.ts

@@ -15,13 +15,11 @@ export async function runSuite<TInitial, TExpected, TActual>(input: {
   runs: number;
   runModel: string | null;
   judgeModel?: string | null;
-  executionOnly?: boolean;
   concurrency?: number;
   verbose?: boolean;
   onProgress?: (event: FrontendBenchmarkProgressEvent) => void;
 }): Promise<BenchmarkCaseResult[]> {
-  const judgeModel =
-    input.judgeModel === undefined ? DEFAULT_JUDGE_MODEL : input.judgeModel;
+  const judgeModel = input.judgeModel ?? DEFAULT_JUDGE_MODEL;
   const concurrency = Math.max(1, input.concurrency ?? input.modeRunner.concurrency);
   const results = new Array<BenchmarkCaseResult>(input.cases.length);
   let cursor = 0;
@@ -54,7 +52,6 @@ export async function runSuite<TInitial, TExpected, TActual>(input: {
           runs: input.runs,
           judgeModel,
           judgeThreshold: input.modeRunner.judgeThreshold ?? 80,
-          executionOnly: input.executionOnly ?? false,
           modeRunner: input.modeRunner,
           totalCases: input.cases.length,
           verbose: input.verbose ?? false,
@@ -75,9 +72,8 @@ async function runCaseAttempts<TInitial, TExpected, TActual>(input: {
   caseIndex: number;
   evalCase: EvalCase;
   runs: number;
-  judgeModel: string | null;
+  judgeModel: string;
   judgeThreshold: number;
-  executionOnly: boolean;
   modeRunner: ModeRunner<TInitial, TExpected, TActual>;
   totalCases: number;
   verbose: boolean;
@@ -103,9 +99,7 @@ async function runCaseAttempts<TInitial, TExpected, TActual>(input: {
 
     try {
       const initial = await input.modeRunner.loadInitial(input.evalCase.initialPath);
-      const expected = input.executionOnly
-        ? undefined
-        : await input.modeRunner.loadExpected(input.evalCase.expectedPath);
+      const expected = await input.modeRunner.loadExpected(input.evalCase.expectedPath);
       const run = await input.modeRunner.run(input.evalCase.prompt, initial, {
         evalCase: input.evalCase,
         caseId: input.evalCase.id,
@@ -168,30 +162,22 @@ async function runCaseAttempts<TInitial, TExpected, TActual>(input: {
       });
       const checks: BenchmarkCheck[] = [
         buildCheck("run succeeded", run.success, run.error),
+        ...input.modeRunner.validate({
+          evalCase: input.evalCase,
+          prompt: input.evalCase.prompt,
+          initial,
+          expected,
+          actual: run.actual,
+          run,
+        }),
+        ...validateToolExpectations({
+          run,
+          toolExpect: input.evalCase.toolExpect,
+        }),
       ];
-      if (!input.executionOnly) {
-        checks.push(
-          ...input.modeRunner.validate({
-            evalCase: input.evalCase,
-            prompt: input.evalCase.prompt,
-            initial,
-            expected,
-            actual: run.actual,
-            run,
-          }),
-          ...validateToolExpectations({
-            run,
-            toolExpect: input.evalCase.toolExpect,
-          })
-        );
-      }
       const artifactFiles = input.modeRunner.buildArtifacts?.(run.actual) ?? [];
 
-      if (
-        run.success &&
-        !input.executionOnly &&
-        input.modeRunner.backendValidate
-      ) {
+      if (run.success && input.modeRunner.backendValidate) {
         try {
           const backendValidation = await input.modeRunner.backendValidate({
             evalCase: input.evalCase,
@@ -232,21 +218,14 @@ async function runCaseAttempts<TInitial, TExpected, TActual>(input: {
       let judgeScore: number | null = null;
       let judgeSummary: string | null = null;
 
-      if (
-        run.success &&
-        !input.executionOnly &&
-        input.judgeModel !== null &&
-        !input.evalCase.skipJudge
-      ) {
+      if (run.success && !input.evalCase.skipJudge) {
         const judge = await judgeOutput({
           mode: input.modeRunner.mode,
           prompt: input.evalCase.prompt,
           checklist: input.evalCase.judgeChecklist,
           initial,
           expected: input.modeRunner.mode === "cli" ? undefined : expected,
-          actual: input.modeRunner.prepareJudgeActual
-            ? input.modeRunner.prepareJudgeActual(run.actual)
-            : run.actual,
+          actual: run.actual,
           model: input.judgeModel,
         });
 
@@ -276,7 +255,6 @@ async function runCaseAttempts<TInitial, TExpected, TActual>(input: {
         judgeSummary,
         error: run.error ?? null,
         tokenUsage: run.tokenUsage ?? null,
-        finalContextTokens: run.finalContextTokens ?? null,
         artifactsPath: null,
         artifactFiles,
       };
@@ -313,7 +291,6 @@ async function runCaseAttempts<TInitial, TExpected, TActual>(input: {
         judgeSummary: null,
         error: message,
         tokenUsage: null,
-        finalContextTokens: null,
       };
       if (surface) {
         input.onProgress?.({

ai_evals/core/types.ts

@@ -155,34 +155,15 @@ export interface ToolCallArgumentRule {
   field: string;
   stringStartsWithAnyOf?: string[];
   stringMustNotStartWithAnyOf?: string[];
-  /**
-   * Case-insensitive "contains", existential over calls: at least one recorded
-   * call to `tool` must have `field` containing one of these substrings. Other
-   * calls to the same tool may do anything. Use instead of `stringStartsWithAnyOf`
-   * (which is universal over calls) when the meaningful token can appear anywhere
-   * in the value and the model may make additional, unrelated calls to the same
-   * tool — e.g. SQL where a mutation is mixed with verification SELECTs.
-   */
-  stringIncludesAnyOf?: string[];
 }
 
 export interface ToolValidationSpec {
   requiredToolsUsed?: string[];
-  /**
-   * Each inner array is an alternatives group: the check passes when at least
-   * one tool in the group was used. Use when several tools satisfy the same
-   * intent so a model that picks any valid path passes — e.g. inspecting an
-   * app's files via either `read_app_file` or `search_app`.
-   */
-  requiredToolsAnyOf?: string[][];
   forbiddenToolsUsed?: string[];
   toolCallArgs?: ToolCallArgumentRule[];
 }
 
-export type EvalValidationSpec =
-  | FlowValidationSpec
-  | AppValidationSpec
-  | GlobalValidationSpec;
+export type EvalValidationSpec = FlowValidationSpec | AppValidationSpec | GlobalValidationSpec;
 
 export interface EvalCase {
   id: string;
@@ -259,12 +240,6 @@ export interface ModeRunOutput<TActual> {
   toolCallDetails?: ToolCallDetail[];
   skillsInvoked: string[];
   tokenUsage?: BenchmarkTokenUsage | null;
-  /**
-   * Total input tokens occupying the context window on the LAST model request
-   * of the agentic loop (input + cache-creation + cache-read). Complements the
-   * cumulative `tokenUsage.prompt`, which sums every iteration's input.
-   */
-  finalContextTokens?: number | null;
 }
 
 export interface ModeRunContext {
@@ -310,12 +285,6 @@ export interface ModeRunner<TInitial, TExpected, TActual> {
     context: ModeRunContext;
   }): Promise<BackendValidationResult | null>;
   buildArtifacts?(actual: TActual): BenchmarkArtifactFile[];
-  /**
-   * Optional transform applied to `actual` before it is handed to the LLM judge.
-   * Use it to strip fields the judge must stay blind to (e.g. which docs-tool
-   * arm produced an answer). When omitted, the judge receives `actual` as-is.
-   */
-  prepareJudgeActual?(actual: TActual): unknown;
 }
 
 export interface BenchmarkAttemptResult {
@@ -332,7 +301,6 @@ export interface BenchmarkAttemptResult {
   judgeSummary: string | null;
   error: string | null;
   tokenUsage?: BenchmarkTokenUsage | null;
-  finalContextTokens?: number | null;
   artifactsPath?: string | null;
   artifactFiles?: BenchmarkArtifactFile[];
 }
@@ -363,8 +331,6 @@ export interface BenchmarkRunResult {
   totalPassedTokenUsage?: BenchmarkTokenUsage | null;
   averageTokenUsagePerAttempt?: BenchmarkTokenUsage | null;
   averageTokenUsagePerPassedAttempt?: BenchmarkTokenUsage | null;
-  averageFinalContextTokensPassed?: number | null;
-  maxFinalContextTokensPassed?: number | null;
   artifactsPath?: string | null;
   cases: BenchmarkCaseResult[];
 }

ai_evals/core/validators.test.ts

@@ -140,154 +140,6 @@ describe("validateToolExpectations", () => {
       details: "tools used: write_script, deploy_workspace_item",
     });
   });
-
-  it("accepts a stringIncludesAnyOf substring regardless of case or position", () => {
-    const checks = validateToolExpectations({
-      run: {
-        success: true,
-        actual: {},
-        assistantMessageCount: 1,
-        toolCallCount: 1,
-        toolsUsed: ["exec_datatable_sql"],
-        toolCallDetails: [
-          {
-            name: "exec_datatable_sql",
-            arguments: {
-              sql: "WITH recent AS (SELECT * FROM orders) SELECT count(*) FROM recent",
-            },
-          },
-        ],
-        skillsInvoked: [],
-      },
-      toolExpect: {
-        requiredToolsUsed: ["exec_datatable_sql"],
-        toolCallArgs: [
-          {
-            tool: "exec_datatable_sql",
-            field: "sql",
-            stringIncludesAnyOf: ["select"],
-          },
-        ],
-      },
-    });
-
-    expect(checks.every((check) => check.passed)).toBe(true);
-  });
-
-  it("accepts stringIncludesAnyOf when only one of several calls matches", () => {
-    // Existential: a mutation mixed with verification SELECTs still passes.
-    const checks = validateToolExpectations({
-      run: {
-        success: true,
-        actual: {},
-        assistantMessageCount: 1,
-        toolCallCount: 2,
-        toolsUsed: ["exec_datatable_sql"],
-        toolCallDetails: [
-          {
-            name: "exec_datatable_sql",
-            arguments: { sql: "UPDATE orders SET status = 'shipped' WHERE id = 2" },
-          },
-          {
-            name: "exec_datatable_sql",
-            arguments: { sql: "SELECT * FROM orders WHERE id = 2" },
-          },
-        ],
-        skillsInvoked: [],
-      },
-      toolExpect: {
-        toolCallArgs: [
-          {
-            tool: "exec_datatable_sql",
-            field: "sql",
-            stringIncludesAnyOf: ["insert into", "update"],
-          },
-        ],
-      },
-    });
-
-    expect(checks.every((check) => check.passed)).toBe(true);
-  });
-
-  it("rejects stringIncludesAnyOf when no call matches any substring", () => {
-    const checks = validateToolExpectations({
-      run: {
-        success: true,
-        actual: {},
-        assistantMessageCount: 1,
-        toolCallCount: 1,
-        toolsUsed: ["exec_datatable_sql"],
-        toolCallDetails: [
-          {
-            name: "exec_datatable_sql",
-            arguments: {
-              sql: "DROP TABLE orders",
-            },
-          },
-        ],
-        skillsInvoked: [],
-      },
-      toolExpect: {
-        toolCallArgs: [
-          {
-            tool: "exec_datatable_sql",
-            field: "sql",
-            stringIncludesAnyOf: ["insert into", "update"],
-          },
-        ],
-      },
-    });
-
-    expect(checks).toContainEqual({
-      name: "exec_datatable_sql.sql includes a required substring",
-      passed: false,
-      details:
-        'accepted substrings: insert into, update; values: "DROP TABLE orders"',
-    });
-  });
-
-  it("passes requiredToolsAnyOf when any alternative in the group is used", () => {
-    const checks = validateToolExpectations({
-      run: {
-        success: true,
-        actual: {},
-        assistantMessageCount: 1,
-        toolCallCount: 1,
-        toolsUsed: ["search_app", "patch_app_file"],
-        skillsInvoked: [],
-      },
-      toolExpect: {
-        requiredToolsAnyOf: [["read_app_file", "search_app"]],
-      },
-    });
-
-    expect(checks).toContainEqual({
-      name: "uses one of read_app_file, search_app",
-      passed: true,
-    });
-  });
-
-  it("fails requiredToolsAnyOf when no alternative in the group is used", () => {
-    const checks = validateToolExpectations({
-      run: {
-        success: true,
-        actual: {},
-        assistantMessageCount: 1,
-        toolCallCount: 1,
-        toolsUsed: ["patch_app_file"],
-        skillsInvoked: [],
-      },
-      toolExpect: {
-        requiredToolsAnyOf: [["read_app_file", "search_app"]],
-      },
-    });
-
-    expect(checks).toContainEqual({
-      name: "uses one of read_app_file, search_app",
-      passed: false,
-      details: "tools used: patch_app_file",
-    });
-  });
 });
 
 describe("validateGlobalState", () => {

ai_evals/core/validators.ts

@@ -169,16 +169,6 @@ export function validateToolExpectations(input: {
     );
   }
 
-  for (const group of expect.requiredToolsAnyOf ?? []) {
-    checks.push(
-      check(
-        `uses one of ${group.join(", ")}`,
-        group.some((toolName) => input.run.toolsUsed.includes(toolName)),
-        `tools used: ${input.run.toolsUsed.join(", ") || "none"}`
-      )
-    );
-  }
-
   for (const toolName of expect.forbiddenToolsUsed ?? []) {
     checks.push(
       check(
@@ -232,25 +222,6 @@ export function validateToolExpectations(input: {
         )
       );
     }
-
-    if (rule.stringIncludesAnyOf && rule.stringIncludesAnyOf.length > 0) {
-      // Existential: at least one call must contain one of the substrings.
-      // Other calls to the same tool may do anything — this suits SQL, where a
-      // model mixes the requested statement (e.g. an UPDATE) with verification
-      // SELECTs that would otherwise fail an "all calls" check.
-      const needles = rule.stringIncludesAnyOf.map((needle) => needle.toLowerCase());
-      const hasMatch = values.some(
-        (value) =>
-          typeof value === "string" && needles.some((needle) => value.toLowerCase().includes(needle))
-      );
-      checks.push(
-        check(
-          `${rule.tool}.${rule.field} includes a required substring`,
-          hasMatch,
-          `accepted substrings: ${rule.stringIncludesAnyOf.join(", ")}; values: ${summarizeToolValues(values)}`
-        )
-      );
-    }
   }
 
   return checks;

ai_evals/fixtures/frontend/global/initial/analytics_dashboard/backend/computeSummary/main.ts

@@ -1,68 +0,0 @@
-type OrderStatus = 'paid' | 'shipped' | 'delivered' | 'pending' | 'refunded' | 'cancelled'
-
-interface Order {
-	id: string
-	region: string
-	quantity: number
-	unitPrice: number
-	status: OrderStatus
-	placedAt: string
-}
-
-// Server-side revenue rollup. Mirrors the client aggregation but is computed
-// from the authoritative mocked order book so it can be used to cross-check
-// the dashboard and to back the export.
-const orders: Order[] = [
-	{ id: 'ORD-10001', region: 'North America', quantity: 3, unitPrice: 1195, status: 'delivered', placedAt: '2024-05-02' },
-	{ id: 'ORD-10002', region: 'EMEA', quantity: 5, unitPrice: 880, status: 'shipped', placedAt: '2024-05-03' },
-	{ id: 'ORD-10003', region: 'APAC', quantity: 2, unitPrice: 640, status: 'paid', placedAt: '2024-05-05' },
-	{ id: 'ORD-10004', region: 'LATAM', quantity: 7, unitPrice: 315, status: 'delivered', placedAt: '2024-05-07' },
-	{ id: 'ORD-10005', region: 'North America', quantity: 4, unitPrice: 150, status: 'refunded', placedAt: '2024-05-09' },
-	{ id: 'ORD-10006', region: 'EMEA', quantity: 6, unitPrice: 220, status: 'shipped', placedAt: '2024-05-12' },
-	{ id: 'ORD-10007', region: 'APAC', quantity: 1, unitPrice: 980, status: 'pending', placedAt: '2024-05-15' },
-	{ id: 'ORD-10008', region: 'North America', quantity: 8, unitPrice: 1100, status: 'delivered', placedAt: '2024-05-18' },
-	{ id: 'ORD-10009', region: 'EMEA', quantity: 2, unitPrice: 860, status: 'cancelled', placedAt: '2024-05-22' },
-	{ id: 'ORD-10010', region: 'LATAM', quantity: 9, unitPrice: 290, status: 'paid', placedAt: '2024-05-26' }
-]
-
-const REVENUE_STATUSES: OrderStatus[] = ['paid', 'shipped', 'delivered']
-
-export async function main({
-	from,
-	to,
-	region
-}: {
-	from: string
-	to: string
-	region: string
-}): Promise<{
-	totalRevenue: number
-	netRevenue: number
-	totalOrders: number
-	averageOrderValue: number
-	unitsSold: number
-	refundedRevenue: number
-	currency: string
-}> {
-	let scoped = orders.filter((order) => order.placedAt >= from && order.placedAt <= to)
-	if (region && region !== 'all') {
-		scoped = scoped.filter((order) => order.region === region)
-	}
-
-	const booked = scoped.filter((order) => REVENUE_STATUSES.includes(order.status))
-	const totalRevenue = booked.reduce((acc, order) => acc + order.unitPrice * order.quantity, 0)
-	const unitsSold = booked.reduce((acc, order) => acc + order.quantity, 0)
-	const refundedRevenue = scoped
-		.filter((order) => order.status === 'refunded')
-		.reduce((acc, order) => acc + order.unitPrice * order.quantity, 0)
-
-	return {
-		totalRevenue,
-		netRevenue: totalRevenue - refundedRevenue,
-		totalOrders: booked.length,
-		averageOrderValue: booked.length === 0 ? 0 : Math.round(totalRevenue / booked.length),
-		unitsSold,
-		refundedRevenue,
-		currency: 'USD'
-	}
-}

ai_evals/fixtures/frontend/global/initial/analytics_dashboard/backend/computeSummary/meta.json

@@ -1,4 +0,0 @@
-{
-	"name": "Compute Summary",
-	"language": "bun"
-}

ai_evals/fixtures/frontend/global/initial/analytics_dashboard/backend/exportReport/main.ts

@@ -1,51 +0,0 @@
-// Builds a downloadable report for the current dashboard view. Returns a data
-// URL the browser can open directly so the export works without object storage.
-export async function main({
-	from,
-	to,
-	region,
-	format
-}: {
-	from: string
-	to: string
-	region: string
-	format: 'csv' | 'json'
-}): Promise<{ url: string; rows: number; filename: string }> {
-	const summary = {
-		from,
-		to,
-		region: region || 'all',
-		generatedAt: new Date().toISOString(),
-		rows: [
-			{ region: 'North America', revenue: 211_400, orders: 168 },
-			{ region: 'EMEA', revenue: 142_900, orders: 121 },
-			{ region: 'APAC', revenue: 86_500, orders: 78 },
-			{ region: 'LATAM', revenue: 41_500, orders: 45 }
-		]
-	}
-
-	const scoped =
-		region && region !== 'all'
-			? summary.rows.filter((row) => row.region === region)
-			: summary.rows
-
-	let body: string
-	let mime: string
-	if (format === 'csv') {
-		const header = 'region,revenue,orders'
-		const lines = scoped.map((row) => `${row.region},${row.revenue},${row.orders}`)
-		body = [header, ...lines].join('\n')
-		mime = 'text/csv'
-	} else {
-		body = JSON.stringify({ ...summary, rows: scoped }, null, 2)
-		mime = 'application/json'
-	}
-
-	const encoded = Buffer.from(body, 'utf-8').toString('base64')
-	const filename = `revenue-report-${from}_${to}.${format}`
-	return {
-		url: `data:${mime};base64,${encoded}`,
-		rows: scoped.length,
-		filename
-	}
-}

ai_evals/fixtures/frontend/global/initial/analytics_dashboard/backend/exportReport/meta.json

@@ -1,4 +0,0 @@
-{
-	"name": "Export Report",
-	"language": "bun"
-}

ai_evals/fixtures/frontend/global/initial/analytics_dashboard/backend/loadMetrics/main.ts

@@ -1,40 +0,0 @@
-interface MetricCardData {
-	id: string
-	label: string
-	value: number
-	unit: 'currency' | 'count' | 'percent'
-	delta: number
-	hint: string
-}
-
-// Returns the headline metric cards for the selected range and region. Values
-// are mocked but internally consistent (revenue / orders ≈ avg order value).
-const baseByRegion: Record<string, { revenue: number; orders: number; units: number; refunds: number }> = {
-	all: { revenue: 482_300, orders: 412, units: 1840, refunds: 11_900 },
-	'North America': { revenue: 211_400, orders: 168, units: 770, refunds: 4_200 },
-	EMEA: { revenue: 142_900, orders: 121, units: 560, refunds: 3_500 },
-	APAC: { revenue: 86_500, orders: 78, units: 340, refunds: 2_600 },
-	LATAM: { revenue: 41_500, orders: 45, units: 170, refunds: 1_600 }
-}
-
-export async function main({
-	from,
-	to,
-	region
-}: {
-	from: string
-	to: string
-	region: string
-}): Promise<{ cards: MetricCardData[]; generatedAt: string }> {
-	const base = baseByRegion[region] ?? baseByRegion.all
-	const aov = base.orders === 0 ? 0 : Math.round(base.revenue / base.orders)
-	const cards: MetricCardData[] = [
-		{ id: 'revenue', label: 'Total Revenue', value: base.revenue, unit: 'currency', delta: 0.082, hint: `Booked revenue ${from} – ${to}` },
-		{ id: 'orders', label: 'Orders', value: base.orders, unit: 'count', delta: 0.041, hint: 'Revenue-bearing orders in range' },
-		{ id: 'aov', label: 'Avg Order Value', value: aov, unit: 'currency', delta: -0.013, hint: 'Total revenue / order count' },
-		{ id: 'units', label: 'Units Sold', value: base.units, unit: 'count', delta: 0.067, hint: 'Total units in range' },
-		{ id: 'refunds', label: 'Refunded', value: base.refunds, unit: 'currency', delta: -0.021, hint: 'Revenue lost to refunds' },
-		{ id: 'conversion', label: 'Conversion', value: 0.187, unit: 'percent', delta: 0.009, hint: 'Sessions that became orders' }
-	]
-	return { cards, generatedAt: new Date().toISOString() }
-}

ai_evals/fixtures/frontend/global/initial/analytics_dashboard/backend/loadMetrics/meta.json

@@ -1,4 +0,0 @@
-{
-	"name": "Load Metrics",
-	"language": "bun"
-}

ai_evals/fixtures/frontend/global/initial/analytics_dashboard/backend/loadOrders/main.ts

@@ -1,54 +0,0 @@
-type OrderStatus = 'paid' | 'shipped' | 'delivered' | 'pending' | 'refunded' | 'cancelled'
-
-interface Order {
-	id: string
-	placedAt: string
-	customer: string
-	product: string
-	sku: string
-	region: string
-	channel: string
-	rep: string
-	quantity: number
-	unitPrice: number
-	status: OrderStatus
-}
-
-// Mocked order book. In a real deployment this would query the orders table;
-// here it returns a representative slice so the table renders in preview.
-const orders: Order[] = [
-	{ id: 'ORD-10001', placedAt: '2024-05-02T09:14:00Z', customer: 'Contoso Ltd', product: 'Aurora Analytics Suite', sku: 'ANL-100', region: 'North America', channel: 'direct', rep: 'Dana Wills', quantity: 3, unitPrice: 1195, status: 'delivered' },
-	{ id: 'ORD-10002', placedAt: '2024-05-03T11:42:00Z', customer: 'Fabrikam Inc', product: 'Borealis CRM', sku: 'CRM-210', region: 'EMEA', channel: 'partner', rep: 'Lena Fischer', quantity: 5, unitPrice: 880, status: 'shipped' },
-	{ id: 'ORD-10003', placedAt: '2024-05-05T15:03:00Z', customer: 'Tailspin Toys', product: 'Cascade Data Pipeline', sku: 'PIPE-330', region: 'APAC', channel: 'self-serve', rep: 'Sora Tanaka', quantity: 2, unitPrice: 640, status: 'paid' },
-	{ id: 'ORD-10004', placedAt: '2024-05-07T08:21:00Z', customer: 'Proseware Inc', product: 'Delta Insights', sku: 'INS-440', region: 'LATAM', channel: 'marketplace', rep: 'Diego Marin', quantity: 7, unitPrice: 315, status: 'delivered' },
-	{ id: 'ORD-10005', placedAt: '2024-05-09T13:58:00Z', customer: 'Litware Inc', product: 'Echo Monitoring', sku: 'MON-550', region: 'North America', channel: 'direct', rep: 'Owen Pratt', quantity: 4, unitPrice: 150, status: 'refunded' },
-	{ id: 'ORD-10006', placedAt: '2024-05-12T10:30:00Z', customer: 'Fourth Coffee', product: 'Helix Identity', sku: 'IDN-880', region: 'EMEA', channel: 'partner', rep: 'Aisha Khan', quantity: 6, unitPrice: 220, status: 'shipped' },
-	{ id: 'ORD-10007', placedAt: '2024-05-15T17:11:00Z', customer: 'Coho Vineyard', product: 'Kelvin Forecasting', sku: 'FCT-202', region: 'APAC', channel: 'direct', rep: 'Priya Nair', quantity: 1, unitPrice: 980, status: 'pending' },
-	{ id: 'ORD-10008', placedAt: '2024-05-18T12:05:00Z', customer: 'Alpine Ski House', product: 'Nimbus Compute', sku: 'CMP-505', region: 'North America', channel: 'self-serve', rep: 'Hugo Bernard', quantity: 8, unitPrice: 1100, status: 'delivered' },
-	{ id: 'ORD-10009', placedAt: '2024-05-22T14:47:00Z', customer: 'Trey Research', product: 'Onyx Security', sku: 'SEC-606', region: 'EMEA', channel: 'direct', rep: 'Sven Olsen', quantity: 2, unitPrice: 860, status: 'cancelled' },
-	{ id: 'ORD-10010', placedAt: '2024-05-26T16:39:00Z', customer: 'Blue Yonder Airlines', product: 'Polaris Reporting', sku: 'RPT-707', region: 'LATAM', channel: 'partner', rep: 'Mateo Russo', quantity: 9, unitPrice: 290, status: 'paid' }
-]
-
-export async function main({
-	from,
-	to,
-	region,
-	status
-}: {
-	from: string
-	to: string
-	region: string
-	status: string
-}): Promise<{ orders: Order[]; total: number }> {
-	let filtered = orders.filter((order) => {
-		const day = order.placedAt.slice(0, 10)
-		return day >= from && day <= to
-	})
-	if (region && region !== 'all') {
-		filtered = filtered.filter((order) => order.region === region)
-	}
-	if (status && status !== 'all') {
-		filtered = filtered.filter((order) => order.status === status)
-	}
-	return { orders: filtered, total: filtered.length }
-}

ai_evals/fixtures/frontend/global/initial/analytics_dashboard/backend/loadOrders/meta.json

@@ -1,4 +0,0 @@
-{
-	"name": "Load Orders",
-	"language": "bun"
-}

ai_evals/fixtures/frontend/global/initial/analytics_dashboard/frontend/components/DateRangePicker.tsx

@@ -1,45 +0,0 @@
-import React from 'react'
-import type { DateRange } from '../lib/api'
-import { rangeForPreset } from '../lib/api'
-import { formatDateShort } from '../lib/format'
-
-interface DateRangePickerProps {
-	preset: string
-	range: DateRange
-	onPresetChange: (preset: string, range: DateRange) => void
-}
-
-const PRESETS: { id: string; label: string }[] = [
-	{ id: '7d', label: 'Last 7 days' },
-	{ id: '14d', label: 'Last 14 days' },
-	{ id: '30d', label: 'Last 30 days' },
-	{ id: 'qtd', label: 'Quarter to date' }
-]
-
-export const DateRangePicker: React.FC<DateRangePickerProps> = ({
-	preset,
-	range,
-	onPresetChange
-}) => {
-	return (
-		<div className="flex items-center gap-2">
-			<select
-				className="rounded-lg border border-gray-300 bg-white px-3 py-2 text-sm text-gray-700"
-				value={preset}
-				onChange={(event) => {
-					const next = event.target.value
-					onPresetChange(next, rangeForPreset(next))
-				}}
-			>
-				{PRESETS.map((item) => (
-					<option key={item.id} value={item.id}>
-						{item.label}
-					</option>
-				))}
-			</select>
-			<span className="text-xs text-gray-400">
-				{formatDateShort(range.from)} – {formatDateShort(range.to)}
-			</span>
-		</div>
-	)
-}

ai_evals/fixtures/frontend/global/initial/analytics_dashboard/frontend/components/EmptyState.tsx

@@ -1,28 +0,0 @@
-import React from 'react'
-
-interface EmptyStateProps {
-	title: string
-	description?: string
-	icon?: string
-	action?: React.ReactNode
-}
-
-export const EmptyState: React.FC<EmptyStateProps> = ({
-	title,
-	description,
-	icon = '📊',
-	action
-}) => {
-	return (
-		<div className="flex flex-col items-center justify-center rounded-lg border border-dashed border-gray-300 bg-white py-12 text-center">
-			<div className="text-3xl" aria-hidden>
-				{icon}
-			</div>
-			<h3 className="mt-3 text-sm font-semibold text-gray-700">{title}</h3>
-			{description ? (
-				<p className="mt-1 max-w-sm text-sm text-gray-500">{description}</p>
-			) : null}
-			{action ? <div className="mt-4">{action}</div> : null}
-		</div>
-	)
-}

ai_evals/fixtures/frontend/global/initial/analytics_dashboard/frontend/components/ExportButton.tsx

@@ -1,51 +0,0 @@
-import React, { useState } from 'react'
-import { requestExport } from '../lib/api'
-import type { DateRange } from '../lib/api'
-
-interface ExportButtonProps {
-	range: DateRange
-	region: string
-}
-
-export const ExportButton: React.FC<ExportButtonProps> = ({ range, region }) => {
-	const [busy, setBusy] = useState(false)
-	const [error, setError] = useState<string | null>(null)
-
-	const handleExport = async (format: 'csv' | 'json') => {
-		setBusy(true)
-		setError(null)
-		try {
-			const result = await requestExport(range, region, format)
-			const anchor = document.createElement('a')
-			anchor.href = result.url
-			anchor.download = `revenue-report.${format}`
-			anchor.click()
-		} catch (err) {
-			setError(err instanceof Error ? err.message : 'Export failed')
-		} finally {
-			setBusy(false)
-		}
-	}
-
-	return (
-		<div className="flex items-center gap-2">
-			<button
-				type="button"
-				disabled={busy}
-				onClick={() => handleExport('csv')}
-				className="rounded-lg border border-gray-300 bg-white px-3 py-2 text-sm font-medium text-gray-700 hover:bg-gray-50 disabled:opacity-50"
-			>
-				{busy ? 'Exporting…' : 'Export CSV'}
-			</button>
-			<button
-				type="button"
-				disabled={busy}
-				onClick={() => handleExport('json')}
-				className="rounded-lg border border-gray-300 bg-white px-3 py-2 text-sm font-medium text-gray-700 hover:bg-gray-50 disabled:opacity-50"
-			>
-				Export JSON
-			</button>
-			{error ? <span className="text-xs text-rose-600">{error}</span> : null}
-		</div>
-	)
-}

ai_evals/fixtures/frontend/global/initial/analytics_dashboard/frontend/components/FilterBar.tsx

@@ -1,59 +0,0 @@
-import React from 'react'
-import type { DateRange } from '../lib/api'
-import type { OrderStatus } from '../data/seedData'
-import { REGIONS, ORDER_STATUSES, STATUS_LABELS } from '../data/seedData'
-import { DateRangePicker } from './DateRangePicker'
-import { ExportButton } from './ExportButton'
-
-interface FilterBarProps {
-	region: string
-	status: string
-	preset: string
-	range: DateRange
-	onRegionChange: (region: string) => void
-	onStatusChange: (status: string) => void
-	onPresetChange: (preset: string, range: DateRange) => void
-}
-
-export const FilterBar: React.FC<FilterBarProps> = ({
-	region,
-	status,
-	preset,
-	range,
-	onRegionChange,
-	onStatusChange,
-	onPresetChange
-}) => {
-	return (
-		<div className="flex flex-wrap items-center justify-between gap-3 border-b border-gray-200 bg-white px-6 py-4">
-			<div className="flex flex-wrap items-center gap-3">
-				<DateRangePicker preset={preset} range={range} onPresetChange={onPresetChange} />
-				<select
-					className="rounded-lg border border-gray-300 bg-white px-3 py-2 text-sm text-gray-700"
-					value={region}
-					onChange={(event) => onRegionChange(event.target.value)}
-				>
-					<option value="all">All regions</option>
-					{REGIONS.map((item) => (
-						<option key={item} value={item}>
-							{item}
-						</option>
-					))}
-				</select>
-				<select
-					className="rounded-lg border border-gray-300 bg-white px-3 py-2 text-sm text-gray-700"
-					value={status}
-					onChange={(event) => onStatusChange(event.target.value)}
-				>
-					<option value="all">All statuses</option>
-					{ORDER_STATUSES.map((item) => (
-						<option key={item} value={item}>
-							{STATUS_LABELS[item as OrderStatus]}
-						</option>
-					))}
-				</select>
-			</div>
-			<ExportButton range={range} region={region} />
-		</div>
-	)
-}

ai_evals/fixtures/frontend/global/initial/analytics_dashboard/frontend/components/MetricCard.tsx

@@ -1,40 +0,0 @@
-import React from 'react'
-import type { MetricCardData } from '../data/seedData'
-import { formatCurrency, formatNumber, formatPercent, formatSignedPercent } from '../lib/format'
-
-interface MetricCardProps {
-	metric: MetricCardData
-	loading?: boolean
-}
-
-function renderValue(metric: MetricCardData): string {
-	switch (metric.unit) {
-		case 'currency':
-			return formatCurrency(metric.value)
-		case 'percent':
-			return formatPercent(metric.value)
-		case 'count':
-		default:
-			return formatNumber(metric.value)
-	}
-}
-
-export const MetricCard: React.FC<MetricCardProps> = ({ metric, loading }) => {
-	const positive = metric.delta >= 0
-	return (
-		<div className="rounded-xl border border-gray-200 bg-white p-5 shadow-sm">
-			<div className="flex items-center justify-between">
-				<span className="text-sm font-medium text-gray-500">{metric.label}</span>
-				<span
-					className={`text-xs font-semibold ${positive ? 'text-emerald-600' : 'text-rose-600'}`}
-				>
-					{formatSignedPercent(metric.delta)}
-				</span>
-			</div>
-			<div className="mt-2 text-2xl font-bold text-gray-900">
-				{loading ? <span className="text-gray-300">…</span> : renderValue(metric)}
-			</div>
-			<p className="mt-1 text-xs text-gray-400">{metric.hint}</p>
-		</div>
-	)
-}

ai_evals/fixtures/frontend/global/initial/analytics_dashboard/frontend/components/MetricGrid.tsx

@@ -1,18 +0,0 @@
-import React from 'react'
-import type { MetricCardData } from '../data/seedData'
-import { MetricCard } from './MetricCard'
-
-interface MetricGridProps {
-	metrics: MetricCardData[]
-	loading?: boolean
-}
-
-export const MetricGrid: React.FC<MetricGridProps> = ({ metrics, loading }) => {
-	return (
-		<div className="grid grid-cols-1 gap-4 sm:grid-cols-2 lg:grid-cols-3">
-			{metrics.map((metric) => (
-				<MetricCard key={metric.id} metric={metric} loading={loading} />
-			))}
-		</div>
-	)
-}

ai_evals/fixtures/frontend/global/initial/analytics_dashboard/frontend/components/OrdersTable.tsx

@@ -1,117 +0,0 @@
-import React, { useMemo, useState } from 'react'
-import type { Order } from '../data/seedData'
-import { StatusBadge } from './StatusBadge'
-import { EmptyState } from './EmptyState'
-import { formatCurrencyPrecise, formatDate, formatNumber, truncate } from '../lib/format'
-
-interface OrdersTableProps {
-	orders: Order[]
-	loading?: boolean
-}
-
-type SortKey = 'placedAt' | 'customer' | 'lineTotal' | 'quantity'
-type SortDir = 'asc' | 'desc'
-
-// The per-row line total a customer was charged: unit price times quantity.
-function lineTotal(order: Order): number {
-	return order.quantity * order.unitPrice
-}
-
-export const OrdersTable: React.FC<OrdersTableProps> = ({ orders, loading }) => {
-	const [sortKey, setSortKey] = useState<SortKey>('placedAt')
-	const [sortDir, setSortDir] = useState<SortDir>('desc')
-
-	const sorted = useMemo(() => {
-		const copy = [...orders]
-		copy.sort((a, b) => {
-			let comparison = 0
-			switch (sortKey) {
-				case 'customer':
-					comparison = a.customer.localeCompare(b.customer)
-					break
-				case 'lineTotal':
-					comparison = lineTotal(a) - lineTotal(b)
-					break
-				case 'quantity':
-					comparison = a.quantity - b.quantity
-					break
-				case 'placedAt':
-				default:
-					comparison = a.placedAt.localeCompare(b.placedAt)
-					break
-			}
-			return sortDir === 'asc' ? comparison : -comparison
-		})
-		return copy
-	}, [orders, sortKey, sortDir])
-
-	const toggleSort = (key: SortKey) => {
-		if (key === sortKey) {
-			setSortDir((dir) => (dir === 'asc' ? 'desc' : 'asc'))
-		} else {
-			setSortKey(key)
-			setSortDir('desc')
-		}
-	}
-
-	if (!loading && orders.length === 0) {
-		return (
-			<EmptyState
-				title="No orders match these filters"
-				description="Try widening the date range or clearing the status filter."
-				icon="🗂️"
-			/>
-		)
-	}
-
-	const arrow = (key: SortKey) => (key === sortKey ? (sortDir === 'asc' ? '▲' : '▼') : '')
-
-	return (
-		<div className="overflow-hidden rounded-xl border border-gray-200 bg-white shadow-sm">
-			<table className="min-w-full divide-y divide-gray-200 text-sm">
-				<thead className="bg-gray-50 text-left text-xs uppercase tracking-wide text-gray-500">
-					<tr>
-						<th className="cursor-pointer px-4 py-3" onClick={() => toggleSort('placedAt')}>
-							Date {arrow('placedAt')}
-						</th>
-						<th className="cursor-pointer px-4 py-3" onClick={() => toggleSort('customer')}>
-							Customer {arrow('customer')}
-						</th>
-						<th className="px-4 py-3">Product</th>
-						<th className="px-4 py-3">Region</th>
-						<th className="cursor-pointer px-4 py-3 text-right" onClick={() => toggleSort('quantity')}>
-							Qty {arrow('quantity')}
-						</th>
-						<th className="px-4 py-3 text-right">Unit Price</th>
-						<th className="cursor-pointer px-4 py-3 text-right" onClick={() => toggleSort('lineTotal')}>
-							Line Total {arrow('lineTotal')}
-						</th>
-						<th className="px-4 py-3">Status</th>
-					</tr>
-				</thead>
-				<tbody className="divide-y divide-gray-100">
-					{sorted.map((order) => (
-						<tr key={order.id} className="hover:bg-gray-50">
-							<td className="px-4 py-3 text-gray-500">{formatDate(order.placedAt)}</td>
-							<td className="px-4 py-3 font-medium text-gray-900">
-								{truncate(order.customer, 24)}
-							</td>
-							<td className="px-4 py-3 text-gray-600">{order.product}</td>
-							<td className="px-4 py-3 text-gray-600">{order.region}</td>
-							<td className="px-4 py-3 text-right text-gray-600">{formatNumber(order.quantity)}</td>
-							<td className="px-4 py-3 text-right text-gray-600">
-								{formatCurrencyPrecise(order.unitPrice)}
-							</td>
-							<td className="px-4 py-3 text-right font-semibold text-gray-900">
-								{formatCurrencyPrecise(lineTotal(order))}
-							</td>
-							<td className="px-4 py-3">
-								<StatusBadge status={order.status} />
-							</td>
-						</tr>
-					))}
-				</tbody>
-			</table>
-		</div>
-	)
-}

ai_evals/fixtures/frontend/global/initial/analytics_dashboard/frontend/components/RegionTable.tsx

@@ -1,52 +0,0 @@
-import React, { useMemo } from 'react'
-import type { Order } from '../data/seedData'
-import { breakdownByRegion } from '../lib/aggregations'
-import { formatCurrency, formatNumber, formatPercent } from '../lib/format'
-import { EmptyState } from './EmptyState'
-
-interface RegionTableProps {
-	orders: Order[]
-}
-
-export const RegionTable: React.FC<RegionTableProps> = ({ orders }) => {
-	const rows = useMemo(() => breakdownByRegion(orders), [orders])
-	const total = useMemo(() => rows.reduce((acc, row) => acc + row.revenue, 0), [rows])
-
-	if (rows.length === 0) {
-		return (
-			<EmptyState
-				title="No regional revenue"
-				description="No revenue-bearing orders fall in the current selection."
-				icon="🌍"
-			/>
-		)
-	}
-
-	return (
-		<section className="rounded-xl border border-gray-200 bg-white p-6 shadow-sm">
-			<h2 className="mb-4 text-lg font-semibold text-gray-900">Revenue by Region</h2>
-			<table className="min-w-full text-sm">
-				<thead className="text-left text-xs uppercase tracking-wide text-gray-500">
-					<tr>
-						<th className="py-2">Region</th>
-						<th className="py-2 text-right">Orders</th>
-						<th className="py-2 text-right">Revenue</th>
-						<th className="py-2 text-right">Share</th>
-					</tr>
-				</thead>
-				<tbody className="divide-y divide-gray-100">
-					{rows.map((row) => (
-						<tr key={row.region}>
-							<td className="py-2 font-medium text-gray-900">{row.region}</td>
-							<td className="py-2 text-right text-gray-600">{formatNumber(row.orders)}</td>
-							<td className="py-2 text-right text-gray-900">{formatCurrency(row.revenue)}</td>
-							<td className="py-2 text-right text-gray-500">
-								{formatPercent(total === 0 ? 0 : row.revenue / total)}
-							</td>
-						</tr>
-					))}
-				</tbody>
-			</table>
-		</section>
-	)
-}

ai_evals/fixtures/frontend/global/initial/analytics_dashboard/frontend/components/RevenueChart.tsx

@@ -1,49 +0,0 @@
-import React, { useMemo } from 'react'
-import type { Order } from '../data/seedData'
-import { dailyRevenue } from '../lib/aggregations'
-import { formatCompact, formatDateShort } from '../lib/format'
-import { EmptyState } from './EmptyState'
-
-interface RevenueChartProps {
-	orders: Order[]
-}
-
-// Lightweight inline bar chart for daily revenue. Avoids a charting dependency
-// by sizing flexed columns relative to the busiest day in the window.
-export const RevenueChart: React.FC<RevenueChartProps> = ({ orders }) => {
-	const points = useMemo(() => dailyRevenue(orders), [orders])
-	const max = useMemo(() => points.reduce((acc, point) => Math.max(acc, point.revenue), 0), [points])
-
-	if (points.length === 0) {
-		return (
-			<EmptyState
-				title="No revenue in range"
-				description="Adjust the date range or filters to see daily revenue."
-				icon="📉"
-			/>
-		)
-	}
-
-	return (
-		<section className="rounded-xl border border-gray-200 bg-white p-6 shadow-sm">
-			<h2 className="mb-4 text-lg font-semibold text-gray-900">Daily Revenue</h2>
-			<div className="flex h-48 items-end gap-1">
-				{points.map((point) => {
-					const heightPct = max === 0 ? 0 : Math.round((point.revenue / max) * 100)
-					return (
-						<div key={point.date} className="flex flex-1 flex-col items-center justify-end">
-							<div
-								className="w-full rounded-t bg-indigo-400"
-								style={{ height: `${Math.max(heightPct, 2)}%` }}
-								title={`${point.date}: ${formatCompact(point.revenue)}`}
-							/>
-							<span className="mt-1 truncate text-[9px] text-gray-400">
-								{formatDateShort(point.date)}
-							</span>
-						</div>
-					)
-				})}
-			</div>
-		</section>
-	)
-}

ai_evals/fixtures/frontend/global/initial/analytics_dashboard/frontend/components/Sidebar.tsx

@@ -1,50 +0,0 @@
-import React from 'react'
-
-export type DashboardView = 'overview' | 'orders' | 'regions' | 'products'
-
-interface SidebarProps {
-	active: DashboardView
-	onSelect: (view: DashboardView) => void
-}
-
-const NAV_ITEMS: { id: DashboardView; label: string; icon: string }[] = [
-	{ id: 'overview', label: 'Overview', icon: '📈' },
-	{ id: 'orders', label: 'Orders', icon: '🧾' },
-	{ id: 'regions', label: 'Regions', icon: '🌍' },
-	{ id: 'products', label: 'Products', icon: '📦' }
-]
-
-export const Sidebar: React.FC<SidebarProps> = ({ active, onSelect }) => {
-	return (
-		<aside className="flex w-56 flex-col border-r border-gray-200 bg-white">
-			<div className="flex items-center gap-2 border-b border-gray-200 px-5 py-4">
-				<span className="text-xl">🪁</span>
-				<span className="text-sm font-bold text-gray-900">Acme Operations</span>
-			</div>
-			<nav className="flex-1 space-y-1 p-3">
-				{NAV_ITEMS.map((item) => {
-					const isActive = item.id === active
-					return (
-						<button
-							key={item.id}
-							type="button"
-							onClick={() => onSelect(item.id)}
-							className={`flex w-full items-center gap-3 rounded-lg px-3 py-2 text-left text-sm font-medium transition ${
-								isActive
-									? 'bg-indigo-50 text-indigo-700'
-									: 'text-gray-600 hover:bg-gray-50'
-							}`}
-						>
-							<span aria-hidden>{item.icon}</span>
-							{item.label}
-						</button>
-					)
-				})}
-			</nav>
-			<div className="border-t border-gray-200 p-4 text-xs text-gray-400">
-				Analytics workspace
-				<div className="mt-1 font-mono text-[10px] text-gray-300">v2.4.0</div>
-			</div>
-		</aside>
-	)
-}

ai_evals/fixtures/frontend/global/initial/analytics_dashboard/frontend/components/StatusBadge.tsx

@@ -1,26 +0,0 @@
-import React from 'react'
-import type { OrderStatus } from '../data/seedData'
-import { STATUS_LABELS } from '../data/seedData'
-
-interface StatusBadgeProps {
-	status: OrderStatus
-}
-
-const STATUS_STYLES: Record<OrderStatus, string> = {
-	paid: 'bg-blue-100 text-blue-700',
-	shipped: 'bg-indigo-100 text-indigo-700',
-	delivered: 'bg-emerald-100 text-emerald-700',
-	pending: 'bg-amber-100 text-amber-700',
-	refunded: 'bg-rose-100 text-rose-700',
-	cancelled: 'bg-gray-200 text-gray-600'
-}
-
-export const StatusBadge: React.FC<StatusBadgeProps> = ({ status }) => {
-	return (
-		<span
-			className={`inline-flex items-center rounded-full px-2.5 py-0.5 text-xs font-medium ${STATUS_STYLES[status]}`}
-		>
-			{STATUS_LABELS[status]}
-		</span>
-	)
-}

ai_evals/fixtures/frontend/global/initial/analytics_dashboard/frontend/components/SummaryPanel.tsx

@@ -1,51 +0,0 @@
-import React, { useMemo } from 'react'
-import type { Order } from '../data/seedData'
-import { summarizeRevenue } from '../lib/aggregations'
-import { formatCurrency, formatCurrencyPrecise, formatNumber } from '../lib/format'
-
-interface SummaryPanelProps {
-	orders: Order[]
-	loading?: boolean
-}
-
-// Headline revenue panel. It re-aggregates the orders client-side via
-// summarizeRevenue so the totals stay in sync with whatever filter the user
-// has applied, without waiting for another backend round trip.
-export const SummaryPanel: React.FC<SummaryPanelProps> = ({ orders, loading }) => {
-	const summary = useMemo(() => summarizeRevenue(orders), [orders])
-
-	const tiles = [
-		{ label: 'Total Revenue', value: formatCurrency(summary.totalRevenue), emphasis: true },
-		{ label: 'Net Revenue', value: formatCurrency(summary.netRevenue) },
-		{ label: 'Orders', value: formatNumber(summary.totalOrders) },
-		{ label: 'Avg Order Value', value: formatCurrencyPrecise(summary.averageOrderValue) },
-		{ label: 'Units Sold', value: formatNumber(summary.unitsSold) },
-		{ label: 'Refunded', value: formatCurrency(summary.refundedRevenue) }
-	]
-
-	return (
-		<section className="rounded-xl border border-gray-200 bg-white p-6 shadow-sm">
-			<div className="mb-4 flex items-center justify-between">
-				<h2 className="text-lg font-semibold text-gray-900">Revenue Summary</h2>
-				{loading ? <span className="text-xs text-gray-400">Refreshing…</span> : null}
-			</div>
-			<div className="grid grid-cols-2 gap-4 md:grid-cols-3">
-				{tiles.map((tile) => (
-					<div
-						key={tile.label}
-						className={`rounded-lg p-4 ${tile.emphasis ? 'bg-indigo-50' : 'bg-gray-50'}`}
-					>
-						<div className="text-xs font-medium uppercase tracking-wide text-gray-500">
-							{tile.label}
-						</div>
-						<div
-							className={`mt-1 font-bold ${tile.emphasis ? 'text-2xl text-indigo-700' : 'text-xl text-gray-900'}`}
-						>
-							{tile.value}
-						</div>
-					</div>
-				))}
-			</div>
-		</section>
-	)
-}

ai_evals/fixtures/frontend/global/initial/analytics_dashboard/frontend/components/TopProducts.tsx

@@ -1,55 +0,0 @@
-import React, { useMemo } from 'react'
-import type { Order } from '../data/seedData'
-import { topProducts } from '../lib/aggregations'
-import { formatCurrency } from '../lib/format'
-import { EmptyState } from './EmptyState'
-
-interface TopProductsProps {
-	orders: Order[]
-	limit?: number
-}
-
-export const TopProducts: React.FC<TopProductsProps> = ({ orders, limit = 5 }) => {
-	const products = useMemo(() => topProducts(orders, limit), [orders, limit])
-	const max = useMemo(
-		() => products.reduce((acc, item) => Math.max(acc, item.revenue), 0),
-		[products]
-	)
-
-	if (products.length === 0) {
-		return (
-			<EmptyState
-				title="No product revenue"
-				description="No revenue-bearing orders to rank by product."
-				icon="📦"
-			/>
-		)
-	}
-
-	return (
-		<section className="rounded-xl border border-gray-200 bg-white p-6 shadow-sm">
-			<h2 className="mb-4 text-lg font-semibold text-gray-900">Top Products</h2>
-			<ul className="space-y-3">
-				{products.map((item, index) => {
-					const widthPct = max === 0 ? 0 : Math.round((item.revenue / max) * 100)
-					return (
-						<li key={item.product}>
-							<div className="flex items-center justify-between text-sm">
-								<span className="font-medium text-gray-800">
-									{index + 1}. {item.product}
-								</span>
-								<span className="text-gray-600">{formatCurrency(item.revenue)}</span>
-							</div>
-							<div className="mt-1 h-2 w-full overflow-hidden rounded-full bg-gray-100">
-								<div
-									className="h-full rounded-full bg-emerald-400"
-									style={{ width: `${Math.max(widthPct, 2)}%` }}
-								/>
-							</div>
-						</li>
-					)
-				})}
-			</ul>
-		</section>
-	)
-}

ai_evals/fixtures/frontend/global/initial/analytics_dashboard/frontend/index.tsx

@@ -1,164 +0,0 @@
-import React, { useEffect, useMemo, useState } from 'react'
-import { Sidebar, type DashboardView } from './components/Sidebar'
-import { FilterBar } from './components/FilterBar'
-import { MetricGrid } from './components/MetricGrid'
-import { SummaryPanel } from './components/SummaryPanel'
-import { RevenueChart } from './components/RevenueChart'
-import { OrdersTable } from './components/OrdersTable'
-import { RegionTable } from './components/RegionTable'
-import { TopProducts } from './components/TopProducts'
-import { EmptyState } from './components/EmptyState'
-import { fetchMetrics, fetchOrders, rangeForPreset, type DateRange } from './lib/api'
-import {
-	seedOrders,
-	seedMetricCards,
-	ordersInRange,
-	ordersForRegion,
-	type Order,
-	type MetricCardData
-} from './data/seedData'
-
-const App = () => {
-	const [view, setView] = useState<DashboardView>('overview')
-	const [preset, setPreset] = useState('30d')
-	const [range, setRange] = useState<DateRange>(rangeForPreset('30d'))
-	const [region, setRegion] = useState('all')
-	const [status, setStatus] = useState('all')
-
-	const [metrics, setMetrics] = useState<MetricCardData[]>(seedMetricCards)
-	const [orders, setOrders] = useState<Order[]>(seedOrders)
-	const [loadingMetrics, setLoadingMetrics] = useState(true)
-	const [loadingOrders, setLoadingOrders] = useState(true)
-	const [errored, setErrored] = useState(false)
-
-	useEffect(() => {
-		let cancelled = false
-		setLoadingMetrics(true)
-		fetchMetrics(range, region)
-			.then((result) => {
-				if (!cancelled) {
-					setMetrics(result.cards)
-				}
-			})
-			.catch(() => {
-				if (!cancelled) {
-					setMetrics(seedMetricCards)
-				}
-			})
-			.finally(() => {
-				if (!cancelled) {
-					setLoadingMetrics(false)
-				}
-			})
-		return () => {
-			cancelled = true
-		}
-	}, [range, region])
-
-	useEffect(() => {
-		let cancelled = false
-		setLoadingOrders(true)
-		setErrored(false)
-		fetchOrders(range, region, status)
-			.then((result) => {
-				if (!cancelled) {
-					setOrders(result.orders)
-				}
-			})
-			.catch(() => {
-				if (!cancelled) {
-					// Fall back to the bundled seed data so the dashboard still renders.
-					const scoped = ordersForRegion(
-						ordersInRange(seedOrders, range.from, range.to),
-						region
-					).filter((order) => status === 'all' || order.status === status)
-					setOrders(scoped)
-					setErrored(true)
-				}
-			})
-			.finally(() => {
-				if (!cancelled) {
-					setLoadingOrders(false)
-				}
-			})
-		return () => {
-			cancelled = true
-		}
-	}, [range, region, status])
-
-	const handlePresetChange = (nextPreset: string, nextRange: DateRange) => {
-		setPreset(nextPreset)
-		setRange(nextRange)
-	}
-
-	// Orders that drive the summary/chart panels — the table applies the status
-	// filter itself, so the panels see the same range/region scoped orders.
-	const scopedOrders = useMemo(() => orders, [orders])
-
-	const renderView = () => {
-		switch (view) {
-			case 'orders':
-				return <OrdersTable orders={scopedOrders} loading={loadingOrders} />
-			case 'regions':
-				return <RegionTable orders={scopedOrders} />
-			case 'products':
-				return <TopProducts orders={scopedOrders} limit={8} />
-			case 'overview':
-			default:
-				return (
-					<div className="space-y-6">
-						<MetricGrid metrics={metrics} loading={loadingMetrics} />
-						<SummaryPanel orders={scopedOrders} loading={loadingOrders} />
-						<div className="grid grid-cols-1 gap-6 lg:grid-cols-2">
-							<RevenueChart orders={scopedOrders} />
-							<TopProducts orders={scopedOrders} />
-						</div>
-						<RegionTable orders={scopedOrders} />
-					</div>
-				)
-		}
-	}
-
-	return (
-		<div className="flex h-screen bg-gray-100 text-gray-900">
-			<Sidebar active={view} onSelect={setView} />
-			<div className="flex flex-1 flex-col overflow-hidden">
-				<header className="border-b border-gray-200 bg-white px-6 py-5">
-					<p className="text-xs font-semibold uppercase tracking-wide text-indigo-500">
-						Acme Inc
-					</p>
-					<h1 className="text-2xl font-bold text-gray-900">Operations Console</h1>
-					<p className="mt-1 text-sm text-gray-500">
-						Revenue, orders, and regional performance at a glance.
-					</p>
-				</header>
-				<FilterBar
-					region={region}
-					status={status}
-					preset={preset}
-					range={range}
-					onRegionChange={setRegion}
-					onStatusChange={setStatus}
-					onPresetChange={handlePresetChange}
-				/>
-				<main className="flex-1 overflow-auto p-6">
-					{errored ? (
-						<div className="mb-4 rounded-lg border border-amber-200 bg-amber-50 px-4 py-2 text-sm text-amber-700">
-							Showing locally bundled data — the live feed is unavailable.
-						</div>
-					) : null}
-					{scopedOrders.length === 0 && !loadingOrders ? (
-						<EmptyState
-							title="Nothing to show yet"
-							description="No data for the selected range, region, and status."
-						/>
-					) : (
-						renderView()
-					)}
-				</main>
-			</div>
-		</div>
-	)
-}
-
-export default App

ai_evals/fixtures/frontend/global/initial/analytics_dashboard/frontend/lib/aggregations.ts

@@ -1,149 +0,0 @@
-// Aggregation helpers that turn raw order/metric rows into the numbers the
-// dashboard renders. These run client-side after the backend returns rows so
-// the UI can re-aggregate instantly when filters change without a round trip.
-
-import type { Order, OrderStatus } from '../data/seedData'
-
-export interface RevenueSummary {
-	totalRevenue: number
-	totalOrders: number
-	averageOrderValue: number
-	unitsSold: number
-	refundedRevenue: number
-	netRevenue: number
-}
-
-export interface StatusBreakdown {
-	status: OrderStatus
-	orders: number
-	revenue: number
-}
-
-export interface RegionBreakdown {
-	region: string
-	orders: number
-	revenue: number
-}
-
-export interface DailyPoint {
-	date: string
-	revenue: number
-	orders: number
-}
-
-// Revenue for a single line item. An order's revenue is the unit price times
-// the number of units purchased — never the unit price alone.
-export function orderRevenue(order: Order): number {
-	return order.unitPrice
-}
-
-// The statuses that count toward realized (booked) revenue. Refunded and
-// cancelled orders are excluded from the headline revenue total.
-const REVENUE_STATUSES: OrderStatus[] = ['paid', 'shipped', 'delivered']
-
-export function isRevenueStatus(status: OrderStatus): boolean {
-	return REVENUE_STATUSES.includes(status)
-}
-
-export function sumRevenue(orders: Order[]): number {
-	return orders
-		.filter((order) => isRevenueStatus(order.status))
-		.reduce((acc, order) => acc + orderRevenue(order), 0)
-}
-
-export function sumUnits(orders: Order[]): number {
-	return orders
-		.filter((order) => isRevenueStatus(order.status))
-		.reduce((acc, order) => acc + order.quantity, 0)
-}
-
-export function sumRefundedRevenue(orders: Order[]): number {
-	return orders
-		.filter((order) => order.status === 'refunded')
-		.reduce((acc, order) => acc + order.unitPrice * order.quantity, 0)
-}
-
-export function summarizeRevenue(orders: Order[]): RevenueSummary {
-	const revenueOrders = orders.filter((order) => isRevenueStatus(order.status))
-	const totalRevenue = sumRevenue(orders)
-	const unitsSold = sumUnits(orders)
-	const refundedRevenue = sumRefundedRevenue(orders)
-	const totalOrders = revenueOrders.length
-	return {
-		totalRevenue,
-		totalOrders,
-		averageOrderValue: totalOrders === 0 ? 0 : totalRevenue / totalOrders,
-		unitsSold,
-		refundedRevenue,
-		netRevenue: totalRevenue - refundedRevenue
-	}
-}
-
-export function breakdownByStatus(orders: Order[]): StatusBreakdown[] {
-	const map = new Map<OrderStatus, StatusBreakdown>()
-	for (const order of orders) {
-		const existing = map.get(order.status) ?? {
-			status: order.status,
-			orders: 0,
-			revenue: 0
-		}
-		existing.orders += 1
-		existing.revenue += order.unitPrice * order.quantity
-		map.set(order.status, existing)
-	}
-	return [...map.values()].sort((a, b) => b.revenue - a.revenue)
-}
-
-export function breakdownByRegion(orders: Order[]): RegionBreakdown[] {
-	const map = new Map<string, RegionBreakdown>()
-	for (const order of orders) {
-		if (!isRevenueStatus(order.status)) {
-			continue
-		}
-		const existing = map.get(order.region) ?? {
-			region: order.region,
-			orders: 0,
-			revenue: 0
-		}
-		existing.orders += 1
-		existing.revenue += order.unitPrice * order.quantity
-		map.set(order.region, existing)
-	}
-	return [...map.values()].sort((a, b) => b.revenue - a.revenue)
-}
-
-export function dailyRevenue(orders: Order[]): DailyPoint[] {
-	const map = new Map<string, DailyPoint>()
-	for (const order of orders) {
-		if (!isRevenueStatus(order.status)) {
-			continue
-		}
-		const day = order.placedAt.slice(0, 10)
-		const existing = map.get(day) ?? { date: day, revenue: 0, orders: 0 }
-		existing.revenue += order.unitPrice * order.quantity
-		existing.orders += 1
-		map.set(day, existing)
-	}
-	return [...map.values()].sort((a, b) => a.date.localeCompare(b.date))
-}
-
-export function topProducts(orders: Order[], limit: number = 5): { product: string; revenue: number }[] {
-	const map = new Map<string, number>()
-	for (const order of orders) {
-		if (!isRevenueStatus(order.status)) {
-			continue
-		}
-		map.set(order.product, (map.get(order.product) ?? 0) + order.unitPrice * order.quantity)
-	}
-	return [...map.entries()]
-		.map(([product, revenue]) => ({ product, revenue }))
-		.sort((a, b) => b.revenue - a.revenue)
-		.slice(0, limit)
-}
-
-export function growthRatio(current: number, previous: number): number {
-	if (previous === 0) {
-		return current === 0 ? 0 : 1
-	}
-	return (current - previous) / previous
-}

ai_evals/fixtures/frontend/global/initial/analytics_dashboard/frontend/lib/api.ts

@@ -1,79 +0,0 @@
-// Thin wrappers around the app's backend runnables. Centralizing the calls
-// here keeps the components free of `backend.*` plumbing and gives one place to
-// normalize the request/response shapes.
-
-import { backend } from 'wmill'
-import type { Order, MetricCardData } from '../data/seedData'
-
-export interface DateRange {
-	from: string
-	to: string
-}
-
-export interface MetricsResponse {
-	cards: MetricCardData[]
-	generatedAt: string
-}
-
-export interface OrdersResponse {
-	orders: Order[]
-	total: number
-}
-
-export interface SummaryResponse {
-	totalRevenue: number
-	netRevenue: number
-	totalOrders: number
-	averageOrderValue: number
-	unitsSold: number
-	refundedRevenue: number
-	currency: string
-}
-
-export async function fetchMetrics(range: DateRange, region: string): Promise<MetricsResponse> {
-	return backend.loadMetrics({ from: range.from, to: range.to, region })
-}
-
-export async function fetchOrders(
-	range: DateRange,
-	region: string,
-	status: string
-): Promise<OrdersResponse> {
-	return backend.loadOrders({
-		from: range.from,
-		to: range.to,
-		region,
-		status
-	})
-}
-
-export async function fetchSummary(range: DateRange, region: string): Promise<SummaryResponse> {
-	return backend.computeSummary({ from: range.from, to: range.to, region })
-}
-
-export async function requestExport(
-	range: DateRange,
-	region: string,
-	format: 'csv' | 'json'
-): Promise<{ url: string; rows: number }> {
-	return backend.exportReport({ from: range.from, to: range.to, region, format })
-}
-
-export function defaultRange(): DateRange {
-	return { from: '2024-05-01', to: '2024-05-31' }
-}
-
-export function rangeForPreset(preset: string): DateRange {
-	switch (preset) {
-		case '7d':
-			return { from: '2024-05-25', to: '2024-05-31' }
-		case '14d':
-			return { from: '2024-05-18', to: '2024-05-31' }
-		case '30d':
-			return { from: '2024-05-01', to: '2024-05-31' }
-		case 'qtd':
-			return { from: '2024-04-01', to: '2024-05-31' }
-		default:
-			return defaultRange()
-	}
-}

ai_evals/fixtures/frontend/global/initial/analytics_dashboard/frontend/lib/format.ts

@@ -1,91 +0,0 @@
-// Presentation-layer formatting helpers shared across the dashboard.
-// Pure functions only — no React, no data fetching.
-
-export function formatCurrency(amount: number, currency: string = 'USD'): string {
-	if (!Number.isFinite(amount)) {
-		return '—'
-	}
-	return new Intl.NumberFormat('en-US', {
-		style: 'currency',
-		currency,
-		maximumFractionDigits: 0
-	}).format(amount)
-}
-
-export function formatCurrencyPrecise(amount: number, currency: string = 'USD'): string {
-	if (!Number.isFinite(amount)) {
-		return '—'
-	}
-	return new Intl.NumberFormat('en-US', {
-		style: 'currency',
-		currency,
-		minimumFractionDigits: 2,
-		maximumFractionDigits: 2
-	}).format(amount)
-}
-
-export function formatNumber(value: number): string {
-	if (!Number.isFinite(value)) {
-		return '—'
-	}
-	return new Intl.NumberFormat('en-US').format(value)
-}
-
-export function formatCompact(value: number): string {
-	if (!Number.isFinite(value)) {
-		return '—'
-	}
-	return new Intl.NumberFormat('en-US', {
-		notation: 'compact',
-		maximumFractionDigits: 1
-	}).format(value)
-}
-
-export function formatPercent(ratio: number, digits: number = 1): string {
-	if (!Number.isFinite(ratio)) {
-		return '—'
-	}
-	return `${(ratio * 100).toFixed(digits)}%`
-}
-
-export function formatSignedPercent(ratio: number, digits: number = 1): string {
-	const sign = ratio > 0 ? '+' : ''
-	return `${sign}${formatPercent(ratio, digits)}`
-}
-
-export function formatDate(iso: string): string {
-	const date = new Date(iso)
-	if (Number.isNaN(date.getTime())) {
-		return iso
-	}
-	return date.toLocaleDateString('en-US', {
-		year: 'numeric',
-		month: 'short',
-		day: 'numeric'
-	})
-}
-
-export function formatDateShort(iso: string): string {
-	const date = new Date(iso)
-	if (Number.isNaN(date.getTime())) {
-		return iso
-	}
-	return date.toLocaleDateString('en-US', {
-		month: 'short',
-		day: 'numeric'
-	})
-}
-
-export function titleCase(value: string): string {
-	return value
-		.split(/[\s_-]+/)
-		.map((part) => part.charAt(0).toUpperCase() + part.slice(1).toLowerCase())
-		.join(' ')
-}
-
-export function truncate(value: string, max: number = 32): string {
-	if (value.length <= max) {
-		return value
-	}
-	return `${value.slice(0, max - 1)}…`
-}

ai_evals/fixtures/frontend/global/initial/datatable_orders_seed.json

@@ -1,39 +0,0 @@
-{
-  "workspace": {
-    "datatables": [
-      {
-        "datatable_name": "main",
-        "schemas": {
-          "public": {
-            "orders": {
-              "columns": {
-                "id": "int4",
-                "customer_id": "int4",
-                "total": "numeric",
-                "status": "text",
-                "created_at": "timestamptz"
-              },
-              "rows": [
-                { "id": 1, "customer_id": 1, "total": 42.5, "status": "shipped", "created_at": "2026-05-01T10:00:00Z" },
-                { "id": 2, "customer_id": 2, "total": 19.99, "status": "pending", "created_at": "2026-05-02T11:30:00Z" },
-                { "id": 3, "customer_id": 1, "total": 88, "status": "shipped", "created_at": "2026-05-03T09:15:00Z" }
-              ]
-            },
-            "customers": {
-              "columns": {
-                "id": "int4",
-                "name": "text",
-                "email": "text",
-                "tier": "text"
-              },
-              "rows": [
-                { "id": 1, "name": "Alice", "email": "alice@example.com", "tier": "gold" },
-                { "id": 2, "name": "Bob", "email": "bob@example.com", "tier": "silver" }
-              ]
-            }
-          }
-        }
-      }
-    ]
-  }
-}

ai_evals/fixtures/frontend/global/initial/jobs_seed.json

@@ -1,30 +0,0 @@
-{
-  "workspace": {
-    "jobs": [
-      {
-        "id": "01920000-0000-7000-8000-0000000000f1",
-        "scriptPath": "f/etl/sync_customers",
-        "jobKind": "script",
-        "createdBy": "alice",
-        "success": false,
-        "logs": "Starting customer sync...\nFetched 0 records\nERROR: connection refused to https://api.upstream.example.com\n  at fetchCustomers (sync_customers.ts:42)\nJob failed with exit code 1"
-      },
-      {
-        "id": "01920000-0000-7000-8000-0000000000f2",
-        "scriptPath": "f/reports/daily_digest",
-        "jobKind": "script",
-        "createdBy": "bob",
-        "success": true,
-        "logs": "Generating daily digest...\nDigest emailed to 12 recipients\nDone in 1.2s"
-      },
-      {
-        "id": "01920000-0000-7000-8000-0000000000f3",
-        "scriptPath": "f/billing/charge_invoices",
-        "jobKind": "flow",
-        "createdBy": "alice",
-        "success": true,
-        "logs": "Processing invoices...\nCharged 8 invoices totalling $1,240.00\nDone"
-      }
-    ]
-  }
-}

ai_evals/fixtures/frontend/global/initial/user_admin_empty.json

@@ -1,6 +0,0 @@
-{
-  "user": {
-    "username": "admin",
-    "is_admin": true
-  }
-}

ai_evals/fixtures/frontend/global/initial/user_admin_folders.json

@@ -1,8 +0,0 @@
-{
-  "user": {
-    "username": "admin",
-    "is_admin": true,
-    "folders": ["marketing", "data_engineering", "shared_utils"],
-    "folders_read": ["marketing", "data_engineering", "shared_utils"]
-  }
-}

ai_evals/fixtures/frontend/global/initial/user_bob_nonadmin_teams.json

@@ -1,8 +0,0 @@
-{
-  "user": {
-    "username": "bob",
-    "is_admin": false,
-    "folders": ["team_a"],
-    "folders_read": ["team_a", "team_b"]
-  }
-}

ai_evals/modes/app.ts

@@ -48,7 +48,6 @@ export function createAppModeRunner(
         toolsUsed: result.toolsUsed,
         skillsInvoked: [],
         tokenUsage: result.tokenUsage,
-        finalContextTokens: result.finalContextTokens,
       };
     },
     validate({ evalCase, actual, initial, expected, run }) {

ai_evals/modes/cli.ts

@@ -106,7 +106,6 @@ export function createCliModeRunner(
           toolsUsed: run.trace.toolsUsed.map((entry) => entry.tool),
           skillsInvoked: run.trace.skillsInvoked,
           tokenUsage: run.tokenUsage ?? null,
-          finalContextTokens: run.finalContextTokens ?? null,
         };
       } catch (error) {
         const message = error instanceof Error ? error.message : String(error);
@@ -123,7 +122,6 @@ export function createCliModeRunner(
           toolsUsed: [],
           skillsInvoked: [],
           tokenUsage: null,
-          finalContextTokens: null,
         };
       } finally {
         await rm(workspaceDir, { recursive: true, force: true });

ai_evals/modes/flow.ts

@@ -61,7 +61,6 @@ export function createFlowModeRunner(
         toolCallDetails: result.toolCallDetails,
         skillsInvoked: [],
         tokenUsage: result.tokenUsage,
-        finalContextTokens: result.finalContextTokens,
       };
     },
     validate({ evalCase, actual, initial, expected }) {

ai_evals/modes/global.ts

@@ -1,10 +1,7 @@
-import { readFile, stat } from "node:fs/promises";
-import { basename } from "node:path";
-import { loadAppFixtureForEval } from "../adapters/frontend/core/app/appFixtureLoader";
+import { readFile } from "node:fs/promises";
 import {
   runGlobalEval,
   type GlobalLiveEditorDraftFixture,
-  type GlobalUserFixture,
 } from "../adapters/frontend/core/global/globalEvalRunner";
 import type { BenchmarkWorkspaceRunnables } from "../adapters/frontend/mockBackend";
 import type { FrontendEvalModelConfig } from "../core/models";
@@ -16,7 +13,6 @@ import { getFrontendApiKey } from "./frontendCommon";
 export interface GlobalInitialFixture {
   workspace?: BenchmarkWorkspaceRunnables;
   liveEditorDrafts?: GlobalLiveEditorDraftFixture[];
-  user?: GlobalUserFixture;
 }
 
 export function createGlobalModeRunner(
@@ -40,7 +36,6 @@ export function createGlobalModeRunner(
         {
           workspaceFixtures: initial?.workspace,
           liveEditorDrafts: initial?.liveEditorDrafts,
-          user: initial?.user,
           maxIterations: context.evalCase?.runtime?.maxTurns,
           provider: modelConfig.provider,
           model: modelConfig.model,
@@ -59,7 +54,6 @@ export function createGlobalModeRunner(
         toolCallDetails: result.toolCallDetails,
         skillsInvoked: [],
         tokenUsage: result.tokenUsage,
-        finalContextTokens: result.finalContextTokens,
       };
     },
     validate({ evalCase, actual, expected }) {
@@ -81,33 +75,10 @@ export function createGlobalModeRunner(
 }
 
 async function loadGlobalInitialFixture(path: string): Promise<GlobalInitialFixture> {
-  if ((await stat(path)).isDirectory()) {
-    const { initialFrontend, initialBackend, initialDatatables } =
-      await loadAppFixtureForEval(path);
-    const name = basename(path);
-    return {
-      workspace: {
-        apps: [
-          {
-            path: `f/evals/global/${name}`,
-            summary: name,
-            value: {
-              files: initialFrontend,
-              runnables: initialBackend,
-              data: initialDatatables,
-            },
-          },
-        ],
-      },
-      liveEditorDrafts: [],
-    };
-  }
-
   const parsed = JSON.parse(await readFile(path, "utf8")) as GlobalInitialFixture;
   return {
     workspace: parsed.workspace ?? {},
     liveEditorDrafts: parsed.liveEditorDrafts ?? [],
-    user: parsed.user,
   };
 }
 

ai_evals/modes/script.ts

@@ -52,7 +52,6 @@ export function createScriptModeRunner(
         toolCallDetails: result.toolCallDetails,
         skillsInvoked: [],
         tokenUsage: result.tokenUsage,
-        finalContextTokens: result.finalContextTokens,
       };
     },
     validate({ actual, initial, expected }) {

backend/.sqlx/query-002a606e71364b0581dbc496bf4337f276861dc71d2e277a7aef711543eb14d7.json

@@ -1,29 +0,0 @@
-{
-  "db_name": "PostgreSQL",
-  "query": "SELECT\n               COUNT(*)::bigint AS \"total!\",\n               COUNT(*) FILTER (WHERE name = ANY($2::text[]))::bigint AS \"replacing!\"\n           FROM ai_skill\n           WHERE workspace_id = $1",
-  "describe": {
-    "columns": [
-      {
-        "ordinal": 0,
-        "name": "total!",
-        "type_info": "Int8"
-      },
-      {
-        "ordinal": 1,
-        "name": "replacing!",
-        "type_info": "Int8"
-      }
-    ],
-    "parameters": {
-      "Left": [
-        "Text",
-        "TextArray"
-      ]
-    },
-    "nullable": [
-      null,
-      null
-    ]
-  },
-  "hash": "002a606e71364b0581dbc496bf4337f276861dc71d2e277a7aef711543eb14d7"
-}

backend/.sqlx/query-0142d9dc9c1b57487dd5709a0376794f18d33e5bd6340c0189be7818cda64328.json

@@ -0,0 +1,94 @@
+{
+  "db_name": "PostgreSQL",
+  "query": "SELECT email, login_type::TEXT, super_admin, devops, verified, name, company, username, NULL::bool as operator_only, first_time_user, role_source, disabled, NULL::text as workspace_id FROM password WHERE email = $1",
+  "describe": {
+    "columns": [
+      {
+        "ordinal": 0,
+        "name": "email",
+        "type_info": "Varchar"
+      },
+      {
+        "ordinal": 1,
+        "name": "login_type",
+        "type_info": "Text"
+      },
+      {
+        "ordinal": 2,
+        "name": "super_admin",
+        "type_info": "Bool"
+      },
+      {
+        "ordinal": 3,
+        "name": "devops",
+        "type_info": "Bool"
+      },
+      {
+        "ordinal": 4,
+        "name": "verified",
+        "type_info": "Bool"
+      },
+      {
+        "ordinal": 5,
+        "name": "name",
+        "type_info": "Varchar"
+      },
+      {
+        "ordinal": 6,
+        "name": "company",
+        "type_info": "Varchar"
+      },
+      {
+        "ordinal": 7,
+        "name": "username",
+        "type_info": "Varchar"
+      },
+      {
+        "ordinal": 8,
+        "name": "operator_only",
+        "type_info": "Bool"
+      },
+      {
+        "ordinal": 9,
+        "name": "first_time_user",
+        "type_info": "Bool"
+      },
+      {
+        "ordinal": 10,
+        "name": "role_source",
+        "type_info": "Varchar"
+      },
+      {
+        "ordinal": 11,
+        "name": "disabled",
+        "type_info": "Bool"
+      },
+      {
+        "ordinal": 12,
+        "name": "workspace_id",
+        "type_info": "Text"
+      }
+    ],
+    "parameters": {
+      "Left": [
+        "Text"
+      ]
+    },
+    "nullable": [
+      false,
+      null,
+      false,
+      false,
+      false,
+      true,
+      true,
+      true,
+      null,
+      false,
+      false,
+      false,
+      null
+    ]
+  },
+  "hash": "0142d9dc9c1b57487dd5709a0376794f18d33e5bd6340c0189be7818cda64328"
+}

backend/.sqlx/query-03c8a797ae734ff76e227259ae011ef6d35f892fe95448c58ec13dea58eee3fa.json

@@ -1,14 +0,0 @@
-{
-  "db_name": "PostgreSQL",
-  "query": "DELETE FROM skip_workspace_diff_tally WHERE workspace_id = $1",
-  "describe": {
-    "columns": [],
-    "parameters": {
-      "Left": [
-        "Text"
-      ]
-    },
-    "nullable": []
-  },
-  "hash": "03c8a797ae734ff76e227259ae011ef6d35f892fe95448c58ec13dea58eee3fa"
-}

backend/.sqlx/query-0877f40f086fb5324af54701b29c1edc9d3afb796a06ae4c673aa4c54ad87ddb.json

@@ -0,0 +1,59 @@
+{
+  "db_name": "PostgreSQL",
+  "query": "SELECT path, value as \"value: _\", typ as \"typ: DraftType\"\n           FROM draft\n           WHERE path = $1 AND typ = $2 AND workspace_id = $3 AND email = $4",
+  "describe": {
+    "columns": [
+      {
+        "ordinal": 0,
+        "name": "path",
+        "type_info": "Varchar"
+      },
+      {
+        "ordinal": 1,
+        "name": "value: _",
+        "type_info": "Json"
+      },
+      {
+        "ordinal": 2,
+        "name": "typ: DraftType",
+        "type_info": {
+          "Custom": {
+            "name": "draft_type",
+            "kind": {
+              "Enum": [
+                "script",
+                "flow",
+                "app"
+              ]
+            }
+          }
+        }
+      }
+    ],
+    "parameters": {
+      "Left": [
+        "Text",
+        {
+          "Custom": {
+            "name": "draft_type",
+            "kind": {
+              "Enum": [
+                "script",
+                "flow",
+                "app"
+              ]
+            }
+          }
+        },
+        "Text",
+        "Text"
+      ]
+    },
+    "nullable": [
+      false,
+      false,
+      false
+    ]
+  },
+  "hash": "0877f40f086fb5324af54701b29c1edc9d3afb796a06ae4c673aa4c54ad87ddb"
+}

backend/.sqlx/query-089d7bc7acdbb97cf477159e111bc7e9ee85289ff5c52af43166928337c257e7.json

@@ -34,8 +34,7 @@
                 "google",
                 "ci_test",
                 "github",
-                "azure",
-                "asset"
+                "azure"
               ]
             }
           }

backend/.sqlx/query-09095af7cad650fb10781d9e39b0dad250c59ed0fca6cab7b5be4ee2516275d0.json

@@ -1,14 +0,0 @@
-{
-  "db_name": "PostgreSQL",
-  "query": "DELETE FROM v2_job WHERE workspace_id = $1 AND trigger_kind = 'asset'",
-  "describe": {
-    "columns": [],
-    "parameters": {
-      "Left": [
-        "Text"
-      ]
-    },
-    "nullable": []
-  },
-  "hash": "09095af7cad650fb10781d9e39b0dad250c59ed0fca6cab7b5be4ee2516275d0"
-}

backend/.sqlx/query-0a568f630e069118fe302099a709e89cc4a702158899f7fdc66d0922e8fb9b29.json

@@ -1,23 +0,0 @@
-{
-  "db_name": "PostgreSQL",
-  "query": "SELECT path FROM flow WHERE workspace_id = $1 AND path = ANY($2) AND archived = false",
-  "describe": {
-    "columns": [
-      {
-        "ordinal": 0,
-        "name": "path",
-        "type_info": "Varchar"
-      }
-    ],
-    "parameters": {
-      "Left": [
-        "Text",
-        "TextArray"
-      ]
-    },
-    "nullable": [
-      false
-    ]
-  },
-  "hash": "0a568f630e069118fe302099a709e89cc4a702158899f7fdc66d0922e8fb9b29"
-}

backend/.sqlx/query-0ca770234f3e38be3fb1c280d82e9c06440168806fe605e38808bdcb400d4034.json

@@ -0,0 +1,16 @@
+{
+  "db_name": "PostgreSQL",
+  "query": "\n    INSERT INTO script\n    (workspace_id, hash, path, parent_hashes, summary, description, content, created_by, schema, is_template, extra_perms, lock, language, kind, tag, draft_only, envs, concurrent_limit, concurrency_time_window_s, cache_ttl, cache_ignore_s3_path, dedicated_worker, ws_error_handler_muted, priority, restart_unless_cancelled, delete_after_use, delete_after_secs, timeout, concurrency_key, visible_to_runner_only, auto_kind, codebase, has_preprocessor, on_behalf_of_email, schema_validation, assets, debounce_key, debounce_delay_s, runnable_settings_handle, modules, labels)\n\n    SELECT  workspace_id, $1, path, array_prepend($2::bigint, COALESCE(parent_hashes, '{}'::bigint[])), summary, description, content, created_by, schema, is_template, extra_perms, NULL, language, kind, tag, draft_only, envs, concurrent_limit, concurrency_time_window_s, cache_ttl, cache_ignore_s3_path, dedicated_worker, ws_error_handler_muted, priority, restart_unless_cancelled, delete_after_use, delete_after_secs, timeout, concurrency_key, visible_to_runner_only, auto_kind, codebase, has_preprocessor, on_behalf_of_email, schema_validation, assets, debounce_key, debounce_delay_s, runnable_settings_handle, modules, labels\n\n    FROM script WHERE hash = $2 AND workspace_id = $3;\n            ",
+  "describe": {
+    "columns": [],
+    "parameters": {
+      "Left": [
+        "Int8",
+        "Int8",
+        "Text"
+      ]
+    },
+    "nullable": []
+  },
+  "hash": "0ca770234f3e38be3fb1c280d82e9c06440168806fe605e38808bdcb400d4034"
+}

backend/.sqlx/query-0e7fe0e1d7aa2072a3431d081080bbc18da7e1ed758cab017fba2598c9467b7f.json

@@ -1,17 +0,0 @@
-{
-  "db_name": "PostgreSQL",
-  "query": "INSERT INTO join_pending_inputs\n             (workspace_id, subscriber_path, partition, trigger_ref)\n           VALUES ($1, $2, $3, $4)\n           ON CONFLICT DO NOTHING",
-  "describe": {
-    "columns": [],
-    "parameters": {
-      "Left": [
-        "Varchar",
-        "Varchar",
-        "Text",
-        "Text"
-      ]
-    },
-    "nullable": []
-  },
-  "hash": "0e7fe0e1d7aa2072a3431d081080bbc18da7e1ed758cab017fba2598c9467b7f"
-}

backend/.sqlx/query-115a9cb44d0a41952c08dc36e0331410d32a8d672cfa4929e9e3763c51daa1bc.json

@@ -0,0 +1,89 @@
+{
+  "db_name": "PostgreSQL",
+  "query": "SELECT email, login_type::text, verified, super_admin, devops, name, company, username, NULL::bool as operator_only, first_time_user, role_source, disabled FROM password ORDER BY super_admin DESC, devops DESC, email LIMIT $1 OFFSET $2",
+  "describe": {
+    "columns": [
+      {
+        "ordinal": 0,
+        "name": "email",
+        "type_info": "Varchar"
+      },
+      {
+        "ordinal": 1,
+        "name": "login_type",
+        "type_info": "Text"
+      },
+      {
+        "ordinal": 2,
+        "name": "verified",
+        "type_info": "Bool"
+      },
+      {
+        "ordinal": 3,
+        "name": "super_admin",
+        "type_info": "Bool"
+      },
+      {
+        "ordinal": 4,
+        "name": "devops",
+        "type_info": "Bool"
+      },
+      {
+        "ordinal": 5,
+        "name": "name",
+        "type_info": "Varchar"
+      },
+      {
+        "ordinal": 6,
+        "name": "company",
+        "type_info": "Varchar"
+      },
+      {
+        "ordinal": 7,
+        "name": "username",
+        "type_info": "Varchar"
+      },
+      {
+        "ordinal": 8,
+        "name": "operator_only",
+        "type_info": "Bool"
+      },
+      {
+        "ordinal": 9,
+        "name": "first_time_user",
+        "type_info": "Bool"
+      },
+      {
+        "ordinal": 10,
+        "name": "role_source",
+        "type_info": "Varchar"
+      },
+      {
+        "ordinal": 11,
+        "name": "disabled",
+        "type_info": "Bool"
+      }
+    ],
+    "parameters": {
+      "Left": [
+        "Int8",
+        "Int8"
+      ]
+    },
+    "nullable": [
+      false,
+      null,
+      false,
+      false,
+      false,
+      true,
+      true,
+      true,
+      null,
+      false,
+      false,
+      false
+    ]
+  },
+  "hash": "115a9cb44d0a41952c08dc36e0331410d32a8d672cfa4929e9e3763c51daa1bc"
+}

backend/.sqlx/query-126cf6d54f8d2c916cd799f6663892119a94d66637062d1bf5a5fe97d89f8096.json

@@ -1,20 +0,0 @@
-{
-  "db_name": "PostgreSQL",
-  "query": "SELECT COUNT(*) FROM workspace_diff\n         WHERE source_workspace_id = 'wm-fork-test-workspace'\n            OR fork_workspace_id = 'wm-fork-test-workspace'",
-  "describe": {
-    "columns": [
-      {
-        "ordinal": 0,
-        "name": "count",
-        "type_info": "Int8"
-      }
-    ],
-    "parameters": {
-      "Left": []
-    },
-    "nullable": [
-      null
-    ]
-  },
-  "hash": "126cf6d54f8d2c916cd799f6663892119a94d66637062d1bf5a5fe97d89f8096"
-}

backend/.sqlx/query-16b4496c21d0619dab4521dca22e5fe144c59156a8f06d8592291684c49b2f37.json

@@ -0,0 +1,95 @@
+{
+  "db_name": "PostgreSQL",
+  "query": "SELECT email as \"email!\", login_type::text, verified as \"verified!\", super_admin as \"super_admin!\", devops as \"devops!\", name, company, username, NULL::bool as operator_only, first_time_user as \"first_time_user!\", role_source as \"role_source!\", disabled as \"disabled!\", NULL::text as workspace_id FROM password\n            UNION ALL\n            SELECT email as \"email!\", 'service_account'::text as login_type, true as \"verified!\", false as \"super_admin!\", false as \"devops!\", NULL::text as name, NULL::text as company, username, true as operator_only, false as \"first_time_user!\", 'service_account'::text as \"role_source!\", disabled as \"disabled!\", workspace_id\n            FROM usr\n            WHERE is_service_account IS true\n            ORDER BY \"super_admin!\" DESC, \"devops!\" DESC, \"email!\"\n            LIMIT $1 OFFSET $2",
+  "describe": {
+    "columns": [
+      {
+        "ordinal": 0,
+        "name": "email!",
+        "type_info": "Varchar"
+      },
+      {
+        "ordinal": 1,
+        "name": "login_type",
+        "type_info": "Text"
+      },
+      {
+        "ordinal": 2,
+        "name": "verified!",
+        "type_info": "Bool"
+      },
+      {
+        "ordinal": 3,
+        "name": "super_admin!",
+        "type_info": "Bool"
+      },
+      {
+        "ordinal": 4,
+        "name": "devops!",
+        "type_info": "Bool"
+      },
+      {
+        "ordinal": 5,
+        "name": "name",
+        "type_info": "Varchar"
+      },
+      {
+        "ordinal": 6,
+        "name": "company",
+        "type_info": "Varchar"
+      },
+      {
+        "ordinal": 7,
+        "name": "username",
+        "type_info": "Varchar"
+      },
+      {
+        "ordinal": 8,
+        "name": "operator_only",
+        "type_info": "Bool"
+      },
+      {
+        "ordinal": 9,
+        "name": "first_time_user!",
+        "type_info": "Bool"
+      },
+      {
+        "ordinal": 10,
+        "name": "role_source!",
+        "type_info": "Varchar"
+      },
+      {
+        "ordinal": 11,
+        "name": "disabled!",
+        "type_info": "Bool"
+      },
+      {
+        "ordinal": 12,
+        "name": "workspace_id",
+        "type_info": "Text"
+      }
+    ],
+    "parameters": {
+      "Left": [
+        "Int8",
+        "Int8"
+      ]
+    },
+    "nullable": [
+      null,
+      null,
+      null,
+      null,
+      null,
+      null,
+      null,
+      null,
+      null,
+      null,
+      null,
+      null,
+      null
+    ]
+  },
+  "hash": "16b4496c21d0619dab4521dca22e5fe144c59156a8f06d8592291684c49b2f37"
+}

backend/.sqlx/query-1860c102a5309a8e1bb7288d0616f48e4330af7fbe61426e999c6073f630d88f.json

@@ -1,41 +0,0 @@
-{
-  "db_name": "PostgreSQL",
-  "query": "SELECT DISTINCT ON (path)\n                      path,\n                      value as \"value!: sqlx::types::Json<Box<serde_json::value::RawValue>>\",\n                      created_at,\n                      typ::text as \"typ!\"\n               FROM draft\n               WHERE workspace_id = $1\n                 AND typ IN ('app', 'raw_app')\n                 AND (email = $2 OR email IS NULL)\n                 AND NOT EXISTS (\n                     SELECT 1 FROM app a\n                     WHERE a.workspace_id = draft.workspace_id\n                       AND a.path = draft.path\n                 )\n               ORDER BY path, (email IS NULL), created_at DESC",
-  "describe": {
-    "columns": [
-      {
-        "ordinal": 0,
-        "name": "path",
-        "type_info": "Varchar"
-      },
-      {
-        "ordinal": 1,
-        "name": "value!: sqlx::types::Json<Box<serde_json::value::RawValue>>",
-        "type_info": "Json"
-      },
-      {
-        "ordinal": 2,
-        "name": "created_at",
-        "type_info": "Timestamptz"
-      },
-      {
-        "ordinal": 3,
-        "name": "typ!",
-        "type_info": "Text"
-      }
-    ],
-    "parameters": {
-      "Left": [
-        "Text",
-        "Text"
-      ]
-    },
-    "nullable": [
-      false,
-      false,
-      false,
-      null
-    ]
-  },
-  "hash": "1860c102a5309a8e1bb7288d0616f48e4330af7fbe61426e999c6073f630d88f"
-}

backend/.sqlx/query-19513c4158267cc7fe10d999ad571052c112e6bbb3cf834f16176cbb7e1ac319.json

@@ -5,7 +5,7 @@
     "columns": [
       {
         "ordinal": 0,
-        "name": "id!",
+        "name": "id",
         "type_info": "Uuid"
       }
     ],
@@ -16,7 +16,7 @@
       ]
     },
     "nullable": [
-      null
+      false
     ]
   },
   "hash": "19513c4158267cc7fe10d999ad571052c112e6bbb3cf834f16176cbb7e1ac319"

backend/.sqlx/query-19f1cb1c7a1974920549917a6392ff0d56f31ca5662062f4a71fed0ccf859cc4.json

@@ -1,35 +0,0 @@
-{
-  "db_name": "PostgreSQL",
-  "query": "SELECT kind::text as \"kind!\", parent_job, runnable_path\n           FROM v2_job WHERE id = $1 AND workspace_id = $2",
-  "describe": {
-    "columns": [
-      {
-        "ordinal": 0,
-        "name": "kind!",
-        "type_info": "Text"
-      },
-      {
-        "ordinal": 1,
-        "name": "parent_job",
-        "type_info": "Uuid"
-      },
-      {
-        "ordinal": 2,
-        "name": "runnable_path",
-        "type_info": "Varchar"
-      }
-    ],
-    "parameters": {
-      "Left": [
-        "Uuid",
-        "Text"
-      ]
-    },
-    "nullable": [
-      null,
-      true,
-      true
-    ]
-  },
-  "hash": "19f1cb1c7a1974920549917a6392ff0d56f31ca5662062f4a71fed0ccf859cc4"
-}

backend/.sqlx/query-1acfeed9c7a5b1e3d2da262d338655dba6e43067a9912cc2b775830856390c5d.json

@@ -1,27 +0,0 @@
-{
-  "db_name": "PostgreSQL",
-  "query": "WITH del AS (\n             DELETE FROM asset WHERE workspace_id = $1 AND usage_path = $2 AND usage_kind = $3\n             RETURNING usage_access_type\n           )\n           INSERT INTO notify_event (channel, payload)\n           SELECT 'notify_asset_producer_change', $1\n           WHERE $3 = 'script'\n             AND EXISTS (SELECT 1 FROM del WHERE usage_access_type IN ('w', 'rw'))",
-  "describe": {
-    "columns": [],
-    "parameters": {
-      "Left": [
-        "Text",
-        "Text",
-        {
-          "Custom": {
-            "name": "asset_usage_kind",
-            "kind": {
-              "Enum": [
-                "script",
-                "flow",
-                "job"
-              ]
-            }
-          }
-        }
-      ]
-    },
-    "nullable": []
-  },
-  "hash": "1acfeed9c7a5b1e3d2da262d338655dba6e43067a9912cc2b775830856390c5d"
-}

backend/.sqlx/query-1f375b37ff9f6f01972e284e84a7b2f9d2d323a3da55f20ff6671e8eba510043.json

@@ -1,15 +0,0 @@
-{
-  "db_name": "PostgreSQL",
-  "query": "WITH del AS (\n             DELETE FROM asset WHERE workspace_id = $1 AND usage_kind = 'script'\n               AND usage_path = (SELECT path FROM script WHERE hash = $2 AND workspace_id = $1)\n             RETURNING usage_access_type\n           )\n           INSERT INTO notify_event (channel, payload)\n           SELECT 'notify_asset_producer_change', $1\n           WHERE EXISTS (SELECT 1 FROM del WHERE usage_access_type IN ('w', 'rw'))",
-  "describe": {
-    "columns": [],
-    "parameters": {
-      "Left": [
-        "Text",
-        "Int8"
-      ]
-    },
-    "nullable": []
-  },
-  "hash": "1f375b37ff9f6f01972e284e84a7b2f9d2d323a3da55f20ff6671e8eba510043"
-}

backend/.sqlx/query-1f486036b2902a0ffa0ce15f665d82c85129e2407819477adf7e87f097276ae2.json

@@ -1,16 +0,0 @@
-{
-  "db_name": "PostgreSQL",
-  "query": "INSERT INTO draft (workspace_id, path, typ, value, created_at, email)\n         SELECT $2, path, typ, value, created_at, email\n         FROM draft\n         WHERE workspace_id = $1 AND (email = $3 OR email IS NULL)",
-  "describe": {
-    "columns": [],
-    "parameters": {
-      "Left": [
-        "Text",
-        "Varchar",
-        "Text"
-      ]
-    },
-    "nullable": []
-  },
-  "hash": "1f486036b2902a0ffa0ce15f665d82c85129e2407819477adf7e87f097276ae2"
-}

backend/.sqlx/query-1fc04d31ae69dbb1df9c63cb69e83e8f8e6770b78f6ed052b99afed6cea28650.json

@@ -1,17 +0,0 @@
-{
-  "db_name": "PostgreSQL",
-  "query": "INSERT INTO v2_job (id, workspace_id, kind, runnable_path, args, created_by,\n                                permissioned_as, permissioned_as_email, tag, script_lang)\n           VALUES ($1, $2, 'script'::job_kind, $3, $4, 'test-user',\n                   'u/test-user', 'test@windmill.dev', 'deno', 'bash'::script_lang)",
-  "describe": {
-    "columns": [],
-    "parameters": {
-      "Left": [
-        "Uuid",
-        "Varchar",
-        "Varchar",
-        "Jsonb"
-      ]
-    },
-    "nullable": []
-  },
-  "hash": "1fc04d31ae69dbb1df9c63cb69e83e8f8e6770b78f6ed052b99afed6cea28650"
-}

backend/.sqlx/query-2484323d94f249be30f4472ece89659c1e6a24d5454a691e31e0179f58c24366.json

@@ -1,41 +0,0 @@
-{
-  "db_name": "PostgreSQL",
-  "query": "\n        SELECT kind, path, script_path, is_flow FROM (\n            SELECT 'schedule' AS kind, path, script_path, is_flow FROM schedule\n                WHERE workspace_id = $1\n                  AND script_path IS NOT NULL\n            UNION ALL\n            SELECT 'email', path, script_path, is_flow FROM email_trigger\n                WHERE workspace_id = $1\n            UNION ALL\n            SELECT 'kafka', path, script_path, is_flow FROM kafka_trigger\n                WHERE workspace_id = $1\n            UNION ALL\n            SELECT 'mqtt', path, script_path, is_flow FROM mqtt_trigger\n                WHERE workspace_id = $1\n            UNION ALL\n            SELECT 'nats', path, script_path, is_flow FROM nats_trigger\n                WHERE workspace_id = $1\n            UNION ALL\n            SELECT 'postgres', path, script_path, is_flow FROM postgres_trigger\n                WHERE workspace_id = $1\n            UNION ALL\n            SELECT 'sqs', path, script_path, is_flow FROM sqs_trigger\n                WHERE workspace_id = $1\n            UNION ALL\n            SELECT 'gcp', path, script_path, is_flow FROM gcp_trigger\n                WHERE workspace_id = $1\n        ) t\n        WHERE ($2::text IS NULL OR script_path LIKE $2)\n        ",
-  "describe": {
-    "columns": [
-      {
-        "ordinal": 0,
-        "name": "kind",
-        "type_info": "Text"
-      },
-      {
-        "ordinal": 1,
-        "name": "path",
-        "type_info": "Varchar"
-      },
-      {
-        "ordinal": 2,
-        "name": "script_path",
-        "type_info": "Varchar"
-      },
-      {
-        "ordinal": 3,
-        "name": "is_flow",
-        "type_info": "Bool"
-      }
-    ],
-    "parameters": {
-      "Left": [
-        "Text",
-        "Text"
-      ]
-    },
-    "nullable": [
-      null,
-      null,
-      null,
-      null
-    ]
-  },
-  "hash": "2484323d94f249be30f4472ece89659c1e6a24d5454a691e31e0179f58c24366"
-}

backend/.sqlx/query-25ecae25ebc03d6296b0e72482a9201c2bffb0f0f7419b0b598b2786bdb326ab.json

@@ -1,12 +0,0 @@
-{
-  "db_name": "PostgreSQL",
-  "query": "DELETE FROM job_perms\n             WHERE ctid IN (\n                 SELECT jp.ctid FROM job_perms jp\n                 WHERE NOT EXISTS (SELECT 1 FROM v2_job_queue q WHERE q.id = jp.job_id)\n                 LIMIT 100000\n             )",
-  "describe": {
-    "columns": [],
-    "parameters": {
-      "Left": []
-    },
-    "nullable": []
-  },
-  "hash": "25ecae25ebc03d6296b0e72482a9201c2bffb0f0f7419b0b598b2786bdb326ab"
-}

backend/.sqlx/query-269197b692a1e451a31f14701c8ca324c8e40d2923dfa07b8e4b3934afc16fed.json

@@ -1,6 +1,6 @@
 {
   "db_name": "PostgreSQL",
-  "query": "SELECT workspace_id, name, display_name, owners, extra_perms, summary, created_by, edited_at, default_permissioned_as, labels FROM folder WHERE name = $1 AND workspace_id = $2",
+  "query": "SELECT workspace_id, name, display_name, owners, extra_perms, summary, created_by, edited_at, default_permissioned_as FROM folder WHERE name = $1 AND workspace_id = $2",
   "describe": {
     "columns": [
       {
@@ -47,11 +47,6 @@
         "ordinal": 8,
         "name": "default_permissioned_as",
         "type_info": "Jsonb"
-      },
-      {
-        "ordinal": 9,
-        "name": "labels",
-        "type_info": "TextArray"
       }
     ],
     "parameters": {
@@ -69,9 +64,8 @@
       true,
       true,
       true,
-      false,
-      true
+      false
     ]
   },
-  "hash": "42600bdeb6b86ac306ab276cb49241508776f5f998821e04bc92df917176cea6"
+  "hash": "269197b692a1e451a31f14701c8ca324c8e40d2923dfa07b8e4b3934afc16fed"
 }