feat(ai-chat): wire llms.txt docs lookup tools into global chat mode

Add list_docs_pages/read_docs_page to globalTools and a docs system-prompt section so the workspace assistant can answer product questions from windmill.dev docs and cite canonical URLs.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

.agents/skills/pr/SKILL.md

@@ -55,55 +55,18 @@ The body MUST be explicit about what changed. Structure:
 
 The harness/tooling that invoked the skill may add its own attribution trailer; the skill itself does not prescribe one.
 
-## Screenshots (required for frontend changes)
-
-If `git diff main...HEAD --name-only` matches `^frontend/`, the PR body **must** include
-screenshots of the affected UI. Skip only when there is no visible UI effect (types,
-tests, build config) — and say so in the body.
-
-1. Verify the change in the browser (AGENTS.md → "Verifying Frontend Changes").
-2. Screenshot each affected page with `mcp__playwright__browser_take_screenshot` (save to a file).
-3. Host each image and get its Markdown embed by pushing to the public
-   `windmill-labs/agent-screenshots-internal` repo. **Pipe base64 through stdin** —
-   passing it as `-f content=…` fails with `argument list too long` on real images:
-
-   ```bash
-   REPO=windmill-labs/agent-screenshots-internal
-   IMG=screenshot.png                                          # repeat per page
-   DEST="shots/$(git branch --show-current)/$(date +%s)-$(basename "$IMG")"
-   base64 -w0 "$IMG" | jq -Rs --arg m "add $DEST" '{message:$m, content:.}' \
-     | gh api -X PUT "repos/$REPO/contents/$DEST" --input - >/dev/null
-   echo "![$(basename "$IMG" .png)](https://raw.githubusercontent.com/$REPO/main/$DEST)"
-   ```
-   Derive `$DEST` from the file name (as above) so distinct pages never collide — a
-   fixed name would make same-second uploads reuse one path, and the second `PUT`
-   then 422s (the Contents API needs the existing file's `sha` to overwrite).
-4. Put the printed `![…](…)` lines under a `## Screenshots` heading in the PR body.
-
-Requires `gh` (`repo` scope), `jq`, `base64` — all in the devShell. The host repo is
-public (so the raw URLs render for reviewers without a token) and its history is
-permanent — **never screenshot pages that show secrets or sensitive values** (workspace
-variables, resource values, instance settings, OAuth/SMTP config); deleting the file
-can't undo an accidental capture. (GitHub's drag-and-drop uploader needs a browser
-session and can't be driven from a token.)
-
-If `gh` can't push to the host repo (e.g. a CI token scoped only to `windmill`), do
-**not** fail the PR or skip silently — hand the upload to the user, who has push access,
-and continue once they confirm it's done.
-
 ## Execution Steps
 
 1. Run `git status` to check for uncommitted changes
 2. Run `git log main..HEAD --oneline` to see all commits in this branch
 3. Run `git diff main...HEAD` to see the full diff against main
 4. **Invoke the `local-review` skill** before creating the PR (`/local-review` in Claude Code, `$local-review` in Codex, `pi --skill local-review` / `/skill:local-review` in Pi). If issues are found, fix them and commit before proceeding. Do not skip this step.
-5. **Screenshots for frontend changes**: if `git diff main...HEAD --name-only` matches `^frontend/`, capture and embed screenshots of the affected UI per "Screenshots" above before writing the PR body (skip only if there is no visible UI effect).
-6. Check if remote branch exists and is up to date:
+5. Check if remote branch exists and is up to date:
    ```bash
    git rev-parse --abbrev-ref --symbolic-full-name @{u} 2>/dev/null || echo "no upstream"
    ```
-7. Push to remote if needed: `git push -u origin HEAD`
-8. Create draft PR using gh CLI:
+6. Push to remote if needed: `git push -u origin HEAD`
+7. Create draft PR using gh CLI:
    ```bash
    gh pr create --draft --title "<type>: <description>" --body "$(cat <<'EOF'
    ## Summary
@@ -119,7 +82,7 @@ and continue once they confirm it's done.
    EOF
    )"
    ```
-9. Return the PR URL to the user
+8. Return the PR URL to the user
 
 ## EE Companion PR (when `*_ee.rs` files were modified)
 

.github/workflows/ai-agent-tests.yml

@@ -1,132 +0,0 @@
-name: AI Agent Integration Tests
-
-# Exercises the AI agent flow path (preview_flow with `aiagent` modules) against
-# real LLM providers. Runs only when AI-agent backend code or the tests change,
-# because each run makes real (paid) LLM calls. To avoid spending on every commit,
-# the PR side triggers only when a PR is marked ready for review (out of draft) —
-# not on `synchronize` — plus push to main and manual dispatch.
-on:
-  workflow_dispatch:
-  push:
-    branches: [main]
-    paths:
-      - "integration_tests/ai_agent_tests/**"
-      - "backend/windmill-ai/**"
-      - "backend/windmill-api/src/ai.rs"
-      - "backend/windmill-worker/src/ai_executor.rs"
-      - "backend/windmill-worker/src/ai/**"
-      - "backend/windmill-worker/src/memory_common.rs"
-      - "backend/windmill-common/src/flow_conversations.rs"
-      - ".github/workflows/ai-agent-tests.yml"
-  pull_request:
-    types: [opened, reopened, ready_for_review]
-    paths:
-      - "integration_tests/ai_agent_tests/**"
-      - "backend/windmill-ai/**"
-      - "backend/windmill-api/src/ai.rs"
-      - "backend/windmill-worker/src/ai_executor.rs"
-      - "backend/windmill-worker/src/ai/**"
-      - "backend/windmill-worker/src/memory_common.rs"
-      - "backend/windmill-common/src/flow_conversations.rs"
-      - ".github/workflows/ai-agent-tests.yml"
-
-concurrency:
-  group: ai-agent-tests-${{ github.ref }}
-  cancel-in-progress: true
-
-jobs:
-  ai_agent_e2e:
-    # Skip draft PRs; the `opened`/`reopened` types would otherwise fire while
-    # still a draft. `ready_for_review` always arrives non-draft.
-    if: github.event_name != 'pull_request' || github.event.pull_request.draft == false
-    runs-on: ubicloud-standard-16
-    services:
-      postgres:
-        image: postgres:16
-        ports:
-          - 5432:5432
-        env:
-          POSTGRES_DB: windmill
-          POSTGRES_PASSWORD: changeme
-        options: >-
-          --health-cmd pg_isready --health-interval 10s --health-timeout 5s
-          --health-retries 5
-    steps:
-      - uses: actions/checkout@v4
-
-      - uses: actions-rust-lang/setup-rust-toolchain@v1
-        with:
-          cache-workspaces: backend
-          toolchain: 1.93.0
-
-      - uses: oven-sh/setup-bun@v2
-        with:
-          bun-version: 1.3.10
-
-      - uses: actions/setup-node@v4
-        with:
-          node-version: "20"
-
-      - uses: actions/setup-python@v5
-        with:
-          python-version: "3.11"
-
-      # CE build (no enterprise/license needed for AI agents). `quickjs` powers
-      # flow input-transform JS eval; `mcp` is required by the deepwiki MCP tool
-      # test. Bun tool scripts run via the always-on worker (BUN_PATH).
-      - name: Build Windmill
-        working-directory: ./backend
-        env:
-          SQLX_OFFLINE: true
-          CARGO_BUILD_JOBS: 12
-          RUSTFLAGS: ""
-        run: cargo build --features quickjs,mcp
-
-      - name: Start Windmill
-        working-directory: ./backend
-        env:
-          DATABASE_URL: postgres://postgres:changeme@localhost:5432/windmill
-          BUN_PATH: bun
-          NODE_BIN_PATH: node
-          RUST_LOG: info
-        run: |
-          mkdir -p ../integration_tests/logs
-          ./target/debug/windmill > ../integration_tests/logs/windmill.log 2>&1 &
-          echo "Waiting for Windmill to be ready..."
-          for i in $(seq 1 60); do
-            if curl -sf http://localhost:8000/api/version > /dev/null 2>&1; then
-              echo "Windmill is ready"
-              break
-            fi
-            sleep 2
-          done
-          curl -sf http://localhost:8000/api/version > /dev/null || { echo "Windmill failed to start"; tail -50 ../integration_tests/logs/windmill.log; exit 1; }
-
-      - name: Run AI agent integration tests
-        timeout-minutes: 20
-        working-directory: ./integration_tests/ai_agent_tests
-        env:
-          WINDMILL_URL: http://localhost:8000
-          # Only the providers we have org secrets for. Other providers
-          # (Azure, Bedrock, OpenRouter) are skipped by conftest when their
-          # keys are absent — see skip_provider_without_credentials.
-          ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }}
-          OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
-          GOOGLE_AI_API_KEY: ${{ secrets.GOOGLE_API_KEY }}
-        run: |
-          python -m venv .venv
-          .venv/bin/pip install -r requirements.txt
-          # The S3/vision-attachment tests need MinIO large-file storage and
-          # image-capable provider setup; out of scope for this cost-controlled
-          # smoke. Add MinIO secrets + a storage service to enable them.
-          .venv/bin/python -m pytest -v \
-            --ignore=test_user_attachments.py \
-            --ignore=test_user_images.py \
-            --ignore=test_image_output.py
-
-      - name: Archive Windmill logs
-        uses: actions/upload-artifact@v4
-        if: always()
-        with:
-          name: ai-agent-tests-windmill-logs
-          path: integration_tests/logs

.github/workflows/ai-evals-test.yml

@@ -1,166 +0,0 @@
-name: AI Evals (global mode)
-
-# Smoke-tests the production global AI chat proxy/frontend execution path via
-# the ai_evals harness, one case across one cheap model per provider. Runs only
-# when the eval harness or the global chat code change, since each run makes real
-# (paid) LLM calls. The backend is built from source purely as the AI proxy the
-# harness routes model calls through; the global tools/drafts run in-process in
-# the Vitest bridge against production frontend code. To avoid spending on every
-# commit, the PR side triggers only when a PR is marked ready for review (out of
-# draft) — not on `synchronize` — plus push to main and manual dispatch.
-on:
-  workflow_dispatch:
-  push:
-    branches: [main]
-    paths:
-      - "ai_evals/**"
-      - "backend/windmill-api/src/ai.rs"
-      - "backend/windmill-ai/**"
-      - "frontend/src/lib/components/copilot/**"
-      # The eval harness runs production frontend code in-process; these are the
-      # AI/draft-specific deps outside copilot/ that the global smoke exercises.
-      - "frontend/src/lib/userDraft.svelte.ts"
-      - "frontend/src/lib/userDraftDbSyncer.svelte.ts"
-      - "frontend/src/lib/infer.ts"
-      - ".github/workflows/ai-evals-test.yml"
-  pull_request:
-    types: [opened, reopened, ready_for_review]
-    paths:
-      - "ai_evals/**"
-      - "backend/windmill-api/src/ai.rs"
-      - "backend/windmill-ai/**"
-      - "frontend/src/lib/components/copilot/**"
-      # The eval harness runs production frontend code in-process; these are the
-      # AI/draft-specific deps outside copilot/ that the global smoke exercises.
-      - "frontend/src/lib/userDraft.svelte.ts"
-      - "frontend/src/lib/userDraftDbSyncer.svelte.ts"
-      - "frontend/src/lib/infer.ts"
-      - ".github/workflows/ai-evals-test.yml"
-
-concurrency:
-  group: ai-evals-test-${{ github.ref }}
-  cancel-in-progress: true
-
-jobs:
-  ai_evals_global:
-    # Provider secrets are unavailable to forked and Dependabot PRs.
-    if: >-
-      github.event_name != 'pull_request' ||
-      (
-        github.event.pull_request.draft == false &&
-        github.event.pull_request.head.repo.full_name == github.repository &&
-        github.event.pull_request.user.login != 'dependabot[bot]'
-      )
-    runs-on: ubicloud-standard-16
-    services:
-      postgres:
-        image: postgres:16
-        ports:
-          - 5432:5432
-        env:
-          POSTGRES_DB: windmill
-          POSTGRES_PASSWORD: changeme
-        options: >-
-          --health-cmd pg_isready --health-interval 10s --health-timeout 5s
-          --health-retries 5
-    steps:
-      - uses: actions/checkout@v4
-
-      - uses: actions-rust-lang/setup-rust-toolchain@v1
-        with:
-          cache-workspaces: backend
-          toolchain: 1.93.0
-
-      - uses: oven-sh/setup-bun@v2
-        with:
-          bun-version: 1.3.10
-
-      - uses: actions/setup-node@v4
-        with:
-          # Node 22.19+ is required by the frontend's undici 8.x, which the
-          # Vitest bridge loads; Node 20 fails with markAsUncloneable.
-          node-version: "22"
-
-      # CE build used only as the AI proxy (login, workspace, provider resource,
-      # /ai/proxy). No worker execution or MCP needed — global tools/drafts run
-      # in the Vitest bridge. quickjs matches the standard CE feature set.
-      - name: Build Windmill (AI proxy)
-        working-directory: ./backend
-        env:
-          SQLX_OFFLINE: true
-          CARGO_BUILD_JOBS: 12
-          RUSTFLAGS: ""
-        run: cargo build --features quickjs
-
-      - name: Start Windmill
-        working-directory: ./backend
-        env:
-          DATABASE_URL: postgres://postgres:changeme@localhost:5432/windmill
-          RUST_LOG: info
-        run: |
-          mkdir -p ../ai_evals/logs
-          ./target/debug/windmill > ../ai_evals/logs/windmill.log 2>&1 &
-          echo "Waiting for Windmill to be ready..."
-          for i in $(seq 1 60); do
-            if curl -sf http://localhost:8000/api/version > /dev/null 2>&1; then
-              echo "Windmill is ready"
-              break
-            fi
-            sleep 2
-          done
-          curl -sf http://localhost:8000/api/version > /dev/null || { echo "Windmill failed to start"; tail -50 ../ai_evals/logs/windmill.log; exit 1; }
-
-      - name: Install frontend deps + generate client
-        working-directory: ./frontend
-        run: |
-          npm ci
-          npm run generate-backend-client
-
-      - name: Run global AI evals
-        timeout-minutes: 20
-        working-directory: ./ai_evals
-        env:
-          WMILL_AI_EVAL_BACKEND_URL: http://localhost:8000
-          WMILL_AI_EVAL_BACKEND_WORKSPACE: integration-tests
-          # Anthropic backs the haiku model. Google AI uses GEMINI_API_KEY.
-          ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }}
-          OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
-          GEMINI_API_KEY: ${{ secrets.GOOGLE_API_KEY }}
-          DEEPSEEK_API_KEY: ${{ secrets.DEEPSEEK_API_KEY }}
-        run: |
-          bun install
-          mkdir -p results
-          # One cheap model per provider (anthropic/openai/googleai/deepseek).
-          fail=0
-          for m in haiku 4o gemini-3-flash-preview deepseek-v4-flash; do
-            echo "::group::global-test1-script-create ($m)"
-            if ! bun run cli -- run global global-test1-script-create \
-                  --model "$m" --execution-only --output "$PWD/results/ci-$m.json"; then
-              echo "$m: harness/proxy errored"
-              fail=1
-              echo "::endgroup::"
-              continue
-            fi
-            # The CLI exits 0 when the harness records failed attempts, so gate
-            # on execution-only pass counts while ignoring model output quality.
-            if jq -e \
-              '.attemptCount > 0 and .passedAttempts == .attemptCount' \
-              "results/ci-$m.json" > /dev/null; then
-              echo "$m: OK — proxy/frontend execution completed"
-            else
-              echo "$m: FAILED proxy/frontend execution"
-              jq -c '.cases[0].attempts[0].checks' "results/ci-$m.json" || true
-              fail=1
-            fi
-            echo "::endgroup::"
-          done
-          [ "$fail" = 0 ] || { echo "ai_evals global smoke failed"; exit 1; }
-
-      - name: Archive logs and results
-        uses: actions/upload-artifact@v4
-        if: always()
-        with:
-          name: ai-evals-global-logs
-          path: |
-            ai_evals/logs
-            ai_evals/results

.github/workflows/claude-plan.yml

@@ -45,7 +45,7 @@ jobs:
           allowed_bots: 'windmill-internal-app[bot]'
           trigger_phrase: '/plan'
           claude_args: |
-            --model claude-opus-4-8
+            --model claude-fable-5
             --system-prompt "# Claude Planning Mode
 
             You are operating in PLANNING MODE ONLY. Your role is to create detailed, structured plans without making any code changes.

.github/workflows/claude.yml

@@ -51,4 +51,4 @@ jobs:
             }
           claude_args: |
             --allowedTools "Bash,WebFetch,WebSearch"
-            --model claude-opus-4-8
+            --model claude-fable-5

.github/workflows/pr-ready-review.yml

@@ -160,4 +160,4 @@ jobs:
             ${{ env.REVIEW_PROMPT }}
           claude_args: |
             --allowedTools "mcp__github_inline_comment__create_inline_comment,Bash(gh pr comment:*),Bash(gh pr diff:*),Bash(gh pr view:*)"
-            --model claude-opus-4-8
+            --model claude-fable-5

.gitignore

@@ -33,5 +33,4 @@ backend/chrome_profiler.json
 .fast-check/
 __pycache__/
 .playwright-mcp/
-.codex
-.claude/scheduled_tasks.lock
+.codex

AGENTS.md

@@ -15,7 +15,6 @@ Open-source platform for internal tools, workflows, API integrations, background
 - **Enterprise**: `docs/enterprise.md` — EE file conventions and PR workflow
 - **Backend patterns**: use the `rust-backend` skill when writing Rust code
 - **Frontend patterns**: use the `svelte-frontend` skill when writing Svelte code. Do NOT edit svelte files unless you have read that skill.
-- **Frontend UUIDs**: do not call `crypto.randomUUID()` in frontend code. Import `randomUUID` from `$lib/utils/uuid` instead.
 - **Code review**: review the current PR or branch against the shared review policy in `REVIEW.md` (severity triage, public-surface checklist, AGENTS.md compliance, test-coverage assessment). The skill at `.agents/skills/local-review/SKILL.md` orchestrates it. All three CLIs auto-discover the same SKILL — Claude reads `.claude/skills/` (symlinked to the canonical `.agents/skills/` file), Codex and Pi read `.agents/skills/` directly. Invoke with `/local-review` in Claude Code, `$local-review` (or `/skills` selector) in Codex, or `pi --skill local-review` / `/skill:local-review` in Pi.
 - **Domain guides**: `.claude/skills/native-trigger/` and `frontend/tutorial-system-guide.mdc`
 - **Brand/UI guidelines**: `frontend/brand-guidelines.md`
@@ -48,8 +47,6 @@ Typical flow:
 5. `mcp__playwright__browser_take_screenshot` for visual confirmation
 6. `mcp__playwright__browser_console_messages` / `browser_network_requests` to surface errors
 
-**Attach the screenshots to the PR.** For any change under `frontend/`, embed screenshots of the affected UI in the PR body — the `pr` skill requires this and carries the upload recipe.
-
 If you cannot exercise a UI change (no dev server, etc.), say so explicitly rather than claiming success.
 
 ## Banned Patterns
@@ -109,5 +106,4 @@ $NAV --root backend callees "X"                           # what does X call?
 - Search for existing code to reuse before writing new code
 - Follow established patterns in the codebase
 - Keep changes focused — don't refactor beyond what's asked
-- **Comments record constraints, not narration.** Write a comment only for what the code can't show: why a non-obvious approach is required, what breaks if it's "simplified" away. State each invariant once, at the place where someone would break it, in ≤4 lines. Don't describe what the next line does, don't repeat the same rationale at multiple sites, and don't address the PR reviewer (justifying a change belongs in the PR description, not the code). Describe the code as it is, never its drafting history: "we no longer do X", "unchanged behavior", "instead of the previous approach" are meaningless to a reader who never saw the earlier iteration — before finishing, reread your comments as if the current state is the only state that ever existed.
 - **Never attribute work to a specific customer, account, or "requested by a customer" in repo-tracked content** (PR descriptions, commit messages, code comments, docs). Describe changes by their technical motivation instead.

CHANGELOG.md

@@ -1,331 +1,5 @@
 # Changelog
 
-## [1.737.0](https://github.com/windmill-labs/windmill/compare/v1.736.0...v1.737.0) (2026-06-23)
-
-
-### Features
-
-* **apps:** opt-in sandbox isolation for published & raw apps (alpha) ([#9420](https://github.com/windmill-labs/windmill/issues/9420)) ([2879cbb](https://github.com/windmill-labs/windmill/commit/2879cbb65a4122c86b4a472206d74d9009b07904))
-
-
-### Bug Fixes
-
-* allow SQL args in managed // materialize scripts ([#9733](https://github.com/windmill-labs/windmill/issues/9733)) ([fa35968](https://github.com/windmill-labs/windmill/commit/fa3596885bf2d7ee8859f295e820ee362c756911))
-* bound orphan-cleanup drain rate with capped multi-batch loop ([#9730](https://github.com/windmill-labs/windmill/issues/9730)) ([31d9215](https://github.com/windmill-labs/windmill/commit/31d9215e5a61f19662cc87be8147007e1d47ebb6))
-* **ext-jwt:** reject external JWT auth for non-existent workspaces ([#9723](https://github.com/windmill-labs/windmill/issues/9723)) ([c644311](https://github.com/windmill-labs/windmill/commit/c644311eca4bcaf4b68058cf1d5d79d4078aee1a))
-* optimize cleanup_job_perms_orphaned and job_result_stream cleanup queries ([#9727](https://github.com/windmill-labs/windmill/issues/9727)) ([6d94865](https://github.com/windmill-labs/windmill/commit/6d9486510933af9109f52011d93b13847dbdbb39))
-* prevent silent audit-partition outage via monitor watchdog + alert ([#9729](https://github.com/windmill-labs/windmill/issues/9729)) ([8dea383](https://github.com/windmill-labs/windmill/commit/8dea38383f884f59b2956c39f1424005a21265bd))
-
-
-### Performance Improvements
-
-* **monitor:** hash active-root exclusion in retention delete (WIN-2088) ([#9732](https://github.com/windmill-labs/windmill/issues/9732)) ([75bafab](https://github.com/windmill-labs/windmill/commit/75bafabeeec76cf6da33eef41f588e37071df011))
-
-## [1.736.0](https://github.com/windmill-labs/windmill/compare/v1.735.0...v1.736.0) (2026-06-23)
-
-
-### Features
-
-* **ai-chat:** workspace AI chat skills (SKILL.md upload + read_skill tool) ([#9648](https://github.com/windmill-labs/windmill/issues/9648)) ([6f4017d](https://github.com/windmill-labs/windmill/commit/6f4017d694494a158ebd0579c93336c378cba0fd))
-
-
-### Bug Fixes
-
-* **drafts:** stop mis-filing workspace-blind legacy drafts on migration ([#9725](https://github.com/windmill-labs/windmill/issues/9725)) ([3bf5b72](https://github.com/windmill-labs/windmill/commit/3bf5b72afab3241ea41a261a40c2434764bdaf72))
-* **frontend:** destroy old WebsocketProvider on workspace switch in MultiplayerMenu ([#9719](https://github.com/windmill-labs/windmill/issues/9719)) ([6e96f90](https://github.com/windmill-labs/windmill/commit/6e96f90065dfe2f6ccc5eb4f85f4facd3515c70c))
-* **frontend:** ensure type:object in test_run_flow tool schema for Anthropic ([#9721](https://github.com/windmill-labs/windmill/issues/9721)) ([d5cb944](https://github.com/windmill-labs/windmill/commit/d5cb944cf92f074b2ee42c876595eacdfa2f4d76))
-* **health:** detect read-only replica via pg_is_in_recovery() ([#9722](https://github.com/windmill-labs/windmill/issues/9722)) ([e16061d](https://github.com/windmill-labs/windmill/commit/e16061df06babeae935a9396de5bdcd46e8119a9))
-* re-enforce scoped API token boundaries across handlers ([#9712](https://github.com/windmill-labs/windmill/issues/9712)) ([e19594d](https://github.com/windmill-labs/windmill/commit/e19594df2ad015a0336ade95e04562f5562ec3f6))
-
-## [1.735.0](https://github.com/windmill-labs/windmill/compare/v1.734.0...v1.735.0) (2026-06-22)
-
-
-### Features
-
-* clarify session draft bar tracks all workspace draft changes ([#9714](https://github.com/windmill-labs/windmill/issues/9714)) ([ed016a5](https://github.com/windmill-labs/windmill/commit/ed016a5edb4527877bf7e6bf92feafc2710669c2))
-* **copilot:** improve global-mode path selection + add path-selection evals ([#9698](https://github.com/windmill-labs/windmill/issues/9698)) ([74a2329](https://github.com/windmill-labs/windmill/commit/74a2329d2e8395141807c43acef07ef132490039))
-* link files & folders to the global AI chat ([#9520](https://github.com/windmill-labs/windmill/issues/9520)) ([84cc043](https://github.com/windmill-labs/windmill/commit/84cc043406d63a1e1472165cf24ce8c09905fc5f))
-* scope default instance db name to workspace (dt_/dl_) ([#9699](https://github.com/windmill-labs/windmill/issues/9699)) ([4a8a724](https://github.com/windmill-labs/windmill/commit/4a8a724895dcecb835e1eb1e4fd7d1bbc8b3e0fb))
-
-
-### Bug Fixes
-
-* enforce job_dir containment when writing module files ([#9703](https://github.com/windmill-labs/windmill/issues/9703)) ([e403f92](https://github.com/windmill-labs/windmill/commit/e403f92d7e84cebc78709dce1a0928048ba2506d))
-* **frontend:** deploy full script/flow draft from AI chat via shared module ([#9642](https://github.com/windmill-labs/windmill/issues/9642)) ([23bf6bf](https://github.com/windmill-labs/windmill/commit/23bf6bf3da01d552d2dfab6dbcfd30f758ed34d2))
-* **frontend:** strip raw-app post-deploy diff noise (raw_app/lock/data) ([#9706](https://github.com/windmill-labs/windmill/issues/9706)) ([e20a277](https://github.com/windmill-labs/windmill/commit/e20a27745a08d552e6d2c5a8bbaf08ccfe89c68f))
-* ignore NotFound errors when deleting log files from object store ([#9707](https://github.com/windmill-labs/windmill/issues/9707)) ([8a0b0ab](https://github.com/windmill-labs/windmill/commit/8a0b0abead71320c4f69eb3007739a19f76d4126))
-* **oauth:** restore bring-your-own CC token URL override ([#9711](https://github.com/windmill-labs/windmill/issues/9711)) ([ef4962e](https://github.com/windmill-labs/windmill/commit/ef4962e52aba0bc79bf72523de9101853c660654))
-* sanitize git credentials from ansible executor errors and logs ([#9697](https://github.com/windmill-labs/windmill/issues/9697)) ([ace7b68](https://github.com/windmill-labs/windmill/commit/ace7b68a28b00d715298ffcb6ae907c1974a74b8))
-
-## [1.734.0](https://github.com/windmill-labs/windmill/compare/v1.733.1...v1.734.0) (2026-06-20)
-
-
-### Features
-
-* ducklake materialization for data pipelines ([#9689](https://github.com/windmill-labs/windmill/issues/9689)) ([3ebf243](https://github.com/windmill-labs/windmill/commit/3ebf24359d66048d6361ce65cd879cdc04b737ed))
-
-
-### Bug Fixes
-
-* **frontend:** clear branch step state when switching outer loop iterations ([#9650](https://github.com/windmill-labs/windmill/issues/9650)) ([09a8004](https://github.com/windmill-labs/windmill/commit/09a80040ca268a4379d5302e3401435ba93247e0))
-
-## [1.733.1](https://github.com/windmill-labs/windmill/compare/v1.733.0...v1.733.1) (2026-06-19)
-
-
-### Bug Fixes
-
-* **backend:** validate ansible vault_id entries before config generation ([#9681](https://github.com/windmill-labs/windmill/issues/9681)) ([c1f31c0](https://github.com/windmill-labs/windmill/commit/c1f31c0e4777bf0cfed0dd7f03249e9a61cd8cb9))
-* **frontend:** group live pipeline runs in the activity panel ([#9684](https://github.com/windmill-labs/windmill/issues/9684)) ([1be4df9](https://github.com/windmill-labs/windmill/commit/1be4df9acb935250d4cc12e83cf67e366d870d5a))
-* require super admin for object storage config test endpoint ([#9683](https://github.com/windmill-labs/windmill/issues/9683)) ([fb44fe7](https://github.com/windmill-labs/windmill/commit/fb44fe7af2b8ebe8ef64ffb0e5acce8580bf4200))
-* validate websocket trigger urls and gate trigger test route ([#9682](https://github.com/windmill-labs/windmill/issues/9682)) ([c39ee07](https://github.com/windmill-labs/windmill/commit/c39ee07c0bcd2249dd19ffa5cd988125eefc6c9f))
-
-## [1.733.0](https://github.com/windmill-labs/windmill/compare/v1.732.0...v1.733.0) (2026-06-19)
-
-
-### Features
-
-* **ai-chat:** cap read_app_file + search_app grep tool to bound context in large raw apps ([#9653](https://github.com/windmill-labs/windmill/issues/9653)) ([4296a6a](https://github.com/windmill-labs/windmill/commit/4296a6ae1f73564de4df54fe1df0a03c1df05dfd))
-* **python, windows:** enable S3 to cache wheels ([#5199](https://github.com/windmill-labs/windmill/issues/5199)) ([ab3bc97](https://github.com/windmill-labs/windmill/commit/ab3bc97cd92b6480327029bcf018280442462af7))
-
-
-### Bug Fixes
-
-* allow users to always discard their own drafts without write permission ([#9659](https://github.com/windmill-labs/windmill/issues/9659)) ([6833a55](https://github.com/windmill-labs/windmill/commit/6833a554aeddb3e63173d3c3140b490c0bf2822b))
-* **backend:** clean up unique_ext_jwt_token on workspace deletion ([#9676](https://github.com/windmill-labs/windmill/issues/9676)) ([9add719](https://github.com/windmill-labs/windmill/commit/9add719d936cdcfb2c4062629e3e1f792694dafe))
-* **backend:** strip NUL bytes from draft values on write ([#9673](https://github.com/windmill-labs/windmill/issues/9673)) ([924f9c7](https://github.com/windmill-labs/windmill/commit/924f9c7e8d8863d9af40aee246a519b4be0e1ea2))
-* **python:** split PIP_TRUSTED_HOST by whitespace to support multiple hosts ([#9675](https://github.com/windmill-labs/windmill/issues/9675)) ([cafb473](https://github.com/windmill-labs/windmill/commit/cafb473494d9cff3a8b2aeaf9f18b015f966e7b3))
-
-## [1.732.0](https://github.com/windmill-labs/windmill/compare/v1.731.0...v1.732.0) (2026-06-19)
-
-
-### Features
-
-* **ansible:** add AI chat and editor bar buttons for ansible ([#9671](https://github.com/windmill-labs/windmill/issues/9671)) ([017c3d3](https://github.com/windmill-labs/windmill/commit/017c3d3343c2577501103be4b2dd8dac9727d80d))
-
-
-### Bug Fixes
-
-* **ai:** emit token usage in gemini proxy streaming translation ([#9669](https://github.com/windmill-labs/windmill/issues/9669)) ([0cc2257](https://github.com/windmill-labs/windmill/commit/0cc2257596a3965cf6db21a0090edcce6e1b8419))
-* **backend:** grant script_trigger access to windmill roles ([#9674](https://github.com/windmill-labs/windmill/issues/9674)) ([3361736](https://github.com/windmill-labs/windmill/commit/33617367d09537667d2ab3f91135c736194b9e7e))
-* **frontend:** ignore hash/assets in script diffs and drafts (WIN-2071) ([#9664](https://github.com/windmill-labs/windmill/issues/9664)) ([3371265](https://github.com/windmill-labs/windmill/commit/33712653821e83f2562dd5f271dbec0188d5d2f8))
-
-## [1.731.0](https://github.com/windmill-labs/windmill/compare/v1.730.0...v1.731.0) (2026-06-19)
-
-
-### Features
-
-* **backend:** auto-reconnect postgres trigger listener with backoff (WIN-2073) ([#9666](https://github.com/windmill-labs/windmill/issues/9666)) ([a425431](https://github.com/windmill-labs/windmill/commit/a425431e9067bcf85474fdc7b7ef7f73e41b9071))
-
-
-### Bug Fixes
-
-* **backend:** grant notify_event access to windmill roles ([#9665](https://github.com/windmill-labs/windmill/issues/9665)) ([a682d02](https://github.com/windmill-labs/windmill/commit/a682d02311a2110bfc0d5e0a5b52e96147fe0dd7))
-* **mcp:** repair invalid type keywords in tool JSON schemas ([#9667](https://github.com/windmill-labs/windmill/issues/9667)) ([c30bdec](https://github.com/windmill-labs/windmill/commit/c30bdecea77ff9b4d74d52961f3101201099b683))
-* trigger flow error handler on unrecoverable (OOM/zombie) step failures ([#9662](https://github.com/windmill-labs/windmill/issues/9662)) ([7e4df02](https://github.com/windmill-labs/windmill/commit/7e4df02bd60c4d6ee8c92d3dfd19f4e587ff9632))
-
-## [1.730.0](https://github.com/windmill-labs/windmill/compare/v1.729.0...v1.730.0) (2026-06-18)
-
-
-### Features
-
-* **ai-chat:** summary-based conversation compaction ([#9645](https://github.com/windmill-labs/windmill/issues/9645)) ([5d553b8](https://github.com/windmill-labs/windmill/commit/5d553b81c06664aab61131a93b198575c088d12d))
-* Data Pipelines alpha ([#9193](https://github.com/windmill-labs/windmill/issues/9193)) ([7155a0b](https://github.com/windmill-labs/windmill/commit/7155a0bb96cf30bd878272a0f4c3c3b02341b261))
-
-
-### Bug Fixes
-
-* **ai-chat:** stop echoing app draft value in global chat write tool results ([#9658](https://github.com/windmill-labs/windmill/issues/9658)) ([2fed808](https://github.com/windmill-labs/windmill/commit/2fed808b9e716d9a44b34c7a073ec0d37374be05))
-* **backend:** include raw_app drafts in list_apps draft_users ([#9647](https://github.com/windmill-labs/windmill/issues/9647)) ([19bc005](https://github.com/windmill-labs/windmill/commit/19bc0052f1069d732231950a0ec958f675d57417))
-* **frontend:** keep ?new_draft flag until first save is confirmed ([#9656](https://github.com/windmill-labs/windmill/issues/9656)) ([9b6b7c3](https://github.com/windmill-labs/windmill/commit/9b6b7c3862d9988e5e91eaab2b967a23f41cdc0d))
-* **frontend:** re-key raw-app autosave on post-deploy navigation ([#9646](https://github.com/windmill-labs/windmill/issues/9646)) ([1058bde](https://github.com/windmill-labs/windmill/commit/1058bdeccdc4c403ef4599db0ee74a65a66c715f))
-* gate agent-worker global setting reads with a blocklist ([#9623](https://github.com/windmill-labs/windmill/issues/9623)) ([fdd82f0](https://github.com/windmill-labs/windmill/commit/fdd82f0c48f29805cd9e219649f27fba45c7fd92))
-* **workspaces:** add instance setting to disable workspace invite/add emails ([#9643](https://github.com/windmill-labs/windmill/issues/9643)) ([796230d](https://github.com/windmill-labs/windmill/commit/796230d90a7e6d1debc15e139ab708881e527862))
-
-## [1.729.0](https://github.com/windmill-labs/windmill/compare/v1.728.1...v1.729.0) (2026-06-18)
-
-
-### Features
-
-* add ducklake schema support to the database manager ([#9633](https://github.com/windmill-labs/windmill/issues/9633)) ([3eeccaf](https://github.com/windmill-labs/windmill/commit/3eeccaf9682b7803fdf5be8dcbc4d243e0ba2e49))
-* **ai-chat:** self-hosted docs tools via windmill.dev llms.txt + ask benchmark ([#9578](https://github.com/windmill-labs/windmill/issues/9578)) ([f4425fc](https://github.com/windmill-labs/windmill/commit/f4425fca9fb0d02b845bd72888ade54905c5a30b))
-* **frontend:** View Diff and in-place Load for other users' drafts ([#9621](https://github.com/windmill-labs/windmill/issues/9621)) ([5508f1d](https://github.com/windmill-labs/windmill/commit/5508f1da9cd04c2583eb3f7ee6bce19d067f2227))
-* per-user draft review & deploy page (gating, badges, rename, raw-app deploy fixes) ([#9625](https://github.com/windmill-labs/windmill/issues/9625)) ([e09cd58](https://github.com/windmill-labs/windmill/commit/e09cd5862cb636e143027fe8d9a5be9c7097b031))
-* queue messages typed while ai chat is streaming ([#9525](https://github.com/windmill-labs/windmill/issues/9525)) ([51bd869](https://github.com/windmill-labs/windmill/commit/51bd8692a482850f7ac8b04dd16db5876336b5b9))
-* zero-setup oauth client credentials for registry providers ([#9559](https://github.com/windmill-labs/windmill/issues/9559)) ([e26a923](https://github.com/windmill-labs/windmill/commit/e26a9239a62a25abf90ef06ade4dde7f36e791bb))
-
-
-### Bug Fixes
-
-* **ai_evals:** adapt global eval harness to DB-backed user drafts ([#9641](https://github.com/windmill-labs/windmill/issues/9641)) ([e87ff79](https://github.com/windmill-labs/windmill/commit/e87ff79ecf6a6e0958916ed1b3756fb3addf719f))
-* **drafts:** preserve original timestamp when migrating localStorage drafts ([#9638](https://github.com/windmill-labs/windmill/issues/9638)) ([8021775](https://github.com/windmill-labs/windmill/commit/8021775f5f961ef6fd01b022639b85855326a1da))
-* **frontend:** don't save drafts on leave when auto-save is off, warn instead ([#9630](https://github.com/windmill-labs/windmill/issues/9630)) ([2523465](https://github.com/windmill-labs/windmill/commit/252346500945a9571af744c839ac0c7d6870504f))
-* **frontend:** render Modal2 dialogs above the AI chat panel ([#9636](https://github.com/windmill-labs/windmill/issues/9636)) ([b67c8cf](https://github.com/windmill-labs/windmill/commit/b67c8cf42b477575fc1bc448058ec0d3b7e54fee))
-* **frontend:** show AI sessions when AI unconfigured, with disabled chat ([#9644](https://github.com/windmill-labs/windmill/issues/9644)) ([ba69d81](https://github.com/windmill-labs/windmill/commit/ba69d8147b615e160cf3d2885fc65a0777b78b71))
-* **git-sync:** bump default sync script to hub/28719 (windmill-cli 1.728.1) for WAC modules ([#9649](https://github.com/windmill-labs/windmill/issues/9649)) ([3c0e38b](https://github.com/windmill-labs/windmill/commit/3c0e38b5890d77983cb5cf5f422a62a73e7a4f22))
-
-## [1.728.1](https://github.com/windmill-labs/windmill/compare/v1.728.0...v1.728.1) (2026-06-17)
-
-
-### Bug Fixes
-
-* **backend:** purge workspace_diff cache on workspace delete ([#9627](https://github.com/windmill-labs/windmill/issues/9627)) ([8a3f69d](https://github.com/windmill-labs/windmill/commit/8a3f69dda8f2088fb859ed8ed6e54458940423d0))
-* **cli:** fall back to esbuild-wasm on native host/binary mismatch ([#9629](https://github.com/windmill-labs/windmill/issues/9629)) ([86d1d16](https://github.com/windmill-labs/windmill/commit/86d1d160f0d3bd9faabdafada07e2956dd98445d))
-* **frontend:** persist session-editor draft path/summary edits + per-line diff tooltips ([#9622](https://github.com/windmill-labs/windmill/issues/9622)) ([e4bfeb2](https://github.com/windmill-labs/windmill/commit/e4bfeb29bc4e89669863b5f6396904a331167658))
-
-## [1.728.0](https://github.com/windmill-labs/windmill/compare/v1.727.0...v1.728.0) (2026-06-16)
-
-
-### Features
-
-* **frontend:** adapt AI-chat/sessions drafts to DB-backed model ([#9601](https://github.com/windmill-labs/windmill/issues/9601)) ([611c70a](https://github.com/windmill-labs/windmill/commit/611c70acd211cf4b8f8308da4a264c670a2f5f43))
-* **frontend:** consolidate draft-migration errors into a single toast + modal ([#9612](https://github.com/windmill-labs/windmill/issues/9612)) ([bc0d5bf](https://github.com/windmill-labs/windmill/commit/bc0d5bf241df3633921bd9d43d171e91034fbfcf))
-* **frontend:** dedup user drafts against the deployed baseline ([#9618](https://github.com/windmill-labs/windmill/issues/9618)) ([a2ce446](https://github.com/windmill-labs/windmill/commit/a2ce44645fdbfa98bf250fac2d15d2b5b26c4b47))
-
-
-### Bug Fixes
-
-* **frontend:** reset deleteWorkspaceForkModal on confirm in SidebarContent ([#9619](https://github.com/windmill-labs/windmill/issues/9619)) ([7cb5c6e](https://github.com/windmill-labs/windmill/commit/7cb5c6e749b2020dee5ee1499f0dc69c5109a6d8))
-* **frontend:** session Drafts drawer uses raw_app kind for the raw-app diff ([#9617](https://github.com/windmill-labs/windmill/issues/9617)) ([46288b6](https://github.com/windmill-labs/windmill/commit/46288b6143efae4dfdf6fe068b97a1e8831fce6a))
-* **nativets:** respect custom CA certs in in-process fetch runtime ([#9615](https://github.com/windmill-labs/windmill/issues/9615)) ([41562c7](https://github.com/windmill-labs/windmill/commit/41562c7d7c708d7d056d9b3d0c39b994a6f4a016))
-* **ResourceForm:** initialize JSON editor when resource type schema is unavailable ([#9611](https://github.com/windmill-labs/windmill/issues/9611)) ([5a24057](https://github.com/windmill-labs/windmill/commit/5a2405743b4622fc1021109114d007057abd5dfd))
-* show folder labels in the folder list table ([#9620](https://github.com/windmill-labs/windmill/issues/9620)) ([651fa13](https://github.com/windmill-labs/windmill/commit/651fa13ee80ff76e5a53ef1ed545b03ce6792294))
-* show last updated date per user in other-users-drafts modal ([#9614](https://github.com/windmill-labs/windmill/issues/9614)) ([f6104ce](https://github.com/windmill-labs/windmill/commit/f6104ce05c4005ffb9fe8112782d1ef6d3065300))
-
-## [1.727.0](https://github.com/windmill-labs/windmill/compare/v1.726.1...v1.727.0) (2026-06-16)
-
-
-### Features
-
-* support temp_script_refs in wmill dev for local relative imports ([#9554](https://github.com/windmill-labs/windmill/issues/9554)) ([33ac287](https://github.com/windmill-labs/windmill/commit/33ac287065742df53f363a5fe09f54f5584a85a6))
-
-
-### Bug Fixes
-
-* **cli:** harden legacy flow lock migration ordering and collision guard ([#9557](https://github.com/windmill-labs/windmill/issues/9557)) ([cd09870](https://github.com/windmill-labs/windmill/commit/cd098700c2cd8d7e9150f760938f4eaf34d188ec))
-* **cli:** include __mod/ folder in gitSyncIncludePattern for scripts ([#9606](https://github.com/windmill-labs/windmill/issues/9606)) ([252c1b3](https://github.com/windmill-labs/windmill/commit/252c1b35fc716c3486109d89615127c588bbe90a))
-* **frontend:** allow same-origin redirects in isValidLogoutRedirect ([#9568](https://github.com/windmill-labs/windmill/issues/9568)) ([8500435](https://github.com/windmill-labs/windmill/commit/8500435e82231e13a1b8a874fd0545f0a0a73fee))
-* **frontend:** make UserDraft read-after-write work without live entry ([#9609](https://github.com/windmill-labs/windmill/issues/9609)) ([51e82d7](https://github.com/windmill-labs/windmill/commit/51e82d7c6d30c66c84236feb743c09929934e564))
-* **frontend:** seed detached user-draft handles so new-item drawers render ([#9608](https://github.com/windmill-labs/windmill/issues/9608)) ([9e3c0de](https://github.com/windmill-labs/windmill/commit/9e3c0decf95378c66055d82215c15cd3bf4a69cb))
-* **frontend:** strip server-managed fields from value diffs ([#9599](https://github.com/windmill-labs/windmill/issues/9599)) ([c213801](https://github.com/windmill-labs/windmill/commit/c213801b5aee54d801c14b9eb31422f2a312ef7e))
-
-## [1.726.1](https://github.com/windmill-labs/windmill/compare/v1.726.0...v1.726.1) (2026-06-15)
-
-
-### Bug Fixes
-
-* **apps:** prevent decision tree graph editor crash on missing graph context ([#9602](https://github.com/windmill-labs/windmill/issues/9602))  ([24f3259](https://github.com/windmill-labs/windmill/commit/24f32596e9ca39c963c19af4a8b14fbcd04e3a78))
-* db-backed draft fixes — review-page UX, legacy drafts, session restore ([#9600](https://github.com/windmill-labs/windmill/issues/9600)) ([4e4b224](https://github.com/windmill-labs/windmill/commit/4e4b2247ef471dada1b8c894974fe921d14b3947))
-
-## [1.726.0](https://github.com/windmill-labs/windmill/compare/v1.725.1...v1.726.0) (2026-06-15)
-
-
-### Features
-
-* **audit:** record workspace archive/unarchive/delete in instance audit log ([#9596](https://github.com/windmill-labs/windmill/issues/9596)) ([9de5708](https://github.com/windmill-labs/windmill/commit/9de57086086bb5626d175c7f926915d1d6ac67ca))
-* **frontend:** add user-level toggle to disable Windmill AI ([#9585](https://github.com/windmill-labs/windmill/issues/9585)) ([5709a56](https://github.com/windmill-labs/windmill/commit/5709a564fbafd9aa91943572ecd8c3e0c45c20b1))
-
-
-### Bug Fixes
-
-* **embeddings:** retry HuggingFace model downloads with backoff ([#9597](https://github.com/windmill-labs/windmill/issues/9597)) ([6a62959](https://github.com/windmill-labs/windmill/commit/6a6295921d681359155d814507908792be405679))
-* resolve release CI failures (pypi bundle, flow serde test, cli windows) ([#9595](https://github.com/windmill-labs/windmill/issues/9595)) ([5ccaae8](https://github.com/windmill-labs/windmill/commit/5ccaae8ab36f2be18b67863ea069763455908029))
-
-## [1.725.1](https://github.com/windmill-labs/windmill/compare/v1.725.0...v1.725.1) (2026-06-15)
-
-
-### Bug Fixes
-
-* **apps:** apply scope-path predicate to app list/search endpoints ([#9581](https://github.com/windmill-labs/windmill/issues/9581)) ([3bf6e10](https://github.com/windmill-labs/windmill/commit/3bf6e102afbdad41e558617bc812012eaaaecd9b))
-* **auth:** add scope checks to scripts/flows list_tokens endpoints ([#9582](https://github.com/windmill-labs/windmill/issues/9582)) ([36c9f86](https://github.com/windmill-labs/windmill/commit/36c9f8612b5778aa2c981454729590b71671ce8d))
-* **cli:** preserve committed script.lock on transient NULL lock during git-sync deploy ([#9593](https://github.com/windmill-labs/windmill/issues/9593)) ([6b916ac](https://github.com/windmill-labs/windmill/commit/6b916ac688e0305284e6cf819bf28803bcca0118))
-* expose parent_hash in MCP createScript tool for updates ([#9586](https://github.com/windmill-labs/windmill/issues/9586)) ([a69505d](https://github.com/windmill-labs/windmill/commit/a69505df9bf25d7c4f11d0528a7450c08dbb422c))
-* **flows:** stop serializing default retry/stop_after_if fields ([#9583](https://github.com/windmill-labs/windmill/issues/9583)) ([e1e2a24](https://github.com/windmill-labs/windmill/commit/e1e2a24b6a6752b3ac779cb0db38061cbc54425e))
-* **security:** sanitize dependency names & connection strings against command/SQL injection ([#9590](https://github.com/windmill-labs/windmill/issues/9590)) ([aff0a4e](https://github.com/windmill-labs/windmill/commit/aff0a4ec189cd8e315282e878bb858ef00635b90))
-
-## [1.725.0](https://github.com/windmill-labs/windmill/compare/v1.724.0...v1.725.0) (2026-06-15)
-
-
-### Features
-
-* Db-backed user drafts ([#9351](https://github.com/windmill-labs/windmill/issues/9351)) ([1fc3557](https://github.com/windmill-labs/windmill/commit/1fc355709c025fd256c5a4035356e15a5a05b23d))
-* scope AI session storage per user, session list in IndexedDB ([#9518](https://github.com/windmill-labs/windmill/issues/9518)) ([aa26c4d](https://github.com/windmill-labs/windmill/commit/aa26c4d9b22b3a353a6c0605eb9a4193e34aa18c))
-
-
-### Bug Fixes
-
-* **powershell:** sanitize module names to prevent command injection (CWE-78) ([#9587](https://github.com/windmill-labs/windmill/issues/9587)) ([6acce7a](https://github.com/windmill-labs/windmill/commit/6acce7a88733683153db534cb18752b31d93af82))
-
-## [1.724.0](https://github.com/windmill-labs/windmill/compare/v1.723.0...v1.724.0) (2026-06-15)
-
-
-### Features
-
-* **cli:** add --yes, --secret/--no-secret and --description to variable add ([#9548](https://github.com/windmill-labs/windmill/issues/9548)) ([4e9e0c0](https://github.com/windmill-labs/windmill/commit/4e9e0c024b4b95f9676b1646591d8f0c662e84ab))
-* **frontend:** improve AI chat cancel and interrupted-turn handling ([#9539](https://github.com/windmill-labs/windmill/issues/9539)) ([114c412](https://github.com/windmill-labs/windmill/commit/114c41251a8c738b58a1a3dd9434d09d33feb6f1))
-* **frontend:** precise AI chat context usage tracking + indicator ([#9551](https://github.com/windmill-labs/windmill/issues/9551)) ([2b47180](https://github.com/windmill-labs/windmill/commit/2b471805bf1c92bb210cfacda217a4341e1f989c))
-* wire chat reasoning effort through gemini and bedrock proxies ([#9545](https://github.com/windmill-labs/windmill/issues/9545)) ([aaf0563](https://github.com/windmill-labs/windmill/commit/aaf05635cedadc73455dc522474b673719f9fd5c))
-
-
-### Bug Fixes
-
-* actually isolate windows job children from CTRL_BREAK_EVENT + reap on worker death ([#9563](https://github.com/windmill-labs/windmill/issues/9563)) ([61f3291](https://github.com/windmill-labs/windmill/commit/61f3291b240bdb5c26bee8947351a9590bc3bd45))
-* **ai:** enforce resource authz when loading MCP tools in agent worker ([#9571](https://github.com/windmill-labs/windmill/issues/9571)) ([317a862](https://github.com/windmill-labs/windmill/commit/317a8629d1c8436d2a6f3443bd25b81d606ce283))
-* append system CA bundle to tracing proxy cert file ([#9549](https://github.com/windmill-labs/windmill/issues/9549)) ([3cf4083](https://github.com/windmill-labs/windmill/commit/3cf40839602e5c3d1df51f0a29b01736bade09da))
-* **cli:** consistent flow inline lock filenames for compound extensions ([#9555](https://github.com/windmill-labs/windmill/issues/9555)) ([f0659a7](https://github.com/windmill-labs/windmill/commit/f0659a755a161420833e3bfdbe04befc6ebeb977))
-* **flows:** skip_if evaluates wrong previous_result during retry ([#9547](https://github.com/windmill-labs/windmill/issues/9547)) ([2aab352](https://github.com/windmill-labs/windmill/commit/2aab35245c362c2f911c60ea435f29bfb1369ebf))
-* **folders:** allow hyphens in folder names ([#9566](https://github.com/windmill-labs/windmill/issues/9566)) ([84df111](https://github.com/windmill-labs/windmill/commit/84df11177f2009bff007e9b722b55a9a5a63c06a)), closes [#8474](https://github.com/windmill-labs/windmill/issues/8474)
-* **frontend:** load resource value in JSON editor when resource type is missing ([#9574](https://github.com/windmill-labs/windmill/issues/9574)) ([251266c](https://github.com/windmill-labs/windmill/commit/251266cd8119dbef314314daed43aa43ab92f1c9))
-* isolate windows job children from worker CTRL_BREAK_EVENT ([#9562](https://github.com/windmill-labs/windmill/issues/9562)) ([1d6191e](https://github.com/windmill-labs/windmill/commit/1d6191ebb75843917eec6c76a4be347f9ac4cb72))
-* stop sending temperature for AI chat across all providers ([#9553](https://github.com/windmill-labs/windmill/issues/9553)) ([3585716](https://github.com/windmill-labs/windmill/commit/358571687296fbe5c378533b3c1662707955c64a))
-
-## [1.723.0](https://github.com/windmill-labs/windmill/compare/v1.722.0...v1.723.0) (2026-06-11)
-
-
-### Features
-
-* add get_app_runtime_logs tool to global chat ([#9502](https://github.com/windmill-labs/windmill/issues/9502)) ([f86d0d7](https://github.com/windmill-labs/windmill/commit/f86d0d79fc6aa23119fd59761330ab79592d0a2a))
-* **cli:** improve agent prompts/skills and workspace fork workflow ([#9531](https://github.com/windmill-labs/windmill/issues/9531)) ([5bdc4f8](https://github.com/windmill-labs/windmill/commit/5bdc4f83ce37302a2c375d0ff73763acfee2aadb))
-* enable native web search in copilot ([#9522](https://github.com/windmill-labs/windmill/issues/9522)) ([d3f5fe1](https://github.com/windmill-labs/windmill/commit/d3f5fe1c8c39ff07d05f0922b8f40aa95756a707))
-
-
-### Bug Fixes
-
-* **frontend:** stop live activity flickering when user has multiple tabs ([#9543](https://github.com/windmill-labs/windmill/issues/9543)) ([57e627e](https://github.com/windmill-labs/windmill/commit/57e627eabf7c4144ce1c07214ad44d026b82f0b4))
-* omit temperature for claude fable 5 ([#9540](https://github.com/windmill-labs/windmill/issues/9540)) ([bd00bee](https://github.com/windmill-labs/windmill/commit/bd00beeac54dbcfa9ab86fd336fca1a8fa289341))
-* refetch license key from settings when in-memory key is invalid ([#9534](https://github.com/windmill-labs/windmill/issues/9534)) ([38c0ccd](https://github.com/windmill-labs/windmill/commit/38c0ccdf563d3655a4cba390ce9372bc9f9c4a9b))
-
-## [1.722.0](https://github.com/windmill-labs/windmill/compare/v1.721.0...v1.722.0) (2026-06-11)
-
-
-### Features
-
-* add reasoning effort control and thinking display to AI chat ([#9511](https://github.com/windmill-labs/windmill/issues/9511)) ([7f987e8](https://github.com/windmill-labs/windmill/commit/7f987e8c9807b72d9cc3901b6e4d02a24c423f50))
-* **ai-chat:** collapse big pastes, cap input height, escape HTML ([#9487](https://github.com/windmill-labs/windmill/issues/9487)) ([365e204](https://github.com/windmill-labs/windmill/commit/365e20410ed528d5b4e967b64fb282bcc1e03ccd))
-* **ai-chat:** quick access to AI prompt settings from chat ([#9508](https://github.com/windmill-labs/windmill/issues/9508)) ([b894f78](https://github.com/windmill-labs/windmill/commit/b894f783f183ab795eab6f57da8654275d9ad82d))
-* **ai:** add list_runs and get_job_logs tools to global chat mode ([#9488](https://github.com/windmill-labs/windmill/issues/9488)) ([cfe5119](https://github.com/windmill-labs/windmill/commit/cfe51190356a9e922f6398dd875abc368accbd15))
-* clear conflict error + force delete when reusing a fork workspace id ([#9499](https://github.com/windmill-labs/windmill/issues/9499)) ([fddabe9](https://github.com/windmill-labs/windmill/commit/fddabe9c5c6f178b4b09854dc11e50adafd44c87))
-* **flow:** support worker tag override on AI agent steps ([#9513](https://github.com/windmill-labs/windmill/issues/9513)) ([a6a5600](https://github.com/windmill-labs/windmill/commit/a6a5600833063e36b35aac7f90dcbdcd3db7c627))
-* folder-level label inheritance for scripts, flows and jobs ([#9524](https://github.com/windmill-labs/windmill/issues/9524)) ([765f50c](https://github.com/windmill-labs/windmill/commit/765f50c474f8abf76550664ed15418ddb3c0b221))
-* **frontend:** show AI sessions in narrow-screen burger menu ([#9523](https://github.com/windmill-labs/windmill/issues/9523)) ([a8f1062](https://github.com/windmill-labs/windmill/commit/a8f1062f37228e7a8257aac1cd3295bd5084ff90))
-* prefer idle worker pods on k8s autoscaling scale-in via pod-deletion-cost ([#9515](https://github.com/windmill-labs/windmill/issues/9515)) ([3c3f157](https://github.com/windmill-labs/windmill/commit/3c3f15722fd22713cc8baa744d9a81207943a33c))
-* prompt browser confirmation on page exit with unsaved changes ([#9503](https://github.com/windmill-labs/windmill/issues/9503)) ([3119e16](https://github.com/windmill-labs/windmill/commit/3119e16ed8df0daec0e019ae2efb2176d7e9e953))
-* **worker:** #ssh directive to run a bash script on a remote SSH host ([#9479](https://github.com/windmill-labs/windmill/issues/9479)) ([afddfe8](https://github.com/windmill-labs/windmill/commit/afddfe84452357b06f4fb815566d4c8269ceafbf))
-* workspace protection rule to restrict anonymous app deployment ([#9509](https://github.com/windmill-labs/windmill/issues/9509)) ([cf9ad54](https://github.com/windmill-labs/windmill/commit/cf9ad54181d38c5d3c3aa05208e17d8d97eae4ef))
-
-
-### Bug Fixes
-
-* **cli:** include lock-relevant script content in lock cache key ([#9528](https://github.com/windmill-labs/windmill/issues/9528)) ([dc60e1a](https://github.com/windmill-labs/windmill/commit/dc60e1aa174f2a7616d2d4a37117a463ef25a9d6))
-* **frontend:** allow copy/paste shortcuts inside ConfirmationModal ([#9505](https://github.com/windmill-labs/windmill/issues/9505)) ([7fc5340](https://github.com/windmill-labs/windmill/commit/7fc5340da39c049d059aa0ae15ceb6a98ca58053))
-* **frontend:** clarify trigger filters match the message parsed as JSON ([#9516](https://github.com/windmill-labs/windmill/issues/9516)) ([0b17843](https://github.com/windmill-labs/windmill/commit/0b178437ce8cd8295be21c0c6077c39e957a3337))
-* **frontend:** enable Apply button when env vars change in worker group config ([#9501](https://github.com/windmill-labs/windmill/issues/9501)) ([c80c6d8](https://github.com/windmill-labs/windmill/commit/c80c6d8fcdfc1de8fe841e9699452f67a25cad23))
-* **frontend:** improve AI chat markdown and typing dots in dark mode ([#9497](https://github.com/windmill-labs/windmill/issues/9497)) ([4e86806](https://github.com/windmill-labs/windmill/commit/4e868062d4c22c5574149d35235b7ad1abe805f3))
-* **frontend:** stop echoing draft values in global chat write tool results ([#9530](https://github.com/windmill-labs/windmill/issues/9530)) ([ce6e2f7](https://github.com/windmill-labs/windmill/commit/ce6e2f7ade25ca91c375f0de5f1be3f1c82ccb47))
-* inherit container NO_PROXY into MITM tracing proxy job exclusions ([#9492](https://github.com/windmill-labs/windmill/issues/9492)) ([4c22e3b](https://github.com/windmill-labs/windmill/commit/4c22e3b712a74828cf654ea7d89aeab5b50cfbd7))
-* make default chat model optional in AI settings ([#9514](https://github.com/windmill-labs/windmill/issues/9514)) ([1d43288](https://github.com/windmill-labs/windmill/commit/1d4328877fcc87352fb56de63f0570739d5a9dc4))
-* **nsjail:** make ansible collections mount non-mandatory ([#9510](https://github.com/windmill-labs/windmill/issues/9510)) ([08da7a1](https://github.com/windmill-labs/windmill/commit/08da7a121b4b835500dfc2bd943c4bdce912c63e))
-* **nsjail:** make ansible uv tools mount non-mandatory ([#9507](https://github.com/windmill-labs/windmill/issues/9507)) ([dc368a9](https://github.com/windmill-labs/windmill/commit/dc368a9669e812c593ad848266f645b6223501e6))
-
 ## [1.721.0](https://github.com/windmill-labs/windmill/compare/v1.720.0...v1.721.0) (2026-06-09)
 
 

ai_evals/README.md

@@ -75,8 +75,6 @@ Public CLI surface:
 - `--model <alias>`: choose the model under test
 - `--models <a,b,c>`: run the same cases sequentially against several model aliases
 - `--verbose`: stream assistant output for frontend runs
-- `--skip-judge`: skip LLM judge scoring for the run
-- `--execution-only`: only require the model/proxy/frontend loop to complete; skip validators, tool expectations, backend artifact validation, and judge scoring
 - `--record`: append a compact tracked summary line to `ai_evals/history/<mode>.jsonl` for full-suite runs only
 - `--backend-validation <mode>`: optional backend smoke validation (`off` or `preview`) for `script` and `flow` evals
 
@@ -101,7 +99,7 @@ Notes:
 - the command also prints accepted alias spellings such as `gpt-4o`, `gpt-55`, `claude-opus-4.6`, and `claude-haiku-4.5`
 - frontend modes (`flow`, `script`, `app`, `global`) can use Anthropic, OpenAI, Gemini, and DeepSeek-backed aliases
 - `cli` mode always uses the Anthropic agent SDK, so only Anthropic aliases are valid there
-- the judge model is separate and currently defaults to `claude-sonnet-4-6`; use `--skip-judge` for deterministic-only runs
+- the judge model is separate and currently defaults to `claude-sonnet-4-6`
 
 ## Case Format
 

ai_evals/adapters/cli/runtime.ts

@@ -16,9 +16,6 @@ export interface PromptRunResult {
   output: string;
   durationMs: number;
   tokenUsage: BenchmarkTokenUsage | null;
-  // Input tokens on the last assistant turn. The SDK `result` message reports
-  // usage cumulatively, so the final context size comes from per-turn usage.
-  finalContextTokens: number | null;
   trace: CliTrace;
 }
 
@@ -147,7 +144,6 @@ export async function runPromptAndCapture(
   let output = "";
   let assistantMessageCount = 0;
   let tokenUsage: BenchmarkTokenUsage | null = null;
-  let finalContextTokens: number | null = null;
   const startedAt = Date.now();
   const stubBinDir = join(cwd, WMILL_STUB_DIR_NAME);
   const wmillLogPath = join(cwd, WMILL_LOG_FILE_NAME);
@@ -170,12 +166,6 @@ export async function runPromptAndCapture(
   for await (const message of query({ prompt, options })) {
     if (message.type === "assistant") {
       assistantMessageCount += 1;
-      const turnContext = anthropicUsageToBenchmarkTokenUsage(
-        message.message?.usage
-      )?.prompt;
-      if (turnContext && turnContext > 0) {
-        finalContextTokens = turnContext;
-      }
       const content = message.message?.content;
       if (Array.isArray(content)) {
         for (const block of content) {
@@ -220,7 +210,6 @@ export async function runPromptAndCapture(
     output,
     durationMs: Date.now() - startedAt,
     tokenUsage,
-    finalContextTokens,
     trace: {
       toolsUsed,
       skillsInvoked,

ai_evals/adapters/frontend/benchmarkRunner.ts

@@ -12,7 +12,7 @@ import { resolveWindmillBackendSettings } from "../../core/windmillBackendSettin
 import { emitFrontendBenchmarkProgress } from "./progress";
 import { DEFAULT_JUDGE_MODEL } from "../../core/judge";
 
-export type FrontendBenchmarkMode = "flow" | "app" | "script" | "global";
+export type FrontendBenchmarkMode = "flow" | "app" | "script" | "global" | "ask";
 
 export async function runFrontendBenchmarkFromEnv(): Promise<BenchmarkRunResult> {
   const mode = parseMode(process.env.WMILL_FRONTEND_AI_EVAL_MODE);
@@ -25,12 +25,6 @@ export async function runFrontendBenchmarkFromEnv(): Promise<BenchmarkRunResult>
   );
   const emitProgress = process.env.WMILL_FRONTEND_AI_EVAL_PROGRESS === "1";
   const verbose = process.env.WMILL_FRONTEND_AI_EVAL_VERBOSE === "1";
-  const executionOnly =
-    process.env.WMILL_FRONTEND_AI_EVAL_EXECUTION_ONLY === "1";
-  const judgeModel =
-    process.env.WMILL_FRONTEND_AI_EVAL_SKIP_JUDGE === "1" || executionOnly
-      ? null
-      : DEFAULT_JUDGE_MODEL;
   const model = resolveEvalModel(
     mode,
     process.env.WMILL_FRONTEND_AI_EVAL_MODEL,
@@ -48,14 +42,17 @@ export async function runFrontendBenchmarkFromEnv(): Promise<BenchmarkRunResult>
     backendValidation,
     backendSettings,
   );
-  const runModel = formatRunModelLabel(mode, model);
+  const docsTool =
+    mode === "ask" ? process.env.WMILL_AI_EVAL_DOCS_TOOL?.trim() || "llmstxt" : undefined;
+  const runModel = docsTool
+    ? `${formatRunModelLabel(mode, model)} ask:${docsTool}`
+    : formatRunModelLabel(mode, model);
   const caseResults = await runSuite({
     modeRunner,
     cases: selectedCases,
     runs,
     runModel,
-    judgeModel,
-    executionOnly,
+    judgeModel: DEFAULT_JUDGE_MODEL,
     concurrency: verbose ? 1 : undefined,
     verbose,
     onProgress: emitProgress
@@ -67,7 +64,7 @@ export async function runFrontendBenchmarkFromEnv(): Promise<BenchmarkRunResult>
     mode,
     runs,
     runModel,
-    judgeModel,
+    judgeModel: DEFAULT_JUDGE_MODEL,
     caseResults,
   });
 }
@@ -99,6 +96,10 @@ async function getModeRunner(
       const { createGlobalModeRunner } = await import("../../modes/global");
       return createGlobalModeRunner(model, backendSettings);
     }
+    case "ask": {
+      const { createAskModeRunner } = await import("../../modes/ask");
+      return createAskModeRunner(model, backendSettings);
+    }
   }
 }
 
@@ -107,7 +108,8 @@ function parseMode(value: string | undefined): FrontendBenchmarkMode {
     value === "flow" ||
     value === "app" ||
     value === "script" ||
-    value === "global"
+    value === "global" ||
+    value === "ask"
   ) {
     return value;
   }

ai_evals/adapters/frontend/core/app/appEvalRunner.ts

@@ -38,7 +38,6 @@ export interface AppEvalResult {
   toolCallCount: number;
   toolsUsed: string[];
   tokenUsage: TokenUsage;
-  finalContextTokens: number | null;
 }
 
 export interface AppEvalOptions {
@@ -114,7 +113,6 @@ export async function runAppEval(
       toolCallCount: rawResult.toolCallsCount,
       toolsUsed: rawResult.toolsCalled,
       tokenUsage: rawResult.tokenUsage,
-      finalContextTokens: rawResult.finalContextTokens,
     };
   } finally {
     await cleanup();

ai_evals/adapters/frontend/core/ask/askEvalRunner.ts

@@ -0,0 +1,175 @@
+import type { ChatCompletionMessageParam } from "openai/resources/chat/completions.mjs";
+import type { AIProvider } from "$lib/gen/types.gen";
+import {
+  getAskTools,
+  prepareAskSystemMessage,
+  prepareAskUserMessage,
+  type DocsToolVariant,
+} from "../../../../../frontend/src/lib/components/copilot/chat/ask/core";
+import type { ModeRunContext } from "../../../../core/types";
+import type { AskAnswerState } from "../../../../core/validators";
+import type { WindmillBackendSettings } from "../../../../core/windmillBackendSettings";
+import { WindmillBackendClient } from "../../windmillBackend";
+import { runEval } from "../shared";
+import type { TokenUsage, ToolCallDetail } from "../shared/types";
+
+export interface AskEvalResult {
+  success: boolean;
+  state: AskAnswerState;
+  error?: string;
+  assistantMessageCount: number;
+  toolCallCount: number;
+  toolsUsed: string[];
+  toolCallDetails: ToolCallDetail[];
+  tokenUsage: TokenUsage;
+}
+
+export interface AskEvalOptions {
+  variant: DocsToolVariant;
+  model?: string;
+  maxIterations?: number;
+  provider?: AIProvider;
+  backend: WindmillBackendSettings;
+  runContext?: ModeRunContext;
+}
+
+const DOCS_TOOL_ENV = "WMILL_AI_EVAL_DOCS_TOOL";
+
+/**
+ * Resolves which docs-tool arm to benchmark. Defaults to the new llms.txt arm.
+ */
+export function resolveDocsToolVariant(
+  value: string | undefined = process.env[DOCS_TOOL_ENV],
+): DocsToolVariant {
+  return value === "inkeep" ? "inkeep" : "llmstxt";
+}
+
+export async function runAskEval(
+  userPrompt: string,
+  apiKey: string,
+  options: AskEvalOptions,
+): Promise<AskEvalResult> {
+  // The production inkeep tool calls fetch('/api/inkeep') with a relative URL,
+  // which has no origin under node. Install a process-wide shim that rewrites
+  // /api/* calls to the eval backend with an auth token. Only needed for the
+  // inkeep arm; the llms.txt arm fetches absolute windmill.dev URLs natively.
+  if (options.variant === "inkeep") {
+    await installInkeepFetchShim(options.backend);
+  }
+
+  const model = options.model ?? "claude-haiku-4-5-20251001";
+
+  const rawResult = await runEval({
+    userPrompt,
+    systemMessage: prepareAskSystemMessage(undefined, options.variant),
+    userMessage: prepareAskUserMessage(userPrompt),
+    tools: getAskTools(options.variant),
+    helpers: {},
+    apiKey,
+    getOutput: () => ({
+      answer: "",
+      docsTool: options.variant,
+      toolsUsed: [],
+      toolCallCount: 0,
+    }),
+    onAssistantMessageStart: options.runContext?.onAssistantMessageStart,
+    onAssistantToken: options.runContext?.onAssistantChunk,
+    onAssistantMessageEnd: options.runContext?.onAssistantMessageEnd,
+    onToolCall: options.runContext?.onToolCall,
+    options: {
+      maxIterations: options.maxIterations,
+      model,
+      provider: options.provider,
+      backend: options.backend,
+      caseId: options.runContext?.caseId,
+      attempt: options.runContext?.attempt,
+    },
+  });
+
+  const answer = extractFinalAnswer(rawResult.messages);
+
+  return {
+    success: rawResult.success,
+    state: {
+      answer,
+      docsTool: options.variant,
+      toolsUsed: rawResult.toolsCalled,
+      toolCallCount: rawResult.toolCallsCount,
+    },
+    error: rawResult.error,
+    assistantMessageCount: rawResult.iterations,
+    toolCallCount: rawResult.toolCallsCount,
+    toolsUsed: rawResult.toolsCalled,
+    toolCallDetails: rawResult.toolCallDetails,
+    tokenUsage: rawResult.tokenUsage,
+  };
+}
+
+/**
+ * The final answer is the last assistant message with non-empty string content.
+ */
+export function extractFinalAnswer(
+  messages: ChatCompletionMessageParam[],
+): string {
+  for (let i = messages.length - 1; i >= 0; i--) {
+    const message = messages[i];
+    if (message.role !== "assistant") {
+      continue;
+    }
+    const content = message.content;
+    if (typeof content === "string" && content.trim().length > 0) {
+      return content;
+    }
+    if (Array.isArray(content)) {
+      const text = content
+        .map((part) =>
+          part && typeof part === "object" && "text" in part
+            ? String((part as { text?: unknown }).text ?? "")
+            : "",
+        )
+        .join("")
+        .trim();
+      if (text.length > 0) {
+        return text;
+      }
+    }
+  }
+  return "";
+}
+
+let inkeepFetchShimInstalled = false;
+
+/**
+ * Installs (once) a process-wide fetch wrapper that rewrites relative /api/*
+ * requests to the eval backend, adding a bearer token. All other URLs are
+ * passed through untouched.
+ */
+async function installInkeepFetchShim(
+  backend: WindmillBackendSettings,
+): Promise<void> {
+  if (inkeepFetchShimInstalled) {
+    return;
+  }
+  inkeepFetchShimInstalled = true;
+
+  const client = new WindmillBackendClient(backend);
+  const originalFetch = globalThis.fetch.bind(globalThis);
+
+  globalThis.fetch = (async (
+    input: RequestInfo | URL,
+    init?: RequestInit,
+  ): Promise<Response> => {
+    if (typeof input === "string" && input.startsWith("/api/")) {
+      const token = await client.getToken();
+      const url = `${backend.baseUrl}${input}`;
+      return await originalFetch(url, {
+        ...init,
+        headers: {
+          ...(init?.headers ?? {}),
+          Authorization: `Bearer ${token}`,
+        },
+      });
+    }
+    return await originalFetch(input as any, init);
+  }) as typeof fetch;
+}

ai_evals/adapters/frontend/core/flow/flowEvalRunner.ts

@@ -39,7 +39,6 @@ export interface FlowEvalResult {
   toolsUsed: string[];
   toolCallDetails: ToolCallDetail[];
   tokenUsage: TokenUsage;
-  finalContextTokens: number | null;
 }
 
 export interface FlowEvalOptions {
@@ -114,7 +113,6 @@ export async function runFlowEval(
       toolsUsed: rawResult.toolsCalled,
       toolCallDetails: rawResult.toolCallDetails,
       tokenUsage: rawResult.tokenUsage,
-      finalContextTokens: rawResult.finalContextTokens,
     };
   } finally {
     await cleanup();

ai_evals/adapters/frontend/core/global/globalEvalRunner.ts

@@ -9,7 +9,6 @@ import {
 } from "../../../../../frontend/src/lib/components/copilot/chat/global/core";
 import {
   clearGlobalDrafts,
-  getGlobalDraft,
   listGlobalDrafts,
 } from "../../../../../frontend/src/lib/components/copilot/chat/global/userDraftAdapter";
 import type { Tool as ProductionTool } from "../../../../../frontend/src/lib/components/copilot/chat/shared";
@@ -19,7 +18,6 @@ import type { GlobalDraftState } from "../../../../core/validators";
 import type { WindmillBackendSettings } from "../../../../core/windmillBackendSettings";
 import {
   registerBenchmarkWorkspaceRunnables,
-  seedBenchmarkDraft,
   unregisterBenchmarkWorkspaceRunnables,
   type BenchmarkWorkspaceRunnables,
 } from "../../mockBackend";
@@ -32,10 +30,6 @@ const MUTATING_GLOBAL_TOOLS = new Set([
 ]);
 const DISABLE_ACTIVE_EDITOR_CONTEXT_ENV =
   "WMILL_AI_EVAL_DISABLE_ACTIVE_EDITOR_CONTEXT";
-// A/B gate for the search_app read tool: set to "1" to run the baseline arm
-// (toolset without search_app) so its token cost can be compared against the arm
-// that offers it.
-const DISABLE_SEARCH_APP_ENV = "WMILL_AI_EVAL_DISABLE_SEARCH_APP";
 
 const LIVE_EDITOR_ITEM_KINDS = {
   script: "script",
@@ -50,20 +44,6 @@ export interface GlobalLiveEditorDraftFixture {
   value?: unknown;
 }
 
-// Identity the global system prompt builds paths from. Production reads
-// `userStore` (whoami) to fill `u/{username}/...`; the eval harness never logs
-// in, so without this the prompt sees an empty username (`u//...`) and no
-// path-selection case is meaningful. Seeded per-case via the initial fixture and
-// passed straight to `prepareGlobalSystemMessage` (no global-store mutation).
-export interface GlobalUserFixture {
-  username: string;
-  is_admin?: boolean;
-  /** Folders the user can write to (the writable set whoami returns). */
-  folders?: string[];
-  /** Folders the user can read; read-only folders = folders_read \ folders. */
-  folders_read?: string[];
-}
-
 export interface GlobalEvalResult {
   success: boolean;
   state: GlobalDraftState;
@@ -73,13 +53,11 @@ export interface GlobalEvalResult {
   toolsUsed: string[];
   toolCallDetails: ToolCallDetail[];
   tokenUsage: TokenUsage;
-  finalContextTokens: number | null;
 }
 
 export interface GlobalEvalOptions {
   workspaceFixtures?: BenchmarkWorkspaceRunnables;
   liveEditorDrafts?: GlobalLiveEditorDraftFixture[];
-  user?: GlobalUserFixture;
   model?: string;
   maxIterations?: number;
   provider?: AIProvider;
@@ -105,11 +83,9 @@ export async function runGlobalEval(
     const model = options.model ?? "claude-haiku-4-5-20251001";
     const injectActiveEditorContext =
       process.env[DISABLE_ACTIVE_EDITOR_CONTEXT_ENV] !== "1";
-    // Pass the seeded identity straight to the prompt builder rather than mutating
-    // the process-global `userStore`, so concurrent cases never race on it.
     const rawResult = await runEval({
       userPrompt,
-      systemMessage: prepareGlobalSystemMessage(undefined, { user: options.user }),
+      systemMessage: prepareGlobalSystemMessage(),
       userMessage: prepareGlobalUserMessage(
         userPrompt,
         [],
@@ -118,7 +94,7 @@ export async function runGlobalEval(
       tools: getGlobalEvalTools(),
       helpers: {},
       apiKey,
-      getOutput: () => collectGlobalDraftState(workspaceRoot),
+      getOutput: () => ({ drafts: listGlobalDrafts(workspaceRoot) }),
       onAssistantMessageStart: options.runContext?.onAssistantMessageStart,
       onAssistantToken: options.runContext?.onAssistantChunk,
       onAssistantMessageEnd: options.runContext?.onAssistantMessageEnd,
@@ -143,7 +119,6 @@ export async function runGlobalEval(
       toolsUsed: rawResult.toolsCalled,
       toolCallDetails: rawResult.toolCallDetails,
       tokenUsage: rawResult.tokenUsage,
-      finalContextTokens: rawResult.finalContextTokens,
     };
   } finally {
     clearGlobalDrafts(workspaceRoot);
@@ -155,32 +130,6 @@ export async function runGlobalEval(
   }
 }
 
-// Build the harness output from the DB-backed drafts. `listGlobalDrafts` returns
-// metadata-only rows for backend drafts (the model's `write_script` etc. persist
-// straight to the backend with no in-tab editor cell), so re-read each such row
-// with `getGlobalDraft` to attach the full value the validators assert on. A row
-// that already carries a value (the production in-tab cell overlay) is kept as-is.
-async function collectGlobalDraftState(
-  workspace: string,
-): Promise<GlobalDraftState> {
-  const items = await listGlobalDrafts(workspace);
-  const drafts = await Promise.all(
-    items.map(async (item) => {
-      if (item.value !== undefined) {
-        return item;
-      }
-      const full = await getGlobalDraft(
-        workspace,
-        item.type,
-        item.path,
-        item.triggerKind,
-      );
-      return full ?? item;
-    }),
-  );
-  return { drafts: drafts as GlobalDraftState["drafts"] };
-}
-
 function seedLiveEditorDrafts(
   workspace: string,
   fixtures: GlobalLiveEditorDraftFixture[],
@@ -189,9 +138,7 @@ function seedLiveEditorDrafts(
     const itemKind = LIVE_EDITOR_ITEM_KINDS[fixture.type];
     const storagePath = fixture.storagePath ?? fixture.effectivePath ?? "";
     if (fixture.value !== undefined) {
-      // Seed as a backend draft row, not an in-tab cell: a cell would shadow the
-      // model's DB-backed edit when the output is read back via listGlobalDrafts.
-      seedBenchmarkDraft(workspace, itemKind, storagePath, fixture.value);
+      UserDraft.save(itemKind, storagePath, fixture.value, { workspace });
     }
     UserDraft.setLiveEditorDraft({
       workspace,
@@ -214,28 +161,25 @@ function clearLiveEditorDrafts(
 }
 
 function getGlobalEvalTools(): ProductionTool<{}>[] {
-  const disableSearchApp = process.env[DISABLE_SEARCH_APP_ENV] === "1";
-  return (globalTools as ProductionTool<{}>[])
-    .filter((tool) => !(disableSearchApp && tool.def.function.name === "search_app"))
-    .map((tool) => {
-      if (!MUTATING_GLOBAL_TOOLS.has(tool.def.function.name)) {
-        return tool;
-      }
+  return (globalTools as ProductionTool<{}>[]).map((tool) => {
+    if (!MUTATING_GLOBAL_TOOLS.has(tool.def.function.name)) {
+      return tool;
+    }
 
-      return {
-        ...tool,
-        requiresConfirmation: false,
-        validateBeforeConfirmation: undefined,
-        fn: async () =>
-          JSON.stringify(
-            {
-              success: false,
-              error:
-                "This mutating workspace tool is disabled during ai_evals global mode.",
-            },
-            null,
-            2,
-          ),
-      };
-    });
+    return {
+      ...tool,
+      requiresConfirmation: false,
+      validateBeforeConfirmation: undefined,
+      fn: async () =>
+        JSON.stringify(
+          {
+            success: false,
+            error:
+              "This mutating workspace tool is disabled during ai_evals global mode.",
+          },
+          null,
+          2,
+        ),
+    };
+  });
 }

ai_evals/adapters/frontend/core/script/scriptEvalRunner.ts

@@ -25,7 +25,6 @@ export interface ScriptEvalResult {
   toolsUsed: string[];
   toolCallDetails: ToolCallDetail[];
   tokenUsage: TokenUsage;
-  finalContextTokens: number | null;
 }
 
 export interface ScriptEvalOptions {
@@ -112,7 +111,6 @@ export async function runScriptEval(
       toolsUsed: rawResult.toolsCalled,
       toolCallDetails: rawResult.toolCallDetails,
       tokenUsage: rawResult.tokenUsage,
-      finalContextTokens: rawResult.finalContextTokens,
     };
   } finally {
     await cleanup();

ai_evals/adapters/frontend/core/shared/baseEvalRunner.ts

@@ -38,9 +38,8 @@ export interface RunEvalParams<THelpers, TOutput> {
   helpers: THelpers;
   /** API key for the provider */
   apiKey: string;
-  /** Function to get the current output state. May be async — global mode reads
-   * DB-backed drafts back through the (mocked) backend to build its output. */
-  getOutput: () => TOutput | Promise<TOutput>;
+  /** Function to get the current output state */
+  getOutput: () => TOutput;
   /** Model and Windmill backend configuration */
   options: EvalRunnerOptions;
   onAssistantMessageStart?: () => void;
@@ -155,10 +154,9 @@ export async function runEval<THelpers, TOutput>(
       if (result.hitMaxIterations) {
         return {
           success: false,
-          output: (await getOutput()) as TOutput,
+          output: getOutput(),
           error: `Reached max turns (${maxIterations})`,
           tokenUsage: result.tokenUsage,
-          finalContextTokens: result.lastIterationUsage?.prompt ?? null,
           toolCallsCount,
           toolsCalled,
           toolCallDetails,
@@ -172,9 +170,8 @@ export async function runEval<THelpers, TOutput>(
 
       return {
         success: true,
-        output: (await getOutput()) as TOutput,
+        output: getOutput(),
         tokenUsage: result.tokenUsage,
-        finalContextTokens: result.lastIterationUsage?.prompt ?? null,
         toolCallsCount,
         toolsCalled,
         toolCallDetails,
@@ -194,10 +191,9 @@ export async function runEval<THelpers, TOutput>(
 
       return {
         success: false,
-        output: (await getOutput()) as TOutput,
+        output: getOutput(),
         error: errorMessage,
         tokenUsage: { prompt: 0, completion: 0, total: 0 },
-        finalContextTokens: null,
         toolCallsCount,
         toolsCalled,
         toolCallDetails,

ai_evals/adapters/frontend/core/shared/types.ts

@@ -28,8 +28,6 @@ export interface RawEvalResult<TOutput> {
   output: TOutput;
   error?: string;
   tokenUsage: TokenUsage;
-  /** Input tokens on the last model request of the loop (see BenchmarkAttemptResult.finalContextTokens). */
-  finalContextTokens: number | null;
   toolCallsCount: number;
   toolsCalled: string[];
   toolCallDetails: ToolCallDetail[];

ai_evals/adapters/frontend/mockBackend.ts

@@ -1,20 +1,9 @@
 import { randomUUID } from 'node:crypto'
-import type {
-	AppWithLastVersion,
-	CompletedJob,
-	Flow,
-	Job,
-	ListableApp,
-	Script
-} from '../../../frontend/src/lib/gen'
+import type { CompletedJob, Flow, Job, Script } from '../../../frontend/src/lib/gen'
 import type {
 	DataTableTables,
 	DataTableTableSchema,
-	GetDraftForUserResponse,
-	ListDraftsResponse,
-	ScriptLang,
-	UpdateDraftResponse,
-	UserDraftItemKind
+	ScriptLang
 } from '../../../frontend/src/lib/gen/types.gen'
 import { buildScriptLintResult } from './core/script/preview'
 import { applyDatatableSql, type BenchmarkDatatableSeed } from './datatableSqlEngine'
@@ -40,18 +29,6 @@ export interface BenchmarkWorkspaceFlow {
 	value: Flow['value']
 }
 
-export interface BenchmarkWorkspaceApp {
-	path: string
-	summary: string
-	value: {
-		files: Record<string, string>
-		runnables: Record<string, unknown>
-		data?: unknown
-		policy?: unknown
-		custom_path?: unknown
-	}
-}
-
 export interface BenchmarkWorkspaceJob {
 	/** Stable id so a case prompt can reference a specific run (e.g. for get_job_logs). */
 	id?: string
@@ -66,7 +43,6 @@ export interface BenchmarkWorkspaceJob {
 export interface BenchmarkWorkspaceRunnables {
 	scripts?: BenchmarkWorkspaceScript[]
 	flows?: BenchmarkWorkspaceFlow[]
-	apps?: BenchmarkWorkspaceApp[]
 	datatables?: BenchmarkDatatableSeed[]
 	jobs?: BenchmarkWorkspaceJob[]
 }
@@ -87,7 +63,6 @@ export function resetBenchmarkMockBackend(): void {
 	benchmarkWorkspaces.clear()
 	benchmarkWorkspaceRunnables.clear()
 	benchmarkJobs.clear()
-	benchmarkDrafts.clear()
 }
 
 export function registerBenchmarkWorkspace(workspace: string): void {
@@ -99,8 +74,6 @@ export function registerBenchmarkWorkspaceRunnables(
 	runnables: BenchmarkWorkspaceRunnables
 ): void {
 	benchmarkWorkspaces.add(workspace)
-	// Fresh case: drop any drafts left from a prior run on this workspace id.
-	clearBenchmarkDrafts(workspace)
 	// Datatables are mutated in place by exec_datatable_sql (a write must be visible
 	// to later reads), so store an isolated deep copy — never mutate the caller's seed.
 	benchmarkWorkspaceRunnables.set(workspace, {
@@ -125,7 +98,6 @@ export function registerBenchmarkWorkspaceRunnables(
 export function unregisterBenchmarkWorkspace(workspace: string): void {
 	benchmarkWorkspaces.delete(workspace)
 	benchmarkWorkspaceRunnables.delete(workspace)
-	clearBenchmarkDrafts(workspace)
 	for (const [jobId, entry] of benchmarkJobs.entries()) {
 		if (entry.workspace === workspace) {
 			benchmarkJobs.delete(jobId)
@@ -181,22 +153,6 @@ export function getBenchmarkFlowByPath(workspace: string, path: string): Flow |
 	return flow ? buildBenchmarkFlow(flow) : null
 }
 
-export function listBenchmarkApps(workspace: string): ListableApp[] | null {
-	const runnables = benchmarkWorkspaceRunnables.get(workspace)
-	if (!runnables) {
-		return null
-	}
-	return (runnables.apps ?? []).map(buildBenchmarkListableApp)
-}
-
-export function getBenchmarkAppByPath(workspace: string, path: string): AppWithLastVersion | null {
-	const app = benchmarkWorkspaceRunnables
-		.get(workspace)
-		?.apps?.find((entry) => entry.path === path)
-
-	return app ? buildBenchmarkApp(app) : null
-}
-
 export function createBenchmarkCompletedJob(input: {
 	workspace: string
 	jobKind: CompletedJob['job_kind']
@@ -282,110 +238,6 @@ export function getBenchmarkJobLogs(workspace: string, jobId: string): string {
 	return job.logs ?? ''
 }
 
-// ============= Drafts (per-user, DB-backed in production) =============
-
-/**
- * In-memory stand-in for the per-user draft backend (`DraftService`). The global
- * AI chat now persists and reads drafts through the backend DB instead of an
- * in-tab `UserDraft` cell, so the eval mocks the three draft endpoints it
- * exercises (`updateDraft` / `getDraftForUser` / `listDrafts`) and keeps the
- * saved values here, keyed by workspace + draft kind + storage path. Mirrors the
- * semantics of the production unit test's mock in
- * `frontend/src/lib/components/copilot/chat/global/core.test.ts`.
- */
-const benchmarkDrafts = new Map<
-	string,
-	{ workspace: string; kind: UserDraftItemKind; path: string; value: unknown }
->()
-
-// Fixed timestamp so artifacts stay deterministic. No eval simulates a
-// concurrent writer, so every save is accepted and the conflict branch is
-// never taken — the syncer just records this as its `last_sync` baseline.
-const BENCHMARK_DRAFT_TIMESTAMP = '1970-01-01T00:00:00.000Z'
-
-function benchmarkDraftKey(workspace: string, kind: string, path: string): string {
-	return `${workspace}::${kind}::${path}`
-}
-
-export function clearBenchmarkDrafts(workspace: string): void {
-	for (const [key, entry] of benchmarkDrafts.entries()) {
-		if (entry.workspace === workspace) {
-			benchmarkDrafts.delete(key)
-		}
-	}
-}
-
-/**
- * Seed a draft straight into the store — used by the eval's live-editor draft
- * fixtures, which model "the user already has this draft open/saved". Writing it
- * here (instead of through `UserDraft.save`) keeps it a backend draft row with no
- * shadowing in-tab cell, so a model edit that persists to the backend is what the
- * output read-back captures — not the stale seed.
- */
-export function seedBenchmarkDraft(
-	workspace: string,
-	kind: UserDraftItemKind,
-	path: string,
-	value: unknown
-): void {
-	benchmarkDrafts.set(benchmarkDraftKey(workspace, kind, path), {
-		workspace,
-		kind,
-		path,
-		value
-	})
-}
-
-/** Mirror `DraftService.updateDraft`: a `null`/omitted value deletes the row. */
-export function updateBenchmarkDraft(input: {
-	workspace: string
-	kind: UserDraftItemKind
-	path: string
-	requestBody?: { value?: unknown }
-}): UpdateDraftResponse {
-	const key = benchmarkDraftKey(input.workspace, input.kind, input.path)
-	const value = input.requestBody?.value
-	if (value == null) {
-		benchmarkDrafts.delete(key)
-	} else {
-		benchmarkDrafts.set(key, {
-			workspace: input.workspace,
-			kind: input.kind,
-			path: input.path,
-			value
-		})
-	}
-	return { status: 'saved', current_timestamp: BENCHMARK_DRAFT_TIMESTAMP }
-}
-
-/** Mirror `DraftService.getDraftForUser`: 404-shaped throw when absent so the
- * adapter's narrowed catch treats it as "no draft" instead of re-throwing. */
-export function getBenchmarkDraftForUser(input: {
-	workspace: string
-	kind: UserDraftItemKind
-	path: string
-}): GetDraftForUserResponse {
-	const entry = benchmarkDrafts.get(benchmarkDraftKey(input.workspace, input.kind, input.path))
-	if (!entry) {
-		throw Object.assign(new Error(`no draft for "${input.path}"`), { status: 404 })
-	}
-	return { value: entry.value, created_at: BENCHMARK_DRAFT_TIMESTAMP }
-}
-
-/** Mirror `DraftService.listDrafts`: metadata rows (no value) for a workspace. */
-export function listBenchmarkDrafts(workspace: string): ListDraftsResponse {
-	return [...benchmarkDrafts.values()]
-		.filter((entry) => entry.workspace === workspace)
-		.map((entry) => ({
-			kind: entry.kind,
-			path: entry.path,
-			summary: (entry.value as { summary?: string } | null)?.summary,
-			draft_only: true,
-			legacy_draft: false,
-			created_at: BENCHMARK_DRAFT_TIMESTAMP
-		}))
-}
-
 // ============= Datatables (best-effort in-memory SQL) =============
 
 /**
@@ -640,35 +492,3 @@ function buildBenchmarkFlow(flow: BenchmarkWorkspaceFlow): Flow {
 		extra_perms: {}
 	} as Flow
 }
-
-function buildBenchmarkListableApp(app: BenchmarkWorkspaceApp): ListableApp {
-	return {
-		id: 0,
-		workspace_id: 'benchmark',
-		path: app.path,
-		summary: app.summary,
-		version: 1,
-		extra_perms: {},
-		edited_at: BENCHMARK_TIMESTAMP,
-		execution_mode: 'viewer',
-		raw_app: true
-	}
-}
-
-function buildBenchmarkApp(app: BenchmarkWorkspaceApp): AppWithLastVersion {
-	return {
-		id: 0,
-		workspace_id: 'benchmark',
-		path: app.path,
-		summary: app.summary,
-		versions: [1],
-		created_by: 'benchmark',
-		created_at: BENCHMARK_TIMESTAMP,
-		value: app.value,
-		policy: (app.value.policy ?? {}) as AppWithLastVersion['policy'],
-		execution_mode: 'viewer',
-		extra_perms: {},
-		custom_path: app.value.custom_path as string | undefined,
-		raw_app: true
-	}
-}

ai_evals/adapters/frontend/mockBackendDrafts.test.ts

@@ -1,94 +0,0 @@
-import { afterEach, beforeEach, describe, expect, it } from 'bun:test'
-import {
-	clearBenchmarkDrafts,
-	getBenchmarkDraftForUser,
-	listBenchmarkDrafts,
-	resetBenchmarkMockBackend,
-	seedBenchmarkDraft,
-	updateBenchmarkDraft
-} from './mockBackend'
-
-const WORKSPACE = 'benchmark-drafts-ws'
-
-// Drives the in-memory stand-in for the per-user draft backend (`DraftService`)
-// that the global AI-chat eval round-trips its drafts through. Mirrors the
-// production-unit-test mock in
-// `frontend/src/lib/components/copilot/chat/global/core.test.ts`.
-describe('mockBackend drafts', () => {
-	beforeEach(() => resetBenchmarkMockBackend())
-	afterEach(() => resetBenchmarkMockBackend())
-
-	it('round-trips a saved draft through update / get / list', () => {
-		const value = { summary: 'Greet a user', content: 'export async function main() {}' }
-		const res = updateBenchmarkDraft({
-			workspace: WORKSPACE,
-			kind: 'script',
-			path: 'f/evals/greet',
-			requestBody: { value }
-		})
-		expect(res.status).toBe('saved')
-
-		expect(getBenchmarkDraftForUser({ workspace: WORKSPACE, kind: 'script', path: 'f/evals/greet' }).value).toEqual(
-			value
-		)
-
-		const rows = listBenchmarkDrafts(WORKSPACE)
-		expect(rows).toHaveLength(1)
-		expect(rows[0]).toMatchObject({ kind: 'script', path: 'f/evals/greet', summary: 'Greet a user', draft_only: true })
-	})
-
-	it('treats a null value as a delete', () => {
-		updateBenchmarkDraft({
-			workspace: WORKSPACE,
-			kind: 'variable',
-			path: 'f/evals/token',
-			requestBody: { value: { summary: 'token' } }
-		})
-		updateBenchmarkDraft({
-			workspace: WORKSPACE,
-			kind: 'variable',
-			path: 'f/evals/token',
-			requestBody: { value: null }
-		})
-
-		expect(listBenchmarkDrafts(WORKSPACE)).toHaveLength(0)
-		expect(() => getBenchmarkDraftForUser({ workspace: WORKSPACE, kind: 'variable', path: 'f/evals/token' })).toThrow()
-	})
-
-	it('throws a 404-shaped error when no draft exists', () => {
-		try {
-			getBenchmarkDraftForUser({ workspace: WORKSPACE, kind: 'script', path: 'f/evals/missing' })
-			throw new Error('expected a throw')
-		} catch (e) {
-			expect((e as { status?: number }).status).toBe(404)
-		}
-	})
-
-	it('seeds a draft as a backend row that a later edit overwrites', () => {
-		seedBenchmarkDraft(WORKSPACE, 'script', 'f/evals/current', { content: 'seed' })
-		expect(getBenchmarkDraftForUser({ workspace: WORKSPACE, kind: 'script', path: 'f/evals/current' }).value).toEqual({
-			content: 'seed'
-		})
-
-		// A model edit persists the same path and must win over the seed.
-		updateBenchmarkDraft({
-			workspace: WORKSPACE,
-			kind: 'script',
-			path: 'f/evals/current',
-			requestBody: { value: { content: 'edited' } }
-		})
-		expect(getBenchmarkDraftForUser({ workspace: WORKSPACE, kind: 'script', path: 'f/evals/current' }).value).toEqual({
-			content: 'edited'
-		})
-	})
-
-	it('clears only the targeted workspace', () => {
-		seedBenchmarkDraft(WORKSPACE, 'script', 'f/a', { content: 'a' })
-		seedBenchmarkDraft('other-ws', 'script', 'f/b', { content: 'b' })
-
-		clearBenchmarkDrafts(WORKSPACE)
-
-		expect(listBenchmarkDrafts(WORKSPACE)).toHaveLength(0)
-		expect(listBenchmarkDrafts('other-ws')).toHaveLength(1)
-	})
-})

ai_evals/adapters/frontend/progress.ts

@@ -1,4 +1,4 @@
-export type FrontendBenchmarkProgressSurface = 'flow' | 'app' | 'script' | 'global'
+export type FrontendBenchmarkProgressSurface = 'flow' | 'app' | 'script' | 'global' | 'ask'
 
 export type FrontendBenchmarkProgressEvent =
 	| {

ai_evals/adapters/frontend/runtime.ts

@@ -16,7 +16,7 @@ const FRONTEND_BENCHMARK_TEST =
 const FRONTEND_BENCHMARK_CONFIG =
   "../ai_evals/adapters/frontend/vitest.config.ts";
 
-export type FrontendMode = "flow" | "app" | "script" | "global";
+export type FrontendMode = "flow" | "app" | "script" | "global" | "ask";
 
 export async function runFrontendBenchmarkAdapter(input: {
   mode: FrontendMode;
@@ -24,9 +24,8 @@ export async function runFrontendBenchmarkAdapter(input: {
   runs: number;
   model?: string;
   verbose?: boolean;
-  skipJudge?: boolean;
-  executionOnly?: boolean;
   backendValidation?: string;
+  docsTool?: string;
 }): Promise<BenchmarkRunResult> {
   const tempDir = await mkdtemp(
     path.join(tmpdir(), "wmill-frontend-benchmark-"),
@@ -42,10 +41,9 @@ export async function runFrontendBenchmarkAdapter(input: {
     WMILL_FRONTEND_AI_EVAL_MODEL: input.model ?? "",
     WMILL_FRONTEND_AI_EVAL_PROGRESS: "1",
     WMILL_FRONTEND_AI_EVAL_VERBOSE: input.verbose ? "1" : "0",
-    WMILL_FRONTEND_AI_EVAL_SKIP_JUDGE:
-      input.skipJudge || input.executionOnly ? "1" : "0",
-    WMILL_FRONTEND_AI_EVAL_EXECUTION_ONLY: input.executionOnly ? "1" : "0",
     WMILL_FRONTEND_AI_EVAL_BACKEND_VALIDATION: input.backendValidation ?? "",
+    // Only meaningful for ask mode; selects the docs-tool arm to benchmark.
+    WMILL_AI_EVAL_DOCS_TOOL: input.docsTool ?? process.env.WMILL_AI_EVAL_DOCS_TOOL ?? "",
   };
 
   try {

ai_evals/adapters/frontend/vitestAdapter.test.ts

@@ -33,19 +33,15 @@ vi.mock('$lib/components/vscode', () => ({}))
 vi.mock('$lib/gen', async () => {
 	const actual = await vi.importActual<any>('$lib/gen')
 	const {
-		getBenchmarkAppByPath,
 		getBenchmarkCompletedJob,
 		getBenchmarkCompletedJobResultMaybe,
 		getBenchmarkDatatableSchema,
-		getBenchmarkDraftForUser,
 		getBenchmarkFlowByPath,
 		getBenchmarkJobLogs,
 		getBenchmarkScriptByHash,
 		getBenchmarkScriptByPath,
 		hasBenchmarkWorkspace,
-		listBenchmarkApps,
 		listBenchmarkDatatables,
-		listBenchmarkDrafts,
 		listBenchmarkFlows,
 		listBenchmarkJobs,
 		listBenchmarkScripts,
@@ -54,8 +50,7 @@ vi.mock('$lib/gen', async () => {
 		previewBenchmarkSchedule,
 		runBenchmarkDatatableSql,
 		runBenchmarkFlowByPath,
-		runBenchmarkScriptPreview,
-		updateBenchmarkDraft
+		runBenchmarkScriptPreview
 	} = await import('./mockBackend')
 
 	function wrapService<T extends object>(target: T, overrides: Record<string, unknown>): T {
@@ -71,25 +66,6 @@ vi.mock('$lib/gen', async () => {
 
 	return {
 		...actual,
-		DraftService: wrapService(actual.DraftService, {
-			updateDraft: async (data: {
-				workspace: string
-				kind: any
-				path: string
-				requestBody?: { value?: unknown }
-			}) =>
-				hasBenchmarkWorkspace(data.workspace)
-					? updateBenchmarkDraft(data)
-					: actual.DraftService.updateDraft(data),
-			getDraftForUser: async (data: { workspace: string; kind: any; path: string }) =>
-				hasBenchmarkWorkspace(data.workspace)
-					? getBenchmarkDraftForUser(data)
-					: actual.DraftService.getDraftForUser(data),
-			listDrafts: async (data: { workspace: string }) =>
-				hasBenchmarkWorkspace(data.workspace)
-					? listBenchmarkDrafts(data.workspace)
-					: actual.DraftService.listDrafts(data)
-		}),
 		ScriptService: wrapService(actual.ScriptService, {
 			listScripts: async (data: { workspace: string }) =>
 				hasBenchmarkWorkspace(data.workspace)
@@ -301,20 +277,12 @@ vi.mock('$lib/gen', async () => {
 		}),
 		AppService: wrapService(actual.AppService, {
 			existsApp: async (data: { workspace: string; path: string }) =>
-				hasBenchmarkWorkspace(data.workspace)
-					? Boolean(getBenchmarkAppByPath(data.workspace, data.path))
-					: actual.AppService.existsApp(data),
+				hasBenchmarkWorkspace(data.workspace) ? false : actual.AppService.existsApp(data),
 			listApps: async (data: { workspace: string }) =>
-				hasBenchmarkWorkspace(data.workspace)
-					? (listBenchmarkApps(data.workspace) ?? [])
-					: actual.AppService.listApps(data),
+				hasBenchmarkWorkspace(data.workspace) ? [] : actual.AppService.listApps(data),
 			getAppByPath: async (data: { workspace: string; path: string }) => {
 				if (hasBenchmarkWorkspace(data.workspace)) {
-					const app = getBenchmarkAppByPath(data.workspace, data.path)
-					if (!app) {
-						throw new Error(`App "${data.path}" not found in benchmark workspace`)
-					}
-					return app
+					throw new Error(`App "${data.path}" not found in benchmark workspace`)
 				}
 				return actual.AppService.getAppByPath(data)
 			}

ai_evals/cases/ask.yaml

@@ -0,0 +1,419 @@
+# AI-chat documentation Q&A cases. Each case asks a realistic user question and
+# is graded on answer quality (judge) plus a deterministic URL-citation check.
+# Ground-truth URLs are derived from https://www.windmill.dev/llms.txt — all
+# `answerIncludesAny` URLs are real docs pages (without the `.md` suffix).
+
+# --- Tier 1: direct lookups -------------------------------------------------
+
+- id: ask-lookup-cron-schedule
+  prompt: How do I run a script automatically every Monday at 9am?
+  runtime:
+    maxTurns: 10
+  judgeChecklist:
+    - explains that recurring runs use a schedule with a CRON expression
+    - mentions a concrete cron-like expression or the schedule configuration
+    - the answer is grounded in the Windmill scheduling documentation
+  validate:
+    answerIncludesAny:
+      - ["windmill.dev/docs/core_concepts/scheduling"]
+
+- id: ask-lookup-secret-in-python
+  prompt: How do I store a secret like an API key and use it inside a Python script?
+  runtime:
+    maxTurns: 10
+  judgeChecklist:
+    - explains storing the value as a secret variable
+    - explains accessing it from a Python script (e.g. wmill.get_variable or get_resource)
+    - the answer is grounded in the variables and secrets documentation
+  validate:
+    answerIncludesAny:
+      - ["windmill.dev/docs/core_concepts/variables_and_secrets"]
+
+- id: ask-lookup-webhook-trigger
+  prompt: How can I trigger one of my scripts from an external service by calling a URL?
+  runtime:
+    maxTurns: 10
+  judgeChecklist:
+    - explains that scripts expose webhook URLs
+    - mentions sync vs async webhook behavior or how to find the webhook URL
+    - the answer is grounded in the webhooks documentation
+  validate:
+    answerIncludesAny:
+      - ["windmill.dev/docs/core_concepts/webhooks"]
+
+- id: ask-lookup-retry-step
+  prompt: One step in my flow calls a flaky API. How do I make it automatically retry if it fails?
+  runtime:
+    maxTurns: 10
+  judgeChecklist:
+    - explains configuring retries on a flow step
+    - mentions the number of attempts and/or delay between attempts
+    - the answer is grounded in the error handling or flow editor documentation
+  validate:
+    answerIncludesAny:
+      - [
+          "windmill.dev/docs/flows/retries",
+          "windmill.dev/docs/core_concepts/error_handling",
+          "windmill.dev/docs/flows/flow_editor",
+        ]
+
+- id: ask-lookup-connect-database
+  prompt: How do I connect to my Postgres database so my scripts can query it?
+  runtime:
+    maxTurns: 10
+  judgeChecklist:
+    - explains creating a resource holding the database connection details
+    - explains referencing that resource from a script
+    - the answer is grounded in the resources documentation
+  validate:
+    answerIncludesAny:
+      - [
+          "windmill.dev/docs/core_concepts/resources_and_types",
+          "windmill.dev/docs/getting_started/scripts_quickstart/sql",
+        ]
+
+- id: ask-lookup-python-deps
+  prompt: How do I add a third-party pip package to a Python script in Windmill?
+  runtime:
+    maxTurns: 10
+  judgeChecklist:
+    - explains that imports are auto-detected and dependencies resolved automatically
+    - mentions how to pin a version or the requirements mechanism
+    - the answer is grounded in the Python scripting documentation
+  validate:
+    answerIncludesAny:
+      - [
+          "windmill.dev/docs/getting_started/scripts_quickstart/python",
+          "windmill.dev/docs/advanced/dependencies_in_python",
+        ]
+
+- id: ask-lookup-concurrency-limit
+  prompt: How do I stop a script from running too many times at once so I don't hit an API rate limit?
+  runtime:
+    maxTurns: 10
+  judgeChecklist:
+    - explains setting a concurrency limit on the script
+    - mentions the max number of concurrent executions and/or time window
+    - the answer is grounded in the concurrency limits documentation
+  validate:
+    answerIncludesAny:
+      - [
+          "windmill.dev/docs/core_concepts/concurrency_limits",
+          "windmill.dev/docs/script_editor/concurrency_limit",
+        ]
+
+- id: ask-lookup-app-table-button
+  prompt: How do I build a simple internal tool with a table and a button using Windmill?
+  runtime:
+    maxTurns: 10
+  judgeChecklist:
+    - explains using the app editor with drag-and-drop components
+    - mentions wiring a component to a backend script/runnable
+    - the answer is grounded in the app editor or apps quickstart documentation
+  validate:
+    answerIncludesAny:
+      - [
+          "windmill.dev/docs/apps/app_editor",
+          "windmill.dev/docs/getting_started/apps_quickstart",
+        ]
+
+# --- Tier 2: conceptual / vocabulary-mismatch -------------------------------
+
+- id: ask-concept-flow-wait-for-approval
+  prompt: How can I make a flow pause and wait for a person to approve before it continues?
+  runtime:
+    maxTurns: 10
+  judgeChecklist:
+    - identifies this as a suspend / approval step in a flow
+    - explains that the flow resumes once approved (e.g. via a resume link or form)
+    - the answer is grounded in the flow editor / approval documentation
+  validate:
+    answerIncludesAny:
+      - [
+          "windmill.dev/docs/flows/flow_editor",
+          "windmill.dev/docs/flows/flow_approval",
+        ]
+
+- id: ask-concept-remember-value-between-runs
+  prompt: Can a script remember a value from its previous run, like the last timestamp it processed?
+  runtime:
+    maxTurns: 10
+  judgeChecklist:
+    - identifies states as the mechanism (getState / setState)
+    - explains state persists between runs of the same script
+    - the answer is grounded in the states / within-windmill documentation
+  validate:
+    answerIncludesAny:
+      - ["windmill.dev/docs/core_concepts/persistent_storage/within_windmill"]
+
+- id: ask-concept-steps-at-same-time
+  prompt: How do I make several steps in my flow run at the same time instead of one after another?
+  runtime:
+    maxTurns: 10
+  judgeChecklist:
+    - identifies running branches in parallel (branchall / parallel branches)
+    - explains the steps execute concurrently
+    - the answer is grounded in the flow editor documentation
+  validate:
+    answerIncludesAny:
+      - [
+          "windmill.dev/docs/flows/flow_editor",
+          "windmill.dev/docs/flows/flow_branches",
+        ]
+
+- id: ask-concept-react-to-db-changes
+  prompt: I want something to happen automatically whenever a new row is inserted into my Postgres table. Is that possible?
+  runtime:
+    maxTurns: 10
+  judgeChecklist:
+    - identifies Postgres triggers reacting to insert/update/delete
+    - mentions it listens to database change events (logical replication)
+    - the answer is grounded in the Postgres triggers documentation
+  validate:
+    answerIncludesAny:
+      - [
+          "windmill.dev/docs/core_concepts/postgres_triggers",
+          "windmill.dev/docs/getting_started/triggers",
+        ]
+
+- id: ask-concept-reuse-config-everywhere
+  prompt: I keep pasting the same base URL and credentials into many scripts. Is there a cleaner way to manage that shared config?
+  runtime:
+    maxTurns: 10
+  judgeChecklist:
+    - identifies variables (and/or resources) as the mechanism for reusable config
+    - explains that the config is defined once and referenced from scripts
+    - the answer is grounded in the variables / resources documentation
+  validate:
+    answerIncludesAny:
+      - [
+          "windmill.dev/docs/core_concepts/variables_and_secrets",
+          "windmill.dev/docs/core_concepts/resources_and_types",
+        ]
+
+- id: ask-concept-let-llm-call-my-scripts
+  prompt: I want my AI assistant in Claude or Cursor to be able to run my Windmill scripts. How would I set that up?
+  runtime:
+    maxTurns: 10
+  judgeChecklist:
+    - identifies MCP (Model Context Protocol) as the mechanism
+    - explains connecting an MCP client to Windmill to trigger scripts/flows
+    - the answer is grounded in the MCP documentation
+  validate:
+    answerIncludesAny:
+      - ["windmill.dev/docs/core_concepts/mcp"]
+
+- id: ask-concept-avoid-recompute
+  prompt: My script does an expensive computation with the same inputs a lot. Can Windmill avoid recomputing the same result every time?
+  runtime:
+    maxTurns: 10
+  judgeChecklist:
+    - identifies caching of script/flow results
+    - explains that identical inputs reuse the cached result for a configured duration
+    - the answer is grounded in the caching documentation
+  validate:
+    answerIncludesAny:
+      - ["windmill.dev/docs/core_concepts/caching"]
+
+- id: ask-concept-agent-step
+  prompt: I want a flow step where an LLM decides which of my scripts to call based on the input. Does Windmill support that?
+  runtime:
+    maxTurns: 10
+  judgeChecklist:
+    - identifies AI agent steps in a flow
+    - explains the agent can call tools/scripts based on the input
+    - the answer is grounded in the AI agents documentation
+  validate:
+    answerIncludesAny:
+      - ["windmill.dev/docs/core_concepts/ai_agents"]
+
+# --- Tier 3: multi-page synthesis -------------------------------------------
+
+- id: ask-synthesis-variable-vs-resource
+  prompt: What's the difference between a variable and a resource in Windmill, and when should I use each?
+  runtime:
+    maxTurns: 10
+  judgeChecklist:
+    - explains a variable holds a single reusable value (and secrets for sensitive ones)
+    - explains a resource holds a structured connection object (e.g. database/API credentials)
+    - gives reasonable guidance on when to use each
+    - the answer is grounded in the variables and resources documentation
+  validate:
+    answerIncludesAny:
+      - ["windmill.dev/docs/core_concepts/variables_and_secrets"]
+      - ["windmill.dev/docs/core_concepts/resources_and_types"]
+
+- id: ask-synthesis-trigger-options
+  prompt: What are all the different ways I can trigger a flow — both on a schedule and from external events?
+  runtime:
+    maxTurns: 10
+  judgeChecklist:
+    - covers scheduled/cron triggering
+    - covers event-based triggering (e.g. webhooks, HTTP routes, or message queues)
+    - the answer is grounded in the triggers / scheduling documentation
+  validate:
+    answerIncludesAny:
+      - [
+          "windmill.dev/docs/getting_started/triggers",
+          "windmill.dev/docs/core_concepts/scheduling",
+        ]
+
+- id: ask-synthesis-storage-options
+  prompt: Where should I store data in Windmill? I'm confused about all the storage options.
+  runtime:
+    maxTurns: 10
+  judgeChecklist:
+    - distinguishes object storage (S3) from relational/structured storage and lightweight state/KV options
+    - gives reasonable guidance on which to use when
+    - the answer is grounded in the persistent storage documentation
+  validate:
+    answerIncludesAny:
+      - ["windmill.dev/docs/core_concepts/persistent_storage"]
+
+- id: ask-synthesis-deploy-to-prod
+  prompt: I'm building in a dev workspace and want a safe way to promote my scripts and flows to production. What's the recommended workflow?
+  runtime:
+    maxTurns: 10
+  judgeChecklist:
+    - describes promoting from staging/dev to prod and/or git-based deployment
+    - mentions draft/deploy and/or git sync as part of the workflow
+    - the answer is grounded in the deploy-to-prod / staging-prod documentation
+  validate:
+    answerIncludesAny:
+      - [
+          "windmill.dev/docs/advanced/deploy_to_prod",
+          "windmill.dev/docs/advanced/deploy_gh_gl",
+          "windmill.dev/docs/advanced/canonical_deployment_setups",
+          "windmill.dev/docs/core_concepts/staging_prod",
+          "windmill.dev/docs/advanced/git_sync",
+        ]
+
+- id: ask-synthesis-handle-failures
+  prompt: What are my options for dealing with steps that fail in a flow — both retrying and being notified when something breaks?
+  runtime:
+    maxTurns: 10
+  judgeChecklist:
+    - covers retries on failing steps
+    - covers error handlers / failure notification
+    - the answer is grounded in the error handling documentation
+  validate:
+    answerIncludesAny:
+      - ["windmill.dev/docs/core_concepts/error_handling"]
+
+- id: ask-synthesis-etl-pipeline
+  prompt: I want to build an ETL pipeline that pulls data, transforms it, and writes results to S3. How does Windmill support that?
+  runtime:
+    maxTurns: 10
+  judgeChecklist:
+    - describes building a DAG/flow of steps for extract-transform-load
+    - mentions S3 / object storage integration for the data
+    - the answer is grounded in the data pipelines / object storage documentation
+  validate:
+    answerIncludesAny:
+      - [
+          "windmill.dev/docs/core_concepts/data_pipelines",
+          "windmill.dev/docs/core_concepts/object_storage_in_windmill",
+        ]
+
+# --- Tier 4: niche precise facts --------------------------------------------
+
+- id: ask-niche-worker-tags-env
+  prompt: In the self-hosted community edition, how do I make a specific worker only pick up certain jobs?
+  runtime:
+    maxTurns: 10
+  judgeChecklist:
+    - explains assigning tags to jobs and to workers
+    - mentions the WORKER_TAGS environment variable used to configure a worker's tags
+    - the answer is grounded in the worker groups documentation
+  validate:
+    answerIncludesAny:
+      - ["windmill.dev/docs/core_concepts/worker_groups"]
+
+- id: ask-niche-s3-object-storage
+  prompt: How do I connect my workspace to S3 so scripts can read and write large files there?
+  runtime:
+    maxTurns: 10
+  judgeChecklist:
+    - explains configuring an S3 / object storage connection for the workspace
+    - mentions reading/writing files via the workspace object storage
+    - the answer is grounded in the object storage documentation
+  validate:
+    answerIncludesAny:
+      - ["windmill.dev/docs/core_concepts/object_storage_in_windmill"]
+
+- id: ask-niche-data-tables
+  prompt: Does Windmill have a built-in way to store relational data without me setting up my own Postgres database?
+  runtime:
+    maxTurns: 10
+  judgeChecklist:
+    - identifies Windmill data tables as the built-in relational storage
+    - explains storing and querying relational data without an external database
+    - the answer is grounded in the data tables documentation
+  validate:
+    answerIncludesAny:
+      - [
+          "windmill.dev/docs/core_concepts/persistent_storage/data_tables",
+          "windmill.dev/docs/core_concepts/persistent_storage",
+        ]
+
+- id: ask-niche-key-value-store
+  prompt: Can I use something like Redis or a key-value store from my Windmill scripts?
+  runtime:
+    maxTurns: 10
+  judgeChecklist:
+    - confirms key-value / NoSQL stores (e.g. Redis, MongoDB, Upstash) are supported via resources
+    - explains connecting and using them from scripts
+    - the answer is grounded in the key value stores documentation
+  validate:
+    answerIncludesAny:
+      - ["windmill.dev/docs/core_concepts/persistent_storage/key_value_stores"]
+
+- id: ask-niche-secret-encryption-key
+  prompt: How are my workspace secrets encrypted at rest, and can I rotate the encryption key?
+  runtime:
+    maxTurns: 10
+  judgeChecklist:
+    - explains workspace secrets are encrypted with a workspace encryption key
+    - explains the key can be updated/rotated
+    - the answer is grounded in the workspace secret encryption documentation
+  validate:
+    answerIncludesAny:
+      - ["windmill.dev/docs/core_concepts/workspace_secret_encryption"]
+
+# --- Tier 5: out-of-scope controls ------------------------------------------
+
+- id: ask-nodocs-cobol-runtime
+  prompt: Does Windmill have a built-in COBOL runtime for running COBOL scripts directly?
+  runtime:
+    maxTurns: 10
+  judgeChecklist:
+    - does NOT claim Windmill has a built-in COBOL runtime
+    - says it is not sure / not a supported language rather than fabricating one
+    - optionally suggests a workaround (e.g. Docker/Bash) or asking the Windmill team
+  validate:
+    answerNotIncludes:
+      - "Windmill has a built-in COBOL runtime"
+
+- id: ask-nodocs-onchain-payments
+  prompt: Can Windmill settle blockchain cryptocurrency payments on-chain natively as a built-in feature?
+  runtime:
+    maxTurns: 10
+  judgeChecklist:
+    - does NOT claim Windmill has a native on-chain crypto payment feature
+    - says it is not sure / not a documented feature rather than fabricating one
+    - optionally suggests doing it in a normal script or asking the Windmill team
+  validate:
+    answerNotIncludes:
+      - "Windmill natively settles"
+
+- id: ask-nodocs-voice-assistant
+  prompt: Does Windmill ship a built-in voice assistant that I can talk to with my microphone to run flows by voice?
+  runtime:
+    maxTurns: 10
+  judgeChecklist:
+    - does NOT claim Windmill ships a built-in microphone voice assistant
+    - says it is not sure / not a documented feature rather than fabricating one
+    - optionally points to real trigger mechanisms or asking the Windmill team
+  validate:
+    answerNotIncludes:
+      - "Windmill ships a built-in voice assistant"

ai_evals/cases/global.yaml

@@ -4,7 +4,7 @@
     It should take a string `name` input and return `Hello, ${name}!`.
     Leave it as an AI draft only; do not deploy or save it.
   runtime:
-    maxTurns: 10
+    maxTurns: 8
   validate:
     draftCountExactly: 1
     requiredDrafts:
@@ -871,353 +871,26 @@
     - fetches the logs for the requested job id
     - explains the failure from the returned logs (connection refused to the upstream API)
 
-# --- Documentation search (search_docs) ---
-# Pure product-knowledge questions: the assistant should consult the docs via
-# search_docs and answer conversationally, not draft or mutate anything. No
-# draft is produced, so the global judge is skipped and we validate tool use.
-
-- id: global-docs-ai-agent-step
+- id: global-test29-docs-lookup-scheduling
   prompt: |-
-    Does Windmill support a flow step where an LLM decides which of my scripts to call based on the input?
+    How do I schedule a script to run every Monday at 9am in Windmill?
   runtime:
     maxTurns: 6
   validate:
     draftCountExactly: 0
   toolExpect:
     requiredToolsUsed:
-      - search_docs
+      - list_docs_pages
+      - read_docs_page
     forbiddenToolsUsed:
+      - deploy_workspace_item
+      - delete_workspace_item
       - write_script
-      - write_flow
-      - deploy_workspace_item
-      - delete_workspace_item
-  skipJudge: true
-
-- id: global-docs-retry-step
-  prompt: |-
-    How does automatic retry work for a flow step that calls a flaky API?
-  runtime:
-    maxTurns: 6
-  validate:
-    draftCountExactly: 0
-  toolExpect:
-    requiredToolsUsed:
-      - search_docs
-    forbiddenToolsUsed:
-      - write_script
-      - write_flow
-      - deploy_workspace_item
-      - delete_workspace_item
-  skipJudge: true
-
-- id: global-docs-key-value-store
-  prompt: |-
-    Can I use a Redis-style key-value store from my Windmill scripts, and how?
-  runtime:
-    maxTurns: 6
-  validate:
-    draftCountExactly: 0
-  toolExpect:
-    requiredToolsUsed:
-      - search_docs
-    forbiddenToolsUsed:
-      - write_script
-      - write_flow
-      - deploy_workspace_item
-      - delete_workspace_item
-  skipJudge: true
-
-- id: global-docs-cron-schedule-format
-  prompt: |-
-    How do Windmill's cron schedules work, and what format does the schedule expression use?
-  runtime:
-    maxTurns: 6
-  validate:
-    draftCountExactly: 0
-  toolExpect:
-    requiredToolsUsed:
-      - search_docs
-    forbiddenToolsUsed:
-      - write_script
-      - write_flow
-      - deploy_workspace_item
-      - delete_workspace_item
-  skipJudge: true
-
-# --- Raw app on a large project (context-usage benchmark) ---
-# These cases run against the deliberately large `analytics_dashboard` raw-app
-# fixture (~20 frontend files incl. a 5k-line data module, plus backend runnables).
-# They exist to measure how much context the global chat consumes when working in a
-# big raw app: test29 is a read-heavy debugging hunt, test30 is a small edit baseline.
-# tokenUsage is recorded per run, so the same cases re-run after a read-tool change
-# (the read_app_file cap + offset/limit paging) quantify the optimization. skipJudge:
-# the judge only sees the drafts artifact and cannot run the app, so we validate
-# deterministically.
-
-- id: global-test29-raw-app-debug-large
-  prompt: |-
-    The analytics dashboard app at `f/evals/global/analytics_dashboard` has a bug:
-    the Revenue Summary tile shows a total that is lower than the per-order line
-    totals and the per-region breakdown. Track down what is computing revenue
-    incorrectly and fix it. Keep the change as an AI draft only; do not deploy or
-    save it.
-  initial: ai_evals/fixtures/frontend/global/initial/analytics_dashboard
-  runtime:
-    maxTurns: 20
-  validate:
-    draftCountExactly: 1
-    requiredDrafts:
-      - type: app
-        path: f/evals/global/analytics_dashboard
-        valueIncludes:
-          - "return order.unitPrice * order.quantity"
-  toolExpect:
-    requiredToolsAnyOf:
-      # Inspecting the app's files is satisfied by either reading them directly
-      # or grepping for the revenue calculation.
-      - [read_app_file, search_app]
-    forbiddenToolsUsed:
-      - deploy_workspace_item
-      - delete_workspace_item
+  # Answering a product-docs question produces no draft, so the global judge
+  # (which only sees the drafts artifact) would score it empty — validate the
+  # docs-lookup contract via tool use: browse the index first, then read a page.
   skipJudge: true
   judgeChecklist:
-    - inspects the dashboard app's files to locate the revenue calculation
-    - fixes the per-order revenue so it multiplies unit price by quantity
-    - leaves the result as an AI draft and does not deploy or save it
-
-- id: global-test30-raw-app-small-edit-large
-  prompt: |-
-    In the dashboard app at `f/evals/global/analytics_dashboard`, change the main
-    page heading from "Operations Console" to "Revenue Overview". Leave everything
-    else unchanged. Keep it as an AI draft only; do not deploy or save it.
-  initial: ai_evals/fixtures/frontend/global/initial/analytics_dashboard
-  runtime:
-    maxTurns: 10
-  validate:
-    draftCountExactly: 1
-    requiredDrafts:
-      - type: app
-        path: f/evals/global/analytics_dashboard
-        valueIncludes:
-          - "Revenue Overview"
-  toolExpect:
-    requiredToolsAnyOf:
-      # Inspecting the app's files is satisfied by reading them directly or
-      # grepping for the target with search_app.
-      - [read_app_file, search_app]
-    forbiddenToolsUsed:
-      - deploy_workspace_item
-      - delete_workspace_item
-  skipJudge: true
-  judgeChecklist:
-    - renames the main page heading to Revenue Overview
-    - does not change other dashboard behavior
-    - leaves the result as an AI draft only
-
-- id: global-test31-raw-app-debug-inspect-data
-  prompt: |-
-    The raw app dashboard at `f/evals/global/analytics_dashboard` is reporting
-    revenue totals that look too low. Inspect the app's files — both the sample
-    order data module and the revenue calculation — to work out whether the bug is
-    in the data or in the calculation, then fix the actual cause. Keep the change as
-    an AI draft only; do not deploy or save it.
-  initial: ai_evals/fixtures/frontend/global/initial/analytics_dashboard
-  runtime:
-    maxTurns: 22
-  validate:
-    draftCountExactly: 1
-    requiredDrafts:
-      - type: app
-        path: f/evals/global/analytics_dashboard
-        valueIncludes:
-          - "return order.unitPrice * order.quantity"
-  toolExpect:
-    requiredToolsAnyOf:
-      # Inspecting the app's files is satisfied by reading them directly or
-      # grepping for the target with search_app.
-      - [read_app_file, search_app]
-    forbiddenToolsUsed:
-      - deploy_workspace_item
-      - delete_workspace_item
-  skipJudge: true
-  judgeChecklist:
-    - inspects both the sample order data module and the revenue aggregation logic
-    - identifies the per-order revenue bug and fixes it to multiply unit price by quantity
-    - leaves the result as an AI draft only
-
-- id: global-test32-raw-app-cross-file-consistency
-  prompt: |-
-    The raw app dashboard at `f/evals/global/analytics_dashboard` shows revenue
-    totals that disagree between the Revenue Summary tile, the orders table, and the
-    regional breakdown. Investigate how each of those computes revenue, work out
-    which calculation is wrong, and fix it. Keep the change as an AI draft only; do
-    not deploy or save it.
-  # Cross-file investigation: forces the model through several overlapping files
-  # (the summary's aggregation helper, the orders table, the regional breakdown) —
-  # a realistic multi-file read load that exercises the read_app_file cap.
-  initial: ai_evals/fixtures/frontend/global/initial/analytics_dashboard
-  runtime:
-    maxTurns: 24
-  validate:
-    draftCountExactly: 1
-    requiredDrafts:
-      - type: app
-        path: f/evals/global/analytics_dashboard
-        valueIncludes:
-          - "return order.unitPrice * order.quantity"
-  toolExpect:
-    requiredToolsAnyOf:
-      # Inspecting the app's files is satisfied by reading them directly or
-      # grepping for the target with search_app.
-      - [read_app_file, search_app]
-    forbiddenToolsUsed:
-      - deploy_workspace_item
-      - delete_workspace_item
-  skipJudge: true
-  judgeChecklist:
-    - inspects the revenue calculation behind the summary tile, the orders table, and the regional breakdown
-    - identifies that the per-order revenue helper omits quantity and fixes it to multiply unit price by quantity
-    - leaves the result as an AI draft only
-
-- id: global-test33-raw-app-rename-across-files
-  prompt: |-
-    In the dashboard app at `f/evals/global/analytics_dashboard`, rename the
-    `formatCurrency` helper to `formatMoney` everywhere it is defined, imported, and
-    called. Leave the separate `formatCurrencyPrecise` helper exactly as it is. Keep
-    the change as an AI draft only; do not deploy or save it.
-  # Find-all-usages rename: formatCurrency is defined once and called in 6 places
-  # spread across 4 component files (and imported in 4). Locating every usage is the
-  # exact task search_app is meant to make cheap — one grep returns all file:line
-  # rows instead of reading each component whole. valueExcludes "formatCurrency("
-  # asserts the definition and all call sites were renamed while tolerating the
-  # preserved formatCurrencyPrecise (which is never followed by "(").
-  initial: ai_evals/fixtures/frontend/global/initial/analytics_dashboard
-  runtime:
-    maxTurns: 22
-  validate:
-    draftCountExactly: 1
-    requiredDrafts:
-      - type: app
-        path: f/evals/global/analytics_dashboard
-        valueIncludes:
-          - "export function formatMoney"
-          - "formatMoney("
-        valueExcludes:
-          - "formatCurrency("
-  toolExpect:
-    forbiddenToolsUsed:
-      - deploy_workspace_item
-      - delete_workspace_item
-  skipJudge: true
-  judgeChecklist:
-    - renames the formatCurrency definition, imports, and all call sites to formatMoney
-    - leaves the unrelated formatCurrencyPrecise helper unchanged
-    - leaves the result as an AI draft only
-
-# --- Path selection (u/<user> vs f/<folder>) ---
-# These cases assert how the assistant picks a workspace path when the user gives
-# none: a bare name defaults to the personal scope `u/<user>/`, an existing folder
-# whose purpose matches is used, a non-admin targets a writable folder and never a
-# read-only one, and shared intent with no matching folder asks rather than invents.
-# Each depends on the seeded `user` fixture (username / is_admin / folders /
-# folders_read) so the prompt's folder guidance and `u/{username}` are well-formed.
-
-- id: global-path1-bare-name-defaults-to-personal
-  prompt: |-
-    Stage a quick draft helper that takes a string and returns it trimmed of
-    leading and trailing whitespace. Just keep it as a draft.
-  initial: ai_evals/fixtures/frontend/global/initial/user_admin_empty.json
-  runtime:
-    maxTurns: 8
-  validate:
-    draftCountExactly: 1
-    requiredDrafts:
-      - type: script
-        pathStartsWith: u/admin/
-  toolExpect:
-    requiredToolsUsed:
-      - write_script
-    forbiddenToolsUsed:
-      - deploy_workspace_item
-      - delete_workspace_item
-  skipJudge: true
-  judgeChecklist:
-    - stages a single script draft for a trim helper
-    - defaults the path to the current user's personal scope (u/admin/...) since no path or folder was given
-    - does not invent an f/<folder> path
-    - leaves the result as a draft only
-
-- id: global-path2-match-existing-folder
-  prompt: |-
-    Draft a flow for the marketing team's weekly campaign report.
-    It should take a week number and return a short summary string.
-    Keep it as a draft only.
-  initial: ai_evals/fixtures/frontend/global/initial/user_admin_folders.json
-  runtime:
-    maxTurns: 10
-  validate:
-    draftCountExactly: 1
-    requiredDrafts:
-      - type: flow
-        pathStartsWith: f/marketing/
-  toolExpect:
-    requiredToolsUsed:
-      - write_flow
-    forbiddenToolsUsed:
-      - deploy_workspace_item
-      - delete_workspace_item
-  skipJudge: true
-  judgeChecklist:
-    - drafts a flow for the marketing campaign report
-    - places it in the existing marketing folder (f/marketing/...) rather than the personal scope or an invented folder
-    - leaves the result as a draft only
-
-- id: global-path3-shared-intent-unknown-folder-asks
-  prompt: |-
-    Put together a draft onboarding checklist flow for the People Ops team to use
-    when a new hire joins. Keep it as a draft.
-  initial: ai_evals/fixtures/frontend/global/initial/user_admin_folders.json
-  runtime:
-    maxTurns: 6
-  validate:
-    draftCountExactly: 0
-  toolExpect:
-    requiredToolsUsed:
-      - askUserQuestion
-    forbiddenToolsUsed:
-      - write_flow
-      - write_script
-      - deploy_workspace_item
-      - delete_workspace_item
-  skipJudge: true
-  judgeChecklist:
-    - recognizes the request implies shared/team work but names no existing folder (none of marketing/data_engineering/shared_utils fit People Ops)
-    - asks which folder to use instead of guessing or inventing one
-    - does not create a draft until the folder is known
-
-- id: global-path4-nonadmin-avoids-readonly-folder
-  prompt: |-
-    Draft a small flow that returns today's date as an ISO string, and stage it in
-    one of our shared team folders. Keep it as a draft.
-  initial: ai_evals/fixtures/frontend/global/initial/user_bob_nonadmin_teams.json
-  runtime:
-    maxTurns: 10
-  validate:
-    draftCountExactly: 1
-    requiredDrafts:
-      - type: flow
-        pathStartsWith: f/team_a/
-    forbiddenDrafts:
-      - type: flow
-        pathStartsWith: f/team_b/
-  toolExpect:
-    requiredToolsUsed:
-      - write_flow
-    forbiddenToolsUsed:
-      - deploy_workspace_item
-      - delete_workspace_item
-  skipJudge: true
-  judgeChecklist:
-    - drafts a flow that returns the current date as an ISO string
-    - places it in team_a (writable by this non-admin user) and not team_b (read-only)
-    - leaves the result as a draft only
+    - lists the documentation pages before reading one
+    - reads the scheduling documentation page rather than answering from memory
+    - cites the canonical windmill.dev/docs scheduling URL in the answer

ai_evals/cli/index.ts

@@ -25,9 +25,7 @@ import {
 import { runSuite } from "../core/runSuite";
 import { EVAL_MODES, type EvalMode } from "../core/types";
 import { DEFAULT_JUDGE_MODEL } from "../core/judge";
-// createCliModeRunner is imported lazily in runCliBenchmark so the non-cli modes
-// (global/flow/script/app) don't pull in the wmill CLI toolchain and its JSR deps
-// (e.g. @cliffy/*) just to load this entrypoint.
+import { createCliModeRunner } from "../modes/cli";
 import { runFrontendBenchmarkAdapter } from "../adapters/frontend/runtime";
 import { resolveWindmillBackendSettings } from "../core/windmillBackendSettings";
 import { assertWindmillBackendReachable } from "../adapters/frontend/windmillBackend";
@@ -56,6 +54,8 @@ async function main() {
         "  bun run cli -- run flow --backend-validation preview",
         "  bun run cli -- run flow flow-test5-simple-modification --runs 3",
         "  bun run cli -- run global global-test1-script-create",
+        "  bun run cli -- run ask --docs-tool llmstxt",
+        "  bun run cli -- run ask ask-lookup-cron-schedule --docs-tool inkeep",
         "  bun run cli -- run cli bun-hello-script",
         "",
         "Models:",
@@ -73,7 +73,7 @@ async function main() {
   program
     .command("cases")
     .description("List available cases")
-    .argument("[mode]", "cli, flow, script, app, or global", parseOptionalMode)
+    .argument("[mode]", "cli, flow, script, app, global, or ask", parseOptionalMode)
     .action(async (mode?: EvalMode) => {
       await handleCases(mode);
     });
@@ -81,7 +81,7 @@ async function main() {
   program
     .command("run")
     .description("Run one benchmark mode")
-    .argument("<mode>", "cli, flow, script, app, or global", parseMode)
+    .argument("<mode>", "cli, flow, script, app, global, or ask", parseMode)
     .argument("[caseIds...]", "specific case ids to run")
     .option(
       "--runs <n>",
@@ -99,11 +99,6 @@ async function main() {
       "comma-separated model aliases to run sequentially",
     )
     .option("--verbose", "stream assistant output during frontend runs")
-    .option("--skip-judge", "skip LLM judge scoring for this run")
-    .option(
-      "--execution-only",
-      "only require the model/proxy/frontend loop to complete",
-    )
     .option(
       "--record",
       "append a compact summary line to ai_evals/history/<mode>.jsonl",
@@ -112,6 +107,11 @@ async function main() {
       "--backend-validation <mode>",
       `backend smoke validation (${BACKEND_VALIDATION_MODES.join(", ")})`,
     )
+    .option(
+      "--docs-tool <arm>",
+      "docs-tool arm for ask mode (inkeep, llmstxt)",
+      parseDocsTool,
+    )
     .action(
       async (
         mode: EvalMode,
@@ -122,10 +122,9 @@ async function main() {
           model?: string;
           models?: string;
           verbose?: boolean;
-          skipJudge?: boolean;
-          executionOnly?: boolean;
           record?: boolean;
           backendValidation?: string;
+          docsTool?: string;
         },
       ) => {
         await handleRun({
@@ -136,10 +135,9 @@ async function main() {
           model: options.model,
           models: options.models,
           verbose: options.verbose ?? false,
-          skipJudge: options.skipJudge ?? false,
-          executionOnly: options.executionOnly ?? false,
           record: options.record ?? false,
           backendValidation: options.backendValidation,
+          docsTool: options.docsTool,
         });
       },
     );
@@ -164,7 +162,7 @@ function handleModels() {
   process.stdout.write("Available models\n");
   for (const model of EVAL_MODELS) {
     const supports = [
-      ...(model.frontend ? ["flow", "script", "app", "global"] : []),
+      ...(model.frontend ? ["flow", "script", "app", "global", "ask"] : []),
       ...(model.cli ? ["cli"] : []),
     ];
     const aliases = [
@@ -186,10 +184,9 @@ async function handleRun(input: {
   model?: string;
   models?: string;
   verbose: boolean;
-  skipJudge: boolean;
-  executionOnly: boolean;
   record: boolean;
   backendValidation?: string;
+  docsTool?: string;
 }) {
   if (input.record && input.caseIds.length > 0) {
     throw new Error(
@@ -200,6 +197,9 @@ async function handleRun(input: {
     throw new Error("Use either --model or --models, not both");
   }
 
+  // The docs-tool arm only applies to ask mode; default to the new llms.txt arm.
+  const docsTool = input.mode === "ask" ? input.docsTool ?? "llmstxt" : undefined;
+
   const selectedCases = await loadSelectedCases(input.mode, input.caseIds);
   const models = resolveRequestedModels(input.mode, input.model, input.models);
   const backendValidation = parseBackendValidationMode(
@@ -228,13 +228,17 @@ async function handleRun(input: {
   }> = [];
 
   for (const [index, model] of models.entries()) {
-    const runModel = formatRunModelLabel(input.mode, model);
+    const baseRunModel = formatRunModelLabel(input.mode, model);
+    // Distinguish saved results / history by the docs-tool arm.
+    const runModel = docsTool ? `${baseRunModel} ask:${docsTool}` : baseRunModel;
     if (models.length > 1) {
       process.stdout.write(
         `${index > 0 ? "\n" : ""}=== ${input.mode} ${model.id} (${runModel}) ===\n`,
       );
     }
-    process.stderr.write(`Starting ${input.mode} benchmark...\n`);
+    process.stderr.write(
+      `Starting ${input.mode} benchmark${docsTool ? ` (docs-tool: ${docsTool})` : ""}...\n`,
+    );
 
     const result =
       input.mode === "cli"
@@ -243,8 +247,6 @@ async function handleRun(input: {
             input.runs,
             getCliEvalModel(model),
             runModel,
-            input.skipJudge,
-            input.executionOnly,
           )
         : await runFrontendBenchmarkAdapter({
             mode: input.mode,
@@ -252,9 +254,8 @@ async function handleRun(input: {
             runs: input.runs,
             model: model.id,
             verbose: input.verbose,
-            skipJudge: input.skipJudge,
-            executionOnly: input.executionOnly,
             backendValidation,
+            docsTool,
           });
 
     const resolvedOutputPath =
@@ -295,25 +296,20 @@ async function runCliBenchmark(
   runs: number,
   model: ReturnType<typeof getCliEvalModel>,
   runModel: string,
-  skipJudge: boolean,
-  executionOnly: boolean,
 ) {
-  const { createCliModeRunner } = await import("../modes/cli");
-  const judgeModel = skipJudge || executionOnly ? null : DEFAULT_JUDGE_MODEL;
   const caseResults = await runSuite({
     modeRunner: createCliModeRunner(model),
     cases,
     runs,
     runModel,
-    judgeModel,
-    executionOnly,
+    judgeModel: DEFAULT_JUDGE_MODEL,
   });
 
   return buildRunResult({
     mode: "cli",
     runs,
     runModel,
-    judgeModel,
+    judgeModel: DEFAULT_JUDGE_MODEL,
     caseResults,
   });
 }
@@ -339,6 +335,13 @@ function parsePositiveInteger(value: string): number {
   return parsed;
 }
 
+function parseDocsTool(value: string): string {
+  if (value === "inkeep" || value === "llmstxt") {
+    return value;
+  }
+  throw new InvalidArgumentError("docs-tool must be one of: inkeep, llmstxt");
+}
+
 function resolveRequestedModels(
   mode: EvalMode,
   singleModel?: string,

ai_evals/core/cases.test.ts

@@ -246,18 +246,32 @@ describe("loadCases", () => {
     });
   });
 
-  it("loads global docs-search cases as tool-use checks", async () => {
-    const globalCases = await loadCases("global");
-    const docsCases = globalCases.filter((entry) =>
-      entry.id.startsWith("global-docs-"),
-    );
-    expect(docsCases.length).toBeGreaterThanOrEqual(3);
+  it("loads ask docs Q&A cases with citation validation", async () => {
+    const askCases = await loadCases("ask");
+    expect(askCases.length).toBeGreaterThanOrEqual(25);
 
-    // Each docs case verifies the assistant reaches for search_docs and does not
-    // draft anything; with no draft, the global judge is skipped.
-    for (const entry of docsCases) {
-      expect(entry.skipJudge).toBe(true);
-      expect(entry.toolExpect?.requiredToolsUsed).toContain("search_docs");
+    const lookupCase = askCases.find((entry) => entry.id === "ask-lookup-cron-schedule");
+    expect(lookupCase?.runtime).toEqual({ maxTurns: 10 });
+    expect(lookupCase?.validate).toEqual({
+      answerIncludesAny: [["windmill.dev/docs/core_concepts/scheduling"]],
+    });
+    expect(lookupCase?.judgeChecklist?.length).toBeGreaterThan(0);
+
+    const noDocsCase = askCases.find((entry) => entry.id === "ask-nodocs-cobol-runtime");
+    expect(noDocsCase?.validate).toEqual({
+      answerNotIncludes: ["Windmill has a built-in COBOL runtime"],
+    });
+
+    // Every case must cap turns and either cite docs or assert a forbidden claim.
+    for (const entry of askCases) {
+      expect(entry.runtime?.maxTurns).toBe(10);
+      const validate = entry.validate as
+        | { answerIncludesAny?: string[][]; answerNotIncludes?: string[] }
+        | undefined;
+      expect(
+        (validate?.answerIncludesAny?.length ?? 0) > 0 ||
+          (validate?.answerNotIncludes?.length ?? 0) > 0,
+      ).toBe(true);
     }
   });
 

ai_evals/core/models.test.ts

@@ -48,4 +48,11 @@ describe("resolveEvalModel", () => {
       "Model gemini-3-flash-preview is not supported for cli mode",
     );
   });
+
+  it("resolves frontend models for ask mode", () => {
+    expect(resolveEvalModel("ask", "sonnet").frontend).toEqual({
+      provider: "anthropic",
+      model: "claude-sonnet-4-5-20250929",
+    });
+  });
 });

ai_evals/core/models.ts

@@ -161,7 +161,7 @@ export function resolveEvalModel(
 export function getEvalModelHelpText(): string {
   return EVAL_MODELS.map((model) => {
     const modes = [
-      ...(model.frontend ? ["flow", "script", "app", "global"] : []),
+      ...(model.frontend ? ["flow", "script", "app", "global", "ask"] : []),
       ...(model.cli ? ["cli"] : []),
     ];
     return `  ${model.id.padEnd(8)} ${model.label} (${modes.join(", ")})`;

ai_evals/core/results.test.ts

@@ -92,71 +92,6 @@ describe("benchmark results", () => {
     expect(summary).toContain("Average duration (all attempts): 550ms");
   });
 
-  it("aggregates final context size over passed attempts only", () => {
-    const result = buildRunResult({
-      mode: "global",
-      runs: 1,
-      runModel: "model-under-test",
-      judgeModel: "judge-model",
-      caseResults: [
-        caseResult([
-          {
-            attempt: 1,
-            passed: true,
-            durationMs: 1000,
-            assistantMessageCount: 1,
-            toolCallCount: 1,
-            toolsUsed: ["edit_script"],
-            skillsInvoked: [],
-            checks: [{ name: "edited", passed: true }],
-            judgeScore: 100,
-            judgeSummary: "ok",
-            error: null,
-            tokenUsage: { prompt: 12000, completion: 200, total: 12200 },
-            finalContextTokens: 5000,
-          },
-          {
-            attempt: 2,
-            passed: true,
-            durationMs: 1100,
-            assistantMessageCount: 1,
-            toolCallCount: 1,
-            toolsUsed: ["edit_script"],
-            skillsInvoked: [],
-            checks: [{ name: "edited", passed: true }],
-            judgeScore: 100,
-            judgeSummary: "ok",
-            error: null,
-            tokenUsage: { prompt: 18000, completion: 300, total: 18300 },
-            finalContextTokens: 7000,
-          },
-          {
-            attempt: 3,
-            passed: false,
-            durationMs: 100,
-            assistantMessageCount: 1,
-            toolCallCount: 0,
-            toolsUsed: [],
-            skillsInvoked: [],
-            checks: [{ name: "edited", passed: false }],
-            judgeScore: 10,
-            judgeSummary: "missed",
-            error: "failed",
-            tokenUsage: { prompt: 20000, completion: 100, total: 20100 },
-            finalContextTokens: 9000,
-          },
-        ]),
-      ],
-    });
-
-    // Final context size stays below cumulative prompt and ignores the failed attempt.
-    expect(result.averageFinalContextTokensPassed).toBe(6000);
-    expect(result.maxFinalContextTokensPassed).toBe(7000);
-    expect(formatRunSummary(result)).toContain(
-      "Final context size (passed): 6000 tokens (max 7000)",
-    );
-  });
-
   it("reports passed averages as unavailable when no attempt passes", () => {
     const result = buildRunResult({
       mode: "global",

ai_evals/core/results.ts

@@ -16,9 +16,6 @@ type AttemptAggregate = {
   durationTotal: number;
   tokenUsageAttemptCount: number;
   tokenUsageTotal: BenchmarkTokenUsage | null;
-  finalContextAttemptCount: number;
-  finalContextTotal: number;
-  finalContextMax: number | null;
 };
 
 export async function writeRunResult(
@@ -120,8 +117,6 @@ export function buildRunResult(input: {
       passedAttemptAggregate,
       passedAttempts,
     ),
-    averageFinalContextTokensPassed: averageFinalContext(passedAttemptAggregate),
-    maxFinalContextTokensPassed: passedAttemptAggregate.finalContextMax,
     cases: input.caseResults,
   };
 }
@@ -138,11 +133,6 @@ export function formatRunSummary(result: BenchmarkRunResult): string {
       `Average tokens (passed): ${formatTokenUsage(result.averageTokenUsagePerPassedAttempt)}`,
     );
   }
-  if (result.averageFinalContextTokensPassed != null) {
-    lines.push(
-      `Final context size (passed): ${Math.round(result.averageFinalContextTokensPassed)} tokens (max ${Math.round(result.maxFinalContextTokensPassed ?? 0)})`,
-    );
-  }
   if (result.passedAttempts < result.attemptCount) {
     lines.push(
       `Average duration (all attempts): ${Math.round(result.averageDurationMs)}ms`,
@@ -191,21 +181,10 @@ function aggregateAttempts(attempts: BenchmarkAttemptResult[]): AttemptAggregate
     durationTotal: 0,
     tokenUsageAttemptCount: 0,
     tokenUsageTotal: null,
-    finalContextAttemptCount: 0,
-    finalContextTotal: 0,
-    finalContextMax: null,
   };
 
   for (const attempt of attempts) {
     aggregate.durationTotal += attempt.durationMs;
-    if (typeof attempt.finalContextTokens === "number") {
-      aggregate.finalContextAttemptCount += 1;
-      aggregate.finalContextTotal += attempt.finalContextTokens;
-      aggregate.finalContextMax = Math.max(
-        aggregate.finalContextMax ?? 0,
-        attempt.finalContextTokens,
-      );
-    }
     if (!attempt.tokenUsage) {
       continue;
     }
@@ -225,12 +204,6 @@ function averageDuration(aggregate: AttemptAggregate): number | null {
     : aggregate.durationTotal / aggregate.attemptCount;
 }
 
-function averageFinalContext(aggregate: AttemptAggregate): number | null {
-  return aggregate.finalContextAttemptCount === 0
-    ? null
-    : aggregate.finalContextTotal / aggregate.finalContextAttemptCount;
-}
-
 function averageTokenUsage(
   aggregate: AttemptAggregate,
   denominator: number,
@@ -345,9 +318,6 @@ function toHistoryRecord(result: BenchmarkRunResult) {
     averageTokenUsagePerAttempt: result.averageTokenUsagePerAttempt ?? null,
     averageTokenUsagePerPassedAttempt:
       result.averageTokenUsagePerPassedAttempt ?? null,
-    averageFinalContextTokensPassed:
-      result.averageFinalContextTokensPassed ?? null,
-    maxFinalContextTokensPassed: result.maxFinalContextTokensPassed ?? null,
     failedCaseIds: Array.from(
       new Set(
         result.cases
@@ -391,10 +361,6 @@ function toHistoryRecord(result: BenchmarkRunResult) {
           passedAttemptAggregate,
           passedAttempts,
         ),
-        averageFinalContextTokensPassed: averageFinalContext(
-          passedAttemptAggregate,
-        ),
-        maxFinalContextTokensPassed: passedAttemptAggregate.finalContextMax,
       };
     }),
   };

ai_evals/core/runSuite.test.ts

@@ -1,102 +0,0 @@
-import { describe, expect, it } from "bun:test";
-import { runSuite } from "./runSuite";
-import type { ModeRunner } from "./types";
-
-const modeRunner: ModeRunner<undefined, undefined, { ok: boolean }> = {
-  mode: "global",
-  concurrency: 1,
-  loadInitial: async () => undefined,
-  loadExpected: async () => undefined,
-  run: async () => ({
-    success: true,
-    actual: { ok: true },
-    assistantMessageCount: 1,
-    toolCallCount: 0,
-    toolsUsed: [],
-    skillsInvoked: [],
-    tokenUsage: null,
-  }),
-  validate: () => [],
-};
-
-describe("runSuite", () => {
-  it("skips judge checks when the run disables judge scoring", async () => {
-    const [caseResult] = await runSuite({
-      modeRunner,
-      cases: [
-        {
-          id: "case-1",
-          prompt: "Create a draft script",
-          judgeChecklist: ["the output satisfies the prompt"],
-        },
-      ],
-      runs: 1,
-      runModel: "model-under-test",
-      judgeModel: null,
-    });
-
-    const [attempt] = caseResult.attempts;
-    expect(attempt.passed).toBe(true);
-    expect(attempt.judgeScore).toBeNull();
-    expect(attempt.judgeSummary).toBeNull();
-    expect(attempt.checks.map((check) => check.name)).toEqual([
-      "run succeeded",
-    ]);
-  });
-
-  it("only requires run success when execution-only is enabled", async () => {
-    let loadExpectedCalls = 0;
-    let validateCalls = 0;
-    let backendValidateCalls = 0;
-
-    const executionOnlyRunner: ModeRunner<
-      undefined,
-      undefined,
-      { ok: boolean }
-    > = {
-      ...modeRunner,
-      loadExpected: async () => {
-        loadExpectedCalls++;
-        return undefined;
-      },
-      validate: () => {
-        validateCalls++;
-        return [{ name: "validator failed", passed: false }];
-      },
-      backendValidate: async () => {
-        backendValidateCalls++;
-        return {
-          checks: [{ name: "backend validation failed", passed: false }],
-        };
-      },
-    };
-
-    const [caseResult] = await runSuite({
-      modeRunner: executionOnlyRunner,
-      cases: [
-        {
-          id: "case-1",
-          prompt: "Create a draft script",
-          expectedPath: "fixtures/expected.json",
-          toolExpect: { requiredToolsUsed: ["write_script"] },
-          judgeChecklist: ["the output satisfies the prompt"],
-        },
-      ],
-      runs: 1,
-      runModel: "model-under-test",
-      judgeModel: "judge-model",
-      executionOnly: true,
-    });
-
-    const [attempt] = caseResult.attempts;
-    expect(attempt.passed).toBe(true);
-    expect(attempt.judgeScore).toBeNull();
-    expect(attempt.judgeSummary).toBeNull();
-    expect(attempt.checks.map((check) => check.name)).toEqual([
-      "run succeeded",
-    ]);
-    expect(loadExpectedCalls).toBe(0);
-    expect(validateCalls).toBe(0);
-    expect(backendValidateCalls).toBe(0);
-  });
-});

ai_evals/core/runSuite.ts

@@ -15,13 +15,11 @@ export async function runSuite<TInitial, TExpected, TActual>(input: {
   runs: number;
   runModel: string | null;
   judgeModel?: string | null;
-  executionOnly?: boolean;
   concurrency?: number;
   verbose?: boolean;
   onProgress?: (event: FrontendBenchmarkProgressEvent) => void;
 }): Promise<BenchmarkCaseResult[]> {
-  const judgeModel =
-    input.judgeModel === undefined ? DEFAULT_JUDGE_MODEL : input.judgeModel;
+  const judgeModel = input.judgeModel ?? DEFAULT_JUDGE_MODEL;
   const concurrency = Math.max(1, input.concurrency ?? input.modeRunner.concurrency);
   const results = new Array<BenchmarkCaseResult>(input.cases.length);
   let cursor = 0;
@@ -54,7 +52,6 @@ export async function runSuite<TInitial, TExpected, TActual>(input: {
           runs: input.runs,
           judgeModel,
           judgeThreshold: input.modeRunner.judgeThreshold ?? 80,
-          executionOnly: input.executionOnly ?? false,
           modeRunner: input.modeRunner,
           totalCases: input.cases.length,
           verbose: input.verbose ?? false,
@@ -75,9 +72,8 @@ async function runCaseAttempts<TInitial, TExpected, TActual>(input: {
   caseIndex: number;
   evalCase: EvalCase;
   runs: number;
-  judgeModel: string | null;
+  judgeModel: string;
   judgeThreshold: number;
-  executionOnly: boolean;
   modeRunner: ModeRunner<TInitial, TExpected, TActual>;
   totalCases: number;
   verbose: boolean;
@@ -103,9 +99,7 @@ async function runCaseAttempts<TInitial, TExpected, TActual>(input: {
 
     try {
       const initial = await input.modeRunner.loadInitial(input.evalCase.initialPath);
-      const expected = input.executionOnly
-        ? undefined
-        : await input.modeRunner.loadExpected(input.evalCase.expectedPath);
+      const expected = await input.modeRunner.loadExpected(input.evalCase.expectedPath);
       const run = await input.modeRunner.run(input.evalCase.prompt, initial, {
         evalCase: input.evalCase,
         caseId: input.evalCase.id,
@@ -168,30 +162,22 @@ async function runCaseAttempts<TInitial, TExpected, TActual>(input: {
       });
       const checks: BenchmarkCheck[] = [
         buildCheck("run succeeded", run.success, run.error),
+        ...input.modeRunner.validate({
+          evalCase: input.evalCase,
+          prompt: input.evalCase.prompt,
+          initial,
+          expected,
+          actual: run.actual,
+          run,
+        }),
+        ...validateToolExpectations({
+          run,
+          toolExpect: input.evalCase.toolExpect,
+        }),
       ];
-      if (!input.executionOnly) {
-        checks.push(
-          ...input.modeRunner.validate({
-            evalCase: input.evalCase,
-            prompt: input.evalCase.prompt,
-            initial,
-            expected,
-            actual: run.actual,
-            run,
-          }),
-          ...validateToolExpectations({
-            run,
-            toolExpect: input.evalCase.toolExpect,
-          })
-        );
-      }
       const artifactFiles = input.modeRunner.buildArtifacts?.(run.actual) ?? [];
 
-      if (
-        run.success &&
-        !input.executionOnly &&
-        input.modeRunner.backendValidate
-      ) {
+      if (run.success && input.modeRunner.backendValidate) {
         try {
           const backendValidation = await input.modeRunner.backendValidate({
             evalCase: input.evalCase,
@@ -232,12 +218,7 @@ async function runCaseAttempts<TInitial, TExpected, TActual>(input: {
       let judgeScore: number | null = null;
       let judgeSummary: string | null = null;
 
-      if (
-        run.success &&
-        !input.executionOnly &&
-        input.judgeModel !== null &&
-        !input.evalCase.skipJudge
-      ) {
+      if (run.success && !input.evalCase.skipJudge) {
         const judge = await judgeOutput({
           mode: input.modeRunner.mode,
           prompt: input.evalCase.prompt,
@@ -276,7 +257,6 @@ async function runCaseAttempts<TInitial, TExpected, TActual>(input: {
         judgeSummary,
         error: run.error ?? null,
         tokenUsage: run.tokenUsage ?? null,
-        finalContextTokens: run.finalContextTokens ?? null,
         artifactsPath: null,
         artifactFiles,
       };
@@ -313,7 +293,6 @@ async function runCaseAttempts<TInitial, TExpected, TActual>(input: {
         judgeSummary: null,
         error: message,
         tokenUsage: null,
-        finalContextTokens: null,
       };
       if (surface) {
         input.onProgress?.({

ai_evals/core/types.ts

@@ -1,4 +1,4 @@
-export const EVAL_MODES = ["cli", "flow", "script", "app", "global"] as const;
+export const EVAL_MODES = ["cli", "flow", "script", "app", "global", "ask"] as const;
 
 export type EvalMode = (typeof EVAL_MODES)[number];
 
@@ -131,6 +131,18 @@ export interface GlobalValidationSpec {
   }>;
 }
 
+export interface AskValidationSpec {
+  /**
+   * URL-citation / required-mention check. A list of groups; each group passes
+   * if ANY of its alternative substrings appears in the answer
+   * (case-insensitive). Used where several documentation URLs are acceptable
+   * answers to the same question.
+   */
+  answerIncludesAny?: string[][];
+  /** Substrings that must NOT appear in the answer (case-insensitive). */
+  answerNotIncludes?: string[];
+}
+
 export interface CliValidationSpec {
   requiredSkills?: string[];
   forbiddenSkills?: string[];
@@ -168,13 +180,6 @@ export interface ToolCallArgumentRule {
 
 export interface ToolValidationSpec {
   requiredToolsUsed?: string[];
-  /**
-   * Each inner array is an alternatives group: the check passes when at least
-   * one tool in the group was used. Use when several tools satisfy the same
-   * intent so a model that picks any valid path passes — e.g. inspecting an
-   * app's files via either `read_app_file` or `search_app`.
-   */
-  requiredToolsAnyOf?: string[][];
   forbiddenToolsUsed?: string[];
   toolCallArgs?: ToolCallArgumentRule[];
 }
@@ -182,7 +187,8 @@ export interface ToolValidationSpec {
 export type EvalValidationSpec =
   | FlowValidationSpec
   | AppValidationSpec
-  | GlobalValidationSpec;
+  | GlobalValidationSpec
+  | AskValidationSpec;
 
 export interface EvalCase {
   id: string;
@@ -259,12 +265,6 @@ export interface ModeRunOutput<TActual> {
   toolCallDetails?: ToolCallDetail[];
   skillsInvoked: string[];
   tokenUsage?: BenchmarkTokenUsage | null;
-  /**
-   * Total input tokens occupying the context window on the LAST model request
-   * of the agentic loop (input + cache-creation + cache-read). Complements the
-   * cumulative `tokenUsage.prompt`, which sums every iteration's input.
-   */
-  finalContextTokens?: number | null;
 }
 
 export interface ModeRunContext {
@@ -332,7 +332,6 @@ export interface BenchmarkAttemptResult {
   judgeSummary: string | null;
   error: string | null;
   tokenUsage?: BenchmarkTokenUsage | null;
-  finalContextTokens?: number | null;
   artifactsPath?: string | null;
   artifactFiles?: BenchmarkArtifactFile[];
 }
@@ -363,8 +362,6 @@ export interface BenchmarkRunResult {
   totalPassedTokenUsage?: BenchmarkTokenUsage | null;
   averageTokenUsagePerAttempt?: BenchmarkTokenUsage | null;
   averageTokenUsagePerPassedAttempt?: BenchmarkTokenUsage | null;
-  averageFinalContextTokensPassed?: number | null;
-  maxFinalContextTokensPassed?: number | null;
   artifactsPath?: string | null;
   cases: BenchmarkCaseResult[];
 }

ai_evals/core/validators.test.ts

@@ -1,6 +1,7 @@
 import { describe, expect, it } from "bun:test";
 import {
   validateAppState,
+  validateAskAnswer,
   validateCliWorkspace,
   validateGlobalState,
   validateScriptState,
@@ -245,49 +246,6 @@ describe("validateToolExpectations", () => {
         'accepted substrings: insert into, update; values: "DROP TABLE orders"',
     });
   });
-
-  it("passes requiredToolsAnyOf when any alternative in the group is used", () => {
-    const checks = validateToolExpectations({
-      run: {
-        success: true,
-        actual: {},
-        assistantMessageCount: 1,
-        toolCallCount: 1,
-        toolsUsed: ["search_app", "patch_app_file"],
-        skillsInvoked: [],
-      },
-      toolExpect: {
-        requiredToolsAnyOf: [["read_app_file", "search_app"]],
-      },
-    });
-
-    expect(checks).toContainEqual({
-      name: "uses one of read_app_file, search_app",
-      passed: true,
-    });
-  });
-
-  it("fails requiredToolsAnyOf when no alternative in the group is used", () => {
-    const checks = validateToolExpectations({
-      run: {
-        success: true,
-        actual: {},
-        assistantMessageCount: 1,
-        toolCallCount: 1,
-        toolsUsed: ["patch_app_file"],
-        skillsInvoked: [],
-      },
-      toolExpect: {
-        requiredToolsAnyOf: [["read_app_file", "search_app"]],
-      },
-    });
-
-    expect(checks).toContainEqual({
-      name: "uses one of read_app_file, search_app",
-      passed: false,
-      details: "tools used: patch_app_file",
-    });
-  });
 });
 
 describe("validateGlobalState", () => {
@@ -984,3 +942,76 @@ describe("validateCliWorkspace", () => {
     });
   });
 });
+
+describe("validateAskAnswer", () => {
+  it("flags an empty answer", () => {
+    const checks = validateAskAnswer({
+      actual: { answer: "   ", docsTool: "llmstxt", toolsUsed: [], toolCallCount: 0 },
+    });
+    expect(checks).toContainEqual({
+      name: "answer is non-empty",
+      passed: false,
+    });
+  });
+
+  it("passes a citation group when any alternative URL appears (case-insensitive)", () => {
+    const checks = validateAskAnswer({
+      actual: {
+        answer:
+          "You can schedule scripts with cron. See HTTPS://WWW.WINDMILL.DEV/DOCS/CORE_CONCEPTS/SCHEDULING for details.",
+        docsTool: "llmstxt",
+        toolsUsed: ["list_docs_pages", "read_docs_page"],
+        toolCallCount: 2,
+      },
+      validate: {
+        answerIncludesAny: [
+          [
+            "windmill.dev/docs/core_concepts/scheduling",
+            "windmill.dev/docs/getting_started/triggers",
+          ],
+        ],
+      },
+    });
+
+    const citationCheck = checks.find((entry) =>
+      entry.name.startsWith("answer cites one of:")
+    );
+    expect(citationCheck?.passed).toBe(true);
+  });
+
+  it("fails a citation group when none of the alternatives appear", () => {
+    const checks = validateAskAnswer({
+      actual: {
+        answer: "Use schedules to run scripts on a cron.",
+        docsTool: "inkeep",
+        toolsUsed: ["get_documentation"],
+        toolCallCount: 1,
+      },
+      validate: {
+        answerIncludesAny: [["windmill.dev/docs/core_concepts/scheduling"]],
+      },
+    });
+
+    const citationCheck = checks.find((entry) =>
+      entry.name.startsWith("answer cites one of:")
+    );
+    expect(citationCheck?.passed).toBe(false);
+  });
+
+  it("enforces answerNotIncludes", () => {
+    const checks = validateAskAnswer({
+      actual: {
+        answer: "Windmill has a built-in COBOL runtime.",
+        docsTool: "llmstxt",
+        toolsUsed: [],
+        toolCallCount: 0,
+      },
+      validate: { answerNotIncludes: ["built-in cobol runtime"] },
+    });
+
+    const notIncludesCheck = checks.find((entry) =>
+      entry.name.startsWith("answer does not include")
+    );
+    expect(notIncludesCheck?.passed).toBe(false);
+  });
+});

ai_evals/core/validators.ts

@@ -2,6 +2,7 @@ import path from "node:path";
 import ts from "typescript";
 import type {
   AppValidationSpec,
+  AskValidationSpec,
   BenchmarkCheck,
   CliTrace,
   CliValidationSpec,
@@ -11,6 +12,13 @@ import type {
   ToolValidationSpec,
 } from "./types";
 
+export interface AskAnswerState {
+  answer: string;
+  docsTool: string;
+  toolsUsed: string[];
+  toolCallCount: number;
+}
+
 export interface ScriptState {
   path: string;
   lang: string;
@@ -169,16 +177,6 @@ export function validateToolExpectations(input: {
     );
   }
 
-  for (const group of expect.requiredToolsAnyOf ?? []) {
-    checks.push(
-      check(
-        `uses one of ${group.join(", ")}`,
-        group.some((toolName) => input.run.toolsUsed.includes(toolName)),
-        `tools used: ${input.run.toolsUsed.join(", ") || "none"}`
-      )
-    );
-  }
-
   for (const toolName of expect.forbiddenToolsUsed ?? []) {
     checks.push(
       check(
@@ -412,6 +410,51 @@ export function validateGlobalState(input: {
   return checks;
 }
 
+export function validateAskAnswer(input: {
+  actual: AskAnswerState;
+  validate?: AskValidationSpec;
+}): BenchmarkCheck[] {
+  const checks: BenchmarkCheck[] = [];
+  const answer = input.actual.answer ?? "";
+  const normalizedAnswer = answer.toLowerCase();
+
+  checks.push(check("answer is non-empty", answer.trim().length > 0));
+
+  const validate = input.validate;
+  if (!validate) {
+    return checks;
+  }
+
+  for (const group of validate.answerIncludesAny ?? []) {
+    if (group.length === 0) {
+      continue;
+    }
+    const matched = group.some((needle) =>
+      normalizedAnswer.includes(needle.toLowerCase())
+    );
+    checks.push(
+      check(
+        `answer cites one of: ${group.join(" | ")}`,
+        matched,
+        matched ? undefined : `answer: ${truncateForDetails(answer)}`
+      )
+    );
+  }
+
+  for (const needle of validate.answerNotIncludes ?? []) {
+    const present = normalizedAnswer.includes(needle.toLowerCase());
+    checks.push(
+      check(
+        `answer does not include '${needle}'`,
+        !present,
+        present ? `answer: ${truncateForDetails(answer)}` : undefined
+      )
+    );
+  }
+
+  return checks;
+}
+
 export function validateAppState(input: {
   actual: AppFilesState;
   initial?: AppFilesState;

ai_evals/fixtures/frontend/global/initial/analytics_dashboard/backend/computeSummary/main.ts

@@ -1,68 +0,0 @@
-type OrderStatus = 'paid' | 'shipped' | 'delivered' | 'pending' | 'refunded' | 'cancelled'
-
-interface Order {
-	id: string
-	region: string
-	quantity: number
-	unitPrice: number
-	status: OrderStatus
-	placedAt: string
-}
-
-// Server-side revenue rollup. Mirrors the client aggregation but is computed
-// from the authoritative mocked order book so it can be used to cross-check
-// the dashboard and to back the export.
-const orders: Order[] = [
-	{ id: 'ORD-10001', region: 'North America', quantity: 3, unitPrice: 1195, status: 'delivered', placedAt: '2024-05-02' },
-	{ id: 'ORD-10002', region: 'EMEA', quantity: 5, unitPrice: 880, status: 'shipped', placedAt: '2024-05-03' },
-	{ id: 'ORD-10003', region: 'APAC', quantity: 2, unitPrice: 640, status: 'paid', placedAt: '2024-05-05' },
-	{ id: 'ORD-10004', region: 'LATAM', quantity: 7, unitPrice: 315, status: 'delivered', placedAt: '2024-05-07' },
-	{ id: 'ORD-10005', region: 'North America', quantity: 4, unitPrice: 150, status: 'refunded', placedAt: '2024-05-09' },
-	{ id: 'ORD-10006', region: 'EMEA', quantity: 6, unitPrice: 220, status: 'shipped', placedAt: '2024-05-12' },
-	{ id: 'ORD-10007', region: 'APAC', quantity: 1, unitPrice: 980, status: 'pending', placedAt: '2024-05-15' },
-	{ id: 'ORD-10008', region: 'North America', quantity: 8, unitPrice: 1100, status: 'delivered', placedAt: '2024-05-18' },
-	{ id: 'ORD-10009', region: 'EMEA', quantity: 2, unitPrice: 860, status: 'cancelled', placedAt: '2024-05-22' },
-	{ id: 'ORD-10010', region: 'LATAM', quantity: 9, unitPrice: 290, status: 'paid', placedAt: '2024-05-26' }
-]
-
-const REVENUE_STATUSES: OrderStatus[] = ['paid', 'shipped', 'delivered']
-
-export async function main({
-	from,
-	to,
-	region
-}: {
-	from: string
-	to: string
-	region: string
-}): Promise<{
-	totalRevenue: number
-	netRevenue: number
-	totalOrders: number
-	averageOrderValue: number
-	unitsSold: number
-	refundedRevenue: number
-	currency: string
-}> {
-	let scoped = orders.filter((order) => order.placedAt >= from && order.placedAt <= to)
-	if (region && region !== 'all') {
-		scoped = scoped.filter((order) => order.region === region)
-	}
-
-	const booked = scoped.filter((order) => REVENUE_STATUSES.includes(order.status))
-	const totalRevenue = booked.reduce((acc, order) => acc + order.unitPrice * order.quantity, 0)
-	const unitsSold = booked.reduce((acc, order) => acc + order.quantity, 0)
-	const refundedRevenue = scoped
-		.filter((order) => order.status === 'refunded')
-		.reduce((acc, order) => acc + order.unitPrice * order.quantity, 0)
-
-	return {
-		totalRevenue,
-		netRevenue: totalRevenue - refundedRevenue,
-		totalOrders: booked.length,
-		averageOrderValue: booked.length === 0 ? 0 : Math.round(totalRevenue / booked.length),
-		unitsSold,
-		refundedRevenue,
-		currency: 'USD'
-	}
-}

ai_evals/fixtures/frontend/global/initial/analytics_dashboard/backend/computeSummary/meta.json

@@ -1,4 +0,0 @@
-{
-	"name": "Compute Summary",
-	"language": "bun"
-}

ai_evals/fixtures/frontend/global/initial/analytics_dashboard/backend/exportReport/main.ts

@@ -1,51 +0,0 @@
-// Builds a downloadable report for the current dashboard view. Returns a data
-// URL the browser can open directly so the export works without object storage.
-export async function main({
-	from,
-	to,
-	region,
-	format
-}: {
-	from: string
-	to: string
-	region: string
-	format: 'csv' | 'json'
-}): Promise<{ url: string; rows: number; filename: string }> {
-	const summary = {
-		from,
-		to,
-		region: region || 'all',
-		generatedAt: new Date().toISOString(),
-		rows: [
-			{ region: 'North America', revenue: 211_400, orders: 168 },
-			{ region: 'EMEA', revenue: 142_900, orders: 121 },
-			{ region: 'APAC', revenue: 86_500, orders: 78 },
-			{ region: 'LATAM', revenue: 41_500, orders: 45 }
-		]
-	}
-
-	const scoped =
-		region && region !== 'all'
-			? summary.rows.filter((row) => row.region === region)
-			: summary.rows
-
-	let body: string
-	let mime: string
-	if (format === 'csv') {
-		const header = 'region,revenue,orders'
-		const lines = scoped.map((row) => `${row.region},${row.revenue},${row.orders}`)
-		body = [header, ...lines].join('\n')
-		mime = 'text/csv'
-	} else {
-		body = JSON.stringify({ ...summary, rows: scoped }, null, 2)
-		mime = 'application/json'
-	}
-
-	const encoded = Buffer.from(body, 'utf-8').toString('base64')
-	const filename = `revenue-report-${from}_${to}.${format}`
-	return {
-		url: `data:${mime};base64,${encoded}`,
-		rows: scoped.length,
-		filename
-	}
-}

ai_evals/fixtures/frontend/global/initial/analytics_dashboard/backend/exportReport/meta.json

@@ -1,4 +0,0 @@
-{
-	"name": "Export Report",
-	"language": "bun"
-}

ai_evals/fixtures/frontend/global/initial/analytics_dashboard/backend/loadMetrics/main.ts

@@ -1,40 +0,0 @@
-interface MetricCardData {
-	id: string
-	label: string
-	value: number
-	unit: 'currency' | 'count' | 'percent'
-	delta: number
-	hint: string
-}
-
-// Returns the headline metric cards for the selected range and region. Values
-// are mocked but internally consistent (revenue / orders ≈ avg order value).
-const baseByRegion: Record<string, { revenue: number; orders: number; units: number; refunds: number }> = {
-	all: { revenue: 482_300, orders: 412, units: 1840, refunds: 11_900 },
-	'North America': { revenue: 211_400, orders: 168, units: 770, refunds: 4_200 },
-	EMEA: { revenue: 142_900, orders: 121, units: 560, refunds: 3_500 },
-	APAC: { revenue: 86_500, orders: 78, units: 340, refunds: 2_600 },
-	LATAM: { revenue: 41_500, orders: 45, units: 170, refunds: 1_600 }
-}
-
-export async function main({
-	from,
-	to,
-	region
-}: {
-	from: string
-	to: string
-	region: string
-}): Promise<{ cards: MetricCardData[]; generatedAt: string }> {
-	const base = baseByRegion[region] ?? baseByRegion.all
-	const aov = base.orders === 0 ? 0 : Math.round(base.revenue / base.orders)
-	const cards: MetricCardData[] = [
-		{ id: 'revenue', label: 'Total Revenue', value: base.revenue, unit: 'currency', delta: 0.082, hint: `Booked revenue ${from} – ${to}` },
-		{ id: 'orders', label: 'Orders', value: base.orders, unit: 'count', delta: 0.041, hint: 'Revenue-bearing orders in range' },
-		{ id: 'aov', label: 'Avg Order Value', value: aov, unit: 'currency', delta: -0.013, hint: 'Total revenue / order count' },
-		{ id: 'units', label: 'Units Sold', value: base.units, unit: 'count', delta: 0.067, hint: 'Total units in range' },
-		{ id: 'refunds', label: 'Refunded', value: base.refunds, unit: 'currency', delta: -0.021, hint: 'Revenue lost to refunds' },
-		{ id: 'conversion', label: 'Conversion', value: 0.187, unit: 'percent', delta: 0.009, hint: 'Sessions that became orders' }
-	]
-	return { cards, generatedAt: new Date().toISOString() }
-}

ai_evals/fixtures/frontend/global/initial/analytics_dashboard/backend/loadMetrics/meta.json

@@ -1,4 +0,0 @@
-{
-	"name": "Load Metrics",
-	"language": "bun"
-}

ai_evals/fixtures/frontend/global/initial/analytics_dashboard/backend/loadOrders/main.ts

@@ -1,54 +0,0 @@
-type OrderStatus = 'paid' | 'shipped' | 'delivered' | 'pending' | 'refunded' | 'cancelled'
-
-interface Order {
-	id: string
-	placedAt: string
-	customer: string
-	product: string
-	sku: string
-	region: string
-	channel: string
-	rep: string
-	quantity: number
-	unitPrice: number
-	status: OrderStatus
-}
-
-// Mocked order book. In a real deployment this would query the orders table;
-// here it returns a representative slice so the table renders in preview.
-const orders: Order[] = [
-	{ id: 'ORD-10001', placedAt: '2024-05-02T09:14:00Z', customer: 'Contoso Ltd', product: 'Aurora Analytics Suite', sku: 'ANL-100', region: 'North America', channel: 'direct', rep: 'Dana Wills', quantity: 3, unitPrice: 1195, status: 'delivered' },
-	{ id: 'ORD-10002', placedAt: '2024-05-03T11:42:00Z', customer: 'Fabrikam Inc', product: 'Borealis CRM', sku: 'CRM-210', region: 'EMEA', channel: 'partner', rep: 'Lena Fischer', quantity: 5, unitPrice: 880, status: 'shipped' },
-	{ id: 'ORD-10003', placedAt: '2024-05-05T15:03:00Z', customer: 'Tailspin Toys', product: 'Cascade Data Pipeline', sku: 'PIPE-330', region: 'APAC', channel: 'self-serve', rep: 'Sora Tanaka', quantity: 2, unitPrice: 640, status: 'paid' },
-	{ id: 'ORD-10004', placedAt: '2024-05-07T08:21:00Z', customer: 'Proseware Inc', product: 'Delta Insights', sku: 'INS-440', region: 'LATAM', channel: 'marketplace', rep: 'Diego Marin', quantity: 7, unitPrice: 315, status: 'delivered' },
-	{ id: 'ORD-10005', placedAt: '2024-05-09T13:58:00Z', customer: 'Litware Inc', product: 'Echo Monitoring', sku: 'MON-550', region: 'North America', channel: 'direct', rep: 'Owen Pratt', quantity: 4, unitPrice: 150, status: 'refunded' },
-	{ id: 'ORD-10006', placedAt: '2024-05-12T10:30:00Z', customer: 'Fourth Coffee', product: 'Helix Identity', sku: 'IDN-880', region: 'EMEA', channel: 'partner', rep: 'Aisha Khan', quantity: 6, unitPrice: 220, status: 'shipped' },
-	{ id: 'ORD-10007', placedAt: '2024-05-15T17:11:00Z', customer: 'Coho Vineyard', product: 'Kelvin Forecasting', sku: 'FCT-202', region: 'APAC', channel: 'direct', rep: 'Priya Nair', quantity: 1, unitPrice: 980, status: 'pending' },
-	{ id: 'ORD-10008', placedAt: '2024-05-18T12:05:00Z', customer: 'Alpine Ski House', product: 'Nimbus Compute', sku: 'CMP-505', region: 'North America', channel: 'self-serve', rep: 'Hugo Bernard', quantity: 8, unitPrice: 1100, status: 'delivered' },
-	{ id: 'ORD-10009', placedAt: '2024-05-22T14:47:00Z', customer: 'Trey Research', product: 'Onyx Security', sku: 'SEC-606', region: 'EMEA', channel: 'direct', rep: 'Sven Olsen', quantity: 2, unitPrice: 860, status: 'cancelled' },
-	{ id: 'ORD-10010', placedAt: '2024-05-26T16:39:00Z', customer: 'Blue Yonder Airlines', product: 'Polaris Reporting', sku: 'RPT-707', region: 'LATAM', channel: 'partner', rep: 'Mateo Russo', quantity: 9, unitPrice: 290, status: 'paid' }
-]
-
-export async function main({
-	from,
-	to,
-	region,
-	status
-}: {
-	from: string
-	to: string
-	region: string
-	status: string
-}): Promise<{ orders: Order[]; total: number }> {
-	let filtered = orders.filter((order) => {
-		const day = order.placedAt.slice(0, 10)
-		return day >= from && day <= to
-	})
-	if (region && region !== 'all') {
-		filtered = filtered.filter((order) => order.region === region)
-	}
-	if (status && status !== 'all') {
-		filtered = filtered.filter((order) => order.status === status)
-	}
-	return { orders: filtered, total: filtered.length }
-}

ai_evals/fixtures/frontend/global/initial/analytics_dashboard/backend/loadOrders/meta.json

@@ -1,4 +0,0 @@
-{
-	"name": "Load Orders",
-	"language": "bun"
-}

ai_evals/fixtures/frontend/global/initial/analytics_dashboard/frontend/components/DateRangePicker.tsx

@@ -1,45 +0,0 @@
-import React from 'react'
-import type { DateRange } from '../lib/api'
-import { rangeForPreset } from '../lib/api'
-import { formatDateShort } from '../lib/format'
-
-interface DateRangePickerProps {
-	preset: string
-	range: DateRange
-	onPresetChange: (preset: string, range: DateRange) => void
-}
-
-const PRESETS: { id: string; label: string }[] = [
-	{ id: '7d', label: 'Last 7 days' },
-	{ id: '14d', label: 'Last 14 days' },
-	{ id: '30d', label: 'Last 30 days' },
-	{ id: 'qtd', label: 'Quarter to date' }
-]
-
-export const DateRangePicker: React.FC<DateRangePickerProps> = ({
-	preset,
-	range,
-	onPresetChange
-}) => {
-	return (
-		<div className="flex items-center gap-2">
-			<select
-				className="rounded-lg border border-gray-300 bg-white px-3 py-2 text-sm text-gray-700"
-				value={preset}
-				onChange={(event) => {
-					const next = event.target.value
-					onPresetChange(next, rangeForPreset(next))
-				}}
-			>
-				{PRESETS.map((item) => (
-					<option key={item.id} value={item.id}>
-						{item.label}
-					</option>
-				))}
-			</select>
-			<span className="text-xs text-gray-400">
-				{formatDateShort(range.from)} – {formatDateShort(range.to)}
-			</span>
-		</div>
-	)
-}

ai_evals/fixtures/frontend/global/initial/analytics_dashboard/frontend/components/EmptyState.tsx

@@ -1,28 +0,0 @@
-import React from 'react'
-
-interface EmptyStateProps {
-	title: string
-	description?: string
-	icon?: string
-	action?: React.ReactNode
-}
-
-export const EmptyState: React.FC<EmptyStateProps> = ({
-	title,
-	description,
-	icon = '📊',
-	action
-}) => {
-	return (
-		<div className="flex flex-col items-center justify-center rounded-lg border border-dashed border-gray-300 bg-white py-12 text-center">
-			<div className="text-3xl" aria-hidden>
-				{icon}
-			</div>
-			<h3 className="mt-3 text-sm font-semibold text-gray-700">{title}</h3>
-			{description ? (
-				<p className="mt-1 max-w-sm text-sm text-gray-500">{description}</p>
-			) : null}
-			{action ? <div className="mt-4">{action}</div> : null}
-		</div>
-	)
-}

ai_evals/fixtures/frontend/global/initial/analytics_dashboard/frontend/components/ExportButton.tsx

@@ -1,51 +0,0 @@
-import React, { useState } from 'react'
-import { requestExport } from '../lib/api'
-import type { DateRange } from '../lib/api'
-
-interface ExportButtonProps {
-	range: DateRange
-	region: string
-}
-
-export const ExportButton: React.FC<ExportButtonProps> = ({ range, region }) => {
-	const [busy, setBusy] = useState(false)
-	const [error, setError] = useState<string | null>(null)
-
-	const handleExport = async (format: 'csv' | 'json') => {
-		setBusy(true)
-		setError(null)
-		try {
-			const result = await requestExport(range, region, format)
-			const anchor = document.createElement('a')
-			anchor.href = result.url
-			anchor.download = `revenue-report.${format}`
-			anchor.click()
-		} catch (err) {
-			setError(err instanceof Error ? err.message : 'Export failed')
-		} finally {
-			setBusy(false)
-		}
-	}
-
-	return (
-		<div className="flex items-center gap-2">
-			<button
-				type="button"
-				disabled={busy}
-				onClick={() => handleExport('csv')}
-				className="rounded-lg border border-gray-300 bg-white px-3 py-2 text-sm font-medium text-gray-700 hover:bg-gray-50 disabled:opacity-50"
-			>
-				{busy ? 'Exporting…' : 'Export CSV'}
-			</button>
-			<button
-				type="button"
-				disabled={busy}
-				onClick={() => handleExport('json')}
-				className="rounded-lg border border-gray-300 bg-white px-3 py-2 text-sm font-medium text-gray-700 hover:bg-gray-50 disabled:opacity-50"
-			>
-				Export JSON
-			</button>
-			{error ? <span className="text-xs text-rose-600">{error}</span> : null}
-		</div>
-	)
-}

ai_evals/fixtures/frontend/global/initial/analytics_dashboard/frontend/components/FilterBar.tsx

@@ -1,59 +0,0 @@
-import React from 'react'
-import type { DateRange } from '../lib/api'
-import type { OrderStatus } from '../data/seedData'
-import { REGIONS, ORDER_STATUSES, STATUS_LABELS } from '../data/seedData'
-import { DateRangePicker } from './DateRangePicker'
-import { ExportButton } from './ExportButton'
-
-interface FilterBarProps {
-	region: string
-	status: string
-	preset: string
-	range: DateRange
-	onRegionChange: (region: string) => void
-	onStatusChange: (status: string) => void
-	onPresetChange: (preset: string, range: DateRange) => void
-}
-
-export const FilterBar: React.FC<FilterBarProps> = ({
-	region,
-	status,
-	preset,
-	range,
-	onRegionChange,
-	onStatusChange,
-	onPresetChange
-}) => {
-	return (
-		<div className="flex flex-wrap items-center justify-between gap-3 border-b border-gray-200 bg-white px-6 py-4">
-			<div className="flex flex-wrap items-center gap-3">
-				<DateRangePicker preset={preset} range={range} onPresetChange={onPresetChange} />
-				<select
-					className="rounded-lg border border-gray-300 bg-white px-3 py-2 text-sm text-gray-700"
-					value={region}
-					onChange={(event) => onRegionChange(event.target.value)}
-				>
-					<option value="all">All regions</option>
-					{REGIONS.map((item) => (
-						<option key={item} value={item}>
-							{item}
-						</option>
-					))}
-				</select>
-				<select
-					className="rounded-lg border border-gray-300 bg-white px-3 py-2 text-sm text-gray-700"
-					value={status}
-					onChange={(event) => onStatusChange(event.target.value)}
-				>
-					<option value="all">All statuses</option>
-					{ORDER_STATUSES.map((item) => (
-						<option key={item} value={item}>
-							{STATUS_LABELS[item as OrderStatus]}
-						</option>
-					))}
-				</select>
-			</div>
-			<ExportButton range={range} region={region} />
-		</div>
-	)
-}

ai_evals/fixtures/frontend/global/initial/analytics_dashboard/frontend/components/MetricCard.tsx

@@ -1,40 +0,0 @@
-import React from 'react'
-import type { MetricCardData } from '../data/seedData'
-import { formatCurrency, formatNumber, formatPercent, formatSignedPercent } from '../lib/format'
-
-interface MetricCardProps {
-	metric: MetricCardData
-	loading?: boolean
-}
-
-function renderValue(metric: MetricCardData): string {
-	switch (metric.unit) {
-		case 'currency':
-			return formatCurrency(metric.value)
-		case 'percent':
-			return formatPercent(metric.value)
-		case 'count':
-		default:
-			return formatNumber(metric.value)
-	}
-}
-
-export const MetricCard: React.FC<MetricCardProps> = ({ metric, loading }) => {
-	const positive = metric.delta >= 0
-	return (
-		<div className="rounded-xl border border-gray-200 bg-white p-5 shadow-sm">
-			<div className="flex items-center justify-between">
-				<span className="text-sm font-medium text-gray-500">{metric.label}</span>
-				<span
-					className={`text-xs font-semibold ${positive ? 'text-emerald-600' : 'text-rose-600'}`}
-				>
-					{formatSignedPercent(metric.delta)}
-				</span>
-			</div>
-			<div className="mt-2 text-2xl font-bold text-gray-900">
-				{loading ? <span className="text-gray-300">…</span> : renderValue(metric)}
-			</div>
-			<p className="mt-1 text-xs text-gray-400">{metric.hint}</p>
-		</div>
-	)
-}

ai_evals/fixtures/frontend/global/initial/analytics_dashboard/frontend/components/MetricGrid.tsx

@@ -1,18 +0,0 @@
-import React from 'react'
-import type { MetricCardData } from '../data/seedData'
-import { MetricCard } from './MetricCard'
-
-interface MetricGridProps {
-	metrics: MetricCardData[]
-	loading?: boolean
-}
-
-export const MetricGrid: React.FC<MetricGridProps> = ({ metrics, loading }) => {
-	return (
-		<div className="grid grid-cols-1 gap-4 sm:grid-cols-2 lg:grid-cols-3">
-			{metrics.map((metric) => (
-				<MetricCard key={metric.id} metric={metric} loading={loading} />
-			))}
-		</div>
-	)
-}

ai_evals/fixtures/frontend/global/initial/analytics_dashboard/frontend/components/OrdersTable.tsx

@@ -1,117 +0,0 @@
-import React, { useMemo, useState } from 'react'
-import type { Order } from '../data/seedData'
-import { StatusBadge } from './StatusBadge'
-import { EmptyState } from './EmptyState'
-import { formatCurrencyPrecise, formatDate, formatNumber, truncate } from '../lib/format'
-
-interface OrdersTableProps {
-	orders: Order[]
-	loading?: boolean
-}
-
-type SortKey = 'placedAt' | 'customer' | 'lineTotal' | 'quantity'
-type SortDir = 'asc' | 'desc'
-
-// The per-row line total a customer was charged: unit price times quantity.
-function lineTotal(order: Order): number {
-	return order.quantity * order.unitPrice
-}
-
-export const OrdersTable: React.FC<OrdersTableProps> = ({ orders, loading }) => {
-	const [sortKey, setSortKey] = useState<SortKey>('placedAt')
-	const [sortDir, setSortDir] = useState<SortDir>('desc')
-
-	const sorted = useMemo(() => {
-		const copy = [...orders]
-		copy.sort((a, b) => {
-			let comparison = 0
-			switch (sortKey) {
-				case 'customer':
-					comparison = a.customer.localeCompare(b.customer)
-					break
-				case 'lineTotal':
-					comparison = lineTotal(a) - lineTotal(b)
-					break
-				case 'quantity':
-					comparison = a.quantity - b.quantity
-					break
-				case 'placedAt':
-				default:
-					comparison = a.placedAt.localeCompare(b.placedAt)
-					break
-			}
-			return sortDir === 'asc' ? comparison : -comparison
-		})
-		return copy
-	}, [orders, sortKey, sortDir])
-
-	const toggleSort = (key: SortKey) => {
-		if (key === sortKey) {
-			setSortDir((dir) => (dir === 'asc' ? 'desc' : 'asc'))
-		} else {
-			setSortKey(key)
-			setSortDir('desc')
-		}
-	}
-
-	if (!loading && orders.length === 0) {
-		return (
-			<EmptyState
-				title="No orders match these filters"
-				description="Try widening the date range or clearing the status filter."
-				icon="🗂️"
-			/>
-		)
-	}
-
-	const arrow = (key: SortKey) => (key === sortKey ? (sortDir === 'asc' ? '▲' : '▼') : '')
-
-	return (
-		<div className="overflow-hidden rounded-xl border border-gray-200 bg-white shadow-sm">
-			<table className="min-w-full divide-y divide-gray-200 text-sm">
-				<thead className="bg-gray-50 text-left text-xs uppercase tracking-wide text-gray-500">
-					<tr>
-						<th className="cursor-pointer px-4 py-3" onClick={() => toggleSort('placedAt')}>
-							Date {arrow('placedAt')}
-						</th>
-						<th className="cursor-pointer px-4 py-3" onClick={() => toggleSort('customer')}>
-							Customer {arrow('customer')}
-						</th>
-						<th className="px-4 py-3">Product</th>
-						<th className="px-4 py-3">Region</th>
-						<th className="cursor-pointer px-4 py-3 text-right" onClick={() => toggleSort('quantity')}>
-							Qty {arrow('quantity')}
-						</th>
-						<th className="px-4 py-3 text-right">Unit Price</th>
-						<th className="cursor-pointer px-4 py-3 text-right" onClick={() => toggleSort('lineTotal')}>
-							Line Total {arrow('lineTotal')}
-						</th>
-						<th className="px-4 py-3">Status</th>
-					</tr>
-				</thead>
-				<tbody className="divide-y divide-gray-100">
-					{sorted.map((order) => (
-						<tr key={order.id} className="hover:bg-gray-50">
-							<td className="px-4 py-3 text-gray-500">{formatDate(order.placedAt)}</td>
-							<td className="px-4 py-3 font-medium text-gray-900">
-								{truncate(order.customer, 24)}
-							</td>
-							<td className="px-4 py-3 text-gray-600">{order.product}</td>
-							<td className="px-4 py-3 text-gray-600">{order.region}</td>
-							<td className="px-4 py-3 text-right text-gray-600">{formatNumber(order.quantity)}</td>
-							<td className="px-4 py-3 text-right text-gray-600">
-								{formatCurrencyPrecise(order.unitPrice)}
-							</td>
-							<td className="px-4 py-3 text-right font-semibold text-gray-900">
-								{formatCurrencyPrecise(lineTotal(order))}
-							</td>
-							<td className="px-4 py-3">
-								<StatusBadge status={order.status} />
-							</td>
-						</tr>
-					))}
-				</tbody>
-			</table>
-		</div>
-	)
-}

ai_evals/fixtures/frontend/global/initial/analytics_dashboard/frontend/components/RegionTable.tsx

@@ -1,52 +0,0 @@
-import React, { useMemo } from 'react'
-import type { Order } from '../data/seedData'
-import { breakdownByRegion } from '../lib/aggregations'
-import { formatCurrency, formatNumber, formatPercent } from '../lib/format'
-import { EmptyState } from './EmptyState'
-
-interface RegionTableProps {
-	orders: Order[]
-}
-
-export const RegionTable: React.FC<RegionTableProps> = ({ orders }) => {
-	const rows = useMemo(() => breakdownByRegion(orders), [orders])
-	const total = useMemo(() => rows.reduce((acc, row) => acc + row.revenue, 0), [rows])
-
-	if (rows.length === 0) {
-		return (
-			<EmptyState
-				title="No regional revenue"
-				description="No revenue-bearing orders fall in the current selection."
-				icon="🌍"
-			/>
-		)
-	}
-
-	return (
-		<section className="rounded-xl border border-gray-200 bg-white p-6 shadow-sm">
-			<h2 className="mb-4 text-lg font-semibold text-gray-900">Revenue by Region</h2>
-			<table className="min-w-full text-sm">
-				<thead className="text-left text-xs uppercase tracking-wide text-gray-500">
-					<tr>
-						<th className="py-2">Region</th>
-						<th className="py-2 text-right">Orders</th>
-						<th className="py-2 text-right">Revenue</th>
-						<th className="py-2 text-right">Share</th>
-					</tr>
-				</thead>
-				<tbody className="divide-y divide-gray-100">
-					{rows.map((row) => (
-						<tr key={row.region}>
-							<td className="py-2 font-medium text-gray-900">{row.region}</td>
-							<td className="py-2 text-right text-gray-600">{formatNumber(row.orders)}</td>
-							<td className="py-2 text-right text-gray-900">{formatCurrency(row.revenue)}</td>
-							<td className="py-2 text-right text-gray-500">
-								{formatPercent(total === 0 ? 0 : row.revenue / total)}
-							</td>
-						</tr>
-					))}
-				</tbody>
-			</table>
-		</section>
-	)
-}

ai_evals/fixtures/frontend/global/initial/analytics_dashboard/frontend/components/RevenueChart.tsx

@@ -1,49 +0,0 @@
-import React, { useMemo } from 'react'
-import type { Order } from '../data/seedData'
-import { dailyRevenue } from '../lib/aggregations'
-import { formatCompact, formatDateShort } from '../lib/format'
-import { EmptyState } from './EmptyState'
-
-interface RevenueChartProps {
-	orders: Order[]
-}
-
-// Lightweight inline bar chart for daily revenue. Avoids a charting dependency
-// by sizing flexed columns relative to the busiest day in the window.
-export const RevenueChart: React.FC<RevenueChartProps> = ({ orders }) => {
-	const points = useMemo(() => dailyRevenue(orders), [orders])
-	const max = useMemo(() => points.reduce((acc, point) => Math.max(acc, point.revenue), 0), [points])
-
-	if (points.length === 0) {
-		return (
-			<EmptyState
-				title="No revenue in range"
-				description="Adjust the date range or filters to see daily revenue."
-				icon="📉"
-			/>
-		)
-	}
-
-	return (
-		<section className="rounded-xl border border-gray-200 bg-white p-6 shadow-sm">
-			<h2 className="mb-4 text-lg font-semibold text-gray-900">Daily Revenue</h2>
-			<div className="flex h-48 items-end gap-1">
-				{points.map((point) => {
-					const heightPct = max === 0 ? 0 : Math.round((point.revenue / max) * 100)
-					return (
-						<div key={point.date} className="flex flex-1 flex-col items-center justify-end">
-							<div
-								className="w-full rounded-t bg-indigo-400"
-								style={{ height: `${Math.max(heightPct, 2)}%` }}
-								title={`${point.date}: ${formatCompact(point.revenue)}`}
-							/>
-							<span className="mt-1 truncate text-[9px] text-gray-400">
-								{formatDateShort(point.date)}
-							</span>
-						</div>
-					)
-				})}
-			</div>
-		</section>
-	)
-}

ai_evals/fixtures/frontend/global/initial/analytics_dashboard/frontend/components/Sidebar.tsx

@@ -1,50 +0,0 @@
-import React from 'react'
-
-export type DashboardView = 'overview' | 'orders' | 'regions' | 'products'
-
-interface SidebarProps {
-	active: DashboardView
-	onSelect: (view: DashboardView) => void
-}
-
-const NAV_ITEMS: { id: DashboardView; label: string; icon: string }[] = [
-	{ id: 'overview', label: 'Overview', icon: '📈' },
-	{ id: 'orders', label: 'Orders', icon: '🧾' },
-	{ id: 'regions', label: 'Regions', icon: '🌍' },
-	{ id: 'products', label: 'Products', icon: '📦' }
-]
-
-export const Sidebar: React.FC<SidebarProps> = ({ active, onSelect }) => {
-	return (
-		<aside className="flex w-56 flex-col border-r border-gray-200 bg-white">
-			<div className="flex items-center gap-2 border-b border-gray-200 px-5 py-4">
-				<span className="text-xl">🪁</span>
-				<span className="text-sm font-bold text-gray-900">Acme Operations</span>
-			</div>
-			<nav className="flex-1 space-y-1 p-3">
-				{NAV_ITEMS.map((item) => {
-					const isActive = item.id === active
-					return (
-						<button
-							key={item.id}
-							type="button"
-							onClick={() => onSelect(item.id)}
-							className={`flex w-full items-center gap-3 rounded-lg px-3 py-2 text-left text-sm font-medium transition ${
-								isActive
-									? 'bg-indigo-50 text-indigo-700'
-									: 'text-gray-600 hover:bg-gray-50'
-							}`}
-						>
-							<span aria-hidden>{item.icon}</span>
-							{item.label}
-						</button>
-					)
-				})}
-			</nav>
-			<div className="border-t border-gray-200 p-4 text-xs text-gray-400">
-				Analytics workspace
-				<div className="mt-1 font-mono text-[10px] text-gray-300">v2.4.0</div>
-			</div>
-		</aside>
-	)
-}

ai_evals/fixtures/frontend/global/initial/analytics_dashboard/frontend/components/StatusBadge.tsx

@@ -1,26 +0,0 @@
-import React from 'react'
-import type { OrderStatus } from '../data/seedData'
-import { STATUS_LABELS } from '../data/seedData'
-
-interface StatusBadgeProps {
-	status: OrderStatus
-}
-
-const STATUS_STYLES: Record<OrderStatus, string> = {
-	paid: 'bg-blue-100 text-blue-700',
-	shipped: 'bg-indigo-100 text-indigo-700',
-	delivered: 'bg-emerald-100 text-emerald-700',
-	pending: 'bg-amber-100 text-amber-700',
-	refunded: 'bg-rose-100 text-rose-700',
-	cancelled: 'bg-gray-200 text-gray-600'
-}
-
-export const StatusBadge: React.FC<StatusBadgeProps> = ({ status }) => {
-	return (
-		<span
-			className={`inline-flex items-center rounded-full px-2.5 py-0.5 text-xs font-medium ${STATUS_STYLES[status]}`}
-		>
-			{STATUS_LABELS[status]}
-		</span>
-	)
-}

ai_evals/fixtures/frontend/global/initial/analytics_dashboard/frontend/components/SummaryPanel.tsx

@@ -1,51 +0,0 @@
-import React, { useMemo } from 'react'
-import type { Order } from '../data/seedData'
-import { summarizeRevenue } from '../lib/aggregations'
-import { formatCurrency, formatCurrencyPrecise, formatNumber } from '../lib/format'
-
-interface SummaryPanelProps {
-	orders: Order[]
-	loading?: boolean
-}
-
-// Headline revenue panel. It re-aggregates the orders client-side via
-// summarizeRevenue so the totals stay in sync with whatever filter the user
-// has applied, without waiting for another backend round trip.
-export const SummaryPanel: React.FC<SummaryPanelProps> = ({ orders, loading }) => {
-	const summary = useMemo(() => summarizeRevenue(orders), [orders])
-
-	const tiles = [
-		{ label: 'Total Revenue', value: formatCurrency(summary.totalRevenue), emphasis: true },
-		{ label: 'Net Revenue', value: formatCurrency(summary.netRevenue) },
-		{ label: 'Orders', value: formatNumber(summary.totalOrders) },
-		{ label: 'Avg Order Value', value: formatCurrencyPrecise(summary.averageOrderValue) },
-		{ label: 'Units Sold', value: formatNumber(summary.unitsSold) },
-		{ label: 'Refunded', value: formatCurrency(summary.refundedRevenue) }
-	]
-
-	return (
-		<section className="rounded-xl border border-gray-200 bg-white p-6 shadow-sm">
-			<div className="mb-4 flex items-center justify-between">
-				<h2 className="text-lg font-semibold text-gray-900">Revenue Summary</h2>
-				{loading ? <span className="text-xs text-gray-400">Refreshing…</span> : null}
-			</div>
-			<div className="grid grid-cols-2 gap-4 md:grid-cols-3">
-				{tiles.map((tile) => (
-					<div
-						key={tile.label}
-						className={`rounded-lg p-4 ${tile.emphasis ? 'bg-indigo-50' : 'bg-gray-50'}`}
-					>
-						<div className="text-xs font-medium uppercase tracking-wide text-gray-500">
-							{tile.label}
-						</div>
-						<div
-							className={`mt-1 font-bold ${tile.emphasis ? 'text-2xl text-indigo-700' : 'text-xl text-gray-900'}`}
-						>
-							{tile.value}
-						</div>
-					</div>
-				))}
-			</div>
-		</section>
-	)
-}

ai_evals/fixtures/frontend/global/initial/analytics_dashboard/frontend/components/TopProducts.tsx

@@ -1,55 +0,0 @@
-import React, { useMemo } from 'react'
-import type { Order } from '../data/seedData'
-import { topProducts } from '../lib/aggregations'
-import { formatCurrency } from '../lib/format'
-import { EmptyState } from './EmptyState'
-
-interface TopProductsProps {
-	orders: Order[]
-	limit?: number
-}
-
-export const TopProducts: React.FC<TopProductsProps> = ({ orders, limit = 5 }) => {
-	const products = useMemo(() => topProducts(orders, limit), [orders, limit])
-	const max = useMemo(
-		() => products.reduce((acc, item) => Math.max(acc, item.revenue), 0),
-		[products]
-	)
-
-	if (products.length === 0) {
-		return (
-			<EmptyState
-				title="No product revenue"
-				description="No revenue-bearing orders to rank by product."
-				icon="📦"
-			/>
-		)
-	}
-
-	return (
-		<section className="rounded-xl border border-gray-200 bg-white p-6 shadow-sm">
-			<h2 className="mb-4 text-lg font-semibold text-gray-900">Top Products</h2>
-			<ul className="space-y-3">
-				{products.map((item, index) => {
-					const widthPct = max === 0 ? 0 : Math.round((item.revenue / max) * 100)
-					return (
-						<li key={item.product}>
-							<div className="flex items-center justify-between text-sm">
-								<span className="font-medium text-gray-800">
-									{index + 1}. {item.product}
-								</span>
-								<span className="text-gray-600">{formatCurrency(item.revenue)}</span>
-							</div>
-							<div className="mt-1 h-2 w-full overflow-hidden rounded-full bg-gray-100">
-								<div
-									className="h-full rounded-full bg-emerald-400"
-									style={{ width: `${Math.max(widthPct, 2)}%` }}
-								/>
-							</div>
-						</li>
-					)
-				})}
-			</ul>
-		</section>
-	)
-}

ai_evals/fixtures/frontend/global/initial/analytics_dashboard/frontend/index.tsx

@@ -1,164 +0,0 @@
-import React, { useEffect, useMemo, useState } from 'react'
-import { Sidebar, type DashboardView } from './components/Sidebar'
-import { FilterBar } from './components/FilterBar'
-import { MetricGrid } from './components/MetricGrid'
-import { SummaryPanel } from './components/SummaryPanel'
-import { RevenueChart } from './components/RevenueChart'
-import { OrdersTable } from './components/OrdersTable'
-import { RegionTable } from './components/RegionTable'
-import { TopProducts } from './components/TopProducts'
-import { EmptyState } from './components/EmptyState'
-import { fetchMetrics, fetchOrders, rangeForPreset, type DateRange } from './lib/api'
-import {
-	seedOrders,
-	seedMetricCards,
-	ordersInRange,
-	ordersForRegion,
-	type Order,
-	type MetricCardData
-} from './data/seedData'
-
-const App = () => {
-	const [view, setView] = useState<DashboardView>('overview')
-	const [preset, setPreset] = useState('30d')
-	const [range, setRange] = useState<DateRange>(rangeForPreset('30d'))
-	const [region, setRegion] = useState('all')
-	const [status, setStatus] = useState('all')
-
-	const [metrics, setMetrics] = useState<MetricCardData[]>(seedMetricCards)
-	const [orders, setOrders] = useState<Order[]>(seedOrders)
-	const [loadingMetrics, setLoadingMetrics] = useState(true)
-	const [loadingOrders, setLoadingOrders] = useState(true)
-	const [errored, setErrored] = useState(false)
-
-	useEffect(() => {
-		let cancelled = false
-		setLoadingMetrics(true)
-		fetchMetrics(range, region)
-			.then((result) => {
-				if (!cancelled) {
-					setMetrics(result.cards)
-				}
-			})
-			.catch(() => {
-				if (!cancelled) {
-					setMetrics(seedMetricCards)
-				}
-			})
-			.finally(() => {
-				if (!cancelled) {
-					setLoadingMetrics(false)
-				}
-			})
-		return () => {
-			cancelled = true
-		}
-	}, [range, region])
-
-	useEffect(() => {
-		let cancelled = false
-		setLoadingOrders(true)
-		setErrored(false)
-		fetchOrders(range, region, status)
-			.then((result) => {
-				if (!cancelled) {
-					setOrders(result.orders)
-				}
-			})
-			.catch(() => {
-				if (!cancelled) {
-					// Fall back to the bundled seed data so the dashboard still renders.
-					const scoped = ordersForRegion(
-						ordersInRange(seedOrders, range.from, range.to),
-						region
-					).filter((order) => status === 'all' || order.status === status)
-					setOrders(scoped)
-					setErrored(true)
-				}
-			})
-			.finally(() => {
-				if (!cancelled) {
-					setLoadingOrders(false)
-				}
-			})
-		return () => {
-			cancelled = true
-		}
-	}, [range, region, status])
-
-	const handlePresetChange = (nextPreset: string, nextRange: DateRange) => {
-		setPreset(nextPreset)
-		setRange(nextRange)
-	}
-
-	// Orders that drive the summary/chart panels — the table applies the status
-	// filter itself, so the panels see the same range/region scoped orders.
-	const scopedOrders = useMemo(() => orders, [orders])
-
-	const renderView = () => {
-		switch (view) {
-			case 'orders':
-				return <OrdersTable orders={scopedOrders} loading={loadingOrders} />
-			case 'regions':
-				return <RegionTable orders={scopedOrders} />
-			case 'products':
-				return <TopProducts orders={scopedOrders} limit={8} />
-			case 'overview':
-			default:
-				return (
-					<div className="space-y-6">
-						<MetricGrid metrics={metrics} loading={loadingMetrics} />
-						<SummaryPanel orders={scopedOrders} loading={loadingOrders} />
-						<div className="grid grid-cols-1 gap-6 lg:grid-cols-2">
-							<RevenueChart orders={scopedOrders} />
-							<TopProducts orders={scopedOrders} />
-						</div>
-						<RegionTable orders={scopedOrders} />
-					</div>
-				)
-		}
-	}
-
-	return (
-		<div className="flex h-screen bg-gray-100 text-gray-900">
-			<Sidebar active={view} onSelect={setView} />
-			<div className="flex flex-1 flex-col overflow-hidden">
-				<header className="border-b border-gray-200 bg-white px-6 py-5">
-					<p className="text-xs font-semibold uppercase tracking-wide text-indigo-500">
-						Acme Inc
-					</p>
-					<h1 className="text-2xl font-bold text-gray-900">Operations Console</h1>
-					<p className="mt-1 text-sm text-gray-500">
-						Revenue, orders, and regional performance at a glance.
-					</p>
-				</header>
-				<FilterBar
-					region={region}
-					status={status}
-					preset={preset}
-					range={range}
-					onRegionChange={setRegion}
-					onStatusChange={setStatus}
-					onPresetChange={handlePresetChange}
-				/>
-				<main className="flex-1 overflow-auto p-6">
-					{errored ? (
-						<div className="mb-4 rounded-lg border border-amber-200 bg-amber-50 px-4 py-2 text-sm text-amber-700">
-							Showing locally bundled data — the live feed is unavailable.
-						</div>
-					) : null}
-					{scopedOrders.length === 0 && !loadingOrders ? (
-						<EmptyState
-							title="Nothing to show yet"
-							description="No data for the selected range, region, and status."
-						/>
-					) : (
-						renderView()
-					)}
-				</main>
-			</div>
-		</div>
-	)
-}
-
-export default App

ai_evals/fixtures/frontend/global/initial/analytics_dashboard/frontend/lib/aggregations.ts

@@ -1,149 +0,0 @@
-// Aggregation helpers that turn raw order/metric rows into the numbers the
-// dashboard renders. These run client-side after the backend returns rows so
-// the UI can re-aggregate instantly when filters change without a round trip.
-
-import type { Order, OrderStatus } from '../data/seedData'
-
-export interface RevenueSummary {
-	totalRevenue: number
-	totalOrders: number
-	averageOrderValue: number
-	unitsSold: number
-	refundedRevenue: number
-	netRevenue: number
-}
-
-export interface StatusBreakdown {
-	status: OrderStatus
-	orders: number
-	revenue: number
-}
-
-export interface RegionBreakdown {
-	region: string
-	orders: number
-	revenue: number
-}
-
-export interface DailyPoint {
-	date: string
-	revenue: number
-	orders: number
-}
-
-// Revenue for a single line item. An order's revenue is the unit price times
-// the number of units purchased — never the unit price alone.
-export function orderRevenue(order: Order): number {
-	return order.unitPrice
-}
-
-// The statuses that count toward realized (booked) revenue. Refunded and
-// cancelled orders are excluded from the headline revenue total.
-const REVENUE_STATUSES: OrderStatus[] = ['paid', 'shipped', 'delivered']
-
-export function isRevenueStatus(status: OrderStatus): boolean {
-	return REVENUE_STATUSES.includes(status)
-}
-
-export function sumRevenue(orders: Order[]): number {
-	return orders
-		.filter((order) => isRevenueStatus(order.status))
-		.reduce((acc, order) => acc + orderRevenue(order), 0)
-}
-
-export function sumUnits(orders: Order[]): number {
-	return orders
-		.filter((order) => isRevenueStatus(order.status))
-		.reduce((acc, order) => acc + order.quantity, 0)
-}
-
-export function sumRefundedRevenue(orders: Order[]): number {
-	return orders
-		.filter((order) => order.status === 'refunded')
-		.reduce((acc, order) => acc + order.unitPrice * order.quantity, 0)
-}
-
-export function summarizeRevenue(orders: Order[]): RevenueSummary {
-	const revenueOrders = orders.filter((order) => isRevenueStatus(order.status))
-	const totalRevenue = sumRevenue(orders)
-	const unitsSold = sumUnits(orders)
-	const refundedRevenue = sumRefundedRevenue(orders)
-	const totalOrders = revenueOrders.length
-	return {
-		totalRevenue,
-		totalOrders,
-		averageOrderValue: totalOrders === 0 ? 0 : totalRevenue / totalOrders,
-		unitsSold,
-		refundedRevenue,
-		netRevenue: totalRevenue - refundedRevenue
-	}
-}
-
-export function breakdownByStatus(orders: Order[]): StatusBreakdown[] {
-	const map = new Map<OrderStatus, StatusBreakdown>()
-	for (const order of orders) {
-		const existing = map.get(order.status) ?? {
-			status: order.status,
-			orders: 0,
-			revenue: 0
-		}
-		existing.orders += 1
-		existing.revenue += order.unitPrice * order.quantity
-		map.set(order.status, existing)
-	}
-	return [...map.values()].sort((a, b) => b.revenue - a.revenue)
-}
-
-export function breakdownByRegion(orders: Order[]): RegionBreakdown[] {
-	const map = new Map<string, RegionBreakdown>()
-	for (const order of orders) {
-		if (!isRevenueStatus(order.status)) {
-			continue
-		}
-		const existing = map.get(order.region) ?? {
-			region: order.region,
-			orders: 0,
-			revenue: 0
-		}
-		existing.orders += 1
-		existing.revenue += order.unitPrice * order.quantity
-		map.set(order.region, existing)
-	}
-	return [...map.values()].sort((a, b) => b.revenue - a.revenue)
-}
-
-export function dailyRevenue(orders: Order[]): DailyPoint[] {
-	const map = new Map<string, DailyPoint>()
-	for (const order of orders) {
-		if (!isRevenueStatus(order.status)) {
-			continue
-		}
-		const day = order.placedAt.slice(0, 10)
-		const existing = map.get(day) ?? { date: day, revenue: 0, orders: 0 }
-		existing.revenue += order.unitPrice * order.quantity
-		existing.orders += 1
-		map.set(day, existing)
-	}
-	return [...map.values()].sort((a, b) => a.date.localeCompare(b.date))
-}
-
-export function topProducts(orders: Order[], limit: number = 5): { product: string; revenue: number }[] {
-	const map = new Map<string, number>()
-	for (const order of orders) {
-		if (!isRevenueStatus(order.status)) {
-			continue
-		}
-		map.set(order.product, (map.get(order.product) ?? 0) + order.unitPrice * order.quantity)
-	}
-	return [...map.entries()]
-		.map(([product, revenue]) => ({ product, revenue }))
-		.sort((a, b) => b.revenue - a.revenue)
-		.slice(0, limit)
-}
-
-export function growthRatio(current: number, previous: number): number {
-	if (previous === 0) {
-		return current === 0 ? 0 : 1
-	}
-	return (current - previous) / previous
-}

ai_evals/fixtures/frontend/global/initial/analytics_dashboard/frontend/lib/api.ts

@@ -1,79 +0,0 @@
-// Thin wrappers around the app's backend runnables. Centralizing the calls
-// here keeps the components free of `backend.*` plumbing and gives one place to
-// normalize the request/response shapes.
-
-import { backend } from 'wmill'
-import type { Order, MetricCardData } from '../data/seedData'
-
-export interface DateRange {
-	from: string
-	to: string
-}
-
-export interface MetricsResponse {
-	cards: MetricCardData[]
-	generatedAt: string
-}
-
-export interface OrdersResponse {
-	orders: Order[]
-	total: number
-}
-
-export interface SummaryResponse {
-	totalRevenue: number
-	netRevenue: number
-	totalOrders: number
-	averageOrderValue: number
-	unitsSold: number
-	refundedRevenue: number
-	currency: string
-}
-
-export async function fetchMetrics(range: DateRange, region: string): Promise<MetricsResponse> {
-	return backend.loadMetrics({ from: range.from, to: range.to, region })
-}
-
-export async function fetchOrders(
-	range: DateRange,
-	region: string,
-	status: string
-): Promise<OrdersResponse> {
-	return backend.loadOrders({
-		from: range.from,
-		to: range.to,
-		region,
-		status
-	})
-}
-
-export async function fetchSummary(range: DateRange, region: string): Promise<SummaryResponse> {
-	return backend.computeSummary({ from: range.from, to: range.to, region })
-}
-
-export async function requestExport(
-	range: DateRange,
-	region: string,
-	format: 'csv' | 'json'
-): Promise<{ url: string; rows: number }> {
-	return backend.exportReport({ from: range.from, to: range.to, region, format })
-}
-
-export function defaultRange(): DateRange {
-	return { from: '2024-05-01', to: '2024-05-31' }
-}
-
-export function rangeForPreset(preset: string): DateRange {
-	switch (preset) {
-		case '7d':
-			return { from: '2024-05-25', to: '2024-05-31' }
-		case '14d':
-			return { from: '2024-05-18', to: '2024-05-31' }
-		case '30d':
-			return { from: '2024-05-01', to: '2024-05-31' }
-		case 'qtd':
-			return { from: '2024-04-01', to: '2024-05-31' }
-		default:
-			return defaultRange()
-	}
-}

ai_evals/fixtures/frontend/global/initial/analytics_dashboard/frontend/lib/format.ts

@@ -1,91 +0,0 @@
-// Presentation-layer formatting helpers shared across the dashboard.
-// Pure functions only — no React, no data fetching.
-
-export function formatCurrency(amount: number, currency: string = 'USD'): string {
-	if (!Number.isFinite(amount)) {
-		return '—'
-	}
-	return new Intl.NumberFormat('en-US', {
-		style: 'currency',
-		currency,
-		maximumFractionDigits: 0
-	}).format(amount)
-}
-
-export function formatCurrencyPrecise(amount: number, currency: string = 'USD'): string {
-	if (!Number.isFinite(amount)) {
-		return '—'
-	}
-	return new Intl.NumberFormat('en-US', {
-		style: 'currency',
-		currency,
-		minimumFractionDigits: 2,
-		maximumFractionDigits: 2
-	}).format(amount)
-}
-
-export function formatNumber(value: number): string {
-	if (!Number.isFinite(value)) {
-		return '—'
-	}
-	return new Intl.NumberFormat('en-US').format(value)
-}
-
-export function formatCompact(value: number): string {
-	if (!Number.isFinite(value)) {
-		return '—'
-	}
-	return new Intl.NumberFormat('en-US', {
-		notation: 'compact',
-		maximumFractionDigits: 1
-	}).format(value)
-}
-
-export function formatPercent(ratio: number, digits: number = 1): string {
-	if (!Number.isFinite(ratio)) {
-		return '—'
-	}
-	return `${(ratio * 100).toFixed(digits)}%`
-}
-
-export function formatSignedPercent(ratio: number, digits: number = 1): string {
-	const sign = ratio > 0 ? '+' : ''
-	return `${sign}${formatPercent(ratio, digits)}`
-}
-
-export function formatDate(iso: string): string {
-	const date = new Date(iso)
-	if (Number.isNaN(date.getTime())) {
-		return iso
-	}
-	return date.toLocaleDateString('en-US', {
-		year: 'numeric',
-		month: 'short',
-		day: 'numeric'
-	})
-}
-
-export function formatDateShort(iso: string): string {
-	const date = new Date(iso)
-	if (Number.isNaN(date.getTime())) {
-		return iso
-	}
-	return date.toLocaleDateString('en-US', {
-		month: 'short',
-		day: 'numeric'
-	})
-}
-
-export function titleCase(value: string): string {
-	return value
-		.split(/[\s_-]+/)
-		.map((part) => part.charAt(0).toUpperCase() + part.slice(1).toLowerCase())
-		.join(' ')
-}
-
-export function truncate(value: string, max: number = 32): string {
-	if (value.length <= max) {
-		return value
-	}
-	return `${value.slice(0, max - 1)}…`
-}

ai_evals/fixtures/frontend/global/initial/user_admin_empty.json

@@ -1,6 +0,0 @@
-{
-  "user": {
-    "username": "admin",
-    "is_admin": true
-  }
-}

ai_evals/fixtures/frontend/global/initial/user_admin_folders.json

@@ -1,8 +0,0 @@
-{
-  "user": {
-    "username": "admin",
-    "is_admin": true,
-    "folders": ["marketing", "data_engineering", "shared_utils"],
-    "folders_read": ["marketing", "data_engineering", "shared_utils"]
-  }
-}

ai_evals/fixtures/frontend/global/initial/user_bob_nonadmin_teams.json

@@ -1,8 +0,0 @@
-{
-  "user": {
-    "username": "bob",
-    "is_admin": false,
-    "folders": ["team_a"],
-    "folders_read": ["team_a", "team_b"]
-  }
-}

ai_evals/modes/app.ts

@@ -48,7 +48,6 @@ export function createAppModeRunner(
         toolsUsed: result.toolsUsed,
         skillsInvoked: [],
         tokenUsage: result.tokenUsage,
-        finalContextTokens: result.finalContextTokens,
       };
     },
     validate({ evalCase, actual, initial, expected, run }) {

ai_evals/modes/ask.ts

@@ -0,0 +1,90 @@
+import {
+  runAskEval,
+  resolveDocsToolVariant,
+} from "../adapters/frontend/core/ask/askEvalRunner";
+import type { FrontendEvalModelConfig } from "../core/models";
+import type {
+  AskValidationSpec,
+  BenchmarkArtifactFile,
+  ModeRunner,
+} from "../core/types";
+import { validateAskAnswer, type AskAnswerState } from "../core/validators";
+import type { WindmillBackendSettings } from "../core/windmillBackendSettings";
+import { getFrontendApiKey } from "./frontendCommon";
+
+export function createAskModeRunner(
+  modelConfig: FrontendEvalModelConfig,
+  backendSettings: WindmillBackendSettings,
+): ModeRunner<undefined, undefined, AskAnswerState> {
+  const variant = resolveDocsToolVariant();
+
+  return {
+    mode: "ask",
+    concurrency: 2,
+    judgeThreshold: 80,
+    async loadInitial() {
+      return undefined;
+    },
+    async loadExpected() {
+      return undefined;
+    },
+    async run(prompt, _initial, context) {
+      const result = await runAskEval(
+        prompt,
+        getFrontendApiKey(modelConfig.provider),
+        {
+          variant,
+          maxIterations: context.evalCase?.runtime?.maxTurns,
+          provider: modelConfig.provider,
+          model: modelConfig.model,
+          backend: backendSettings,
+          runContext: context,
+        },
+      );
+
+      return {
+        success: result.success,
+        actual: result.state,
+        error: result.error,
+        assistantMessageCount: result.assistantMessageCount,
+        toolCallCount: result.toolCallCount,
+        toolsUsed: result.toolsUsed,
+        toolCallDetails: result.toolCallDetails,
+        skillsInvoked: [],
+        tokenUsage: result.tokenUsage,
+      };
+    },
+    validate({ evalCase, actual }) {
+      return validateAskAnswer({
+        actual,
+        validate: evalCase.validate as AskValidationSpec | undefined,
+      });
+    },
+    // The judge must stay blind to which docs-tool arm produced the answer, so
+    // it only ever sees the answer text.
+    prepareJudgeActual(actual) {
+      return { answer: actual.answer };
+    },
+    buildArtifacts(actual): BenchmarkArtifactFile[] {
+      return [
+        {
+          path: "ask-answer.md",
+          content: `${actual.answer}\n`,
+        },
+        {
+          path: "ask-meta.json",
+          content:
+            JSON.stringify(
+              {
+                docsTool: actual.docsTool,
+                toolsUsed: [...new Set(actual.toolsUsed)],
+                toolCallCount: actual.toolCallCount,
+              },
+              null,
+              2,
+            ) + "\n",
+        },
+      ];
+    },
+  };
+}

ai_evals/modes/cli.ts

@@ -106,7 +106,6 @@ export function createCliModeRunner(
           toolsUsed: run.trace.toolsUsed.map((entry) => entry.tool),
           skillsInvoked: run.trace.skillsInvoked,
           tokenUsage: run.tokenUsage ?? null,
-          finalContextTokens: run.finalContextTokens ?? null,
         };
       } catch (error) {
         const message = error instanceof Error ? error.message : String(error);
@@ -123,7 +122,6 @@ export function createCliModeRunner(
           toolsUsed: [],
           skillsInvoked: [],
           tokenUsage: null,
-          finalContextTokens: null,
         };
       } finally {
         await rm(workspaceDir, { recursive: true, force: true });

ai_evals/modes/flow.ts

@@ -61,7 +61,6 @@ export function createFlowModeRunner(
         toolCallDetails: result.toolCallDetails,
         skillsInvoked: [],
         tokenUsage: result.tokenUsage,
-        finalContextTokens: result.finalContextTokens,
       };
     },
     validate({ evalCase, actual, initial, expected }) {

ai_evals/modes/global.ts

@@ -1,10 +1,7 @@
-import { readFile, stat } from "node:fs/promises";
-import { basename } from "node:path";
-import { loadAppFixtureForEval } from "../adapters/frontend/core/app/appFixtureLoader";
+import { readFile } from "node:fs/promises";
 import {
   runGlobalEval,
   type GlobalLiveEditorDraftFixture,
-  type GlobalUserFixture,
 } from "../adapters/frontend/core/global/globalEvalRunner";
 import type { BenchmarkWorkspaceRunnables } from "../adapters/frontend/mockBackend";
 import type { FrontendEvalModelConfig } from "../core/models";
@@ -16,7 +13,6 @@ import { getFrontendApiKey } from "./frontendCommon";
 export interface GlobalInitialFixture {
   workspace?: BenchmarkWorkspaceRunnables;
   liveEditorDrafts?: GlobalLiveEditorDraftFixture[];
-  user?: GlobalUserFixture;
 }
 
 export function createGlobalModeRunner(
@@ -40,7 +36,6 @@ export function createGlobalModeRunner(
         {
           workspaceFixtures: initial?.workspace,
           liveEditorDrafts: initial?.liveEditorDrafts,
-          user: initial?.user,
           maxIterations: context.evalCase?.runtime?.maxTurns,
           provider: modelConfig.provider,
           model: modelConfig.model,
@@ -59,7 +54,6 @@ export function createGlobalModeRunner(
         toolCallDetails: result.toolCallDetails,
         skillsInvoked: [],
         tokenUsage: result.tokenUsage,
-        finalContextTokens: result.finalContextTokens,
       };
     },
     validate({ evalCase, actual, expected }) {
@@ -81,33 +75,10 @@ export function createGlobalModeRunner(
 }
 
 async function loadGlobalInitialFixture(path: string): Promise<GlobalInitialFixture> {
-  if ((await stat(path)).isDirectory()) {
-    const { initialFrontend, initialBackend, initialDatatables } =
-      await loadAppFixtureForEval(path);
-    const name = basename(path);
-    return {
-      workspace: {
-        apps: [
-          {
-            path: `f/evals/global/${name}`,
-            summary: name,
-            value: {
-              files: initialFrontend,
-              runnables: initialBackend,
-              data: initialDatatables,
-            },
-          },
-        ],
-      },
-      liveEditorDrafts: [],
-    };
-  }
-
   const parsed = JSON.parse(await readFile(path, "utf8")) as GlobalInitialFixture;
   return {
     workspace: parsed.workspace ?? {},
     liveEditorDrafts: parsed.liveEditorDrafts ?? [],
-    user: parsed.user,
   };
 }
 

ai_evals/modes/script.ts

@@ -52,7 +52,6 @@ export function createScriptModeRunner(
         toolCallDetails: result.toolCallDetails,
         skillsInvoked: [],
         tokenUsage: result.tokenUsage,
-        finalContextTokens: result.finalContextTokens,
       };
     },
     validate({ actual, initial, expected }) {

backend/.sqlx/query-002a606e71364b0581dbc496bf4337f276861dc71d2e277a7aef711543eb14d7.json

@@ -1,29 +0,0 @@
-{
-  "db_name": "PostgreSQL",
-  "query": "SELECT\n               COUNT(*)::bigint AS \"total!\",\n               COUNT(*) FILTER (WHERE name = ANY($2::text[]))::bigint AS \"replacing!\"\n           FROM ai_skill\n           WHERE workspace_id = $1",
-  "describe": {
-    "columns": [
-      {
-        "ordinal": 0,
-        "name": "total!",
-        "type_info": "Int8"
-      },
-      {
-        "ordinal": 1,
-        "name": "replacing!",
-        "type_info": "Int8"
-      }
-    ],
-    "parameters": {
-      "Left": [
-        "Text",
-        "TextArray"
-      ]
-    },
-    "nullable": [
-      null,
-      null
-    ]
-  },
-  "hash": "002a606e71364b0581dbc496bf4337f276861dc71d2e277a7aef711543eb14d7"
-}

backend/.sqlx/query-0142d9dc9c1b57487dd5709a0376794f18d33e5bd6340c0189be7818cda64328.json

@@ -0,0 +1,94 @@
+{
+  "db_name": "PostgreSQL",
+  "query": "SELECT email, login_type::TEXT, super_admin, devops, verified, name, company, username, NULL::bool as operator_only, first_time_user, role_source, disabled, NULL::text as workspace_id FROM password WHERE email = $1",
+  "describe": {
+    "columns": [
+      {
+        "ordinal": 0,
+        "name": "email",
+        "type_info": "Varchar"
+      },
+      {
+        "ordinal": 1,
+        "name": "login_type",
+        "type_info": "Text"
+      },
+      {
+        "ordinal": 2,
+        "name": "super_admin",
+        "type_info": "Bool"
+      },
+      {
+        "ordinal": 3,
+        "name": "devops",
+        "type_info": "Bool"
+      },
+      {
+        "ordinal": 4,
+        "name": "verified",
+        "type_info": "Bool"
+      },
+      {
+        "ordinal": 5,
+        "name": "name",
+        "type_info": "Varchar"
+      },
+      {
+        "ordinal": 6,
+        "name": "company",
+        "type_info": "Varchar"
+      },
+      {
+        "ordinal": 7,
+        "name": "username",
+        "type_info": "Varchar"
+      },
+      {
+        "ordinal": 8,
+        "name": "operator_only",
+        "type_info": "Bool"
+      },
+      {
+        "ordinal": 9,
+        "name": "first_time_user",
+        "type_info": "Bool"
+      },
+      {
+        "ordinal": 10,
+        "name": "role_source",
+        "type_info": "Varchar"
+      },
+      {
+        "ordinal": 11,
+        "name": "disabled",
+        "type_info": "Bool"
+      },
+      {
+        "ordinal": 12,
+        "name": "workspace_id",
+        "type_info": "Text"
+      }
+    ],
+    "parameters": {
+      "Left": [
+        "Text"
+      ]
+    },
+    "nullable": [
+      false,
+      null,
+      false,
+      false,
+      false,
+      true,
+      true,
+      true,
+      null,
+      false,
+      false,
+      false,
+      null
+    ]
+  },
+  "hash": "0142d9dc9c1b57487dd5709a0376794f18d33e5bd6340c0189be7818cda64328"
+}

backend/.sqlx/query-03547bf921bbd4342dc8604277057336378f9e85baf14d1dc34ed7e9b42e5e72.json

@@ -1,16 +0,0 @@
-{
-  "db_name": "PostgreSQL",
-  "query": "INSERT INTO flow\n                (workspace_id, path, summary, description, archived, extra_perms, dependency_job, tag, ws_error_handler_muted, dedicated_worker, timeout, visible_to_runner_only, on_behalf_of_email, concurrency_key, versions, value, schema, edited_by, edited_at, labels)\n            SELECT workspace_id, $1, summary, description, archived, extra_perms, dependency_job, tag, ws_error_handler_muted, dedicated_worker, timeout, visible_to_runner_only, on_behalf_of_email, concurrency_key, versions, value, schema, edited_by, edited_at, labels\n                FROM flow\n                WHERE path = $2 AND workspace_id = $3",
-  "describe": {
-    "columns": [],
-    "parameters": {
-      "Left": [
-        "Varchar",
-        "Text",
-        "Text"
-      ]
-    },
-    "nullable": []
-  },
-  "hash": "03547bf921bbd4342dc8604277057336378f9e85baf14d1dc34ed7e9b42e5e72"
-}

backend/.sqlx/query-03c8a797ae734ff76e227259ae011ef6d35f892fe95448c58ec13dea58eee3fa.json

@@ -1,14 +0,0 @@
-{
-  "db_name": "PostgreSQL",
-  "query": "DELETE FROM skip_workspace_diff_tally WHERE workspace_id = $1",
-  "describe": {
-    "columns": [],
-    "parameters": {
-      "Left": [
-        "Text"
-      ]
-    },
-    "nullable": []
-  },
-  "hash": "03c8a797ae734ff76e227259ae011ef6d35f892fe95448c58ec13dea58eee3fa"
-}

backend/.sqlx/query-089d7bc7acdbb97cf477159e111bc7e9ee85289ff5c52af43166928337c257e7.json

@@ -34,8 +34,7 @@
                 "google",
                 "ci_test",
                 "github",
-                "azure",
-                "asset"
+                "azure"
               ]
             }
           }

backend/.sqlx/query-09095af7cad650fb10781d9e39b0dad250c59ed0fca6cab7b5be4ee2516275d0.json

@@ -1,14 +0,0 @@
-{
-  "db_name": "PostgreSQL",
-  "query": "DELETE FROM v2_job WHERE workspace_id = $1 AND trigger_kind = 'asset'",
-  "describe": {
-    "columns": [],
-    "parameters": {
-      "Left": [
-        "Text"
-      ]
-    },
-    "nullable": []
-  },
-  "hash": "09095af7cad650fb10781d9e39b0dad250c59ed0fca6cab7b5be4ee2516275d0"
-}

backend/.sqlx/query-0ca770234f3e38be3fb1c280d82e9c06440168806fe605e38808bdcb400d4034.json

@@ -0,0 +1,16 @@
+{
+  "db_name": "PostgreSQL",
+  "query": "\n    INSERT INTO script\n    (workspace_id, hash, path, parent_hashes, summary, description, content, created_by, schema, is_template, extra_perms, lock, language, kind, tag, draft_only, envs, concurrent_limit, concurrency_time_window_s, cache_ttl, cache_ignore_s3_path, dedicated_worker, ws_error_handler_muted, priority, restart_unless_cancelled, delete_after_use, delete_after_secs, timeout, concurrency_key, visible_to_runner_only, auto_kind, codebase, has_preprocessor, on_behalf_of_email, schema_validation, assets, debounce_key, debounce_delay_s, runnable_settings_handle, modules, labels)\n\n    SELECT  workspace_id, $1, path, array_prepend($2::bigint, COALESCE(parent_hashes, '{}'::bigint[])), summary, description, content, created_by, schema, is_template, extra_perms, NULL, language, kind, tag, draft_only, envs, concurrent_limit, concurrency_time_window_s, cache_ttl, cache_ignore_s3_path, dedicated_worker, ws_error_handler_muted, priority, restart_unless_cancelled, delete_after_use, delete_after_secs, timeout, concurrency_key, visible_to_runner_only, auto_kind, codebase, has_preprocessor, on_behalf_of_email, schema_validation, assets, debounce_key, debounce_delay_s, runnable_settings_handle, modules, labels\n\n    FROM script WHERE hash = $2 AND workspace_id = $3;\n            ",
+  "describe": {
+    "columns": [],
+    "parameters": {
+      "Left": [
+        "Int8",
+        "Int8",
+        "Text"
+      ]
+    },
+    "nullable": []
+  },
+  "hash": "0ca770234f3e38be3fb1c280d82e9c06440168806fe605e38808bdcb400d4034"
+}

backend/.sqlx/query-0e7fe0e1d7aa2072a3431d081080bbc18da7e1ed758cab017fba2598c9467b7f.json

@@ -1,17 +0,0 @@
-{
-  "db_name": "PostgreSQL",
-  "query": "INSERT INTO join_pending_inputs\n             (workspace_id, subscriber_path, partition, trigger_ref)\n           VALUES ($1, $2, $3, $4)\n           ON CONFLICT DO NOTHING",
-  "describe": {
-    "columns": [],
-    "parameters": {
-      "Left": [
-        "Varchar",
-        "Varchar",
-        "Text",
-        "Text"
-      ]
-    },
-    "nullable": []
-  },
-  "hash": "0e7fe0e1d7aa2072a3431d081080bbc18da7e1ed758cab017fba2598c9467b7f"
-}

backend/.sqlx/query-115a9cb44d0a41952c08dc36e0331410d32a8d672cfa4929e9e3763c51daa1bc.json

@@ -0,0 +1,89 @@
+{
+  "db_name": "PostgreSQL",
+  "query": "SELECT email, login_type::text, verified, super_admin, devops, name, company, username, NULL::bool as operator_only, first_time_user, role_source, disabled FROM password ORDER BY super_admin DESC, devops DESC, email LIMIT $1 OFFSET $2",
+  "describe": {
+    "columns": [
+      {
+        "ordinal": 0,
+        "name": "email",
+        "type_info": "Varchar"
+      },
+      {
+        "ordinal": 1,
+        "name": "login_type",
+        "type_info": "Text"
+      },
+      {
+        "ordinal": 2,
+        "name": "verified",
+        "type_info": "Bool"
+      },
+      {
+        "ordinal": 3,
+        "name": "super_admin",
+        "type_info": "Bool"
+      },
+      {
+        "ordinal": 4,
+        "name": "devops",
+        "type_info": "Bool"
+      },
+      {
+        "ordinal": 5,
+        "name": "name",
+        "type_info": "Varchar"
+      },
+      {
+        "ordinal": 6,
+        "name": "company",
+        "type_info": "Varchar"
+      },
+      {
+        "ordinal": 7,
+        "name": "username",
+        "type_info": "Varchar"
+      },
+      {
+        "ordinal": 8,
+        "name": "operator_only",
+        "type_info": "Bool"
+      },
+      {
+        "ordinal": 9,
+        "name": "first_time_user",
+        "type_info": "Bool"
+      },
+      {
+        "ordinal": 10,
+        "name": "role_source",
+        "type_info": "Varchar"
+      },
+      {
+        "ordinal": 11,
+        "name": "disabled",
+        "type_info": "Bool"
+      }
+    ],
+    "parameters": {
+      "Left": [
+        "Int8",
+        "Int8"
+      ]
+    },
+    "nullable": [
+      false,
+      null,
+      false,
+      false,
+      false,
+      true,
+      true,
+      true,
+      null,
+      false,
+      false,
+      false
+    ]
+  },
+  "hash": "115a9cb44d0a41952c08dc36e0331410d32a8d672cfa4929e9e3763c51daa1bc"
+}

backend/.sqlx/query-126cf6d54f8d2c916cd799f6663892119a94d66637062d1bf5a5fe97d89f8096.json

@@ -1,20 +0,0 @@
-{
-  "db_name": "PostgreSQL",
-  "query": "SELECT COUNT(*) FROM workspace_diff\n         WHERE source_workspace_id = 'wm-fork-test-workspace'\n            OR fork_workspace_id = 'wm-fork-test-workspace'",
-  "describe": {
-    "columns": [
-      {
-        "ordinal": 0,
-        "name": "count",
-        "type_info": "Int8"
-      }
-    ],
-    "parameters": {
-      "Left": []
-    },
-    "nullable": [
-      null
-    ]
-  },
-  "hash": "126cf6d54f8d2c916cd799f6663892119a94d66637062d1bf5a5fe97d89f8096"
-}

backend/.sqlx/query-16b4496c21d0619dab4521dca22e5fe144c59156a8f06d8592291684c49b2f37.json

@@ -0,0 +1,95 @@
+{
+  "db_name": "PostgreSQL",
+  "query": "SELECT email as \"email!\", login_type::text, verified as \"verified!\", super_admin as \"super_admin!\", devops as \"devops!\", name, company, username, NULL::bool as operator_only, first_time_user as \"first_time_user!\", role_source as \"role_source!\", disabled as \"disabled!\", NULL::text as workspace_id FROM password\n            UNION ALL\n            SELECT email as \"email!\", 'service_account'::text as login_type, true as \"verified!\", false as \"super_admin!\", false as \"devops!\", NULL::text as name, NULL::text as company, username, true as operator_only, false as \"first_time_user!\", 'service_account'::text as \"role_source!\", disabled as \"disabled!\", workspace_id\n            FROM usr\n            WHERE is_service_account IS true\n            ORDER BY \"super_admin!\" DESC, \"devops!\" DESC, \"email!\"\n            LIMIT $1 OFFSET $2",
+  "describe": {
+    "columns": [
+      {
+        "ordinal": 0,
+        "name": "email!",
+        "type_info": "Varchar"
+      },
+      {
+        "ordinal": 1,
+        "name": "login_type",
+        "type_info": "Text"
+      },
+      {
+        "ordinal": 2,
+        "name": "verified!",
+        "type_info": "Bool"
+      },
+      {
+        "ordinal": 3,
+        "name": "super_admin!",
+        "type_info": "Bool"
+      },
+      {
+        "ordinal": 4,
+        "name": "devops!",
+        "type_info": "Bool"
+      },
+      {
+        "ordinal": 5,
+        "name": "name",
+        "type_info": "Varchar"
+      },
+      {
+        "ordinal": 6,
+        "name": "company",
+        "type_info": "Varchar"
+      },
+      {
+        "ordinal": 7,
+        "name": "username",
+        "type_info": "Varchar"
+      },
+      {
+        "ordinal": 8,
+        "name": "operator_only",
+        "type_info": "Bool"
+      },
+      {
+        "ordinal": 9,
+        "name": "first_time_user!",
+        "type_info": "Bool"
+      },
+      {
+        "ordinal": 10,
+        "name": "role_source!",
+        "type_info": "Varchar"
+      },
+      {
+        "ordinal": 11,
+        "name": "disabled!",
+        "type_info": "Bool"
+      },
+      {
+        "ordinal": 12,
+        "name": "workspace_id",
+        "type_info": "Text"
+      }
+    ],
+    "parameters": {
+      "Left": [
+        "Int8",
+        "Int8"
+      ]
+    },
+    "nullable": [
+      null,
+      null,
+      null,
+      null,
+      null,
+      null,
+      null,
+      null,
+      null,
+      null,
+      null,
+      null,
+      null
+    ]
+  },
+  "hash": "16b4496c21d0619dab4521dca22e5fe144c59156a8f06d8592291684c49b2f37"
+}

backend/.sqlx/query-1860c102a5309a8e1bb7288d0616f48e4330af7fbe61426e999c6073f630d88f.json

@@ -1,41 +0,0 @@
-{
-  "db_name": "PostgreSQL",
-  "query": "SELECT DISTINCT ON (path)\n                      path,\n                      value as \"value!: sqlx::types::Json<Box<serde_json::value::RawValue>>\",\n                      created_at,\n                      typ::text as \"typ!\"\n               FROM draft\n               WHERE workspace_id = $1\n                 AND typ IN ('app', 'raw_app')\n                 AND (email = $2 OR email IS NULL)\n                 AND NOT EXISTS (\n                     SELECT 1 FROM app a\n                     WHERE a.workspace_id = draft.workspace_id\n                       AND a.path = draft.path\n                 )\n               ORDER BY path, (email IS NULL), created_at DESC",
-  "describe": {
-    "columns": [
-      {
-        "ordinal": 0,
-        "name": "path",
-        "type_info": "Varchar"
-      },
-      {
-        "ordinal": 1,
-        "name": "value!: sqlx::types::Json<Box<serde_json::value::RawValue>>",
-        "type_info": "Json"
-      },
-      {
-        "ordinal": 2,
-        "name": "created_at",
-        "type_info": "Timestamptz"
-      },
-      {
-        "ordinal": 3,
-        "name": "typ!",
-        "type_info": "Text"
-      }
-    ],
-    "parameters": {
-      "Left": [
-        "Text",
-        "Text"
-      ]
-    },
-    "nullable": [
-      false,
-      false,
-      false,
-      null
-    ]
-  },
-  "hash": "1860c102a5309a8e1bb7288d0616f48e4330af7fbe61426e999c6073f630d88f"
-}

backend/.sqlx/query-19513c4158267cc7fe10d999ad571052c112e6bbb3cf834f16176cbb7e1ac319.json

@@ -5,7 +5,7 @@
     "columns": [
       {
         "ordinal": 0,
-        "name": "id!",
+        "name": "id",
         "type_info": "Uuid"
       }
     ],
@@ -16,7 +16,7 @@
       ]
     },
     "nullable": [
-      null
+      false
     ]
   },
   "hash": "19513c4158267cc7fe10d999ad571052c112e6bbb3cf834f16176cbb7e1ac319"

backend/.sqlx/query-19f1cb1c7a1974920549917a6392ff0d56f31ca5662062f4a71fed0ccf859cc4.json

@@ -1,35 +0,0 @@
-{
-  "db_name": "PostgreSQL",
-  "query": "SELECT kind::text as \"kind!\", parent_job, runnable_path\n           FROM v2_job WHERE id = $1 AND workspace_id = $2",
-  "describe": {
-    "columns": [
-      {
-        "ordinal": 0,
-        "name": "kind!",
-        "type_info": "Text"
-      },
-      {
-        "ordinal": 1,
-        "name": "parent_job",
-        "type_info": "Uuid"
-      },
-      {
-        "ordinal": 2,
-        "name": "runnable_path",
-        "type_info": "Varchar"
-      }
-    ],
-    "parameters": {
-      "Left": [
-        "Uuid",
-        "Text"
-      ]
-    },
-    "nullable": [
-      null,
-      true,
-      true
-    ]
-  },
-  "hash": "19f1cb1c7a1974920549917a6392ff0d56f31ca5662062f4a71fed0ccf859cc4"
-}

backend/.sqlx/query-1acfeed9c7a5b1e3d2da262d338655dba6e43067a9912cc2b775830856390c5d.json

@@ -1,27 +0,0 @@
-{
-  "db_name": "PostgreSQL",
-  "query": "WITH del AS (\n             DELETE FROM asset WHERE workspace_id = $1 AND usage_path = $2 AND usage_kind = $3\n             RETURNING usage_access_type\n           )\n           INSERT INTO notify_event (channel, payload)\n           SELECT 'notify_asset_producer_change', $1\n           WHERE $3 = 'script'\n             AND EXISTS (SELECT 1 FROM del WHERE usage_access_type IN ('w', 'rw'))",
-  "describe": {
-    "columns": [],
-    "parameters": {
-      "Left": [
-        "Text",
-        "Text",
-        {
-          "Custom": {
-            "name": "asset_usage_kind",
-            "kind": {
-              "Enum": [
-                "script",
-                "flow",
-                "job"
-              ]
-            }
-          }
-        }
-      ]
-    },
-    "nullable": []
-  },
-  "hash": "1acfeed9c7a5b1e3d2da262d338655dba6e43067a9912cc2b775830856390c5d"
-}

backend/.sqlx/query-1f375b37ff9f6f01972e284e84a7b2f9d2d323a3da55f20ff6671e8eba510043.json

@@ -1,15 +0,0 @@
-{
-  "db_name": "PostgreSQL",
-  "query": "WITH del AS (\n             DELETE FROM asset WHERE workspace_id = $1 AND usage_kind = 'script'\n               AND usage_path = (SELECT path FROM script WHERE hash = $2 AND workspace_id = $1)\n             RETURNING usage_access_type\n           )\n           INSERT INTO notify_event (channel, payload)\n           SELECT 'notify_asset_producer_change', $1\n           WHERE EXISTS (SELECT 1 FROM del WHERE usage_access_type IN ('w', 'rw'))",
-  "describe": {
-    "columns": [],
-    "parameters": {
-      "Left": [
-        "Text",
-        "Int8"
-      ]
-    },
-    "nullable": []
-  },
-  "hash": "1f375b37ff9f6f01972e284e84a7b2f9d2d323a3da55f20ff6671e8eba510043"
-}

backend/.sqlx/query-1f486036b2902a0ffa0ce15f665d82c85129e2407819477adf7e87f097276ae2.json

@@ -1,16 +0,0 @@
-{
-  "db_name": "PostgreSQL",
-  "query": "INSERT INTO draft (workspace_id, path, typ, value, created_at, email)\n         SELECT $2, path, typ, value, created_at, email\n         FROM draft\n         WHERE workspace_id = $1 AND (email = $3 OR email IS NULL)",
-  "describe": {
-    "columns": [],
-    "parameters": {
-      "Left": [
-        "Text",
-        "Varchar",
-        "Text"
-      ]
-    },
-    "nullable": []
-  },
-  "hash": "1f486036b2902a0ffa0ce15f665d82c85129e2407819477adf7e87f097276ae2"
-}

backend/.sqlx/query-1fc04d31ae69dbb1df9c63cb69e83e8f8e6770b78f6ed052b99afed6cea28650.json

@@ -1,17 +0,0 @@
-{
-  "db_name": "PostgreSQL",
-  "query": "INSERT INTO v2_job (id, workspace_id, kind, runnable_path, args, created_by,\n                                permissioned_as, permissioned_as_email, tag, script_lang)\n           VALUES ($1, $2, 'script'::job_kind, $3, $4, 'test-user',\n                   'u/test-user', 'test@windmill.dev', 'deno', 'bash'::script_lang)",
-  "describe": {
-    "columns": [],
-    "parameters": {
-      "Left": [
-        "Uuid",
-        "Varchar",
-        "Varchar",
-        "Jsonb"
-      ]
-    },
-    "nullable": []
-  },
-  "hash": "1fc04d31ae69dbb1df9c63cb69e83e8f8e6770b78f6ed052b99afed6cea28650"
-}

backend/.sqlx/query-2484323d94f249be30f4472ece89659c1e6a24d5454a691e31e0179f58c24366.json

@@ -1,41 +0,0 @@
-{
-  "db_name": "PostgreSQL",
-  "query": "\n        SELECT kind, path, script_path, is_flow FROM (\n            SELECT 'schedule' AS kind, path, script_path, is_flow FROM schedule\n                WHERE workspace_id = $1\n                  AND script_path IS NOT NULL\n            UNION ALL\n            SELECT 'email', path, script_path, is_flow FROM email_trigger\n                WHERE workspace_id = $1\n            UNION ALL\n            SELECT 'kafka', path, script_path, is_flow FROM kafka_trigger\n                WHERE workspace_id = $1\n            UNION ALL\n            SELECT 'mqtt', path, script_path, is_flow FROM mqtt_trigger\n                WHERE workspace_id = $1\n            UNION ALL\n            SELECT 'nats', path, script_path, is_flow FROM nats_trigger\n                WHERE workspace_id = $1\n            UNION ALL\n            SELECT 'postgres', path, script_path, is_flow FROM postgres_trigger\n                WHERE workspace_id = $1\n            UNION ALL\n            SELECT 'sqs', path, script_path, is_flow FROM sqs_trigger\n                WHERE workspace_id = $1\n            UNION ALL\n            SELECT 'gcp', path, script_path, is_flow FROM gcp_trigger\n                WHERE workspace_id = $1\n        ) t\n        WHERE ($2::text IS NULL OR script_path LIKE $2)\n        ",
-  "describe": {
-    "columns": [
-      {
-        "ordinal": 0,
-        "name": "kind",
-        "type_info": "Text"
-      },
-      {
-        "ordinal": 1,
-        "name": "path",
-        "type_info": "Varchar"
-      },
-      {
-        "ordinal": 2,
-        "name": "script_path",
-        "type_info": "Varchar"
-      },
-      {
-        "ordinal": 3,
-        "name": "is_flow",
-        "type_info": "Bool"
-      }
-    ],
-    "parameters": {
-      "Left": [
-        "Text",
-        "Text"
-      ]
-    },
-    "nullable": [
-      null,
-      null,
-      null,
-      null
-    ]
-  },
-  "hash": "2484323d94f249be30f4472ece89659c1e6a24d5454a691e31e0179f58c24366"
-}

backend/.sqlx/query-25ecae25ebc03d6296b0e72482a9201c2bffb0f0f7419b0b598b2786bdb326ab.json

@@ -1,12 +0,0 @@
-{
-  "db_name": "PostgreSQL",
-  "query": "DELETE FROM job_perms\n             WHERE ctid IN (\n                 SELECT jp.ctid FROM job_perms jp\n                 WHERE NOT EXISTS (SELECT 1 FROM v2_job_queue q WHERE q.id = jp.job_id)\n                 LIMIT 100000\n             )",
-  "describe": {
-    "columns": [],
-    "parameters": {
-      "Left": []
-    },
-    "nullable": []
-  },
-  "hash": "25ecae25ebc03d6296b0e72482a9201c2bffb0f0f7419b0b598b2786bdb326ab"
-}

backend/.sqlx/query-269197b692a1e451a31f14701c8ca324c8e40d2923dfa07b8e4b3934afc16fed.json

@@ -1,6 +1,6 @@
 {
   "db_name": "PostgreSQL",
-  "query": "SELECT workspace_id, name, display_name, owners, extra_perms, summary, created_by, edited_at, default_permissioned_as, labels FROM folder WHERE name = $1 AND workspace_id = $2",
+  "query": "SELECT workspace_id, name, display_name, owners, extra_perms, summary, created_by, edited_at, default_permissioned_as FROM folder WHERE name = $1 AND workspace_id = $2",
   "describe": {
     "columns": [
       {
@@ -47,11 +47,6 @@
         "ordinal": 8,
         "name": "default_permissioned_as",
         "type_info": "Jsonb"
-      },
-      {
-        "ordinal": 9,
-        "name": "labels",
-        "type_info": "TextArray"
       }
     ],
     "parameters": {
@@ -69,9 +64,8 @@
       true,
       true,
       true,
-      false,
-      true
+      false
     ]
   },
-  "hash": "42600bdeb6b86ac306ab276cb49241508776f5f998821e04bc92df917176cea6"
+  "hash": "269197b692a1e451a31f14701c8ca324c8e40d2923dfa07b8e4b3934afc16fed"
 }

backend/.sqlx/query-26e63135fcd8e7d48e25de190a2f72ece70ec92c5b04baad2622639850445900.json

@@ -1,97 +0,0 @@
-{
-  "db_name": "PostgreSQL",
-  "query": "SELECT\n              subscriber_path AS \"subscriber_path!\",\n              asset_kind AS \"asset_kind!: windmill_common::assets::AssetKind\",\n              asset_path AS \"asset_path!\",\n              outcome::text AS \"outcome!\",\n              child_job_id,\n              partition,\n              received_inputs,\n              required_inputs,\n              debounce_s,\n              reason,\n              created_at AS \"created_at!\"\n           FROM dispatch_event\n           WHERE producer_job_id = $1 AND workspace_id = $2\n           ORDER BY id",
-  "describe": {
-    "columns": [
-      {
-        "ordinal": 0,
-        "name": "subscriber_path!",
-        "type_info": "Varchar"
-      },
-      {
-        "ordinal": 1,
-        "name": "asset_kind!: windmill_common::assets::AssetKind",
-        "type_info": {
-          "Custom": {
-            "name": "asset_kind",
-            "kind": {
-              "Enum": [
-                "s3object",
-                "resource",
-                "variable",
-                "ducklake",
-                "datatable",
-                "volume"
-              ]
-            }
-          }
-        }
-      },
-      {
-        "ordinal": 2,
-        "name": "asset_path!",
-        "type_info": "Text"
-      },
-      {
-        "ordinal": 3,
-        "name": "outcome!",
-        "type_info": "Text"
-      },
-      {
-        "ordinal": 4,
-        "name": "child_job_id",
-        "type_info": "Uuid"
-      },
-      {
-        "ordinal": 5,
-        "name": "partition",
-        "type_info": "Text"
-      },
-      {
-        "ordinal": 6,
-        "name": "received_inputs",
-        "type_info": "Int4"
-      },
-      {
-        "ordinal": 7,
-        "name": "required_inputs",
-        "type_info": "Int4"
-      },
-      {
-        "ordinal": 8,
-        "name": "debounce_s",
-        "type_info": "Int4"
-      },
-      {
-        "ordinal": 9,
-        "name": "reason",
-        "type_info": "Text"
-      },
-      {
-        "ordinal": 10,
-        "name": "created_at!",
-        "type_info": "Timestamptz"
-      }
-    ],
-    "parameters": {
-      "Left": [
-        "Uuid",
-        "Text"
-      ]
-    },
-    "nullable": [
-      false,
-      false,
-      false,
-      null,
-      true,
-      true,
-      true,
-      true,
-      true,
-      true,
-      false
-    ]
-  },
-  "hash": "26e63135fcd8e7d48e25de190a2f72ece70ec92c5b04baad2622639850445900"
-}

backend/.sqlx/query-27b0c827467cc92979f094620957bc0edfa295d6c2292e509a5536765d120bd8.json

@@ -1,22 +1,23 @@
 {
   "db_name": "PostgreSQL",
-  "query": "SELECT deleted FROM workspace WHERE id = $1",
+  "query": "SELECT draft_only FROM app WHERE path = $1 AND workspace_id = $2",
   "describe": {
     "columns": [
       {
         "ordinal": 0,
-        "name": "deleted",
+        "name": "draft_only",
         "type_info": "Bool"
       }
     ],
     "parameters": {
       "Left": [
+        "Text",
         "Text"
       ]
     },
     "nullable": [
-      false
+      true
     ]
   },
-  "hash": "ebc8f4840335dc3f4c8274fcf106825ecfdf7fbca59ea26541c5b4c0350b5502"
+  "hash": "27b0c827467cc92979f094620957bc0edfa295d6c2292e509a5536765d120bd8"
 }

backend/.sqlx/query-285c136fc92ce63417e4c65e657d914a6e158d636b854248e4028ed35326f3c6.json

@@ -0,0 +1,23 @@
+{
+  "db_name": "PostgreSQL",
+  "query": "\n                SELECT\n                    path\n                FROM\n                    flow_version\n                WHERE\n                    id = $1 AND\n                    workspace_id = $2\n            ",
+  "describe": {
+    "columns": [
+      {
+        "ordinal": 0,
+        "name": "path",
+        "type_info": "Varchar"
+      }
+    ],
+    "parameters": {
+      "Left": [
+        "Int8",
+        "Text"
+      ]
+    },
+    "nullable": [
+      false
+    ]
+  },
+  "hash": "285c136fc92ce63417e4c65e657d914a6e158d636b854248e4028ed35326f3c6"
+}

backend/.sqlx/query-28f1ecca40c8b81cc59dffb75e2913c889b374999ece04173b2e67dc74005f60.json

@@ -1,11 +1,11 @@
 {
   "db_name": "PostgreSQL",
-  "query": "SELECT policy->>'execution_mode' = 'anonymous' FROM app WHERE path = $1 AND workspace_id = $2 FOR UPDATE",
+  "query": "SELECT draft_only FROM flow WHERE path = $1 AND workspace_id = $2 AND archived = false",
   "describe": {
     "columns": [
       {
         "ordinal": 0,
-        "name": "?column?",
+        "name": "draft_only",
         "type_info": "Bool"
       }
     ],
@@ -16,8 +16,8 @@
       ]
     },
     "nullable": [
-      null
+      true
     ]
   },
-  "hash": "3e82929b365a6aa7ccc39fc5615c4110e1d740e0cb0509b1c58bc6636d83cafd"
+  "hash": "28f1ecca40c8b81cc59dffb75e2913c889b374999ece04173b2e67dc74005f60"
 }

backend/.sqlx/query-2b5fc0500beb2f4c7cf5997f9aea48f77e2abe4523180c507a9a90570127be6d.json

@@ -0,0 +1,60 @@
+{
+  "db_name": "PostgreSQL",
+  "query": "SELECT label, token_prefix, expiration, created_at, last_used_at, scopes, workspace_id FROM token WHERE email = $1\n            ORDER BY created_at DESC LIMIT $2 OFFSET $3",
+  "describe": {
+    "columns": [
+      {
+        "ordinal": 0,
+        "name": "label",
+        "type_info": "Varchar"
+      },
+      {
+        "ordinal": 1,
+        "name": "token_prefix",
+        "type_info": "Varchar"
+      },
+      {
+        "ordinal": 2,
+        "name": "expiration",
+        "type_info": "Timestamptz"
+      },
+      {
+        "ordinal": 3,
+        "name": "created_at",
+        "type_info": "Timestamptz"
+      },
+      {
+        "ordinal": 4,
+        "name": "last_used_at",
+        "type_info": "Timestamptz"
+      },
+      {
+        "ordinal": 5,
+        "name": "scopes",
+        "type_info": "TextArray"
+      },
+      {
+        "ordinal": 6,
+        "name": "workspace_id",
+        "type_info": "Varchar"
+      }
+    ],
+    "parameters": {
+      "Left": [
+        "Text",
+        "Int8",
+        "Int8"
+      ]
+    },
+    "nullable": [
+      true,
+      false,
+      true,
+      false,
+      false,
+      true,
+      true
+    ]
+  },
+  "hash": "2b5fc0500beb2f4c7cf5997f9aea48f77e2abe4523180c507a9a90570127be6d"
+}

backend/.sqlx/query-2bed492ef32edf36e60e8a03268fa25bfb67dd641153d1ca23f7d0d2ae73624e.json

@@ -1,60 +0,0 @@
-{
-  "db_name": "PostgreSQL",
-  "query": "DELETE FROM draft\n               WHERE workspace_id = $1\n                 AND email IS NOT DISTINCT FROM (CASE WHEN $7::bool THEN NULL::text ELSE $2 END)\n                 AND path = $3\n                 AND typ = $4\n                 AND ($6::bool = true\n                      OR $5::timestamptz IS NULL\n                      OR created_at <= $5::timestamptz)\n               RETURNING now() as \"now!\"",
-  "describe": {
-    "columns": [
-      {
-        "ordinal": 0,
-        "name": "now!",
-        "type_info": "Timestamptz"
-      }
-    ],
-    "parameters": {
-      "Left": [
-        "Text",
-        "Text",
-        "Text",
-        {
-          "Custom": {
-            "name": "draft_kind",
-            "kind": {
-              "Enum": [
-                "script",
-                "flow",
-                "app",
-                "raw_app",
-                "resource",
-                "variable",
-                "trigger_schedule",
-                "trigger_webhook",
-                "trigger_default_email",
-                "trigger_email",
-                "trigger_http",
-                "trigger_websocket",
-                "trigger_postgres",
-                "trigger_kafka",
-                "trigger_nats",
-                "trigger_mqtt",
-                "trigger_sqs",
-                "trigger_gcp",
-                "trigger_azure",
-                "trigger_poll",
-                "trigger_cli",
-                "trigger_nextcloud",
-                "trigger_google",
-                "trigger_github"
-              ]
-            }
-          }
-        },
-        "Timestamptz",
-        "Bool",
-        "Bool"
-      ]
-    },
-    "nullable": [
-      null
-    ]
-  },
-  "hash": "2bed492ef32edf36e60e8a03268fa25bfb67dd641153d1ca23f7d0d2ae73624e"
-}

backend/.sqlx/query-311de4a5d2fb3066dc9e49693b9a1dd8e8e4a09200768a73c844810975741894.json

@@ -0,0 +1,23 @@
+{
+  "db_name": "PostgreSQL",
+  "query": "\n            SELECT\n                path\n            FROM\n                flow_version\n            WHERE\n                id = $1 AND\n                workspace_id = $2\n            ",
+  "describe": {
+    "columns": [
+      {
+        "ordinal": 0,
+        "name": "path",
+        "type_info": "Varchar"
+      }
+    ],
+    "parameters": {
+      "Left": [
+        "Int8",
+        "Text"
+      ]
+    },
+    "nullable": [
+      false
+    ]
+  },
+  "hash": "311de4a5d2fb3066dc9e49693b9a1dd8e8e4a09200768a73c844810975741894"
+}