starred/windmill
1
0
Fork 0
mirror of https://github.com/windmill-labs/windmill.git synced 2026-10-07 08:02:20 +00:00
Code Issues Packages Projects Releases Wiki Activity

Merge remote-tracking branch 'origin/main' into explore-git-sync-improvements

Browse Source 
# Conflicts:
#	backend/ee-repo-ref.txt
#	backend/windmill-worker/src/result_processor.rs
This commit is contained in:
hugocasa
2026-06-29 19:24:54 +02:00
parent d03045168e 2493eaf031
commit 8e5f43f87d
1133 changed files with 108061 additions and 20842 deletions
Download Patch File Download Diff File Expand all files Collapse all files
.agents/skills/ai-chat/SKILL.md
+40
View File
@@ -0,0 +1,40 @@
---
name: ai-chat
description: Guidance for improving the Windmill AI chat (copilot), especially global mode — tools, prompts, and context-window discipline. Use when editing chat tools, system prompts, or tool-result shapes under frontend/src/lib/components/copilot/chat, or when changing how the chat manages its context window.
---
## Always benchmark before and after
No context or behavior change ships without an `ai_evals` A/B on the affected mode.
Add or adjust cases for exactly what you changed — see the `ai-evals` skill for
authoring and the full run reference.
Run the affected mode **before** your change and **after**, same model(s), same cases.
## Measure the window first, and cumulative second
Optimize **`finalContextTokens`** (window occupancy — what drives overflow and
compaction), then cumulative prompt tokens.
## Context discipline
The dominant fixed cost is per-iteration overhead: the system prompt **plus every
tool schema** is re-sent on every loop iteration. So:
- **Every tool and every parameter is a permanent tax.** Justify each one and measure
it; an extra "locate" round-trip can cost more than the reads it saves. Strip dead
params rather than leaving them in the schema.
- **Tool results return the minimum.** Never echo content the model already has. The
canonical mistake: a write tool that returns the whole edited artifact right after
the model authored it — return `{ success, message }` instead. When you touch a
*shared* write helper (e.g. `finishAppDraftWrite` in `global/core.ts`), re-check
this invariant for **all** the write tools routing through it — the echo has
regressed before via a shared refactor.
## Prompts and tool descriptions are part of the surface
The system prompt and tool descriptions steer behavior as much as the tools
themselves, and are benchmarkable the same way. A description that advertises
truncation makes the model self-limit; the path-conventions block changes where
drafts land. Treat prompt/description edits as real changes and A/B them — a
pure-prompt change is a legitimate, measurable improvement.
.agents/skills/ai-evals/SKILL.md
+87
View File
@@ -0,0 +1,87 @@
---
name: ai-evals
description: Author and run black-box benchmark cases for the Windmill AI generation modes (flow/app/script/cli/global) in ai_evals/. Use when adding or changing eval cases, or when running before/after benchmarks for AI chat / copilot changes.
---
# AI evals — authoring and running benchmark cases
`ai_evals/` is a black-box benchmark runner for the Windmill AI generation modes:
`flow`, `app`, `script`, `cli`, `global`. It always tests the **current** production
prompts, tools, and guidance in this checkout. Each attempt runs the real production
path, deterministic validation, then LLM judging.
The goal is to test current production guidance with realistic user requests — **not**
to pin one exact implementation shape.
## Running benchmarks
```bash
cd ai_evals
bun install # first time; frontend modes also need `cd frontend && bun install`
bun run cli -- models # list model aliases
bun run cli -- cases global # list cases for a mode
bun run cli -- run global global-test1-script-create --model sonnet
```
Frontend modes (`flow`/`script`/`app`/`global`) route model calls through a Windmill
backend's `/api/w/<ws>/ai/proxy`, so you need **any** reachable backend:
```bash
WMILL_AI_EVAL_BACKEND_URL=http://127.0.0.1:<port> WMILL_AI_EVAL_BACKEND_WORKSPACE=integration-tests \
bun run cli -- run global <caseIds...> --models sonnet,gpt-5.5,gemini-3.1-pro-preview
```
- **Reuse an existing workspace.** CE builds cap workspaces, so temp-workspace
creation 400s ("reached workspace limit"). Always set
`WMILL_AI_EVAL_BACKEND_WORKSPACE=integration-tests` (or any existing workspace) to
reuse one. The only side effect of a run is upserting an `f/evals/ai/<provider>`
resource there.
- Provider keys live in `ai_evals/.env` and are auto-loaded by bun. The judge is a
separate Anthropic call (default `claude-sonnet-4-6`) regardless of the model under
test.
## Authoring core rules
1. Write prompts like a real user request.
2. Prefer behavior, inputs, constraints, and outcomes over internal implementation.
3. Keep deterministic validation narrow and hard.
4. Put semantic expectations in `judgeChecklist`.
5. Use `expected` fixtures only when exact structure really matters.
### Prompt writing
Prompts should sound like something a user would naturally ask. Do not write prompts
as if the user knows Windmill internals unless the case explicitly tests a power-user
workflow.
Good:
- "Create a flow that routes support requests based on customer tier."
- "Add a reset button that sets the counter back to 0."
- "Create a flow that reuses the existing greeting script instead of duplicating the logic."
Bad:
- "Use `branchone` with 3 branches and a default branch."
- "Create a `rawscript` step with this exact topology."
- "This is a benchmark harness."
### Deterministic validation
Use deterministic checks only for hard failures: missing required files; unexpected
extra files when the prompt says not to create them; syntax errors; unresolved flow
refs; missing required special modules or suspend config; obvious corruption.
Do **not** encode one preferred implementation. Bad hard checks: exact step topology
for a creation flow; exact branch structure when the prompt only asked for routing;
exact input shape when multiple reasonable shapes are acceptable.
### Judge checklist
Every non-trivial case should have a `judgeChecklist` capturing user-visible behavior
that must be present, important constraints, and key completion criteria — not
low-level implementation details unless truly required.
Good: "the flow calculates the order total with 8% tax"; "the flow reuses the existing
workspace script instead of rewriting the logic". Bad: "uses `branchone`"; "contains a
`rawscript` node".
See `ai_evals/README.md` for the full case format, fields, and fixture details.
.claude/skills/ai-chat/SKILL.md
Symlink
+1
View File
@@ -0,0 +1 @@
../../../.agents/skills/ai-chat/SKILL.md
.claude/skills/ai-evals/SKILL.md
Symlink
+1
View File
@@ -0,0 +1 @@
../../../.agents/skills/ai-evals/SKILL.md
.github/DockerfileBackendTests
+1 -1
View File
@@ -28,7 +28,7 @@ ENV PATH="${PATH}:/usr/local/go/bin"
ENV GO_PATH=/usr/local/go/bin/go
# UV
RUN curl --proto '=https' --tlsv1.2 -LsSf https://github.com/astral-sh/uv/releases/download/0.9.25/uv-installer.sh | sh && mv /usr/local/cargo/bin/uv /usr/local/bin/uv
RUN curl --proto '=https' --tlsv1.2 -LsSf https://github.com/astral-sh/uv/releases/download/0.11.24/uv-installer.sh | sh && mv /usr/local/cargo/bin/uv /usr/local/bin/uv
ENV TZ=Etc/UTC
.github/workflows/ai-agent-tests.yml
+132
View File
@@ -0,0 +1,132 @@
name: AI Agent Integration Tests
# Exercises the AI agent flow path (preview_flow with `aiagent` modules) against
# real LLM providers. Runs only when AI-agent backend code or the tests change,
# because each run makes real (paid) LLM calls. To avoid spending on every commit,
# the PR side triggers only when a PR is marked ready for review (out of draft) —
# not on `synchronize` — plus push to main and manual dispatch.
on:
workflow_dispatch:
push:
branches: [main]
paths:
- "integration_tests/ai_agent_tests/**"
- "backend/windmill-ai/**"
- "backend/windmill-api/src/ai.rs"
- "backend/windmill-worker/src/ai_executor.rs"
- "backend/windmill-worker/src/ai/**"
- "backend/windmill-worker/src/memory_common.rs"
- "backend/windmill-common/src/flow_conversations.rs"
- ".github/workflows/ai-agent-tests.yml"
pull_request:
types: [opened, reopened, ready_for_review]
paths:
- "integration_tests/ai_agent_tests/**"
- "backend/windmill-ai/**"
- "backend/windmill-api/src/ai.rs"
- "backend/windmill-worker/src/ai_executor.rs"
- "backend/windmill-worker/src/ai/**"
- "backend/windmill-worker/src/memory_common.rs"
- "backend/windmill-common/src/flow_conversations.rs"
- ".github/workflows/ai-agent-tests.yml"
concurrency:
group: ai-agent-tests-${{ github.ref }}
cancel-in-progress: true
jobs:
ai_agent_e2e:
# Skip draft PRs; the `opened`/`reopened` types would otherwise fire while
# still a draft. `ready_for_review` always arrives non-draft.
if: github.event_name != 'pull_request' || github.event.pull_request.draft == false
runs-on: ubicloud-standard-16
services:
postgres:
image: postgres:16
ports:
- 5432:5432
env:
POSTGRES_DB: windmill
POSTGRES_PASSWORD: changeme
options: >-
--health-cmd pg_isready --health-interval 10s --health-timeout 5s
--health-retries 5
steps:
- uses: actions/checkout@v4
- uses: actions-rust-lang/setup-rust-toolchain@v1
with:
cache-workspaces: backend
toolchain: 1.93.0
- uses: oven-sh/setup-bun@v2
with:
bun-version: 1.3.10
- uses: actions/setup-node@v4
with:
node-version: "20"
- uses: actions/setup-python@v5
with:
python-version: "3.11"
# CE build (no enterprise/license needed for AI agents). `quickjs` powers
# flow input-transform JS eval; `mcp` is required by the deepwiki MCP tool
# test. Bun tool scripts run via the always-on worker (BUN_PATH).
- name: Build Windmill
working-directory: ./backend
env:
SQLX_OFFLINE: true
CARGO_BUILD_JOBS: 12
RUSTFLAGS: ""
run: cargo build --features quickjs,mcp
- name: Start Windmill
working-directory: ./backend
env:
DATABASE_URL: postgres://postgres:changeme@localhost:5432/windmill
BUN_PATH: bun
NODE_BIN_PATH: node
RUST_LOG: info
run: |
mkdir -p ../integration_tests/logs
./target/debug/windmill > ../integration_tests/logs/windmill.log 2>&1 &
echo "Waiting for Windmill to be ready..."
for i in $(seq 1 60); do
if curl -sf http://localhost:8000/api/version > /dev/null 2>&1; then
echo "Windmill is ready"
break
fi
sleep 2
done
curl -sf http://localhost:8000/api/version > /dev/null || { echo "Windmill failed to start"; tail -50 ../integration_tests/logs/windmill.log; exit 1; }
- name: Run AI agent integration tests
timeout-minutes: 20
working-directory: ./integration_tests/ai_agent_tests
env:
WINDMILL_URL: http://localhost:8000
# Only the providers we have org secrets for. Other providers
# (Azure, Bedrock, OpenRouter) are skipped by conftest when their
# keys are absent — see skip_provider_without_credentials.
ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }}
OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
GOOGLE_AI_API_KEY: ${{ secrets.GOOGLE_API_KEY }}
run: |
python -m venv .venv
.venv/bin/pip install -r requirements.txt
# The S3/vision-attachment tests need MinIO large-file storage and
# image-capable provider setup; out of scope for this cost-controlled
# smoke. Add MinIO secrets + a storage service to enable them.
.venv/bin/python -m pytest -v \
--ignore=test_user_attachments.py \
--ignore=test_user_images.py \
--ignore=test_image_output.py
- name: Archive Windmill logs
uses: actions/upload-artifact@v4
if: always()
with:
name: ai-agent-tests-windmill-logs
path: integration_tests/logs
.github/workflows/ai-evals-test.yml
+166
View File
@@ -0,0 +1,166 @@
name: AI Evals (global mode)
# Smoke-tests the production global AI chat proxy/frontend execution path via
# the ai_evals harness, one case across one cheap model per provider. Runs only
# when the eval harness or the global chat code change, since each run makes real
# (paid) LLM calls. The backend is built from source purely as the AI proxy the
# harness routes model calls through; the global tools/drafts run in-process in
# the Vitest bridge against production frontend code. To avoid spending on every
# commit, the PR side triggers only when a PR is marked ready for review (out of
# draft) — not on `synchronize` — plus push to main and manual dispatch.
on:
workflow_dispatch:
push:
branches: [main]
paths:
- "ai_evals/**"
- "backend/windmill-api/src/ai.rs"
- "backend/windmill-ai/**"
- "frontend/src/lib/components/copilot/**"
# The eval harness runs production frontend code in-process; these are the
# AI/draft-specific deps outside copilot/ that the global smoke exercises.
- "frontend/src/lib/userDraft.svelte.ts"
- "frontend/src/lib/userDraftDbSyncer.svelte.ts"
- "frontend/src/lib/infer.ts"
- ".github/workflows/ai-evals-test.yml"
pull_request:
types: [opened, reopened, ready_for_review]
paths:
- "ai_evals/**"
- "backend/windmill-api/src/ai.rs"
- "backend/windmill-ai/**"
- "frontend/src/lib/components/copilot/**"
# The eval harness runs production frontend code in-process; these are the
# AI/draft-specific deps outside copilot/ that the global smoke exercises.
- "frontend/src/lib/userDraft.svelte.ts"
- "frontend/src/lib/userDraftDbSyncer.svelte.ts"
- "frontend/src/lib/infer.ts"
- ".github/workflows/ai-evals-test.yml"
concurrency:
group: ai-evals-test-${{ github.ref }}
cancel-in-progress: true
jobs:
ai_evals_global:
# Provider secrets are unavailable to forked and Dependabot PRs.
if: >-
github.event_name != 'pull_request' ||
(
github.event.pull_request.draft == false &&
github.event.pull_request.head.repo.full_name == github.repository &&
github.event.pull_request.user.login != 'dependabot[bot]'
)
runs-on: ubicloud-standard-16
services:
postgres:
image: postgres:16
ports:
- 5432:5432
env:
POSTGRES_DB: windmill
POSTGRES_PASSWORD: changeme
options: >-
--health-cmd pg_isready --health-interval 10s --health-timeout 5s
--health-retries 5
steps:
- uses: actions/checkout@v4
- uses: actions-rust-lang/setup-rust-toolchain@v1
with:
cache-workspaces: backend
toolchain: 1.93.0
- uses: oven-sh/setup-bun@v2
with:
bun-version: 1.3.10
- uses: actions/setup-node@v4
with:
# Node 22.19+ is required by the frontend's undici 8.x, which the
# Vitest bridge loads; Node 20 fails with markAsUncloneable.
node-version: "22"
# CE build used only as the AI proxy (login, workspace, provider resource,
# /ai/proxy). No worker execution or MCP needed — global tools/drafts run
# in the Vitest bridge. quickjs matches the standard CE feature set.
- name: Build Windmill (AI proxy)
working-directory: ./backend
env:
SQLX_OFFLINE: true
CARGO_BUILD_JOBS: 12
RUSTFLAGS: ""
run: cargo build --features quickjs
- name: Start Windmill
working-directory: ./backend
env:
DATABASE_URL: postgres://postgres:changeme@localhost:5432/windmill
RUST_LOG: info
run: |
mkdir -p ../ai_evals/logs
./target/debug/windmill > ../ai_evals/logs/windmill.log 2>&1 &
echo "Waiting for Windmill to be ready..."
for i in $(seq 1 60); do
if curl -sf http://localhost:8000/api/version > /dev/null 2>&1; then
echo "Windmill is ready"
break
fi
sleep 2
done
curl -sf http://localhost:8000/api/version > /dev/null || { echo "Windmill failed to start"; tail -50 ../ai_evals/logs/windmill.log; exit 1; }
- name: Install frontend deps + generate client
working-directory: ./frontend
run: |
npm ci
npm run generate-backend-client
- name: Run global AI evals
timeout-minutes: 20
working-directory: ./ai_evals
env:
WMILL_AI_EVAL_BACKEND_URL: http://localhost:8000
WMILL_AI_EVAL_BACKEND_WORKSPACE: integration-tests
# Anthropic backs the haiku model. Google AI uses GEMINI_API_KEY.
ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }}
OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
GEMINI_API_KEY: ${{ secrets.GOOGLE_API_KEY }}
DEEPSEEK_API_KEY: ${{ secrets.DEEPSEEK_API_KEY }}
run: |
bun install
mkdir -p results
# One cheap model per provider (anthropic/openai/googleai/deepseek).
fail=0
for m in haiku 4o gemini-3-flash-preview deepseek-v4-flash; do
echo "::group::global-test1-script-create ($m)"
if ! bun run cli -- run global global-test1-script-create \
--model "$m" --execution-only --output "$PWD/results/ci-$m.json"; then
echo "$m: harness/proxy errored"
fail=1
echo "::endgroup::"
continue
fi
# The CLI exits 0 when the harness records failed attempts, so gate
# on execution-only pass counts while ignoring model output quality.
if jq -e \
'.attemptCount > 0 and .passedAttempts == .attemptCount' \
"results/ci-$m.json" > /dev/null; then
echo "$m: OK — proxy/frontend execution completed"
else
echo "$m: FAILED proxy/frontend execution"
jq -c '.cases[0].attempts[0].checks' "results/ci-$m.json" || true
fail=1
fi
echo "::endgroup::"
done
[ "$fail" = 0 ] || { echo "ai_evals global smoke failed"; exit 1; }
- name: Archive logs and results
uses: actions/upload-artifact@v4
if: always()
with:
name: ai-evals-global-logs
path: |
ai_evals/logs
ai_evals/results
.github/workflows/backend-test-windows.yml
+4 -2
View File
@@ -58,7 +58,9 @@ jobs:
- uses: denoland/setup-deno@v2
with:
deno-version: v2.x
# Pin to the Deno version shipped in the runtime image (Dockerfile) so CI
# tests what production runs, instead of floating on the latest v2.x.
deno-version: 2.2.1
- uses: actions/setup-go@v2
with:
@@ -74,7 +76,7 @@ jobs:
- uses: astral-sh/setup-uv@v6.2.1
with:
version: "0.9.25"
version: "0.11.24"
- uses: shivammathur/setup-php@v2
with:
.github/workflows/backend-test.yml
+4 -2
View File
@@ -50,7 +50,9 @@ jobs:
dotnet-version: "9.0.x"
- uses: denoland/setup-deno@v2
with:
deno-version: v2.x
# Pin to the Deno version shipped in the runtime image (Dockerfile) so CI
# tests what production runs, instead of floating on the latest v2.x.
deno-version: 2.2.1
- uses: actions/setup-go@v2
with:
go-version: 1.21.5
@@ -62,7 +64,7 @@ jobs:
node-version: "20"
- uses: astral-sh/setup-uv@v6.2.1
with:
version: "0.9.25"
version: "0.11.24"
- uses: shivammathur/setup-php@v2
with:
php-version: "8.3"
.github/workflows/claude-plan.yml
+1 -1
View File
@@ -45,7 +45,7 @@ jobs:
allowed_bots: 'windmill-internal-app[bot]'
trigger_phrase: '/plan'
claude_args: |
--model claude-fable-5
--model claude-opus-4-8
--system-prompt "# Claude Planning Mode
You are operating in PLANNING MODE ONLY. Your role is to create detailed, structured plans without making any code changes.
.github/workflows/claude.yml
+39 -1
View File
@@ -37,6 +37,44 @@ jobs:
with:
fetch-depth: 1
# Make the EE source (the *_ee.rs files in the companion repo) available so the
# reviewer can see EE-only code (e.g. windmill-queue/src/jobs_ee.rs), not just the
# CE surface. The EE ref is read from the PR head's backend/ee-repo-ref.txt (via the
# API, so it reflects the PR's EE pin regardless of which ref is checked out here).
- name: Check EE access
id: ee
env:
EE_TOKEN: ${{ secrets.WINDMILL_EE_PRIVATE_ACCESS }}
GH_TOKEN: ${{ github.token }}
PR_NUMBER: ${{ github.event.issue.number || github.event.pull_request.number }}
run: |
if [ -z "$EE_TOKEN" ] || [ -z "$PR_NUMBER" ]; then
echo "available=false" >> "$GITHUB_OUTPUT"
exit 0
fi
HEAD_SHA=$(gh api "repos/${{ github.repository }}/pulls/$PR_NUMBER" --jq .head.sha)
REF=$(gh api "repos/${{ github.repository }}/contents/backend/ee-repo-ref.txt?ref=$HEAD_SHA" --jq .content | base64 -d | tr -d '[:space:]')
if [ -z "$REF" ]; then
echo "available=false" >> "$GITHUB_OUTPUT"
else
echo "available=true" >> "$GITHUB_OUTPUT"
echo "ee_repo_ref=$REF" >> "$GITHUB_OUTPUT"
fi
- name: Checkout EE repository
if: steps.ee.outputs.available == 'true'
uses: actions/checkout@v4
with:
repository: windmill-labs/windmill-ee-private
path: ./windmill-ee-private
ref: ${{ steps.ee.outputs.ee_repo_ref }}
token: ${{ secrets.WINDMILL_EE_PRIVATE_ACCESS }}
fetch-depth: 1
- name: Substitute EE code
if: steps.ee.outputs.available == 'true'
run: ./backend/substitute_ee_code.sh --copy --dir ./windmill-ee-private
- name: Run Claude PR Action
uses: anthropics/claude-code-action@v1
with:
@@ -51,4 +89,4 @@ jobs:
}
claude_args: |
--allowedTools "Bash,WebFetch,WebSearch"
--model claude-fable-5
--model claude-opus-4-8
.github/workflows/pr-ready-review.yml
+1 -1
View File
@@ -160,4 +160,4 @@ jobs:
${{ env.REVIEW_PROMPT }}
claude_args: |
--allowedTools "mcp__github_inline_comment__create_inline_comment,Bash(gh pr comment:*),Bash(gh pr diff:*),Bash(gh pr view:*)"
--model claude-fable-5
--model claude-opus-4-8
.github/workflows/refresh-docs-snapshot.yml
+52
View File
@@ -0,0 +1,52 @@
name: Refresh docs snapshot
# The backend embeds a vendored docs snapshot (backend/windmill-api/docs_snapshot/*.gz)
# so in-product docs search works with no runtime egress. This job re-fetches it from
# windmill.dev on a schedule and opens a PR when it changed, keeping the embedded docs
# fresh independently of the release cadence (the binary embeds whatever is on the
# source tree at build time, so a merged refresh rides into the next release build).
on:
schedule:
- cron: "0 6 * * 1" # Mondays 06:00 UTC
workflow_dispatch:
jobs:
refresh:
runs-on: ubicloud
permissions:
contents: write
pull-requests: write
steps:
- uses: actions/create-github-app-token@v2
id: app
with:
app-id: ${{ vars.INTERNAL_APP_ID }}
private-key: ${{ secrets.INTERNAL_APP_KEY }}
- uses: actions/checkout@v4
with:
token: ${{ steps.app.outputs.token }}
- name: Fetch + re-gzip docs snapshot
run: cd backend/windmill-api/docs_snapshot && ./fetch.sh
- name: Sanity-check the fetched corpus
# curl -f in fetch.sh rejects HTTP errors, but not a valid-but-garbage 200
# (truncated file, error page). Guard against embedding a broken snapshot.
run: |
cd backend/windmill-api/docs_snapshot
test "$(wc -c < llms-full.txt.gz)" -gt 100000
test "$(wc -c < llms.txt.gz)" -gt 1000
pages=$(gzip -dc llms-full.txt.gz | grep -c '^Source:' || true)
echo "pages in snapshot: $pages"
test "${pages:-0}" -ge 200
- uses: peter-evans/create-pull-request@v6
with:
token: ${{ steps.app.outputs.token }}
branch: chore/refresh-docs-snapshot
add-paths: backend/windmill-api/docs_snapshot/*.gz
commit-message: "chore: refresh vendored docs snapshot"
title: "chore: refresh vendored docs snapshot"
body: |
Automated refresh of the embedded docs snapshot
(`backend/windmill-api/docs_snapshot/*.gz`) from windmill.dev.
Review the diff for unexpected churn (a bad upstream docs deploy would
show up as a large drop in pages or content) before merging.
.gitignore
+2 -1
View File
@@ -33,4 +33,5 @@ backend/chrome_profiler.json
.fast-check/
__pycache__/
.playwright-mcp/
.codex
.codex
.claude/scheduled_tasks.lock
AGENTS.md
+1
View File
@@ -109,4 +109,5 @@ $NAV --root backend callees "X" # what does X call?
- Search for existing code to reuse before writing new code
- Follow established patterns in the codebase
- Keep changes focused — don't refactor beyond what's asked
- **Comments record constraints, not narration.** Write a comment only for what the code can't show: why a non-obvious approach is required, what breaks if it's "simplified" away. State each invariant once, at the place where someone would break it, in ≤4 lines. Don't describe what the next line does, don't repeat the same rationale at multiple sites, and don't address the PR reviewer (justifying a change belongs in the PR description, not the code). Describe the code as it is, never its drafting history: "we no longer do X", "unchanged behavior", "instead of the previous approach" are meaningless to a reader who never saw the earlier iteration — before finishing, reread your comments as if the current state is the only state that ever existed.
- **Never attribute work to a specific customer, account, or "requested by a customer" in repo-tracked content** (PR descriptions, commit messages, code comments, docs). Describe changes by their technical motivation instead.
CHANGELOG.md
+414
View File
@@ -1,5 +1,419 @@
# Changelog
## [1.742.0](https://github.com/windmill-labs/windmill/compare/v1.741.0...v1.742.0) (2026-06-28)
### Features
* **apps:** add labels input to app editor deploy drawer ([#9828](https://github.com/windmill-labs/windmill/issues/9828)) ([da45e69](https://github.com/windmill-labs/windmill/commit/da45e699c8aefeede172c90769ef4f4b182fec0c))
* column-level lineage for DuckLake pipelines (SQL-AST inferred + traceable) ([#9814](https://github.com/windmill-labs/windmill/issues/9814)) ([003a262](https://github.com/windmill-labs/windmill/commit/003a262a4e9d6c2a63ada01aa8429aea1fbb6031))
### Bug Fixes
* **audit:** don't read pg_authid from an elevated context in S3 export migration ([#9832](https://github.com/windmill-labs/windmill/issues/9832)) ([75ba81b](https://github.com/windmill-labs/windmill/commit/75ba81b2d27fb0722095780312064cb93d20287e))
* close unauthenticated DAP debugger program-mode launch bypass ([#9829](https://github.com/windmill-labs/windmill/issues/9829)) ([c0768de](https://github.com/windmill-labs/windmill/commit/c0768de0acdf63eaba5fb97d04bfc64f2f03b93d))
* redeploy older app version from deployment history ([#9826](https://github.com/windmill-labs/windmill/issues/9826)) ([c479afa](https://github.com/windmill-labs/windmill/commit/c479afab8ebceccbee050e923dc5c27a6712ea62))
## [1.741.0](https://github.com/windmill-labs/windmill/compare/v1.740.0...v1.741.0) (2026-06-26)
### Features
* **ai-chat:** add create_folder tool to global chat ([#9819](https://github.com/windmill-labs/windmill/issues/9819)) ([44c25de](https://github.com/windmill-labs/windmill/commit/44c25de418612ab98341adb15d5671222b54367e))
* **ai-chat:** hint /compact in context usage tooltip ([#9777](https://github.com/windmill-labs/windmill/issues/9777)) ([aadfb62](https://github.com/windmill-labs/windmill/commit/aadfb620c0b7dcd7e94367b761875e14ef9abe69))
* **ai-chat:** let global chat edit the user's personal instructions ([#9771](https://github.com/windmill-labs/windmill/issues/9771)) ([3be2752](https://github.com/windmill-labs/windmill/commit/3be27521b05de33e48582e80c6651071f889f048))
* **ai-chat:** surface raw apps in the @-mention context picker ([#9800](https://github.com/windmill-labs/windmill/issues/9800)) ([1602244](https://github.com/windmill-labs/windmill/commit/16022447c7b445be753b9545b10b4c67da0893d5))
* capture managed-materialize output schema as asset metadata ([#2](https://github.com/windmill-labs/windmill/issues/2)a) ([#9812](https://github.com/windmill-labs/windmill/issues/9812)) ([ade74b2](https://github.com/windmill-labs/windmill/commit/ade74b297f6a03441e700a20ffc2d7291c8a85fd))
* **sdk:** allow overriding worker tag when running jobs (WIN-2105) ([#9807](https://github.com/windmill-labs/windmill/issues/9807)) ([52fc7bf](https://github.com/windmill-labs/windmill/commit/52fc7bf94cf3f87f68d9dba9884944d87e7d5d57))
### Bug Fixes
* apply step timeout to 'Test this step' preview ([#9810](https://github.com/windmill-labs/windmill/issues/9810)) ([d04062b](https://github.com/windmill-labs/windmill/commit/d04062bff58c9c4c79ce542a4321e71bcbcf0e98))
* **flows:** reject corrupt step paths at deploy + atomic cache writes ([#9751](https://github.com/windmill-labs/windmill/issues/9751)) ([#9813](https://github.com/windmill-labs/windmill/issues/9813)) ([3cda447](https://github.com/windmill-labs/windmill/commit/3cda44762148bcd2ee5c0ea821db884950376ead))
* **frontend:** clarify instance data table unavailable on cloud ([#9806](https://github.com/windmill-labs/windmill/issues/9806)) ([c3e8c78](https://github.com/windmill-labs/windmill/commit/c3e8c789ac05c9c28991d9ab6f2358f61fa87971))
* hide GCS service account key behind a reveal in object storage settings ([#9815](https://github.com/windmill-labs/windmill/issues/9815)) ([0ec5061](https://github.com/windmill-labs/windmill/commit/0ec5061270749ed078e01f5a4bc7397a1755ca32))
* ping job during volume setup to prevent false zombie restarts ([#9803](https://github.com/windmill-labs/windmill/issues/9803)) ([43bb676](https://github.com/windmill-labs/windmill/commit/43bb676dc5652cb06fe1414b8d3aacf295bae36b))
* skipped suspend step no longer parks the flow forever ([#9821](https://github.com/windmill-labs/windmill/issues/9821)) ([40110bc](https://github.com/windmill-labs/windmill/commit/40110bc7158bc42c3d84bd4637a12b82fcd72a9a))
### Performance Improvements
* **audit:** re-anchor S3 audit export on enable + opt-in backfill ([#9818](https://github.com/windmill-labs/windmill/issues/9818)) ([577ceee](https://github.com/windmill-labs/windmill/commit/577ceeee8679f054c6898d1a7889df30ab830f8f))
## [1.740.0](https://github.com/windmill-labs/windmill/compare/v1.739.0...v1.740.0) (2026-06-25)
### Features
* **api:** add structured endpoint for flow logs ([#9797](https://github.com/windmill-labs/windmill/issues/9797)) ([ba768fe](https://github.com/windmill-labs/windmill/commit/ba768fee888682cb50142d6e76c0422c40307f46))
* bounded-cascade selective execution for pipelines (UI + CLI) ([#9695](https://github.com/windmill-labs/windmill/issues/9695)) ([248540a](https://github.com/windmill-labs/windmill/commit/248540ac4d6e4ee9ee7c3e6f2cc822c63cc6426e))
* data tests for ducklake pipeline materialization ([#9708](https://github.com/windmill-labs/windmill/issues/9708)) ([f6998ec](https://github.com/windmill-labs/windmill/commit/f6998ec54cba2507703790bf33427e7567d42c4b))
* detect and guard against deploying stale drafts ([#9768](https://github.com/windmill-labs/windmill/issues/9768)) ([d865518](https://github.com/windmill-labs/windmill/commit/d8655189347f58df9d17e83dc55798baf7964279))
* ducklake time-travel UX (snapshot history + AT VERSION reads) ([#9709](https://github.com/windmill-labs/windmill/issues/9709)) ([d131d75](https://github.com/windmill-labs/windmill/commit/d131d754e1fc9674abf5de383d2bc93596df9bd1))
* self-host docs search for chat, mcp, cli; drop inkeep ([#9772](https://github.com/windmill-labs/windmill/issues/9772)) ([9d61e4e](https://github.com/windmill-labs/windmill/commit/9d61e4e59e4101de84217f7c7846f1aa94e84d89))
### Bug Fixes
* allow hyphens in postgresql database name validation ([#9782](https://github.com/windmill-labs/windmill/issues/9782)) ([170cd79](https://github.com/windmill-labs/windmill/commit/170cd79aaf92152fc3c0f675f155853c7f0e5b25))
* **debounce:** never supersede a running debounce survivor ([#9780](https://github.com/windmill-labs/windmill/issues/9780)) ([5549bdc](https://github.com/windmill-labs/windmill/commit/5549bdc67a5559a764616c44b1018543bc0568fe))
* decrypt secret variables via external backend in common resolvers ([#9784](https://github.com/windmill-labs/windmill/issues/9784)) ([cd42c6c](https://github.com/windmill-labs/windmill/commit/cd42c6ca18261328055554788932c3fe876a4a5b))
* enforce containment of python module dir for preview jobs ([#9704](https://github.com/windmill-labs/windmill/issues/9704)) ([88fca6a](https://github.com/windmill-labs/windmill/commit/88fca6a8c130b9e3b0f0cd410e422d4e074fc11f))
* **frontend:** apply script editor timeout to preview/Test runs ([#9794](https://github.com/windmill-labs/windmill/issues/9794)) ([6664ce6](https://github.com/windmill-labs/windmill/commit/6664ce6dc0c5fbc283303148de06d6bb85e4acf7))
* **frontend:** nested-loop "Test this step" resolves iter to innermost loop ([#9778](https://github.com/windmill-labs/windmill/issues/9778)) ([74ebfc6](https://github.com/windmill-labs/windmill/commit/74ebfc67f069047875db738926865bd4bd6fe9e9))
* opt out of Deno minimum-dependency-age for private npm registries ([#9802](https://github.com/windmill-labs/windmill/issues/9802)) ([b28f974](https://github.com/windmill-labs/windmill/commit/b28f974e5069f635419d9ea56fad6a0e417894e8))
* pass SSL cert env vars to `uv python install` ([#9790](https://github.com/windmill-labs/windmill/issues/9790)) ([962758c](https://github.com/windmill-labs/windmill/commit/962758c02de5f6d962c681fe9c39769b99429e8d))
* **python:** re-verify wheel RECORD on local cache reuse (once per worker) ([#9775](https://github.com/windmill-labs/windmill/issues/9775)) ([6c71c33](https://github.com/windmill-labs/windmill/commit/6c71c33470e3ea547f3b994db829eb4d04882443))
* **python:** serialize concurrent installs into shared wheel cache dir ([#9787](https://github.com/windmill-labs/windmill/issues/9787)) ([11d83ab](https://github.com/windmill-labs/windmill/commit/11d83ab1ec559be5d3263010228e6db65358e04b))
* re-pin stale-draft fork base when restoring an app deployment ([#9792](https://github.com/windmill-labs/windmill/issues/9792)) ([b9711e5](https://github.com/windmill-labs/windmill/commit/b9711e5ace8585315a1c2b85bb25ac8dd7832d6f))
* restore libargon2-1 for PHP runtime in server image ([#9795](https://github.com/windmill-labs/windmill/issues/9795)) ([e9cb806](https://github.com/windmill-labs/windmill/commit/e9cb80639b2dec63ede69fc3a4e3720bb1a3c319))
* use transaction for parallel_monitor_lock DELETE in last-iteration path ([#9789](https://github.com/windmill-labs/windmill/issues/9789)) ([754cae9](https://github.com/windmill-labs/windmill/commit/754cae956ac8d431ddeb055453835e249cdd07b7))
### Performance Improvements
* drop v2_job side-table ON DELETE CASCADE FKs to speed retention deletes ([#9786](https://github.com/windmill-labs/windmill/issues/9786)) ([aa098c7](https://github.com/windmill-labs/windmill/commit/aa098c70c0271b2b1917749d1f607c0559cf04de))
* eliminate dual-connection DB pool contention across worker, queue, and api ([#9798](https://github.com/windmill-labs/windmill/issues/9798)) ([0dbd9c1](https://github.com/windmill-labs/windmill/commit/0dbd9c1231b00d4693af68835fe1d9e7c8869b43))
## [1.739.0](https://github.com/windmill-labs/windmill/compare/v1.738.0...v1.739.0) (2026-06-24)
### Features
* add /compact session chat command ([#9764](https://github.com/windmill-labs/windmill/issues/9764)) ([83cc553](https://github.com/windmill-labs/windmill/commit/83cc5533ee92e59356a117eefeaf42dad23287f6))
* add session chat slash commands ([#9748](https://github.com/windmill-labs/windmill/issues/9748)) ([24b95e9](https://github.com/windmill-labs/windmill/commit/24b95e9fe12ba4abdfe1ff6e9f9fe42cb2ded011))
* **ai-chat:** add /clear session command to start a fresh conversation ([#9769](https://github.com/windmill-labs/windmill/issues/9769)) ([3fafac2](https://github.com/windmill-labs/windmill/commit/3fafac275d2100a6f89924040650cf959945d209))
* **ai-chat:** context usage gauge + unified model settings menu ([#9763](https://github.com/windmill-labs/windmill/issues/9763)) ([2e020b2](https://github.com/windmill-labs/windmill/commit/2e020b2ccc7a649a5923bff72a98f07d4fc85381))
* **apps:** show raw-app fork diffs as per-file tree items ([#9491](https://github.com/windmill-labs/windmill/issues/9491)) ([e98df38](https://github.com/windmill-labs/windmill/commit/e98df38ac43823ee85209a4b09cd70690469302d))
* **frontend:** add filter submenu to collapsed AI sessions popover ([#9757](https://github.com/windmill-labs/windmill/issues/9757)) ([3d48ba7](https://github.com/windmill-labs/windmill/commit/3d48ba7738c3d3356539b5fc44a871f6b7f9d548))
* **frontend:** restore raw app 'open preview in separate window' ([#9765](https://github.com/windmill-labs/windmill/issues/9765)) ([a116715](https://github.com/windmill-labs/windmill/commit/a116715c418c39d48a91e6c0b4484a31537dff38))
* **frontend:** show approval wait as a distinct segment in flow timeline ([#9756](https://github.com/windmill-labs/windmill/issues/9756)) ([2a70ccc](https://github.com/windmill-labs/windmill/commit/2a70ccc38675c7c2353807a4f85764a8a35224e2))
* scope AI sessions per workspace root with lifecycle reconcile ([#9734](https://github.com/windmill-labs/windmill/issues/9734)) ([42c5e7a](https://github.com/windmill-labs/windmill/commit/42c5e7a3fc9b74256de8806ec0d7b62e8bbf029c))
### Bug Fixes
* **ai-chat:** strip unclosed &lt;summary&gt; tag leaking into compaction summary ([#9750](https://github.com/windmill-labs/windmill/issues/9750)) ([250a05f](https://github.com/windmill-labs/windmill/commit/250a05f544ae397bb91af5fc83bf408cfe1c554d))
* **apps:** realign legacy raw-app drafts to raw_app draft kind ([#9761](https://github.com/windmill-labs/windmill/issues/9761)) ([288318a](https://github.com/windmill-labs/windmill/commit/288318ac269714fc03b15622dbb86b1c28268a36))
* **backend:** resolve folder_labels search_path on non-public (PG_SCHEMA) schemas ([#9758](https://github.com/windmill-labs/windmill/issues/9758)) ([f582878](https://github.com/windmill-labs/windmill/commit/f5828780fd6a8be070b2933ebd41ee6dff98a9e1))
* forbid superadmin job tokens from global user and token management ([#9715](https://github.com/windmill-labs/windmill/issues/9715)) ([043c2c0](https://github.com/windmill-labs/windmill/commit/043c2c05b7678c49faca0ccb28e5f6393567ba4d))
* **frontend:** highlight the runtime-chosen branch in flow graph viewer ([#9755](https://github.com/windmill-labs/windmill/issues/9755)) ([de6192b](https://github.com/windmill-labs/windmill/commit/de6192bec1695883a07452f7db2fb51c94dbfd43))
* **frontend:** keep #content portal target present on AI-session route ([#9754](https://github.com/windmill-labs/windmill/issues/9754)) ([5e09c50](https://github.com/windmill-labs/windmill/commit/5e09c501713ebbe05b28ce0084eca641f0dbe95c))
* **frontend:** show AI skills settings only when global mode enabled ([#9747](https://github.com/windmill-labs/windmill/issues/9747)) ([c017f7f](https://github.com/windmill-labs/windmill/commit/c017f7f8919a51292ddf01574961d1774bc1ba23))
* **frontend:** stop flow step id generation from being poisoned by non-canonical keys ([#9766](https://github.com/windmill-labs/windmill/issues/9766)) ([4dbf873](https://github.com/windmill-labs/windmill/commit/4dbf8737238ccc4dc2c67365e6d43f04f46c75b5))
* persist on-behalf-of user across app deploy paths ([#9773](https://github.com/windmill-labs/windmill/issues/9773)) ([f99781c](https://github.com/windmill-labs/windmill/commit/f99781ca5f77248206c951935cc44acfa5f072eb))
* reject symlink traversal in job-dir path validation ([#9713](https://github.com/windmill-labs/windmill/issues/9713)) ([b5bd824](https://github.com/windmill-labs/windmill/commit/b5bd8245d81b84fc14d3ea955bf1e66ac576bf37))
### Performance Improvements
* **audit:** adaptive timestamp floor for S3 audit-log export ([#9752](https://github.com/windmill-labs/windmill/issues/9752)) ([55bed4a](https://github.com/windmill-labs/windmill/commit/55bed4abcfce2a611b16054573980d2eb613ccb3))
* **monitor:** vacuum job_perms/job_result_stream right after each orphan sweep ([#9753](https://github.com/windmill-labs/windmill/issues/9753)) ([8912e21](https://github.com/windmill-labs/windmill/commit/8912e21d1571e57b5cf21b7d4d9520e20a28e70d))
## [1.738.0](https://github.com/windmill-labs/windmill/compare/v1.737.0...v1.738.0) (2026-06-23)
### Features
* add resource and infrastructure telemetry ([#9737](https://github.com/windmill-labs/windmill/issues/9737)) ([9793d01](https://github.com/windmill-labs/windmill/commit/9793d01575415963a89609a1baf2cd64f0d050cc))
* render mermaid diagrams in chat code blocks ([#9738](https://github.com/windmill-labs/windmill/issues/9738)) ([cfb9f1d](https://github.com/windmill-labs/windmill/commit/cfb9f1dbc23110ecf8f91bb3c8c81fc6e35dc09b))
### Bug Fixes
* **ai-chat:** Fix incorrect editor edits from ai chat [#1](https://github.com/windmill-labs/windmill/issues/1) ([#9741](https://github.com/windmill-labs/windmill/issues/9741)) ([fc797a3](https://github.com/windmill-labs/windmill/commit/fc797a35fe7885630c81453df0fc94769e73873a))
* allow object storage test for non-super-admins, harden on cloud ([#9739](https://github.com/windmill-labs/windmill/issues/9739)) ([24446e8](https://github.com/windmill-labs/windmill/commit/24446e80093ade349f7fbf65063d2d1cb5551c1e))
* **frontend:** debounce external code→Monaco sync in Editor ([#9743](https://github.com/windmill-labs/windmill/issues/9743)) ([29c67ce](https://github.com/windmill-labs/windmill/commit/29c67ced97bf2919584986f9d9eceb4337c34ad9))
* **frontend:** preserve editor content when closing instance settings drawer ([#9740](https://github.com/windmill-labs/windmill/issues/9740)) ([11d0e65](https://github.com/windmill-labs/windmill/commit/11d0e65f3af9a048bc1921bbdd3d676a07483a57))
* pipeline annotation false-positives from body comments ([#9736](https://github.com/windmill-labs/windmill/issues/9736)) ([984ea72](https://github.com/windmill-labs/windmill/commit/984ea728d98649b66b1cae899bdab9af3176caa7))
* preserve fork parent linkage on workspace id change ([#9716](https://github.com/windmill-labs/windmill/issues/9716)) ([cbf54d4](https://github.com/windmill-labs/windmill/commit/cbf54d4eb432638e27f67c4c8b879cbcc0291da3))
* prevent variable push from corrupting is_secret variables ([#9705](https://github.com/windmill-labs/windmill/issues/9705)) ([ba4b368](https://github.com/windmill-labs/windmill/commit/ba4b368706e95e22f346a10e5fe145b0795ac3f6))
### Performance Improvements
* **monitor:** skip protected prefix in retention delete via cross-batch watermark (WIN-2088) ([#9744](https://github.com/windmill-labs/windmill/issues/9744)) ([e90b2be](https://github.com/windmill-labs/windmill/commit/e90b2be8fade1eb78cd685890291f5a4553a6a10))
## [1.737.0](https://github.com/windmill-labs/windmill/compare/v1.736.0...v1.737.0) (2026-06-23)
### Features
* **apps:** opt-in sandbox isolation for published & raw apps (alpha) ([#9420](https://github.com/windmill-labs/windmill/issues/9420)) ([2879cbb](https://github.com/windmill-labs/windmill/commit/2879cbb65a4122c86b4a472206d74d9009b07904))
### Bug Fixes
* allow SQL args in managed // materialize scripts ([#9733](https://github.com/windmill-labs/windmill/issues/9733)) ([fa35968](https://github.com/windmill-labs/windmill/commit/fa3596885bf2d7ee8859f295e820ee362c756911))
* bound orphan-cleanup drain rate with capped multi-batch loop ([#9730](https://github.com/windmill-labs/windmill/issues/9730)) ([31d9215](https://github.com/windmill-labs/windmill/commit/31d9215e5a61f19662cc87be8147007e1d47ebb6))
* **ext-jwt:** reject external JWT auth for non-existent workspaces ([#9723](https://github.com/windmill-labs/windmill/issues/9723)) ([c644311](https://github.com/windmill-labs/windmill/commit/c644311eca4bcaf4b68058cf1d5d79d4078aee1a))
* optimize cleanup_job_perms_orphaned and job_result_stream cleanup queries ([#9727](https://github.com/windmill-labs/windmill/issues/9727)) ([6d94865](https://github.com/windmill-labs/windmill/commit/6d9486510933af9109f52011d93b13847dbdbb39))
* prevent silent audit-partition outage via monitor watchdog + alert ([#9729](https://github.com/windmill-labs/windmill/issues/9729)) ([8dea383](https://github.com/windmill-labs/windmill/commit/8dea38383f884f59b2956c39f1424005a21265bd))
### Performance Improvements
* **monitor:** hash active-root exclusion in retention delete (WIN-2088) ([#9732](https://github.com/windmill-labs/windmill/issues/9732)) ([75bafab](https://github.com/windmill-labs/windmill/commit/75bafabeeec76cf6da33eef41f588e37071df011))
## [1.736.0](https://github.com/windmill-labs/windmill/compare/v1.735.0...v1.736.0) (2026-06-23)
### Features
* **ai-chat:** workspace AI chat skills (SKILL.md upload + read_skill tool) ([#9648](https://github.com/windmill-labs/windmill/issues/9648)) ([6f4017d](https://github.com/windmill-labs/windmill/commit/6f4017d694494a158ebd0579c93336c378cba0fd))
### Bug Fixes
* **drafts:** stop mis-filing workspace-blind legacy drafts on migration ([#9725](https://github.com/windmill-labs/windmill/issues/9725)) ([3bf5b72](https://github.com/windmill-labs/windmill/commit/3bf5b72afab3241ea41a261a40c2434764bdaf72))
* **frontend:** destroy old WebsocketProvider on workspace switch in MultiplayerMenu ([#9719](https://github.com/windmill-labs/windmill/issues/9719)) ([6e96f90](https://github.com/windmill-labs/windmill/commit/6e96f90065dfe2f6ccc5eb4f85f4facd3515c70c))
* **frontend:** ensure type:object in test_run_flow tool schema for Anthropic ([#9721](https://github.com/windmill-labs/windmill/issues/9721)) ([d5cb944](https://github.com/windmill-labs/windmill/commit/d5cb944cf92f074b2ee42c876595eacdfa2f4d76))
* **health:** detect read-only replica via pg_is_in_recovery() ([#9722](https://github.com/windmill-labs/windmill/issues/9722)) ([e16061d](https://github.com/windmill-labs/windmill/commit/e16061df06babeae935a9396de5bdcd46e8119a9))
* re-enforce scoped API token boundaries across handlers ([#9712](https://github.com/windmill-labs/windmill/issues/9712)) ([e19594d](https://github.com/windmill-labs/windmill/commit/e19594df2ad015a0336ade95e04562f5562ec3f6))
## [1.735.0](https://github.com/windmill-labs/windmill/compare/v1.734.0...v1.735.0) (2026-06-22)
### Features
* clarify session draft bar tracks all workspace draft changes ([#9714](https://github.com/windmill-labs/windmill/issues/9714)) ([ed016a5](https://github.com/windmill-labs/windmill/commit/ed016a5edb4527877bf7e6bf92feafc2710669c2))
* **copilot:** improve global-mode path selection + add path-selection evals ([#9698](https://github.com/windmill-labs/windmill/issues/9698)) ([74a2329](https://github.com/windmill-labs/windmill/commit/74a2329d2e8395141807c43acef07ef132490039))
* link files & folders to the global AI chat ([#9520](https://github.com/windmill-labs/windmill/issues/9520)) ([84cc043](https://github.com/windmill-labs/windmill/commit/84cc043406d63a1e1472165cf24ce8c09905fc5f))
* scope default instance db name to workspace (dt_/dl_) ([#9699](https://github.com/windmill-labs/windmill/issues/9699)) ([4a8a724](https://github.com/windmill-labs/windmill/commit/4a8a724895dcecb835e1eb1e4fd7d1bbc8b3e0fb))
### Bug Fixes
* enforce job_dir containment when writing module files ([#9703](https://github.com/windmill-labs/windmill/issues/9703)) ([e403f92](https://github.com/windmill-labs/windmill/commit/e403f92d7e84cebc78709dce1a0928048ba2506d))
* **frontend:** deploy full script/flow draft from AI chat via shared module ([#9642](https://github.com/windmill-labs/windmill/issues/9642)) ([23bf6bf](https://github.com/windmill-labs/windmill/commit/23bf6bf3da01d552d2dfab6dbcfd30f758ed34d2))
* **frontend:** strip raw-app post-deploy diff noise (raw_app/lock/data) ([#9706](https://github.com/windmill-labs/windmill/issues/9706)) ([e20a277](https://github.com/windmill-labs/windmill/commit/e20a27745a08d552e6d2c5a8bbaf08ccfe89c68f))
* ignore NotFound errors when deleting log files from object store ([#9707](https://github.com/windmill-labs/windmill/issues/9707)) ([8a0b0ab](https://github.com/windmill-labs/windmill/commit/8a0b0abead71320c4f69eb3007739a19f76d4126))
* **oauth:** restore bring-your-own CC token URL override ([#9711](https://github.com/windmill-labs/windmill/issues/9711)) ([ef4962e](https://github.com/windmill-labs/windmill/commit/ef4962e52aba0bc79bf72523de9101853c660654))
* sanitize git credentials from ansible executor errors and logs ([#9697](https://github.com/windmill-labs/windmill/issues/9697)) ([ace7b68](https://github.com/windmill-labs/windmill/commit/ace7b68a28b00d715298ffcb6ae907c1974a74b8))
## [1.734.0](https://github.com/windmill-labs/windmill/compare/v1.733.1...v1.734.0) (2026-06-20)
### Features
* ducklake materialization for data pipelines ([#9689](https://github.com/windmill-labs/windmill/issues/9689)) ([3ebf243](https://github.com/windmill-labs/windmill/commit/3ebf24359d66048d6361ce65cd879cdc04b737ed))
### Bug Fixes
* **frontend:** clear branch step state when switching outer loop iterations ([#9650](https://github.com/windmill-labs/windmill/issues/9650)) ([09a8004](https://github.com/windmill-labs/windmill/commit/09a80040ca268a4379d5302e3401435ba93247e0))
## [1.733.1](https://github.com/windmill-labs/windmill/compare/v1.733.0...v1.733.1) (2026-06-19)
### Bug Fixes
* **backend:** validate ansible vault_id entries before config generation ([#9681](https://github.com/windmill-labs/windmill/issues/9681)) ([c1f31c0](https://github.com/windmill-labs/windmill/commit/c1f31c0e4777bf0cfed0dd7f03249e9a61cd8cb9))
* **frontend:** group live pipeline runs in the activity panel ([#9684](https://github.com/windmill-labs/windmill/issues/9684)) ([1be4df9](https://github.com/windmill-labs/windmill/commit/1be4df9acb935250d4cc12e83cf67e366d870d5a))
* require super admin for object storage config test endpoint ([#9683](https://github.com/windmill-labs/windmill/issues/9683)) ([fb44fe7](https://github.com/windmill-labs/windmill/commit/fb44fe7af2b8ebe8ef64ffb0e5acce8580bf4200))
* validate websocket trigger urls and gate trigger test route ([#9682](https://github.com/windmill-labs/windmill/issues/9682)) ([c39ee07](https://github.com/windmill-labs/windmill/commit/c39ee07c0bcd2249dd19ffa5cd988125eefc6c9f))
## [1.733.0](https://github.com/windmill-labs/windmill/compare/v1.732.0...v1.733.0) (2026-06-19)
### Features
* **ai-chat:** cap read_app_file + search_app grep tool to bound context in large raw apps ([#9653](https://github.com/windmill-labs/windmill/issues/9653)) ([4296a6a](https://github.com/windmill-labs/windmill/commit/4296a6ae1f73564de4df54fe1df0a03c1df05dfd))
* **python, windows:** enable S3 to cache wheels ([#5199](https://github.com/windmill-labs/windmill/issues/5199)) ([ab3bc97](https://github.com/windmill-labs/windmill/commit/ab3bc97cd92b6480327029bcf018280442462af7))
### Bug Fixes
* allow users to always discard their own drafts without write permission ([#9659](https://github.com/windmill-labs/windmill/issues/9659)) ([6833a55](https://github.com/windmill-labs/windmill/commit/6833a554aeddb3e63173d3c3140b490c0bf2822b))
* **backend:** clean up unique_ext_jwt_token on workspace deletion ([#9676](https://github.com/windmill-labs/windmill/issues/9676)) ([9add719](https://github.com/windmill-labs/windmill/commit/9add719d936cdcfb2c4062629e3e1f792694dafe))
* **backend:** strip NUL bytes from draft values on write ([#9673](https://github.com/windmill-labs/windmill/issues/9673)) ([924f9c7](https://github.com/windmill-labs/windmill/commit/924f9c7e8d8863d9af40aee246a519b4be0e1ea2))
* **python:** split PIP_TRUSTED_HOST by whitespace to support multiple hosts ([#9675](https://github.com/windmill-labs/windmill/issues/9675)) ([cafb473](https://github.com/windmill-labs/windmill/commit/cafb473494d9cff3a8b2aeaf9f18b015f966e7b3))
## [1.732.0](https://github.com/windmill-labs/windmill/compare/v1.731.0...v1.732.0) (2026-06-19)
### Features
* **ansible:** add AI chat and editor bar buttons for ansible ([#9671](https://github.com/windmill-labs/windmill/issues/9671)) ([017c3d3](https://github.com/windmill-labs/windmill/commit/017c3d3343c2577501103be4b2dd8dac9727d80d))
### Bug Fixes
* **ai:** emit token usage in gemini proxy streaming translation ([#9669](https://github.com/windmill-labs/windmill/issues/9669)) ([0cc2257](https://github.com/windmill-labs/windmill/commit/0cc2257596a3965cf6db21a0090edcce6e1b8419))
* **backend:** grant script_trigger access to windmill roles ([#9674](https://github.com/windmill-labs/windmill/issues/9674)) ([3361736](https://github.com/windmill-labs/windmill/commit/33617367d09537667d2ab3f91135c736194b9e7e))
* **frontend:** ignore hash/assets in script diffs and drafts (WIN-2071) ([#9664](https://github.com/windmill-labs/windmill/issues/9664)) ([3371265](https://github.com/windmill-labs/windmill/commit/33712653821e83f2562dd5f271dbec0188d5d2f8))
## [1.731.0](https://github.com/windmill-labs/windmill/compare/v1.730.0...v1.731.0) (2026-06-19)
### Features
* **backend:** auto-reconnect postgres trigger listener with backoff (WIN-2073) ([#9666](https://github.com/windmill-labs/windmill/issues/9666)) ([a425431](https://github.com/windmill-labs/windmill/commit/a425431e9067bcf85474fdc7b7ef7f73e41b9071))
### Bug Fixes
* **backend:** grant notify_event access to windmill roles ([#9665](https://github.com/windmill-labs/windmill/issues/9665)) ([a682d02](https://github.com/windmill-labs/windmill/commit/a682d02311a2110bfc0d5e0a5b52e96147fe0dd7))
* **mcp:** repair invalid type keywords in tool JSON schemas ([#9667](https://github.com/windmill-labs/windmill/issues/9667)) ([c30bdec](https://github.com/windmill-labs/windmill/commit/c30bdecea77ff9b4d74d52961f3101201099b683))
* trigger flow error handler on unrecoverable (OOM/zombie) step failures ([#9662](https://github.com/windmill-labs/windmill/issues/9662)) ([7e4df02](https://github.com/windmill-labs/windmill/commit/7e4df02bd60c4d6ee8c92d3dfd19f4e587ff9632))
## [1.730.0](https://github.com/windmill-labs/windmill/compare/v1.729.0...v1.730.0) (2026-06-18)
### Features
* **ai-chat:** summary-based conversation compaction ([#9645](https://github.com/windmill-labs/windmill/issues/9645)) ([5d553b8](https://github.com/windmill-labs/windmill/commit/5d553b81c06664aab61131a93b198575c088d12d))
* Data Pipelines alpha ([#9193](https://github.com/windmill-labs/windmill/issues/9193)) ([7155a0b](https://github.com/windmill-labs/windmill/commit/7155a0bb96cf30bd878272a0f4c3c3b02341b261))
### Bug Fixes
* **ai-chat:** stop echoing app draft value in global chat write tool results ([#9658](https://github.com/windmill-labs/windmill/issues/9658)) ([2fed808](https://github.com/windmill-labs/windmill/commit/2fed808b9e716d9a44b34c7a073ec0d37374be05))
* **backend:** include raw_app drafts in list_apps draft_users ([#9647](https://github.com/windmill-labs/windmill/issues/9647)) ([19bc005](https://github.com/windmill-labs/windmill/commit/19bc0052f1069d732231950a0ec958f675d57417))
* **frontend:** keep ?new_draft flag until first save is confirmed ([#9656](https://github.com/windmill-labs/windmill/issues/9656)) ([9b6b7c3](https://github.com/windmill-labs/windmill/commit/9b6b7c3862d9988e5e91eaab2b967a23f41cdc0d))
* **frontend:** re-key raw-app autosave on post-deploy navigation ([#9646](https://github.com/windmill-labs/windmill/issues/9646)) ([1058bde](https://github.com/windmill-labs/windmill/commit/1058bdeccdc4c403ef4599db0ee74a65a66c715f))
* gate agent-worker global setting reads with a blocklist ([#9623](https://github.com/windmill-labs/windmill/issues/9623)) ([fdd82f0](https://github.com/windmill-labs/windmill/commit/fdd82f0c48f29805cd9e219649f27fba45c7fd92))
* **workspaces:** add instance setting to disable workspace invite/add emails ([#9643](https://github.com/windmill-labs/windmill/issues/9643)) ([796230d](https://github.com/windmill-labs/windmill/commit/796230d90a7e6d1debc15e139ab708881e527862))
## [1.729.0](https://github.com/windmill-labs/windmill/compare/v1.728.1...v1.729.0) (2026-06-18)
### Features
* add ducklake schema support to the database manager ([#9633](https://github.com/windmill-labs/windmill/issues/9633)) ([3eeccaf](https://github.com/windmill-labs/windmill/commit/3eeccaf9682b7803fdf5be8dcbc4d243e0ba2e49))
* **ai-chat:** self-hosted docs tools via windmill.dev llms.txt + ask benchmark ([#9578](https://github.com/windmill-labs/windmill/issues/9578)) ([f4425fc](https://github.com/windmill-labs/windmill/commit/f4425fca9fb0d02b845bd72888ade54905c5a30b))
* **frontend:** View Diff and in-place Load for other users' drafts ([#9621](https://github.com/windmill-labs/windmill/issues/9621)) ([5508f1d](https://github.com/windmill-labs/windmill/commit/5508f1da9cd04c2583eb3f7ee6bce19d067f2227))
* per-user draft review & deploy page (gating, badges, rename, raw-app deploy fixes) ([#9625](https://github.com/windmill-labs/windmill/issues/9625)) ([e09cd58](https://github.com/windmill-labs/windmill/commit/e09cd5862cb636e143027fe8d9a5be9c7097b031))
* queue messages typed while ai chat is streaming ([#9525](https://github.com/windmill-labs/windmill/issues/9525)) ([51bd869](https://github.com/windmill-labs/windmill/commit/51bd8692a482850f7ac8b04dd16db5876336b5b9))
* zero-setup oauth client credentials for registry providers ([#9559](https://github.com/windmill-labs/windmill/issues/9559)) ([e26a923](https://github.com/windmill-labs/windmill/commit/e26a9239a62a25abf90ef06ade4dde7f36e791bb))
### Bug Fixes
* **ai_evals:** adapt global eval harness to DB-backed user drafts ([#9641](https://github.com/windmill-labs/windmill/issues/9641)) ([e87ff79](https://github.com/windmill-labs/windmill/commit/e87ff79ecf6a6e0958916ed1b3756fb3addf719f))
* **drafts:** preserve original timestamp when migrating localStorage drafts ([#9638](https://github.com/windmill-labs/windmill/issues/9638)) ([8021775](https://github.com/windmill-labs/windmill/commit/8021775f5f961ef6fd01b022639b85855326a1da))
* **frontend:** don't save drafts on leave when auto-save is off, warn instead ([#9630](https://github.com/windmill-labs/windmill/issues/9630)) ([2523465](https://github.com/windmill-labs/windmill/commit/252346500945a9571af744c839ac0c7d6870504f))
* **frontend:** render Modal2 dialogs above the AI chat panel ([#9636](https://github.com/windmill-labs/windmill/issues/9636)) ([b67c8cf](https://github.com/windmill-labs/windmill/commit/b67c8cf42b477575fc1bc448058ec0d3b7e54fee))
* **frontend:** show AI sessions when AI unconfigured, with disabled chat ([#9644](https://github.com/windmill-labs/windmill/issues/9644)) ([ba69d81](https://github.com/windmill-labs/windmill/commit/ba69d8147b615e160cf3d2885fc65a0777b78b71))
* **git-sync:** bump default sync script to hub/28719 (windmill-cli 1.728.1) for WAC modules ([#9649](https://github.com/windmill-labs/windmill/issues/9649)) ([3c0e38b](https://github.com/windmill-labs/windmill/commit/3c0e38b5890d77983cb5cf5f422a62a73e7a4f22))
## [1.728.1](https://github.com/windmill-labs/windmill/compare/v1.728.0...v1.728.1) (2026-06-17)
### Bug Fixes
* **backend:** purge workspace_diff cache on workspace delete ([#9627](https://github.com/windmill-labs/windmill/issues/9627)) ([8a3f69d](https://github.com/windmill-labs/windmill/commit/8a3f69dda8f2088fb859ed8ed6e54458940423d0))
* **cli:** fall back to esbuild-wasm on native host/binary mismatch ([#9629](https://github.com/windmill-labs/windmill/issues/9629)) ([86d1d16](https://github.com/windmill-labs/windmill/commit/86d1d160f0d3bd9faabdafada07e2956dd98445d))
* **frontend:** persist session-editor draft path/summary edits + per-line diff tooltips ([#9622](https://github.com/windmill-labs/windmill/issues/9622)) ([e4bfeb2](https://github.com/windmill-labs/windmill/commit/e4bfeb29bc4e89669863b5f6396904a331167658))
## [1.728.0](https://github.com/windmill-labs/windmill/compare/v1.727.0...v1.728.0) (2026-06-16)
### Features
* **frontend:** adapt AI-chat/sessions drafts to DB-backed model ([#9601](https://github.com/windmill-labs/windmill/issues/9601)) ([611c70a](https://github.com/windmill-labs/windmill/commit/611c70acd211cf4b8f8308da4a264c670a2f5f43))
* **frontend:** consolidate draft-migration errors into a single toast + modal ([#9612](https://github.com/windmill-labs/windmill/issues/9612)) ([bc0d5bf](https://github.com/windmill-labs/windmill/commit/bc0d5bf241df3633921bd9d43d171e91034fbfcf))
* **frontend:** dedup user drafts against the deployed baseline ([#9618](https://github.com/windmill-labs/windmill/issues/9618)) ([a2ce446](https://github.com/windmill-labs/windmill/commit/a2ce44645fdbfa98bf250fac2d15d2b5b26c4b47))
### Bug Fixes
* **frontend:** reset deleteWorkspaceForkModal on confirm in SidebarContent ([#9619](https://github.com/windmill-labs/windmill/issues/9619)) ([7cb5c6e](https://github.com/windmill-labs/windmill/commit/7cb5c6e749b2020dee5ee1499f0dc69c5109a6d8))
* **frontend:** session Drafts drawer uses raw_app kind for the raw-app diff ([#9617](https://github.com/windmill-labs/windmill/issues/9617)) ([46288b6](https://github.com/windmill-labs/windmill/commit/46288b6143efae4dfdf6fe068b97a1e8831fce6a))
* **nativets:** respect custom CA certs in in-process fetch runtime ([#9615](https://github.com/windmill-labs/windmill/issues/9615)) ([41562c7](https://github.com/windmill-labs/windmill/commit/41562c7d7c708d7d056d9b3d0c39b994a6f4a016))
* **ResourceForm:** initialize JSON editor when resource type schema is unavailable ([#9611](https://github.com/windmill-labs/windmill/issues/9611)) ([5a24057](https://github.com/windmill-labs/windmill/commit/5a2405743b4622fc1021109114d007057abd5dfd))
* show folder labels in the folder list table ([#9620](https://github.com/windmill-labs/windmill/issues/9620)) ([651fa13](https://github.com/windmill-labs/windmill/commit/651fa13ee80ff76e5a53ef1ed545b03ce6792294))
* show last updated date per user in other-users-drafts modal ([#9614](https://github.com/windmill-labs/windmill/issues/9614)) ([f6104ce](https://github.com/windmill-labs/windmill/commit/f6104ce05c4005ffb9fe8112782d1ef6d3065300))
## [1.727.0](https://github.com/windmill-labs/windmill/compare/v1.726.1...v1.727.0) (2026-06-16)
### Features
* support temp_script_refs in wmill dev for local relative imports ([#9554](https://github.com/windmill-labs/windmill/issues/9554)) ([33ac287](https://github.com/windmill-labs/windmill/commit/33ac287065742df53f363a5fe09f54f5584a85a6))
### Bug Fixes
* **cli:** harden legacy flow lock migration ordering and collision guard ([#9557](https://github.com/windmill-labs/windmill/issues/9557)) ([cd09870](https://github.com/windmill-labs/windmill/commit/cd098700c2cd8d7e9150f760938f4eaf34d188ec))
* **cli:** include __mod/ folder in gitSyncIncludePattern for scripts ([#9606](https://github.com/windmill-labs/windmill/issues/9606)) ([252c1b3](https://github.com/windmill-labs/windmill/commit/252c1b35fc716c3486109d89615127c588bbe90a))
* **frontend:** allow same-origin redirects in isValidLogoutRedirect ([#9568](https://github.com/windmill-labs/windmill/issues/9568)) ([8500435](https://github.com/windmill-labs/windmill/commit/8500435e82231e13a1b8a874fd0545f0a0a73fee))
* **frontend:** make UserDraft read-after-write work without live entry ([#9609](https://github.com/windmill-labs/windmill/issues/9609)) ([51e82d7](https://github.com/windmill-labs/windmill/commit/51e82d7c6d30c66c84236feb743c09929934e564))
* **frontend:** seed detached user-draft handles so new-item drawers render ([#9608](https://github.com/windmill-labs/windmill/issues/9608)) ([9e3c0de](https://github.com/windmill-labs/windmill/commit/9e3c0decf95378c66055d82215c15cd3bf4a69cb))
* **frontend:** strip server-managed fields from value diffs ([#9599](https://github.com/windmill-labs/windmill/issues/9599)) ([c213801](https://github.com/windmill-labs/windmill/commit/c213801b5aee54d801c14b9eb31422f2a312ef7e))
## [1.726.1](https://github.com/windmill-labs/windmill/compare/v1.726.0...v1.726.1) (2026-06-15)
### Bug Fixes
* **apps:** prevent decision tree graph editor crash on missing graph context ([#9602](https://github.com/windmill-labs/windmill/issues/9602)) ([24f3259](https://github.com/windmill-labs/windmill/commit/24f32596e9ca39c963c19af4a8b14fbcd04e3a78))
* db-backed draft fixes — review-page UX, legacy drafts, session restore ([#9600](https://github.com/windmill-labs/windmill/issues/9600)) ([4e4b224](https://github.com/windmill-labs/windmill/commit/4e4b2247ef471dada1b8c894974fe921d14b3947))
## [1.726.0](https://github.com/windmill-labs/windmill/compare/v1.725.1...v1.726.0) (2026-06-15)
### Features
* **audit:** record workspace archive/unarchive/delete in instance audit log ([#9596](https://github.com/windmill-labs/windmill/issues/9596)) ([9de5708](https://github.com/windmill-labs/windmill/commit/9de57086086bb5626d175c7f926915d1d6ac67ca))
* **frontend:** add user-level toggle to disable Windmill AI ([#9585](https://github.com/windmill-labs/windmill/issues/9585)) ([5709a56](https://github.com/windmill-labs/windmill/commit/5709a564fbafd9aa91943572ecd8c3e0c45c20b1))
### Bug Fixes
* **embeddings:** retry HuggingFace model downloads with backoff ([#9597](https://github.com/windmill-labs/windmill/issues/9597)) ([6a62959](https://github.com/windmill-labs/windmill/commit/6a6295921d681359155d814507908792be405679))
* resolve release CI failures (pypi bundle, flow serde test, cli windows) ([#9595](https://github.com/windmill-labs/windmill/issues/9595)) ([5ccaae8](https://github.com/windmill-labs/windmill/commit/5ccaae8ab36f2be18b67863ea069763455908029))
## [1.725.1](https://github.com/windmill-labs/windmill/compare/v1.725.0...v1.725.1) (2026-06-15)
### Bug Fixes
* **apps:** apply scope-path predicate to app list/search endpoints ([#9581](https://github.com/windmill-labs/windmill/issues/9581)) ([3bf6e10](https://github.com/windmill-labs/windmill/commit/3bf6e102afbdad41e558617bc812012eaaaecd9b))
* **auth:** add scope checks to scripts/flows list_tokens endpoints ([#9582](https://github.com/windmill-labs/windmill/issues/9582)) ([36c9f86](https://github.com/windmill-labs/windmill/commit/36c9f8612b5778aa2c981454729590b71671ce8d))
* **cli:** preserve committed script.lock on transient NULL lock during git-sync deploy ([#9593](https://github.com/windmill-labs/windmill/issues/9593)) ([6b916ac](https://github.com/windmill-labs/windmill/commit/6b916ac688e0305284e6cf819bf28803bcca0118))
* expose parent_hash in MCP createScript tool for updates ([#9586](https://github.com/windmill-labs/windmill/issues/9586)) ([a69505d](https://github.com/windmill-labs/windmill/commit/a69505df9bf25d7c4f11d0528a7450c08dbb422c))
* **flows:** stop serializing default retry/stop_after_if fields ([#9583](https://github.com/windmill-labs/windmill/issues/9583)) ([e1e2a24](https://github.com/windmill-labs/windmill/commit/e1e2a24b6a6752b3ac779cb0db38061cbc54425e))
* **security:** sanitize dependency names & connection strings against command/SQL injection ([#9590](https://github.com/windmill-labs/windmill/issues/9590)) ([aff0a4e](https://github.com/windmill-labs/windmill/commit/aff0a4ec189cd8e315282e878bb858ef00635b90))
## [1.725.0](https://github.com/windmill-labs/windmill/compare/v1.724.0...v1.725.0) (2026-06-15)
### Features
* Db-backed user drafts ([#9351](https://github.com/windmill-labs/windmill/issues/9351)) ([1fc3557](https://github.com/windmill-labs/windmill/commit/1fc355709c025fd256c5a4035356e15a5a05b23d))
* scope AI session storage per user, session list in IndexedDB ([#9518](https://github.com/windmill-labs/windmill/issues/9518)) ([aa26c4d](https://github.com/windmill-labs/windmill/commit/aa26c4d9b22b3a353a6c0605eb9a4193e34aa18c))
### Bug Fixes
* **powershell:** sanitize module names to prevent command injection (CWE-78) ([#9587](https://github.com/windmill-labs/windmill/issues/9587)) ([6acce7a](https://github.com/windmill-labs/windmill/commit/6acce7a88733683153db534cb18752b31d93af82))
## [1.724.0](https://github.com/windmill-labs/windmill/compare/v1.723.0...v1.724.0) (2026-06-15)
### Features
* **cli:** add --yes, --secret/--no-secret and --description to variable add ([#9548](https://github.com/windmill-labs/windmill/issues/9548)) ([4e9e0c0](https://github.com/windmill-labs/windmill/commit/4e9e0c024b4b95f9676b1646591d8f0c662e84ab))
* **frontend:** improve AI chat cancel and interrupted-turn handling ([#9539](https://github.com/windmill-labs/windmill/issues/9539)) ([114c412](https://github.com/windmill-labs/windmill/commit/114c41251a8c738b58a1a3dd9434d09d33feb6f1))
* **frontend:** precise AI chat context usage tracking + indicator ([#9551](https://github.com/windmill-labs/windmill/issues/9551)) ([2b47180](https://github.com/windmill-labs/windmill/commit/2b471805bf1c92bb210cfacda217a4341e1f989c))
* wire chat reasoning effort through gemini and bedrock proxies ([#9545](https://github.com/windmill-labs/windmill/issues/9545)) ([aaf0563](https://github.com/windmill-labs/windmill/commit/aaf05635cedadc73455dc522474b673719f9fd5c))
### Bug Fixes
* actually isolate windows job children from CTRL_BREAK_EVENT + reap on worker death ([#9563](https://github.com/windmill-labs/windmill/issues/9563)) ([61f3291](https://github.com/windmill-labs/windmill/commit/61f3291b240bdb5c26bee8947351a9590bc3bd45))
* **ai:** enforce resource authz when loading MCP tools in agent worker ([#9571](https://github.com/windmill-labs/windmill/issues/9571)) ([317a862](https://github.com/windmill-labs/windmill/commit/317a8629d1c8436d2a6f3443bd25b81d606ce283))
* append system CA bundle to tracing proxy cert file ([#9549](https://github.com/windmill-labs/windmill/issues/9549)) ([3cf4083](https://github.com/windmill-labs/windmill/commit/3cf40839602e5c3d1df51f0a29b01736bade09da))
* **cli:** consistent flow inline lock filenames for compound extensions ([#9555](https://github.com/windmill-labs/windmill/issues/9555)) ([f0659a7](https://github.com/windmill-labs/windmill/commit/f0659a755a161420833e3bfdbe04befc6ebeb977))
* **flows:** skip_if evaluates wrong previous_result during retry ([#9547](https://github.com/windmill-labs/windmill/issues/9547)) ([2aab352](https://github.com/windmill-labs/windmill/commit/2aab35245c362c2f911c60ea435f29bfb1369ebf))
* **folders:** allow hyphens in folder names ([#9566](https://github.com/windmill-labs/windmill/issues/9566)) ([84df111](https://github.com/windmill-labs/windmill/commit/84df11177f2009bff007e9b722b55a9a5a63c06a)), closes [#8474](https://github.com/windmill-labs/windmill/issues/8474)
* **frontend:** load resource value in JSON editor when resource type is missing ([#9574](https://github.com/windmill-labs/windmill/issues/9574)) ([251266c](https://github.com/windmill-labs/windmill/commit/251266cd8119dbef314314daed43aa43ab92f1c9))
* isolate windows job children from worker CTRL_BREAK_EVENT ([#9562](https://github.com/windmill-labs/windmill/issues/9562)) ([1d6191e](https://github.com/windmill-labs/windmill/commit/1d6191ebb75843917eec6c76a4be347f9ac4cb72))
* stop sending temperature for AI chat across all providers ([#9553](https://github.com/windmill-labs/windmill/issues/9553)) ([3585716](https://github.com/windmill-labs/windmill/commit/358571687296fbe5c378533b3c1662707955c64a))
## [1.723.0](https://github.com/windmill-labs/windmill/compare/v1.722.0...v1.723.0) (2026-06-11)
Dockerfile
+5 -5
View File
@@ -163,14 +163,14 @@ ENV PATH /usr/local/bin:/root/.local/bin:/tmp/.local/bin:$PATH
RUN apt-get update \
&& apt-get install -y --no-install-recommends netbase tzdata ca-certificates wget curl jq unzip build-essential unixodbc xmlsec1 software-properties-common tini gnupg lsb-release \
&& apt-get install -y --no-install-recommends netbase tzdata ca-certificates wget curl jq unzip build-essential unixodbc xmlsec1 tini gnupg libargon2-1 \
&& if echo "$features" | grep -q "ee"; then apt-get install -y --no-install-recommends libsasl2-modules-gssapi-mit krb5-user; fi \
&& apt-get clean \
&& rm -rf /var/lib/apt/lists/*
# Install latest PostgreSQL client (pg_dump) from official PostgreSQL apt repository
RUN curl -fsSL https://www.postgresql.org/media/keys/ACCC4CF8.asc | gpg --dearmor -o /usr/share/keyrings/postgresql-archive-keyring.gpg \
&& echo "deb [signed-by=/usr/share/keyrings/postgresql-archive-keyring.gpg] https://apt.postgresql.org/pub/repos/apt $(lsb_release -cs)-pgdg main" > /etc/apt/sources.list.d/pgdg.list \
&& echo "deb [signed-by=/usr/share/keyrings/postgresql-archive-keyring.gpg] https://apt.postgresql.org/pub/repos/apt $(. /etc/os-release; echo "$VERSION_CODENAME")-pgdg main" > /etc/apt/sources.list.d/pgdg.list \
&& apt-get update \
&& apt-get install -y --no-install-recommends postgresql-client \
&& apt-get clean \
@@ -184,11 +184,11 @@ RUN if [ "$WITH_GIT" = "true" ]; then \
else echo 'Building the image without git'; fi;
RUN if [ "$WITH_POWERSHELL" = "true" ]; then \
if [ "$TARGETPLATFORM" = "linux/amd64" ]; then apt-get update -y && apt install libicu-dev -y && wget -O 'pwsh.deb' "https://github.com/PowerShell/PowerShell/releases/download/v${POWERSHELL_VERSION}/powershell_${POWERSHELL_DEB_VERSION}.deb_amd64.deb" && apt-get clean \
if [ "$TARGETPLATFORM" = "linux/amd64" ]; then apt-get update -y && apt install libicu72 -y && wget -O 'pwsh.deb' "https://github.com/PowerShell/PowerShell/releases/download/v${POWERSHELL_VERSION}/powershell_${POWERSHELL_DEB_VERSION}.deb_amd64.deb" && apt-get clean \
&& rm -rf /var/lib/apt/lists/* && \
dpkg --install 'pwsh.deb' && \
rm 'pwsh.deb'; \
elif [ "$TARGETPLATFORM" = "linux/arm64" ]; then apt-get update -y && apt install libicu-dev -y && wget -O powershell.tar.gz "https://github.com/PowerShell/PowerShell/releases/download/v${POWERSHELL_VERSION}/powershell-${POWERSHELL_VERSION}-linux-arm64.tar.gz" && apt-get clean \
elif [ "$TARGETPLATFORM" = "linux/arm64" ]; then apt-get update -y && apt install libicu72 -y && wget -O powershell.tar.gz "https://github.com/PowerShell/PowerShell/releases/download/v${POWERSHELL_VERSION}/powershell-${POWERSHELL_VERSION}-linux-arm64.tar.gz" && apt-get clean \
&& rm -rf /var/lib/apt/lists/* && \
mkdir -p /opt/microsoft/powershell/7 && \
tar zxf powershell.tar.gz -C /opt/microsoft/powershell/7 && \
@@ -233,7 +233,7 @@ ENV PATH="${PATH}:/usr/local/go/bin"
ENV GO_PATH=/usr/local/go/bin/go
# Install UV
RUN curl --proto '=https' --tlsv1.2 -LsSf https://github.com/astral-sh/uv/releases/download/0.9.25/uv-installer.sh | sh && mv /root/.local/bin/uv /usr/local/bin/uv
RUN curl --proto '=https' --tlsv1.2 -LsSf https://github.com/astral-sh/uv/releases/download/0.11.24/uv-installer.sh | sh && mv /root/.local/bin/uv /usr/local/bin/uv
# Preinstall python runtimes to temp build location (will copy with world-writable perms later)
# --compile-bytecode precompiles the stdlib to .pyc so jobs don't recompile it on every run
ai_evals/AGENTS.md
+10 -204
View File
@@ -1,208 +1,14 @@
# AI Evals Authoring Guide
# AI Evals
This folder contains black-box benchmark cases for:
Black-box benchmark cases for the Windmill AI generation modes (`flow`, `app`,
`script`, `cli`, `global`).
- `flow`
- `app`
- `script`
- `cli`
- `global`
**Authoring and running cases is documented in the `ai-evals` skill** — load it
before adding/changing a case or running a benchmark. Claude Code reads
`.claude/skills/ai-evals/SKILL.md`; Codex and Pi read
`.agents/skills/ai-evals/SKILL.md` (same canonical file). Invoke with `/ai-evals` in
Claude Code, `$ai-evals` in Codex, or `pi --skill ai-evals`.
The goal is to test the current production prompts and guidance with realistic user requests, not to test one exact implementation shape.
For AI chat / copilot changes that these evals measure, see the `ai-chat` skill.
## Core rules
1. Write prompts like a real user request.
2. Prefer behavior, inputs, constraints, and outcomes over internal implementation details.
3. Keep deterministic validation narrow and hard.
4. Put semantic expectations in `judgeChecklist`.
5. Use `expected` fixtures only when exact structure really matters.
## Prompt writing
Prompts should sound like something a user would naturally ask.
Good:
- "Create a flow that routes support requests based on customer tier."
- "Add a reset button that sets the counter back to 0."
- "Create a flow that reuses the existing greeting script instead of duplicating the logic."
Bad:
- "Use `branchone` with 3 branches and a default branch."
- "Create a `rawscript` step with this exact topology."
- "This is a benchmark harness."
Do not write prompts as if the user knows Windmill internals unless the case is explicitly testing a power-user workflow.
## Flow-specific rules
This is the main principle you asked for:
- flow prompts should read like requests from a user who does not know the product internals
- the user should ask for behavior, not for `branchone`, `branchall`, `rawscript`, `preprocessor_module`, `failure_module`, exact graph topology, or other internal constructs
That means:
- creation cases should describe the business behavior and expected result
- modification cases may mention existing step names, because the user can see the current flow
- only mention special Windmill constructs when the case is explicitly about those constructs
Examples:
- acceptable creation prompt:
"Create a purchase approval flow that pauses for approval and asks the approver for a comment."
- avoid:
"Create a suspend step with one required event and a resume form."
For flow cases, do not fail a case just because the model chose a different valid topology.
## App-specific rules
App prompts should focus on user-visible behavior:
- what the UI should let the user do
- what should persist
- what backend behavior is needed
Avoid prompting in terms of React structure, component names, or implementation unless the case is specifically about editing an existing app.
## CLI-specific rules
CLI prompts can be more explicit about paths and file names because real CLI users often do specify them.
Still, avoid benchmark phrasing. The prompt should read like a repo task, not a harness instruction.
When relevant, ask the assistant to tell the user which `wmill` commands to run next. That is part of the benchmarked behavior.
## Global-specific rules
Global prompts should exercise workspace-level drafting behavior:
- inspecting existing scripts, flows, apps, schedules, triggers, resources, and variables when relevant
- writing AI drafts rather than saving or deploying by default
- producing coherent multi-artifact changes when the request crosses artifact boundaries
Keep deterministic validation focused on the draft contract: required draft type/path, required content snippets, forbidden draft paths, and forbidden mutating tools such as deploy/delete unless the case explicitly asks for them.
Datatable cases should set `skipJudge: true` and validate through tool-use
(`requiredToolsUsed` / `forbiddenToolsUsed`) and SQL-argument assertions
(`toolCallArgs` with `stringIncludesAnyOf`, e.g. `['select']`, `['create table']`,
`['update', 'insert into']`). Two reasons the judge is unreliable here:
- `list_datatables`, `get_datatable_table_schema`, and `exec_datatable_sql`
produce no drafts, and the global judge only sees the drafts artifact — it
scores a no-draft conversational answer as empty (same as the
`askUserQuestion` cases).
- Even a case that *does* produce a draft (a script reading the data table via
`wmill.datatable()` at runtime) is mis-judged: the judge has no datatable SDK
reference and penalizes correct `wmill.datatable()` usage as wrong. Verify the
SDK call deterministically instead — `requiredDrafts.valueIncludes: ['wmill.datatable(']`
plus forbidding `exec_datatable_sql` (keeping chat-time SQL distinct from
runtime SDK use).
`stringIncludesAnyOf` is existential over calls (at least one matching call), so a
mutation case still passes when the model mixes its UPDATE/INSERT with
verification SELECTs. The in-memory engine (`datatableSqlEngine.ts`) is stateful
within a case — writes persist, so a model that re-queries to verify its
CREATE/UPDATE sees the change and does not loop. But the engine is best-effort
(SELECT returns all rows of the referenced/first table with no WHERE/projection),
so still never assert specific returned row values. Seed data via
`workspace.datatables` in the `initial` fixture (see README).
## Deterministic validation
Use deterministic validation only for hard failures such as:
- missing required files
- unexpected extra files when the prompt says not to create them
- syntax errors
- unresolved flow refs
- missing required special modules or suspend config
- obvious artifact corruption
Do not use deterministic validation to enforce one preferred implementation for broad creation tasks.
Examples of bad hard checks:
- exact step topology for a creation flow
- exact branch structure when the prompt only asked for routing behavior
- exact input shape when multiple reasonable shapes are acceptable
## Judge checklist
Every non-trivial case should have a `judgeChecklist`.
The checklist should capture:
- the user-visible behavior that must be present
- important constraints
- key completion criteria
The checklist should not duplicate low-level implementation details unless they are truly required by the task.
Good checklist items:
- "the flow calculates the order total with 8% tax"
- "the app persists recipes appropriately for a raw Windmill app"
- "the flow reuses the existing workspace script instead of rewriting the logic"
Bad checklist items:
- "uses `branchone`"
- "contains a `rawscript` node"
## When to use `expected`
Use `expected` fixtures when the case is structure-sensitive, for example:
- exact file creation
- exact script content
- modification cases where a specific file must change in a specific way
- cases where preserving an existing structure is part of the requirement
Do not use a full `expected` artifact as the semantic oracle for broad creation tasks when multiple valid outputs should pass.
## When to use `initial`
Use `initial` when the benchmark is about:
- editing an existing artifact
- reusing existing workspace assets
- preserving existing behavior while adding a change
If the case is greenfield, prefer no `initial`.
## Case design ladder
Prefer suites that get gradually harder:
1. trivial create case
2. realistic create case
3. reuse-existing-assets case
4. modification case
5. refactor case
6. edge-case or niche product behavior
The last cases in a suite should cover unusual or product-specific behavior.
## Anti-patterns
Avoid these:
- benchmark framing in prompts
- over-specified internal topology for creation tasks
- judge checklists that just restate implementation details
- deterministic validation that encodes one preferred solution
- fixtures that are so minimal or brittle that they create false negatives
## Before adding a case
Ask:
1. Would a real user plausibly write this prompt?
2. If the model solves it in a different valid way, would the case still pass?
3. Are the hard deterministic checks only catching objectively broken output?
4. Does the `judgeChecklist` describe the real success criteria?
5. If this case fails, will the reason be understandable from the saved artifacts?
The full case format, fields, and fixture details remain in `ai_evals/README.md`.
ai_evals/README.md
+3 -1
View File
@@ -75,6 +75,8 @@ Public CLI surface:
- `--model <alias>`: choose the model under test
- `--models <a,b,c>`: run the same cases sequentially against several model aliases
- `--verbose`: stream assistant output for frontend runs
- `--skip-judge`: skip LLM judge scoring for the run
- `--execution-only`: only require the model/proxy/frontend loop to complete; skip validators, tool expectations, backend artifact validation, and judge scoring
- `--record`: append a compact tracked summary line to `ai_evals/history/<mode>.jsonl` for full-suite runs only
- `--backend-validation <mode>`: optional backend smoke validation (`off` or `preview`) for `script` and `flow` evals
@@ -99,7 +101,7 @@ Notes:
- the command also prints accepted alias spellings such as `gpt-4o`, `gpt-55`, `claude-opus-4.6`, and `claude-haiku-4.5`
- frontend modes (`flow`, `script`, `app`, `global`) can use Anthropic, OpenAI, Gemini, and DeepSeek-backed aliases
- `cli` mode always uses the Anthropic agent SDK, so only Anthropic aliases are valid there
- the judge model is separate and currently defaults to `claude-sonnet-4-6`
- the judge model is separate and currently defaults to `claude-sonnet-4-6`; use `--skip-judge` for deterministic-only runs
## Case Format
ai_evals/adapters/cli/runtime.ts
+11
View File
@@ -16,6 +16,9 @@ export interface PromptRunResult {
output: string;
durationMs: number;
tokenUsage: BenchmarkTokenUsage | null;
// Input tokens on the last assistant turn. The SDK `result` message reports
// usage cumulatively, so the final context size comes from per-turn usage.
finalContextTokens: number | null;
trace: CliTrace;
}
@@ -144,6 +147,7 @@ export async function runPromptAndCapture(
let output = "";
let assistantMessageCount = 0;
let tokenUsage: BenchmarkTokenUsage | null = null;
let finalContextTokens: number | null = null;
const startedAt = Date.now();
const stubBinDir = join(cwd, WMILL_STUB_DIR_NAME);
const wmillLogPath = join(cwd, WMILL_LOG_FILE_NAME);
@@ -166,6 +170,12 @@ export async function runPromptAndCapture(
for await (const message of query({ prompt, options })) {
if (message.type === "assistant") {
assistantMessageCount += 1;
const turnContext = anthropicUsageToBenchmarkTokenUsage(
message.message?.usage
)?.prompt;
if (turnContext && turnContext > 0) {
finalContextTokens = turnContext;
}
const content = message.message?.content;
if (Array.isArray(content)) {
for (const block of content) {
@@ -210,6 +220,7 @@ export async function runPromptAndCapture(
output,
durationMs: Date.now() - startedAt,
tokenUsage,
finalContextTokens,
trace: {
toolsUsed,
skillsInvoked,
ai_evals/adapters/frontend/benchmarkRunner.ts
+15 -3
View File
@@ -25,6 +25,12 @@ export async function runFrontendBenchmarkFromEnv(): Promise<BenchmarkRunResult>
);
const emitProgress = process.env.WMILL_FRONTEND_AI_EVAL_PROGRESS === "1";
const verbose = process.env.WMILL_FRONTEND_AI_EVAL_VERBOSE === "1";
const executionOnly =
process.env.WMILL_FRONTEND_AI_EVAL_EXECUTION_ONLY === "1";
const judgeModel =
process.env.WMILL_FRONTEND_AI_EVAL_SKIP_JUDGE === "1" || executionOnly
? null
: DEFAULT_JUDGE_MODEL;
const model = resolveEvalModel(
mode,
process.env.WMILL_FRONTEND_AI_EVAL_MODEL,
@@ -48,7 +54,8 @@ export async function runFrontendBenchmarkFromEnv(): Promise<BenchmarkRunResult>
cases: selectedCases,
runs,
runModel,
judgeModel: DEFAULT_JUDGE_MODEL,
judgeModel,
executionOnly,
concurrency: verbose ? 1 : undefined,
verbose,
onProgress: emitProgress
@@ -60,7 +67,7 @@ export async function runFrontendBenchmarkFromEnv(): Promise<BenchmarkRunResult>
mode,
runs,
runModel,
judgeModel: DEFAULT_JUDGE_MODEL,
judgeModel,
caseResults,
});
}
@@ -96,7 +103,12 @@ async function getModeRunner(
}
function parseMode(value: string | undefined): FrontendBenchmarkMode {
if (value === "flow" || value === "app" || value === "script" || value === "global") {
if (
value === "flow" ||
value === "app" ||
value === "script" ||
value === "global"
) {
return value;
}
throw new Error(`Unsupported frontend benchmark mode: ${String(value)}`);
ai_evals/adapters/frontend/core/app/appEvalRunner.ts
+2
View File
@@ -38,6 +38,7 @@ export interface AppEvalResult {
toolCallCount: number;
toolsUsed: string[];
tokenUsage: TokenUsage;
finalContextTokens: number | null;
}
export interface AppEvalOptions {
@@ -113,6 +114,7 @@ export async function runAppEval(
toolCallCount: rawResult.toolCallsCount,
toolsUsed: rawResult.toolsCalled,
tokenUsage: rawResult.tokenUsage,
finalContextTokens: rawResult.finalContextTokens,
};
} finally {
await cleanup();
ai_evals/adapters/frontend/core/flow/flowEvalRunner.ts
+2
View File
@@ -39,6 +39,7 @@ export interface FlowEvalResult {
toolsUsed: string[];
toolCallDetails: ToolCallDetail[];
tokenUsage: TokenUsage;
finalContextTokens: number | null;
}
export interface FlowEvalOptions {
@@ -113,6 +114,7 @@ export async function runFlowEval(
toolsUsed: rawResult.toolsCalled,
toolCallDetails: rawResult.toolCallDetails,
tokenUsage: rawResult.tokenUsage,
finalContextTokens: rawResult.finalContextTokens,
};
} finally {
await cleanup();
ai_evals/adapters/frontend/core/global/globalEvalRunner.ts
+79 -23
View File
@@ -9,6 +9,7 @@ import {
} from "../../../../../frontend/src/lib/components/copilot/chat/global/core";
import {
clearGlobalDrafts,
getGlobalDraft,
listGlobalDrafts,
} from "../../../../../frontend/src/lib/components/copilot/chat/global/userDraftAdapter";
import type { Tool as ProductionTool } from "../../../../../frontend/src/lib/components/copilot/chat/shared";
@@ -18,6 +19,7 @@ import type { GlobalDraftState } from "../../../../core/validators";
import type { WindmillBackendSettings } from "../../../../core/windmillBackendSettings";
import {
registerBenchmarkWorkspaceRunnables,
seedBenchmarkDraft,
unregisterBenchmarkWorkspaceRunnables,
type BenchmarkWorkspaceRunnables,
} from "../../mockBackend";
@@ -30,6 +32,10 @@ const MUTATING_GLOBAL_TOOLS = new Set([
]);
const DISABLE_ACTIVE_EDITOR_CONTEXT_ENV =
"WMILL_AI_EVAL_DISABLE_ACTIVE_EDITOR_CONTEXT";
// A/B gate for the search_app read tool: set to "1" to run the baseline arm
// (toolset without search_app) so its token cost can be compared against the arm
// that offers it.
const DISABLE_SEARCH_APP_ENV = "WMILL_AI_EVAL_DISABLE_SEARCH_APP";
const LIVE_EDITOR_ITEM_KINDS = {
script: "script",
@@ -44,6 +50,20 @@ export interface GlobalLiveEditorDraftFixture {
value?: unknown;
}
// Identity the global system prompt builds paths from. Production reads
// `userStore` (whoami) to fill `u/{username}/...`; the eval harness never logs
// in, so without this the prompt sees an empty username (`u//...`) and no
// path-selection case is meaningful. Seeded per-case via the initial fixture and
// passed straight to `prepareGlobalSystemMessage` (no global-store mutation).
export interface GlobalUserFixture {
username: string;
is_admin?: boolean;
/** Folders the user can write to (the writable set whoami returns). */
folders?: string[];
/** Folders the user can read; read-only folders = folders_read \ folders. */
folders_read?: string[];
}
export interface GlobalEvalResult {
success: boolean;
state: GlobalDraftState;
@@ -53,11 +73,13 @@ export interface GlobalEvalResult {
toolsUsed: string[];
toolCallDetails: ToolCallDetail[];
tokenUsage: TokenUsage;
finalContextTokens: number | null;
}
export interface GlobalEvalOptions {
workspaceFixtures?: BenchmarkWorkspaceRunnables;
liveEditorDrafts?: GlobalLiveEditorDraftFixture[];
user?: GlobalUserFixture;
model?: string;
maxIterations?: number;
provider?: AIProvider;
@@ -83,9 +105,11 @@ export async function runGlobalEval(
const model = options.model ?? "claude-haiku-4-5-20251001";
const injectActiveEditorContext =
process.env[DISABLE_ACTIVE_EDITOR_CONTEXT_ENV] !== "1";
// Pass the seeded identity straight to the prompt builder rather than mutating
// the process-global `userStore`, so concurrent cases never race on it.
const rawResult = await runEval({
userPrompt,
systemMessage: prepareGlobalSystemMessage(),
systemMessage: prepareGlobalSystemMessage(undefined, { user: options.user }),
userMessage: prepareGlobalUserMessage(
userPrompt,
[],
@@ -94,7 +118,7 @@ export async function runGlobalEval(
tools: getGlobalEvalTools(),
helpers: {},
apiKey,
getOutput: () => ({ drafts: listGlobalDrafts(workspaceRoot) }),
getOutput: () => collectGlobalDraftState(workspaceRoot),
onAssistantMessageStart: options.runContext?.onAssistantMessageStart,
onAssistantToken: options.runContext?.onAssistantChunk,
onAssistantMessageEnd: options.runContext?.onAssistantMessageEnd,
@@ -119,6 +143,7 @@ export async function runGlobalEval(
toolsUsed: rawResult.toolsCalled,
toolCallDetails: rawResult.toolCallDetails,
tokenUsage: rawResult.tokenUsage,
finalContextTokens: rawResult.finalContextTokens,
};
} finally {
clearGlobalDrafts(workspaceRoot);
@@ -130,6 +155,32 @@ export async function runGlobalEval(
}
}
// Build the harness output from the DB-backed drafts. `listGlobalDrafts` returns
// metadata-only rows for backend drafts (the model's `write_script` etc. persist
// straight to the backend with no in-tab editor cell), so re-read each such row
// with `getGlobalDraft` to attach the full value the validators assert on. A row
// that already carries a value (the production in-tab cell overlay) is kept as-is.
async function collectGlobalDraftState(
workspace: string,
): Promise<GlobalDraftState> {
const items = await listGlobalDrafts(workspace);
const drafts = await Promise.all(
items.map(async (item) => {
if (item.value !== undefined) {
return item;
}
const full = await getGlobalDraft(
workspace,
item.type,
item.path,
item.triggerKind,
);
return full ?? item;
}),
);
return { drafts: drafts as GlobalDraftState["drafts"] };
}
function seedLiveEditorDrafts(
workspace: string,
fixtures: GlobalLiveEditorDraftFixture[],
@@ -138,7 +189,9 @@ function seedLiveEditorDrafts(
const itemKind = LIVE_EDITOR_ITEM_KINDS[fixture.type];
const storagePath = fixture.storagePath ?? fixture.effectivePath ?? "";
if (fixture.value !== undefined) {
UserDraft.save(itemKind, storagePath, fixture.value, { workspace });
// Seed as a backend draft row, not an in-tab cell: a cell would shadow the
// model's DB-backed edit when the output is read back via listGlobalDrafts.
seedBenchmarkDraft(workspace, itemKind, storagePath, fixture.value);
}
UserDraft.setLiveEditorDraft({
workspace,
@@ -161,25 +214,28 @@ function clearLiveEditorDrafts(
}
function getGlobalEvalTools(): ProductionTool<{}>[] {
return (globalTools as ProductionTool<{}>[]).map((tool) => {
if (!MUTATING_GLOBAL_TOOLS.has(tool.def.function.name)) {
return tool;
}
const disableSearchApp = process.env[DISABLE_SEARCH_APP_ENV] === "1";
return (globalTools as ProductionTool<{}>[])
.filter((tool) => !(disableSearchApp && tool.def.function.name === "search_app"))
.map((tool) => {
if (!MUTATING_GLOBAL_TOOLS.has(tool.def.function.name)) {
return tool;
}
return {
...tool,
requiresConfirmation: false,
validateBeforeConfirmation: undefined,
fn: async () =>
JSON.stringify(
{
success: false,
error:
"This mutating workspace tool is disabled during ai_evals global mode.",
},
null,
2,
),
};
});
return {
...tool,
requiresConfirmation: false,
validateBeforeConfirmation: undefined,
fn: async () =>
JSON.stringify(
{
success: false,
error:
"This mutating workspace tool is disabled during ai_evals global mode.",
},
null,
2,
),
};
});
}
ai_evals/adapters/frontend/core/script/scriptEvalRunner.ts
+2
View File
@@ -25,6 +25,7 @@ export interface ScriptEvalResult {
toolsUsed: string[];
toolCallDetails: ToolCallDetail[];
tokenUsage: TokenUsage;
finalContextTokens: number | null;
}
export interface ScriptEvalOptions {
@@ -111,6 +112,7 @@ export async function runScriptEval(
toolsUsed: rawResult.toolsCalled,
toolCallDetails: rawResult.toolCallDetails,
tokenUsage: rawResult.tokenUsage,
finalContextTokens: rawResult.finalContextTokens,
};
} finally {
await cleanup();
ai_evals/adapters/frontend/core/shared/baseEvalRunner.ts
+9 -5
View File
@@ -38,8 +38,9 @@ export interface RunEvalParams<THelpers, TOutput> {
helpers: THelpers;
/** API key for the provider */
apiKey: string;
/** Function to get the current output state */
getOutput: () => TOutput;
/** Function to get the current output state. May be async — global mode reads
* DB-backed drafts back through the (mocked) backend to build its output. */
getOutput: () => TOutput | Promise<TOutput>;
/** Model and Windmill backend configuration */
options: EvalRunnerOptions;
onAssistantMessageStart?: () => void;
@@ -154,9 +155,10 @@ export async function runEval<THelpers, TOutput>(
if (result.hitMaxIterations) {
return {
success: false,
output: getOutput(),
output: (await getOutput()) as TOutput,
error: `Reached max turns (${maxIterations})`,
tokenUsage: result.tokenUsage,
finalContextTokens: result.lastIterationUsage?.prompt ?? null,
toolCallsCount,
toolsCalled,
toolCallDetails,
@@ -170,8 +172,9 @@ export async function runEval<THelpers, TOutput>(
return {
success: true,
output: getOutput(),
output: (await getOutput()) as TOutput,
tokenUsage: result.tokenUsage,
finalContextTokens: result.lastIterationUsage?.prompt ?? null,
toolCallsCount,
toolsCalled,
toolCallDetails,
@@ -191,9 +194,10 @@ export async function runEval<THelpers, TOutput>(
return {
success: false,
output: getOutput(),
output: (await getOutput()) as TOutput,
error: errorMessage,
tokenUsage: { prompt: 0, completion: 0, total: 0 },
finalContextTokens: null,
toolCallsCount,
toolsCalled,
toolCallDetails,
ai_evals/adapters/frontend/core/shared/types.ts
+2
View File
@@ -28,6 +28,8 @@ export interface RawEvalResult<TOutput> {
output: TOutput;
error?: string;
tokenUsage: TokenUsage;
/** Input tokens on the last model request of the loop (see BenchmarkAttemptResult.finalContextTokens). */
finalContextTokens: number | null;
toolCallsCount: number;
toolsCalled: string[];
toolCallDetails: ToolCallDetail[];
ai_evals/adapters/frontend/mockBackend.ts
+189 -2
View File
@@ -1,9 +1,20 @@
import { randomUUID } from 'node:crypto'
import type { CompletedJob, Flow, Job, Script } from '../../../frontend/src/lib/gen'
import type {
AppWithLastVersion,
CompletedJob,
Flow,
Job,
ListableApp,
Script
} from '../../../frontend/src/lib/gen'
import type {
DataTableTables,
DataTableTableSchema,
ScriptLang
GetDraftForUserResponse,
ListDraftsResponse,
ScriptLang,
UpdateDraftResponse,
UserDraftItemKind
} from '../../../frontend/src/lib/gen/types.gen'
import { buildScriptLintResult } from './core/script/preview'
import { applyDatatableSql, type BenchmarkDatatableSeed } from './datatableSqlEngine'
@@ -29,6 +40,18 @@ export interface BenchmarkWorkspaceFlow {
value: Flow['value']
}
export interface BenchmarkWorkspaceApp {
path: string
summary: string
value: {
files: Record<string, string>
runnables: Record<string, unknown>
data?: unknown
policy?: unknown
custom_path?: unknown
}
}
export interface BenchmarkWorkspaceJob {
/** Stable id so a case prompt can reference a specific run (e.g. for get_job_logs). */
id?: string
@@ -43,6 +66,7 @@ export interface BenchmarkWorkspaceJob {
export interface BenchmarkWorkspaceRunnables {
scripts?: BenchmarkWorkspaceScript[]
flows?: BenchmarkWorkspaceFlow[]
apps?: BenchmarkWorkspaceApp[]
datatables?: BenchmarkDatatableSeed[]
jobs?: BenchmarkWorkspaceJob[]
}
@@ -63,6 +87,14 @@ export function resetBenchmarkMockBackend(): void {
benchmarkWorkspaces.clear()
benchmarkWorkspaceRunnables.clear()
benchmarkJobs.clear()
benchmarkDrafts.clear()
}
// Stand-in for FolderService.createFolder so the global create_folder tool runs in
// memory instead of mutating the real backend. Folders aren't otherwise modelled
// (no folder-listing in evals), so this just echoes the created name.
export function createBenchmarkFolder(_workspace: string, name: string): string {
return name
}
export function registerBenchmarkWorkspace(workspace: string): void {
@@ -74,6 +106,8 @@ export function registerBenchmarkWorkspaceRunnables(
runnables: BenchmarkWorkspaceRunnables
): void {
benchmarkWorkspaces.add(workspace)
// Fresh case: drop any drafts left from a prior run on this workspace id.
clearBenchmarkDrafts(workspace)
// Datatables are mutated in place by exec_datatable_sql (a write must be visible
// to later reads), so store an isolated deep copy — never mutate the caller's seed.
benchmarkWorkspaceRunnables.set(workspace, {
@@ -98,6 +132,7 @@ export function registerBenchmarkWorkspaceRunnables(
export function unregisterBenchmarkWorkspace(workspace: string): void {
benchmarkWorkspaces.delete(workspace)
benchmarkWorkspaceRunnables.delete(workspace)
clearBenchmarkDrafts(workspace)
for (const [jobId, entry] of benchmarkJobs.entries()) {
if (entry.workspace === workspace) {
benchmarkJobs.delete(jobId)
@@ -153,6 +188,22 @@ export function getBenchmarkFlowByPath(workspace: string, path: string): Flow |
return flow ? buildBenchmarkFlow(flow) : null
}
export function listBenchmarkApps(workspace: string): ListableApp[] | null {
const runnables = benchmarkWorkspaceRunnables.get(workspace)
if (!runnables) {
return null
}
return (runnables.apps ?? []).map(buildBenchmarkListableApp)
}
export function getBenchmarkAppByPath(workspace: string, path: string): AppWithLastVersion | null {
const app = benchmarkWorkspaceRunnables
.get(workspace)
?.apps?.find((entry) => entry.path === path)
return app ? buildBenchmarkApp(app) : null
}
export function createBenchmarkCompletedJob(input: {
workspace: string
jobKind: CompletedJob['job_kind']
@@ -238,6 +289,110 @@ export function getBenchmarkJobLogs(workspace: string, jobId: string): string {
return job.logs ?? ''
}
// ============= Drafts (per-user, DB-backed in production) =============
/**
* In-memory stand-in for the per-user draft backend (`DraftService`). The global
* AI chat now persists and reads drafts through the backend DB instead of an
* in-tab `UserDraft` cell, so the eval mocks the three draft endpoints it
* exercises (`updateDraft` / `getDraftForUser` / `listDrafts`) and keeps the
* saved values here, keyed by workspace + draft kind + storage path. Mirrors the
* semantics of the production unit test's mock in
* `frontend/src/lib/components/copilot/chat/global/core.test.ts`.
*/
const benchmarkDrafts = new Map<
string,
{ workspace: string; kind: UserDraftItemKind; path: string; value: unknown }
>()
// Fixed timestamp so artifacts stay deterministic. No eval simulates a
// concurrent writer, so every save is accepted and the conflict branch is
// never taken — the syncer just records this as its `last_sync` baseline.
const BENCHMARK_DRAFT_TIMESTAMP = '1970-01-01T00:00:00.000Z'
function benchmarkDraftKey(workspace: string, kind: string, path: string): string {
return `${workspace}::${kind}::${path}`
}
export function clearBenchmarkDrafts(workspace: string): void {
for (const [key, entry] of benchmarkDrafts.entries()) {
if (entry.workspace === workspace) {
benchmarkDrafts.delete(key)
}
}
}
/**
* Seed a draft straight into the store — used by the eval's live-editor draft
* fixtures, which model "the user already has this draft open/saved". Writing it
* here (instead of through `UserDraft.save`) keeps it a backend draft row with no
* shadowing in-tab cell, so a model edit that persists to the backend is what the
* output read-back captures — not the stale seed.
*/
export function seedBenchmarkDraft(
workspace: string,
kind: UserDraftItemKind,
path: string,
value: unknown
): void {
benchmarkDrafts.set(benchmarkDraftKey(workspace, kind, path), {
workspace,
kind,
path,
value
})
}
/** Mirror `DraftService.updateDraft`: a `null`/omitted value deletes the row. */
export function updateBenchmarkDraft(input: {
workspace: string
kind: UserDraftItemKind
path: string
requestBody?: { value?: unknown }
}): UpdateDraftResponse {
const key = benchmarkDraftKey(input.workspace, input.kind, input.path)
const value = input.requestBody?.value
if (value == null) {
benchmarkDrafts.delete(key)
} else {
benchmarkDrafts.set(key, {
workspace: input.workspace,
kind: input.kind,
path: input.path,
value
})
}
return { status: 'saved', current_timestamp: BENCHMARK_DRAFT_TIMESTAMP }
}
/** Mirror `DraftService.getDraftForUser`: 404-shaped throw when absent so the
* adapter's narrowed catch treats it as "no draft" instead of re-throwing. */
export function getBenchmarkDraftForUser(input: {
workspace: string
kind: UserDraftItemKind
path: string
}): GetDraftForUserResponse {
const entry = benchmarkDrafts.get(benchmarkDraftKey(input.workspace, input.kind, input.path))
if (!entry) {
throw Object.assign(new Error(`no draft for "${input.path}"`), { status: 404 })
}
return { value: entry.value, created_at: BENCHMARK_DRAFT_TIMESTAMP }
}
/** Mirror `DraftService.listDrafts`: metadata rows (no value) for a workspace. */
export function listBenchmarkDrafts(workspace: string): ListDraftsResponse {
return [...benchmarkDrafts.values()]
.filter((entry) => entry.workspace === workspace)
.map((entry) => ({
kind: entry.kind,
path: entry.path,
summary: (entry.value as { summary?: string } | null)?.summary,
draft_only: true,
legacy_draft: false,
created_at: BENCHMARK_DRAFT_TIMESTAMP
}))
}
// ============= Datatables (best-effort in-memory SQL) =============
/**
@@ -492,3 +647,35 @@ function buildBenchmarkFlow(flow: BenchmarkWorkspaceFlow): Flow {
extra_perms: {}
} as Flow
}
function buildBenchmarkListableApp(app: BenchmarkWorkspaceApp): ListableApp {
return {
id: 0,
workspace_id: 'benchmark',
path: app.path,
summary: app.summary,
version: 1,
extra_perms: {},
edited_at: BENCHMARK_TIMESTAMP,
execution_mode: 'viewer',
raw_app: true
}
}
function buildBenchmarkApp(app: BenchmarkWorkspaceApp): AppWithLastVersion {
return {
id: 0,
workspace_id: 'benchmark',
path: app.path,
summary: app.summary,
versions: [1],
created_by: 'benchmark',
created_at: BENCHMARK_TIMESTAMP,
value: app.value,
policy: (app.value.policy ?? {}) as AppWithLastVersion['policy'],
execution_mode: 'viewer',
extra_perms: {},
custom_path: app.value.custom_path as string | undefined,
raw_app: true
}
}
ai_evals/adapters/frontend/mockBackendDrafts.test.ts
+94
View File
@@ -0,0 +1,94 @@
import { afterEach, beforeEach, describe, expect, it } from 'bun:test'
import {
clearBenchmarkDrafts,
getBenchmarkDraftForUser,
listBenchmarkDrafts,
resetBenchmarkMockBackend,
seedBenchmarkDraft,
updateBenchmarkDraft
} from './mockBackend'
const WORKSPACE = 'benchmark-drafts-ws'
// Drives the in-memory stand-in for the per-user draft backend (`DraftService`)
// that the global AI-chat eval round-trips its drafts through. Mirrors the
// production-unit-test mock in
// `frontend/src/lib/components/copilot/chat/global/core.test.ts`.
describe('mockBackend drafts', () => {
beforeEach(() => resetBenchmarkMockBackend())
afterEach(() => resetBenchmarkMockBackend())
it('round-trips a saved draft through update / get / list', () => {
const value = { summary: 'Greet a user', content: 'export async function main() {}' }
const res = updateBenchmarkDraft({
workspace: WORKSPACE,
kind: 'script',
path: 'f/evals/greet',
requestBody: { value }
})
expect(res.status).toBe('saved')
expect(getBenchmarkDraftForUser({ workspace: WORKSPACE, kind: 'script', path: 'f/evals/greet' }).value).toEqual(
value
)
const rows = listBenchmarkDrafts(WORKSPACE)
expect(rows).toHaveLength(1)
expect(rows[0]).toMatchObject({ kind: 'script', path: 'f/evals/greet', summary: 'Greet a user', draft_only: true })
})
it('treats a null value as a delete', () => {
updateBenchmarkDraft({
workspace: WORKSPACE,
kind: 'variable',
path: 'f/evals/token',
requestBody: { value: { summary: 'token' } }
})
updateBenchmarkDraft({
workspace: WORKSPACE,
kind: 'variable',
path: 'f/evals/token',
requestBody: { value: null }
})
expect(listBenchmarkDrafts(WORKSPACE)).toHaveLength(0)
expect(() => getBenchmarkDraftForUser({ workspace: WORKSPACE, kind: 'variable', path: 'f/evals/token' })).toThrow()
})
it('throws a 404-shaped error when no draft exists', () => {
try {
getBenchmarkDraftForUser({ workspace: WORKSPACE, kind: 'script', path: 'f/evals/missing' })
throw new Error('expected a throw')
} catch (e) {
expect((e as { status?: number }).status).toBe(404)
}
})
it('seeds a draft as a backend row that a later edit overwrites', () => {
seedBenchmarkDraft(WORKSPACE, 'script', 'f/evals/current', { content: 'seed' })
expect(getBenchmarkDraftForUser({ workspace: WORKSPACE, kind: 'script', path: 'f/evals/current' }).value).toEqual({
content: 'seed'
})
// A model edit persists the same path and must win over the seed.
updateBenchmarkDraft({
workspace: WORKSPACE,
kind: 'script',
path: 'f/evals/current',
requestBody: { value: { content: 'edited' } }
})
expect(getBenchmarkDraftForUser({ workspace: WORKSPACE, kind: 'script', path: 'f/evals/current' }).value).toEqual({
content: 'edited'
})
})
it('clears only the targeted workspace', () => {
seedBenchmarkDraft(WORKSPACE, 'script', 'f/a', { content: 'a' })
seedBenchmarkDraft('other-ws', 'script', 'f/b', { content: 'b' })
clearBenchmarkDrafts(WORKSPACE)
expect(listBenchmarkDrafts(WORKSPACE)).toHaveLength(0)
expect(listBenchmarkDrafts('other-ws')).toHaveLength(1)
})
})
ai_evals/adapters/frontend/runtime.ts
+5
View File
@@ -24,6 +24,8 @@ export async function runFrontendBenchmarkAdapter(input: {
runs: number;
model?: string;
verbose?: boolean;
skipJudge?: boolean;
executionOnly?: boolean;
backendValidation?: string;
}): Promise<BenchmarkRunResult> {
const tempDir = await mkdtemp(
@@ -40,6 +42,9 @@ export async function runFrontendBenchmarkAdapter(input: {
WMILL_FRONTEND_AI_EVAL_MODEL: input.model ?? "",
WMILL_FRONTEND_AI_EVAL_PROGRESS: "1",
WMILL_FRONTEND_AI_EVAL_VERBOSE: input.verbose ? "1" : "0",
WMILL_FRONTEND_AI_EVAL_SKIP_JUDGE:
input.skipJudge || input.executionOnly ? "1" : "0",
WMILL_FRONTEND_AI_EVAL_EXECUTION_ONLY: input.executionOnly ? "1" : "0",
WMILL_FRONTEND_AI_EVAL_BACKEND_VALIDATION: input.backendValidation ?? "",
};
ai_evals/adapters/frontend/vitestAdapter.test.ts
+45 -5
View File
@@ -33,24 +33,30 @@ vi.mock('$lib/components/vscode', () => ({}))
vi.mock('$lib/gen', async () => {
const actual = await vi.importActual<any>('$lib/gen')
const {
getBenchmarkAppByPath,
getBenchmarkCompletedJob,
getBenchmarkCompletedJobResultMaybe,
getBenchmarkDatatableSchema,
getBenchmarkDraftForUser,
getBenchmarkFlowByPath,
getBenchmarkJobLogs,
getBenchmarkScriptByHash,
getBenchmarkScriptByPath,
hasBenchmarkWorkspace,
listBenchmarkApps,
listBenchmarkDatatables,
listBenchmarkDrafts,
listBenchmarkFlows,
listBenchmarkJobs,
listBenchmarkScripts,
createBenchmarkFolder,
createBenchmarkHttpTrigger,
createBenchmarkSchedule,
previewBenchmarkSchedule,
runBenchmarkDatatableSql,
runBenchmarkFlowByPath,
runBenchmarkScriptPreview
runBenchmarkScriptPreview,
updateBenchmarkDraft
} = await import('./mockBackend')
function wrapService<T extends object>(target: T, overrides: Record<string, unknown>): T {
@@ -66,6 +72,31 @@ vi.mock('$lib/gen', async () => {
return {
...actual,
DraftService: wrapService(actual.DraftService, {
updateDraft: async (data: {
workspace: string
kind: any
path: string
requestBody?: { value?: unknown }
}) =>
hasBenchmarkWorkspace(data.workspace)
? updateBenchmarkDraft(data)
: actual.DraftService.updateDraft(data),
getDraftForUser: async (data: { workspace: string; kind: any; path: string }) =>
hasBenchmarkWorkspace(data.workspace)
? getBenchmarkDraftForUser(data)
: actual.DraftService.getDraftForUser(data),
listDrafts: async (data: { workspace: string }) =>
hasBenchmarkWorkspace(data.workspace)
? listBenchmarkDrafts(data.workspace)
: actual.DraftService.listDrafts(data)
}),
FolderService: wrapService(actual.FolderService, {
createFolder: async (data: { workspace: string; requestBody: { name: string } }) =>
hasBenchmarkWorkspace(data.workspace)
? createBenchmarkFolder(data.workspace, data.requestBody.name)
: actual.FolderService.createFolder(data)
}),
ScriptService: wrapService(actual.ScriptService, {
listScripts: async (data: { workspace: string }) =>
hasBenchmarkWorkspace(data.workspace)
@@ -277,12 +308,20 @@ vi.mock('$lib/gen', async () => {
}),
AppService: wrapService(actual.AppService, {
existsApp: async (data: { workspace: string; path: string }) =>
hasBenchmarkWorkspace(data.workspace) ? false : actual.AppService.existsApp(data),
hasBenchmarkWorkspace(data.workspace)
? Boolean(getBenchmarkAppByPath(data.workspace, data.path))
: actual.AppService.existsApp(data),
listApps: async (data: { workspace: string }) =>
hasBenchmarkWorkspace(data.workspace) ? [] : actual.AppService.listApps(data),
hasBenchmarkWorkspace(data.workspace)
? (listBenchmarkApps(data.workspace) ?? [])
: actual.AppService.listApps(data),
getAppByPath: async (data: { workspace: string; path: string }) => {
if (hasBenchmarkWorkspace(data.workspace)) {
throw new Error(`App "${data.path}" not found in benchmark workspace`)
const app = getBenchmarkAppByPath(data.workspace, data.path)
if (!app) {
throw new Error(`App "${data.path}" not found in benchmark workspace`)
}
return app
}
return actual.AppService.getAppByPath(data)
}
@@ -434,5 +473,6 @@ benchmarkIt(
resetBenchmarkMockBackend()
}
},
600_000
// Full-suite runs (30+ cases at concurrency 2-3) routinely exceed 10 minutes.
7_200_000
)
ai_evals/cases/global.yaml
+379 -1
View File
@@ -3,8 +3,9 @@
Create a draft Bun script at `f/evals/global/greet_user`.
It should take a string `name` input and return `Hello, ${name}!`.
Leave it as an AI draft only; do not deploy or save it.
initial: ai_evals/fixtures/frontend/global/initial/user_admin_evals_folder.json
runtime:
maxTurns: 8
maxTurns: 10
validate:
draftCountExactly: 1
requiredDrafts:
@@ -870,3 +871,380 @@
judgeChecklist:
- fetches the logs for the requested job id
- explains the failure from the returned logs (connection refused to the upstream API)
# --- Documentation search (search_docs) ---
# Pure product-knowledge questions: the assistant should consult the docs via
# search_docs and answer conversationally, not draft or mutate anything. No
# draft is produced, so the global judge is skipped and we validate tool use.
- id: global-docs-ai-agent-step
prompt: |-
Does Windmill support a flow step where an LLM decides which of my scripts to call based on the input?
runtime:
maxTurns: 6
validate:
draftCountExactly: 0
toolExpect:
requiredToolsUsed:
- search_docs
forbiddenToolsUsed:
- write_script
- write_flow
- deploy_workspace_item
- delete_workspace_item
skipJudge: true
- id: global-docs-retry-step
prompt: |-
How does automatic retry work for a flow step that calls a flaky API?
runtime:
maxTurns: 6
validate:
draftCountExactly: 0
toolExpect:
requiredToolsUsed:
- search_docs
forbiddenToolsUsed:
- write_script
- write_flow
- deploy_workspace_item
- delete_workspace_item
skipJudge: true
- id: global-docs-key-value-store
prompt: |-
Can I use a Redis-style key-value store from my Windmill scripts, and how?
runtime:
maxTurns: 6
validate:
draftCountExactly: 0
toolExpect:
requiredToolsUsed:
- search_docs
forbiddenToolsUsed:
- write_script
- write_flow
- deploy_workspace_item
- delete_workspace_item
skipJudge: true
- id: global-docs-cron-schedule-format
prompt: |-
How do Windmill's cron schedules work, and what format does the schedule expression use?
runtime:
maxTurns: 6
validate:
draftCountExactly: 0
toolExpect:
requiredToolsUsed:
- search_docs
forbiddenToolsUsed:
- write_script
- write_flow
- deploy_workspace_item
- delete_workspace_item
skipJudge: true
# --- Raw app on a large project (context-usage benchmark) ---
# These cases run against the deliberately large `analytics_dashboard` raw-app
# fixture (~20 frontend files incl. a 5k-line data module, plus backend runnables).
# They exist to measure how much context the global chat consumes when working in a
# big raw app: test29 is a read-heavy debugging hunt, test30 is a small edit baseline.
# tokenUsage is recorded per run, so the same cases re-run after a read-tool change
# (the read_app_file cap + offset/limit paging) quantify the optimization. skipJudge:
# the judge only sees the drafts artifact and cannot run the app, so we validate
# deterministically.
- id: global-test29-raw-app-debug-large
prompt: |-
The analytics dashboard app at `f/evals/global/analytics_dashboard` has a bug:
the Revenue Summary tile shows a total that is lower than the per-order line
totals and the per-region breakdown. Track down what is computing revenue
incorrectly and fix it. Keep the change as an AI draft only; do not deploy or
save it.
initial: ai_evals/fixtures/frontend/global/initial/analytics_dashboard
runtime:
maxTurns: 20
validate:
draftCountExactly: 1
requiredDrafts:
- type: app
path: f/evals/global/analytics_dashboard
valueIncludes:
- "return order.unitPrice * order.quantity"
toolExpect:
requiredToolsAnyOf:
# Inspecting the app's files is satisfied by either reading them directly
# or grepping for the revenue calculation.
- [read_app_file, search_app]
forbiddenToolsUsed:
- deploy_workspace_item
- delete_workspace_item
skipJudge: true
judgeChecklist:
- inspects the dashboard app's files to locate the revenue calculation
- fixes the per-order revenue so it multiplies unit price by quantity
- leaves the result as an AI draft and does not deploy or save it
- id: global-test30-raw-app-small-edit-large
prompt: |-
In the dashboard app at `f/evals/global/analytics_dashboard`, change the main
page heading from "Operations Console" to "Revenue Overview". Leave everything
else unchanged. Keep it as an AI draft only; do not deploy or save it.
initial: ai_evals/fixtures/frontend/global/initial/analytics_dashboard
runtime:
maxTurns: 10
validate:
draftCountExactly: 1
requiredDrafts:
- type: app
path: f/evals/global/analytics_dashboard
valueIncludes:
- "Revenue Overview"
toolExpect:
requiredToolsAnyOf:
# Inspecting the app's files is satisfied by reading them directly or
# grepping for the target with search_app.
- [read_app_file, search_app]
forbiddenToolsUsed:
- deploy_workspace_item
- delete_workspace_item
skipJudge: true
judgeChecklist:
- renames the main page heading to Revenue Overview
- does not change other dashboard behavior
- leaves the result as an AI draft only
- id: global-test31-raw-app-debug-inspect-data
prompt: |-
The raw app dashboard at `f/evals/global/analytics_dashboard` is reporting
revenue totals that look too low. Inspect the app's files — both the sample
order data module and the revenue calculation — to work out whether the bug is
in the data or in the calculation, then fix the actual cause. Keep the change as
an AI draft only; do not deploy or save it.
initial: ai_evals/fixtures/frontend/global/initial/analytics_dashboard
runtime:
maxTurns: 22
validate:
draftCountExactly: 1
requiredDrafts:
- type: app
path: f/evals/global/analytics_dashboard
valueIncludes:
- "return order.unitPrice * order.quantity"
toolExpect:
requiredToolsAnyOf:
# Inspecting the app's files is satisfied by reading them directly or
# grepping for the target with search_app.
- [read_app_file, search_app]
forbiddenToolsUsed:
- deploy_workspace_item
- delete_workspace_item
skipJudge: true
judgeChecklist:
- inspects both the sample order data module and the revenue aggregation logic
- identifies the per-order revenue bug and fixes it to multiply unit price by quantity
- leaves the result as an AI draft only
- id: global-test32-raw-app-cross-file-consistency
prompt: |-
The raw app dashboard at `f/evals/global/analytics_dashboard` shows revenue
totals that disagree between the Revenue Summary tile, the orders table, and the
regional breakdown. Investigate how each of those computes revenue, work out
which calculation is wrong, and fix it. Keep the change as an AI draft only; do
not deploy or save it.
# Cross-file investigation: forces the model through several overlapping files
# (the summary's aggregation helper, the orders table, the regional breakdown) —
# a realistic multi-file read load that exercises the read_app_file cap.
initial: ai_evals/fixtures/frontend/global/initial/analytics_dashboard
runtime:
maxTurns: 24
validate:
draftCountExactly: 1
requiredDrafts:
- type: app
path: f/evals/global/analytics_dashboard
valueIncludes:
- "return order.unitPrice * order.quantity"
toolExpect:
requiredToolsAnyOf:
# Inspecting the app's files is satisfied by reading them directly or
# grepping for the target with search_app.
- [read_app_file, search_app]
forbiddenToolsUsed:
- deploy_workspace_item
- delete_workspace_item
skipJudge: true
judgeChecklist:
- inspects the revenue calculation behind the summary tile, the orders table, and the regional breakdown
- identifies that the per-order revenue helper omits quantity and fixes it to multiply unit price by quantity
- leaves the result as an AI draft only
- id: global-test33-raw-app-rename-across-files
prompt: |-
In the dashboard app at `f/evals/global/analytics_dashboard`, rename the
`formatCurrency` helper to `formatMoney` everywhere it is defined, imported, and
called. Leave the separate `formatCurrencyPrecise` helper exactly as it is. Keep
the change as an AI draft only; do not deploy or save it.
# Find-all-usages rename: formatCurrency is defined once and called in 6 places
# spread across 4 component files (and imported in 4). Locating every usage is the
# exact task search_app is meant to make cheap — one grep returns all file:line
# rows instead of reading each component whole. valueExcludes "formatCurrency("
# asserts the definition and all call sites were renamed while tolerating the
# preserved formatCurrencyPrecise (which is never followed by "(").
initial: ai_evals/fixtures/frontend/global/initial/analytics_dashboard
runtime:
maxTurns: 22
validate:
draftCountExactly: 1
requiredDrafts:
- type: app
path: f/evals/global/analytics_dashboard
valueIncludes:
- "export function formatMoney"
- "formatMoney("
valueExcludes:
- "formatCurrency("
toolExpect:
forbiddenToolsUsed:
- deploy_workspace_item
- delete_workspace_item
skipJudge: true
judgeChecklist:
- renames the formatCurrency definition, imports, and all call sites to formatMoney
- leaves the unrelated formatCurrencyPrecise helper unchanged
- leaves the result as an AI draft only
# --- Path selection (u/<user> vs f/<folder>) ---
# These cases assert how the assistant picks a workspace path when the user gives
# none: a bare name defaults to the personal scope `u/<user>/`, an existing folder
# whose purpose matches is used, a non-admin targets a writable folder and never a
# read-only one, and shared intent with no matching folder asks rather than invents.
# Each depends on the seeded `user` fixture (username / is_admin / folders /
# folders_read) so the prompt's folder guidance and `u/{username}` are well-formed.
- id: global-path1-bare-name-defaults-to-personal
prompt: |-
Stage a quick draft helper that takes a string and returns it trimmed of
leading and trailing whitespace. Just keep it as a draft.
initial: ai_evals/fixtures/frontend/global/initial/user_admin_empty.json
runtime:
maxTurns: 8
validate:
draftCountExactly: 1
requiredDrafts:
- type: script
pathStartsWith: u/admin/
toolExpect:
requiredToolsUsed:
- write_script
forbiddenToolsUsed:
- deploy_workspace_item
- delete_workspace_item
skipJudge: true
judgeChecklist:
- stages a single script draft for a trim helper
- defaults the path to the current user's personal scope (u/admin/...) since no path or folder was given
- does not invent an f/<folder> path
- leaves the result as a draft only
- id: global-path2-match-existing-folder
prompt: |-
Draft a flow for the marketing team's weekly campaign report.
It should take a week number and return a short summary string.
Keep it as a draft only.
initial: ai_evals/fixtures/frontend/global/initial/user_admin_folders.json
runtime:
maxTurns: 10
validate:
draftCountExactly: 1
requiredDrafts:
- type: flow
pathStartsWith: f/marketing/
toolExpect:
requiredToolsUsed:
- write_flow
forbiddenToolsUsed:
- deploy_workspace_item
- delete_workspace_item
skipJudge: true
judgeChecklist:
- drafts a flow for the marketing campaign report
- places it in the existing marketing folder (f/marketing/...) rather than the personal scope or an invented folder
- leaves the result as a draft only
- id: global-path3-shared-intent-unknown-folder-asks
prompt: |-
Put together a draft onboarding checklist flow for the People Ops team to use
when a new hire joins. Keep it as a draft.
initial: ai_evals/fixtures/frontend/global/initial/user_admin_folders.json
runtime:
maxTurns: 8
validate:
draftCountExactly: 0
toolExpect:
requiredToolsUsed:
- askUserQuestion
forbiddenToolsUsed:
- write_flow
- write_script
- deploy_workspace_item
- delete_workspace_item
skipJudge: true
judgeChecklist:
- recognizes the request implies shared/team work but names no existing folder (none of marketing/data_engineering/shared_utils fit People Ops)
- asks which folder to use instead of guessing or inventing one
- does not create a draft until the folder is known
- id: global-path4-nonadmin-avoids-readonly-folder
prompt: |-
Draft a small flow that returns today's date as an ISO string, and stage it in
one of our shared team folders. Keep it as a draft.
initial: ai_evals/fixtures/frontend/global/initial/user_bob_nonadmin_teams.json
runtime:
maxTurns: 10
validate:
draftCountExactly: 1
requiredDrafts:
- type: flow
pathStartsWith: f/team_a/
forbiddenDrafts:
- type: flow
pathStartsWith: f/team_b/
toolExpect:
requiredToolsUsed:
- write_flow
forbiddenToolsUsed:
- deploy_workspace_item
- delete_workspace_item
skipJudge: true
judgeChecklist:
- drafts a flow that returns the current date as an ISO string
- places it in team_a (writable by this non-admin user) and not team_b (read-only)
- leaves the result as a draft only
- id: global-path5-create-folder-then-draft
prompt: |-
Create a new shared folder called "analytics" for our data work, then draft a
script in it that returns the current timestamp as an ISO string. Keep the
script as a draft.
initial: ai_evals/fixtures/frontend/global/initial/user_admin_empty.json
runtime:
maxTurns: 10
validate:
draftCountExactly: 1
requiredDrafts:
- type: script
pathStartsWith: f/analytics/
toolExpect:
requiredToolsUsed:
- create_folder
- write_script
forbiddenToolsUsed:
- deploy_workspace_item
- delete_workspace_item
skipJudge: true
judgeChecklist:
- creates a new shared folder named "analytics" via create_folder
- drafts a script placed in that folder (f/analytics/...) returning an ISO timestamp
- leaves the script as a draft only
ai_evals/cli/index.ts
+25 -3
View File
@@ -25,7 +25,9 @@ import {
import { runSuite } from "../core/runSuite";
import { EVAL_MODES, type EvalMode } from "../core/types";
import { DEFAULT_JUDGE_MODEL } from "../core/judge";
import { createCliModeRunner } from "../modes/cli";
// createCliModeRunner is imported lazily in runCliBenchmark so the non-cli modes
// (global/flow/script/app) don't pull in the wmill CLI toolchain and its JSR deps
// (e.g. @cliffy/*) just to load this entrypoint.
import { runFrontendBenchmarkAdapter } from "../adapters/frontend/runtime";
import { resolveWindmillBackendSettings } from "../core/windmillBackendSettings";
import { assertWindmillBackendReachable } from "../adapters/frontend/windmillBackend";
@@ -97,6 +99,11 @@ async function main() {
"comma-separated model aliases to run sequentially",
)
.option("--verbose", "stream assistant output during frontend runs")
.option("--skip-judge", "skip LLM judge scoring for this run")
.option(
"--execution-only",
"only require the model/proxy/frontend loop to complete",
)
.option(
"--record",
"append a compact summary line to ai_evals/history/<mode>.jsonl",
@@ -115,6 +122,8 @@ async function main() {
model?: string;
models?: string;
verbose?: boolean;
skipJudge?: boolean;
executionOnly?: boolean;
record?: boolean;
backendValidation?: string;
},
@@ -127,6 +136,8 @@ async function main() {
model: options.model,
models: options.models,
verbose: options.verbose ?? false,
skipJudge: options.skipJudge ?? false,
executionOnly: options.executionOnly ?? false,
record: options.record ?? false,
backendValidation: options.backendValidation,
});
@@ -175,6 +186,8 @@ async function handleRun(input: {
model?: string;
models?: string;
verbose: boolean;
skipJudge: boolean;
executionOnly: boolean;
record: boolean;
backendValidation?: string;
}) {
@@ -230,6 +243,8 @@ async function handleRun(input: {
input.runs,
getCliEvalModel(model),
runModel,
input.skipJudge,
input.executionOnly,
)
: await runFrontendBenchmarkAdapter({
mode: input.mode,
@@ -237,6 +252,8 @@ async function handleRun(input: {
runs: input.runs,
model: model.id,
verbose: input.verbose,
skipJudge: input.skipJudge,
executionOnly: input.executionOnly,
backendValidation,
});
@@ -278,20 +295,25 @@ async function runCliBenchmark(
runs: number,
model: ReturnType<typeof getCliEvalModel>,
runModel: string,
skipJudge: boolean,
executionOnly: boolean,
) {
const { createCliModeRunner } = await import("../modes/cli");
const judgeModel = skipJudge || executionOnly ? null : DEFAULT_JUDGE_MODEL;
const caseResults = await runSuite({
modeRunner: createCliModeRunner(model),
cases,
runs,
runModel,
judgeModel: DEFAULT_JUDGE_MODEL,
judgeModel,
executionOnly,
});
return buildRunResult({
mode: "cli",
runs,
runModel,
judgeModel: DEFAULT_JUDGE_MODEL,
judgeModel,
caseResults,
});
}
ai_evals/core/cases.test.ts
+18
View File
@@ -212,6 +212,9 @@ describe("loadCases", () => {
},
],
});
expect(caseEntry?.initialPath).toContain(
"ai_evals/fixtures/frontend/global/initial/user_admin_evals_folder.json"
);
expect(caseEntry?.toolExpect).toMatchObject({
requiredToolsUsed: ["write_script"],
forbiddenToolsUsed: ["deploy_workspace_item", "delete_workspace_item"],
@@ -246,6 +249,21 @@ describe("loadCases", () => {
});
});
it("loads global docs-search cases as tool-use checks", async () => {
const globalCases = await loadCases("global");
const docsCases = globalCases.filter((entry) =>
entry.id.startsWith("global-docs-"),
);
expect(docsCases.length).toBeGreaterThanOrEqual(3);
// Each docs case verifies the assistant reaches for search_docs and does not
// draft anything; with no draft, the global judge is skipped.
for (const entry of docsCases) {
expect(entry.skipJudge).toBe(true);
expect(entry.toolExpect?.requiredToolsUsed).toContain("search_docs");
}
});
it("loads tool expectations for workspace mutation cases", async () => {
const scriptCases = await loadCases("script");
const caseEntry = scriptCases.find(
ai_evals/core/results.test.ts
+65
View File
@@ -92,6 +92,71 @@ describe("benchmark results", () => {
expect(summary).toContain("Average duration (all attempts): 550ms");
});
it("aggregates final context size over passed attempts only", () => {
const result = buildRunResult({
mode: "global",
runs: 1,
runModel: "model-under-test",
judgeModel: "judge-model",
caseResults: [
caseResult([
{
attempt: 1,
passed: true,
durationMs: 1000,
assistantMessageCount: 1,
toolCallCount: 1,
toolsUsed: ["edit_script"],
skillsInvoked: [],
checks: [{ name: "edited", passed: true }],
judgeScore: 100,
judgeSummary: "ok",
error: null,
tokenUsage: { prompt: 12000, completion: 200, total: 12200 },
finalContextTokens: 5000,
},
{
attempt: 2,
passed: true,
durationMs: 1100,
assistantMessageCount: 1,
toolCallCount: 1,
toolsUsed: ["edit_script"],
skillsInvoked: [],
checks: [{ name: "edited", passed: true }],
judgeScore: 100,
judgeSummary: "ok",
error: null,
tokenUsage: { prompt: 18000, completion: 300, total: 18300 },
finalContextTokens: 7000,
},
{
attempt: 3,
passed: false,
durationMs: 100,
assistantMessageCount: 1,
toolCallCount: 0,
toolsUsed: [],
skillsInvoked: [],
checks: [{ name: "edited", passed: false }],
judgeScore: 10,
judgeSummary: "missed",
error: "failed",
tokenUsage: { prompt: 20000, completion: 100, total: 20100 },
finalContextTokens: 9000,
},
]),
],
});
// Final context size stays below cumulative prompt and ignores the failed attempt.
expect(result.averageFinalContextTokensPassed).toBe(6000);
expect(result.maxFinalContextTokensPassed).toBe(7000);
expect(formatRunSummary(result)).toContain(
"Final context size (passed): 6000 tokens (max 7000)",
);
});
it("reports passed averages as unavailable when no attempt passes", () => {
const result = buildRunResult({
mode: "global",
ai_evals/core/results.ts
+34
View File
@@ -16,6 +16,9 @@ type AttemptAggregate = {
durationTotal: number;
tokenUsageAttemptCount: number;
tokenUsageTotal: BenchmarkTokenUsage | null;
finalContextAttemptCount: number;
finalContextTotal: number;
finalContextMax: number | null;
};
export async function writeRunResult(
@@ -117,6 +120,8 @@ export function buildRunResult(input: {
passedAttemptAggregate,
passedAttempts,
),
averageFinalContextTokensPassed: averageFinalContext(passedAttemptAggregate),
maxFinalContextTokensPassed: passedAttemptAggregate.finalContextMax,
cases: input.caseResults,
};
}
@@ -133,6 +138,11 @@ export function formatRunSummary(result: BenchmarkRunResult): string {
`Average tokens (passed): ${formatTokenUsage(result.averageTokenUsagePerPassedAttempt)}`,
);
}
if (result.averageFinalContextTokensPassed != null) {
lines.push(
`Final context size (passed): ${Math.round(result.averageFinalContextTokensPassed)} tokens (max ${Math.round(result.maxFinalContextTokensPassed ?? 0)})`,
);
}
if (result.passedAttempts < result.attemptCount) {
lines.push(
`Average duration (all attempts): ${Math.round(result.averageDurationMs)}ms`,
@@ -181,10 +191,21 @@ function aggregateAttempts(attempts: BenchmarkAttemptResult[]): AttemptAggregate
durationTotal: 0,
tokenUsageAttemptCount: 0,
tokenUsageTotal: null,
finalContextAttemptCount: 0,
finalContextTotal: 0,
finalContextMax: null,
};
for (const attempt of attempts) {
aggregate.durationTotal += attempt.durationMs;
if (typeof attempt.finalContextTokens === "number") {
aggregate.finalContextAttemptCount += 1;
aggregate.finalContextTotal += attempt.finalContextTokens;
aggregate.finalContextMax = Math.max(
aggregate.finalContextMax ?? 0,
attempt.finalContextTokens,
);
}
if (!attempt.tokenUsage) {
continue;
}
@@ -204,6 +225,12 @@ function averageDuration(aggregate: AttemptAggregate): number | null {
: aggregate.durationTotal / aggregate.attemptCount;
}
function averageFinalContext(aggregate: AttemptAggregate): number | null {
return aggregate.finalContextAttemptCount === 0
? null
: aggregate.finalContextTotal / aggregate.finalContextAttemptCount;
}
function averageTokenUsage(
aggregate: AttemptAggregate,
denominator: number,
@@ -318,6 +345,9 @@ function toHistoryRecord(result: BenchmarkRunResult) {
averageTokenUsagePerAttempt: result.averageTokenUsagePerAttempt ?? null,
averageTokenUsagePerPassedAttempt:
result.averageTokenUsagePerPassedAttempt ?? null,
averageFinalContextTokensPassed:
result.averageFinalContextTokensPassed ?? null,
maxFinalContextTokensPassed: result.maxFinalContextTokensPassed ?? null,
failedCaseIds: Array.from(
new Set(
result.cases
@@ -361,6 +391,10 @@ function toHistoryRecord(result: BenchmarkRunResult) {
passedAttemptAggregate,
passedAttempts,
),
averageFinalContextTokensPassed: averageFinalContext(
passedAttemptAggregate,
),
maxFinalContextTokensPassed: passedAttemptAggregate.finalContextMax,
};
}),
};
ai_evals/core/runSuite.test.ts
+102
View File
@@ -0,0 +1,102 @@
import { describe, expect, it } from "bun:test";
import { runSuite } from "./runSuite";
import type { ModeRunner } from "./types";
const modeRunner: ModeRunner<undefined, undefined, { ok: boolean }> = {
mode: "global",
concurrency: 1,
loadInitial: async () => undefined,
loadExpected: async () => undefined,
run: async () => ({
success: true,
actual: { ok: true },
assistantMessageCount: 1,
toolCallCount: 0,
toolsUsed: [],
skillsInvoked: [],
tokenUsage: null,
}),
validate: () => [],
};
describe("runSuite", () => {
it("skips judge checks when the run disables judge scoring", async () => {
const [caseResult] = await runSuite({
modeRunner,
cases: [
{
id: "case-1",
prompt: "Create a draft script",
judgeChecklist: ["the output satisfies the prompt"],
},
],
runs: 1,
runModel: "model-under-test",
judgeModel: null,
});
const [attempt] = caseResult.attempts;
expect(attempt.passed).toBe(true);
expect(attempt.judgeScore).toBeNull();
expect(attempt.judgeSummary).toBeNull();
expect(attempt.checks.map((check) => check.name)).toEqual([
"run succeeded",
]);
});
it("only requires run success when execution-only is enabled", async () => {
let loadExpectedCalls = 0;
let validateCalls = 0;
let backendValidateCalls = 0;
const executionOnlyRunner: ModeRunner<
undefined,
undefined,
{ ok: boolean }
> = {
...modeRunner,
loadExpected: async () => {
loadExpectedCalls++;
return undefined;
},
validate: () => {
validateCalls++;
return [{ name: "validator failed", passed: false }];
},
backendValidate: async () => {
backendValidateCalls++;
return {
checks: [{ name: "backend validation failed", passed: false }],
};
},
};
const [caseResult] = await runSuite({
modeRunner: executionOnlyRunner,
cases: [
{
id: "case-1",
prompt: "Create a draft script",
expectedPath: "fixtures/expected.json",
toolExpect: { requiredToolsUsed: ["write_script"] },
judgeChecklist: ["the output satisfies the prompt"],
},
],
runs: 1,
runModel: "model-under-test",
judgeModel: "judge-model",
executionOnly: true,
});
const [attempt] = caseResult.attempts;
expect(attempt.passed).toBe(true);
expect(attempt.judgeScore).toBeNull();
expect(attempt.judgeSummary).toBeNull();
expect(attempt.checks.map((check) => check.name)).toEqual([
"run succeeded",
]);
expect(loadExpectedCalls).toBe(0);
expect(validateCalls).toBe(0);
expect(backendValidateCalls).toBe(0);
});
});
ai_evals/core/runSuite.ts
+41 -18
View File
@@ -15,11 +15,13 @@ export async function runSuite<TInitial, TExpected, TActual>(input: {
runs: number;
runModel: string | null;
judgeModel?: string | null;
executionOnly?: boolean;
concurrency?: number;
verbose?: boolean;
onProgress?: (event: FrontendBenchmarkProgressEvent) => void;
}): Promise<BenchmarkCaseResult[]> {
const judgeModel = input.judgeModel ?? DEFAULT_JUDGE_MODEL;
const judgeModel =
input.judgeModel === undefined ? DEFAULT_JUDGE_MODEL : input.judgeModel;
const concurrency = Math.max(1, input.concurrency ?? input.modeRunner.concurrency);
const results = new Array<BenchmarkCaseResult>(input.cases.length);
let cursor = 0;
@@ -52,6 +54,7 @@ export async function runSuite<TInitial, TExpected, TActual>(input: {
runs: input.runs,
judgeModel,
judgeThreshold: input.modeRunner.judgeThreshold ?? 80,
executionOnly: input.executionOnly ?? false,
modeRunner: input.modeRunner,
totalCases: input.cases.length,
verbose: input.verbose ?? false,
@@ -72,8 +75,9 @@ async function runCaseAttempts<TInitial, TExpected, TActual>(input: {
caseIndex: number;
evalCase: EvalCase;
runs: number;
judgeModel: string;
judgeModel: string | null;
judgeThreshold: number;
executionOnly: boolean;
modeRunner: ModeRunner<TInitial, TExpected, TActual>;
totalCases: number;
verbose: boolean;
@@ -99,7 +103,9 @@ async function runCaseAttempts<TInitial, TExpected, TActual>(input: {
try {
const initial = await input.modeRunner.loadInitial(input.evalCase.initialPath);
const expected = await input.modeRunner.loadExpected(input.evalCase.expectedPath);
const expected = input.executionOnly
? undefined
: await input.modeRunner.loadExpected(input.evalCase.expectedPath);
const run = await input.modeRunner.run(input.evalCase.prompt, initial, {
evalCase: input.evalCase,
caseId: input.evalCase.id,
@@ -162,22 +168,30 @@ async function runCaseAttempts<TInitial, TExpected, TActual>(input: {
});
const checks: BenchmarkCheck[] = [
buildCheck("run succeeded", run.success, run.error),
...input.modeRunner.validate({
evalCase: input.evalCase,
prompt: input.evalCase.prompt,
initial,
expected,
actual: run.actual,
run,
}),
...validateToolExpectations({
run,
toolExpect: input.evalCase.toolExpect,
}),
];
if (!input.executionOnly) {
checks.push(
...input.modeRunner.validate({
evalCase: input.evalCase,
prompt: input.evalCase.prompt,
initial,
expected,
actual: run.actual,
run,
}),
...validateToolExpectations({
run,
toolExpect: input.evalCase.toolExpect,
})
);
}
const artifactFiles = input.modeRunner.buildArtifacts?.(run.actual) ?? [];
if (run.success && input.modeRunner.backendValidate) {
if (
run.success &&
!input.executionOnly &&
input.modeRunner.backendValidate
) {
try {
const backendValidation = await input.modeRunner.backendValidate({
evalCase: input.evalCase,
@@ -218,14 +232,21 @@ async function runCaseAttempts<TInitial, TExpected, TActual>(input: {
let judgeScore: number | null = null;
let judgeSummary: string | null = null;
if (run.success && !input.evalCase.skipJudge) {
if (
run.success &&
!input.executionOnly &&
input.judgeModel !== null &&
!input.evalCase.skipJudge
) {
const judge = await judgeOutput({
mode: input.modeRunner.mode,
prompt: input.evalCase.prompt,
checklist: input.evalCase.judgeChecklist,
initial,
expected: input.modeRunner.mode === "cli" ? undefined : expected,
actual: run.actual,
actual: input.modeRunner.prepareJudgeActual
? input.modeRunner.prepareJudgeActual(run.actual)
: run.actual,
model: input.judgeModel,
});
@@ -255,6 +276,7 @@ async function runCaseAttempts<TInitial, TExpected, TActual>(input: {
judgeSummary,
error: run.error ?? null,
tokenUsage: run.tokenUsage ?? null,
finalContextTokens: run.finalContextTokens ?? null,
artifactsPath: null,
artifactFiles,
};
@@ -291,6 +313,7 @@ async function runCaseAttempts<TInitial, TExpected, TActual>(input: {
judgeSummary: null,
error: message,
tokenUsage: null,
finalContextTokens: null,
};
if (surface) {
input.onProgress?.({
ai_evals/core/types.ts
+26 -1
View File
@@ -168,11 +168,21 @@ export interface ToolCallArgumentRule {
export interface ToolValidationSpec {
requiredToolsUsed?: string[];
/**
* Each inner array is an alternatives group: the check passes when at least
* one tool in the group was used. Use when several tools satisfy the same
* intent so a model that picks any valid path passes — e.g. inspecting an
* app's files via either `read_app_file` or `search_app`.
*/
requiredToolsAnyOf?: string[][];
forbiddenToolsUsed?: string[];
toolCallArgs?: ToolCallArgumentRule[];
}
export type EvalValidationSpec = FlowValidationSpec | AppValidationSpec | GlobalValidationSpec;
export type EvalValidationSpec =
| FlowValidationSpec
| AppValidationSpec
| GlobalValidationSpec;
export interface EvalCase {
id: string;
@@ -249,6 +259,12 @@ export interface ModeRunOutput<TActual> {
toolCallDetails?: ToolCallDetail[];
skillsInvoked: string[];
tokenUsage?: BenchmarkTokenUsage | null;
/**
* Total input tokens occupying the context window on the LAST model request
* of the agentic loop (input + cache-creation + cache-read). Complements the
* cumulative `tokenUsage.prompt`, which sums every iteration's input.
*/
finalContextTokens?: number | null;
}
export interface ModeRunContext {
@@ -294,6 +310,12 @@ export interface ModeRunner<TInitial, TExpected, TActual> {
context: ModeRunContext;
}): Promise<BackendValidationResult | null>;
buildArtifacts?(actual: TActual): BenchmarkArtifactFile[];
/**
* Optional transform applied to `actual` before it is handed to the LLM judge.
* Use it to strip fields the judge must stay blind to (e.g. which docs-tool
* arm produced an answer). When omitted, the judge receives `actual` as-is.
*/
prepareJudgeActual?(actual: TActual): unknown;
}
export interface BenchmarkAttemptResult {
@@ -310,6 +332,7 @@ export interface BenchmarkAttemptResult {
judgeSummary: string | null;
error: string | null;
tokenUsage?: BenchmarkTokenUsage | null;
finalContextTokens?: number | null;
artifactsPath?: string | null;
artifactFiles?: BenchmarkArtifactFile[];
}
@@ -340,6 +363,8 @@ export interface BenchmarkRunResult {
totalPassedTokenUsage?: BenchmarkTokenUsage | null;
averageTokenUsagePerAttempt?: BenchmarkTokenUsage | null;
averageTokenUsagePerPassedAttempt?: BenchmarkTokenUsage | null;
averageFinalContextTokensPassed?: number | null;
maxFinalContextTokensPassed?: number | null;
artifactsPath?: string | null;
cases: BenchmarkCaseResult[];
}
ai_evals/core/validators.test.ts
+43
View File
@@ -245,6 +245,49 @@ describe("validateToolExpectations", () => {
'accepted substrings: insert into, update; values: "DROP TABLE orders"',
});
});
it("passes requiredToolsAnyOf when any alternative in the group is used", () => {
const checks = validateToolExpectations({
run: {
success: true,
actual: {},
assistantMessageCount: 1,
toolCallCount: 1,
toolsUsed: ["search_app", "patch_app_file"],
skillsInvoked: [],
},
toolExpect: {
requiredToolsAnyOf: [["read_app_file", "search_app"]],
},
});
expect(checks).toContainEqual({
name: "uses one of read_app_file, search_app",
passed: true,
});
});
it("fails requiredToolsAnyOf when no alternative in the group is used", () => {
const checks = validateToolExpectations({
run: {
success: true,
actual: {},
assistantMessageCount: 1,
toolCallCount: 1,
toolsUsed: ["patch_app_file"],
skillsInvoked: [],
},
toolExpect: {
requiredToolsAnyOf: [["read_app_file", "search_app"]],
},
});
expect(checks).toContainEqual({
name: "uses one of read_app_file, search_app",
passed: false,
details: "tools used: patch_app_file",
});
});
});
describe("validateGlobalState", () => {
ai_evals/core/validators.ts
+10
View File
@@ -169,6 +169,16 @@ export function validateToolExpectations(input: {
);
}
for (const group of expect.requiredToolsAnyOf ?? []) {
checks.push(
check(
`uses one of ${group.join(", ")}`,
group.some((toolName) => input.run.toolsUsed.includes(toolName)),
`tools used: ${input.run.toolsUsed.join(", ") || "none"}`
)
);
}
for (const toolName of expect.forbiddenToolsUsed ?? []) {
checks.push(
check(
ai_evals/fixtures/frontend/global/initial/analytics_dashboard/backend/computeSummary/main.ts
+68
View File
@@ -0,0 +1,68 @@
type OrderStatus = 'paid' | 'shipped' | 'delivered' | 'pending' | 'refunded' | 'cancelled'
interface Order {
id: string
region: string
quantity: number
unitPrice: number
status: OrderStatus
placedAt: string
}
// Server-side revenue rollup. Mirrors the client aggregation but is computed
// from the authoritative mocked order book so it can be used to cross-check
// the dashboard and to back the export.
const orders: Order[] = [
{ id: 'ORD-10001', region: 'North America', quantity: 3, unitPrice: 1195, status: 'delivered', placedAt: '2024-05-02' },
{ id: 'ORD-10002', region: 'EMEA', quantity: 5, unitPrice: 880, status: 'shipped', placedAt: '2024-05-03' },
{ id: 'ORD-10003', region: 'APAC', quantity: 2, unitPrice: 640, status: 'paid', placedAt: '2024-05-05' },
{ id: 'ORD-10004', region: 'LATAM', quantity: 7, unitPrice: 315, status: 'delivered', placedAt: '2024-05-07' },
{ id: 'ORD-10005', region: 'North America', quantity: 4, unitPrice: 150, status: 'refunded', placedAt: '2024-05-09' },
{ id: 'ORD-10006', region: 'EMEA', quantity: 6, unitPrice: 220, status: 'shipped', placedAt: '2024-05-12' },
{ id: 'ORD-10007', region: 'APAC', quantity: 1, unitPrice: 980, status: 'pending', placedAt: '2024-05-15' },
{ id: 'ORD-10008', region: 'North America', quantity: 8, unitPrice: 1100, status: 'delivered', placedAt: '2024-05-18' },
{ id: 'ORD-10009', region: 'EMEA', quantity: 2, unitPrice: 860, status: 'cancelled', placedAt: '2024-05-22' },
{ id: 'ORD-10010', region: 'LATAM', quantity: 9, unitPrice: 290, status: 'paid', placedAt: '2024-05-26' }
]
const REVENUE_STATUSES: OrderStatus[] = ['paid', 'shipped', 'delivered']
export async function main({
from,
to,
region
}: {
from: string
to: string
region: string
}): Promise<{
totalRevenue: number
netRevenue: number
totalOrders: number
averageOrderValue: number
unitsSold: number
refundedRevenue: number
currency: string
}> {
let scoped = orders.filter((order) => order.placedAt >= from && order.placedAt <= to)
if (region && region !== 'all') {
scoped = scoped.filter((order) => order.region === region)
}
const booked = scoped.filter((order) => REVENUE_STATUSES.includes(order.status))
const totalRevenue = booked.reduce((acc, order) => acc + order.unitPrice * order.quantity, 0)
const unitsSold = booked.reduce((acc, order) => acc + order.quantity, 0)
const refundedRevenue = scoped
.filter((order) => order.status === 'refunded')
.reduce((acc, order) => acc + order.unitPrice * order.quantity, 0)
return {
totalRevenue,
netRevenue: totalRevenue - refundedRevenue,
totalOrders: booked.length,
averageOrderValue: booked.length === 0 ? 0 : Math.round(totalRevenue / booked.length),
unitsSold,
refundedRevenue,
currency: 'USD'
}
}
ai_evals/fixtures/frontend/global/initial/analytics_dashboard/backend/computeSummary/meta.json
+4
View File
@@ -0,0 +1,4 @@
{
"name": "Compute Summary",
"language": "bun"
}
ai_evals/fixtures/frontend/global/initial/analytics_dashboard/backend/exportReport/main.ts
+51
View File
@@ -0,0 +1,51 @@
// Builds a downloadable report for the current dashboard view. Returns a data
// URL the browser can open directly so the export works without object storage.
export async function main({
from,
to,
region,
format
}: {
from: string
to: string
region: string
format: 'csv' | 'json'
}): Promise<{ url: string; rows: number; filename: string }> {
const summary = {
from,
to,
region: region || 'all',
generatedAt: new Date().toISOString(),
rows: [
{ region: 'North America', revenue: 211_400, orders: 168 },
{ region: 'EMEA', revenue: 142_900, orders: 121 },
{ region: 'APAC', revenue: 86_500, orders: 78 },
{ region: 'LATAM', revenue: 41_500, orders: 45 }
]
}
const scoped =
region && region !== 'all'
? summary.rows.filter((row) => row.region === region)
: summary.rows
let body: string
let mime: string
if (format === 'csv') {
const header = 'region,revenue,orders'
const lines = scoped.map((row) => `${row.region},${row.revenue},${row.orders}`)
body = [header, ...lines].join('\n')
mime = 'text/csv'
} else {
body = JSON.stringify({ ...summary, rows: scoped }, null, 2)
mime = 'application/json'
}
const encoded = Buffer.from(body, 'utf-8').toString('base64')
const filename = `revenue-report-${from}_${to}.${format}`
return {
url: `data:${mime};base64,${encoded}`,
rows: scoped.length,
filename
}
}
ai_evals/fixtures/frontend/global/initial/analytics_dashboard/backend/exportReport/meta.json
+4
View File
@@ -0,0 +1,4 @@
{
"name": "Export Report",
"language": "bun"
}
ai_evals/fixtures/frontend/global/initial/analytics_dashboard/backend/loadMetrics/main.ts
+40
View File
@@ -0,0 +1,40 @@
interface MetricCardData {
id: string
label: string
value: number
unit: 'currency' | 'count' | 'percent'
delta: number
hint: string
}
// Returns the headline metric cards for the selected range and region. Values
// are mocked but internally consistent (revenue / orders ≈ avg order value).
const baseByRegion: Record<string, { revenue: number; orders: number; units: number; refunds: number }> = {
all: { revenue: 482_300, orders: 412, units: 1840, refunds: 11_900 },
'North America': { revenue: 211_400, orders: 168, units: 770, refunds: 4_200 },
EMEA: { revenue: 142_900, orders: 121, units: 560, refunds: 3_500 },
APAC: { revenue: 86_500, orders: 78, units: 340, refunds: 2_600 },
LATAM: { revenue: 41_500, orders: 45, units: 170, refunds: 1_600 }
}
export async function main({
from,
to,
region
}: {
from: string
to: string
region: string
}): Promise<{ cards: MetricCardData[]; generatedAt: string }> {
const base = baseByRegion[region] ?? baseByRegion.all
const aov = base.orders === 0 ? 0 : Math.round(base.revenue / base.orders)
const cards: MetricCardData[] = [
{ id: 'revenue', label: 'Total Revenue', value: base.revenue, unit: 'currency', delta: 0.082, hint: `Booked revenue ${from} – ${to}` },
{ id: 'orders', label: 'Orders', value: base.orders, unit: 'count', delta: 0.041, hint: 'Revenue-bearing orders in range' },
{ id: 'aov', label: 'Avg Order Value', value: aov, unit: 'currency', delta: -0.013, hint: 'Total revenue / order count' },
{ id: 'units', label: 'Units Sold', value: base.units, unit: 'count', delta: 0.067, hint: 'Total units in range' },
{ id: 'refunds', label: 'Refunded', value: base.refunds, unit: 'currency', delta: -0.021, hint: 'Revenue lost to refunds' },
{ id: 'conversion', label: 'Conversion', value: 0.187, unit: 'percent', delta: 0.009, hint: 'Sessions that became orders' }
]
return { cards, generatedAt: new Date().toISOString() }
}
ai_evals/fixtures/frontend/global/initial/analytics_dashboard/backend/loadMetrics/meta.json
+4
View File
@@ -0,0 +1,4 @@
{
"name": "Load Metrics",
"language": "bun"
}
ai_evals/fixtures/frontend/global/initial/analytics_dashboard/backend/loadOrders/main.ts
+54
View File
@@ -0,0 +1,54 @@
type OrderStatus = 'paid' | 'shipped' | 'delivered' | 'pending' | 'refunded' | 'cancelled'
interface Order {
id: string
placedAt: string
customer: string
product: string
sku: string
region: string
channel: string
rep: string
quantity: number
unitPrice: number
status: OrderStatus
}
// Mocked order book. In a real deployment this would query the orders table;
// here it returns a representative slice so the table renders in preview.
const orders: Order[] = [
{ id: 'ORD-10001', placedAt: '2024-05-02T09:14:00Z', customer: 'Contoso Ltd', product: 'Aurora Analytics Suite', sku: 'ANL-100', region: 'North America', channel: 'direct', rep: 'Dana Wills', quantity: 3, unitPrice: 1195, status: 'delivered' },
{ id: 'ORD-10002', placedAt: '2024-05-03T11:42:00Z', customer: 'Fabrikam Inc', product: 'Borealis CRM', sku: 'CRM-210', region: 'EMEA', channel: 'partner', rep: 'Lena Fischer', quantity: 5, unitPrice: 880, status: 'shipped' },
{ id: 'ORD-10003', placedAt: '2024-05-05T15:03:00Z', customer: 'Tailspin Toys', product: 'Cascade Data Pipeline', sku: 'PIPE-330', region: 'APAC', channel: 'self-serve', rep: 'Sora Tanaka', quantity: 2, unitPrice: 640, status: 'paid' },
{ id: 'ORD-10004', placedAt: '2024-05-07T08:21:00Z', customer: 'Proseware Inc', product: 'Delta Insights', sku: 'INS-440', region: 'LATAM', channel: 'marketplace', rep: 'Diego Marin', quantity: 7, unitPrice: 315, status: 'delivered' },
{ id: 'ORD-10005', placedAt: '2024-05-09T13:58:00Z', customer: 'Litware Inc', product: 'Echo Monitoring', sku: 'MON-550', region: 'North America', channel: 'direct', rep: 'Owen Pratt', quantity: 4, unitPrice: 150, status: 'refunded' },
{ id: 'ORD-10006', placedAt: '2024-05-12T10:30:00Z', customer: 'Fourth Coffee', product: 'Helix Identity', sku: 'IDN-880', region: 'EMEA', channel: 'partner', rep: 'Aisha Khan', quantity: 6, unitPrice: 220, status: 'shipped' },
{ id: 'ORD-10007', placedAt: '2024-05-15T17:11:00Z', customer: 'Coho Vineyard', product: 'Kelvin Forecasting', sku: 'FCT-202', region: 'APAC', channel: 'direct', rep: 'Priya Nair', quantity: 1, unitPrice: 980, status: 'pending' },
{ id: 'ORD-10008', placedAt: '2024-05-18T12:05:00Z', customer: 'Alpine Ski House', product: 'Nimbus Compute', sku: 'CMP-505', region: 'North America', channel: 'self-serve', rep: 'Hugo Bernard', quantity: 8, unitPrice: 1100, status: 'delivered' },
{ id: 'ORD-10009', placedAt: '2024-05-22T14:47:00Z', customer: 'Trey Research', product: 'Onyx Security', sku: 'SEC-606', region: 'EMEA', channel: 'direct', rep: 'Sven Olsen', quantity: 2, unitPrice: 860, status: 'cancelled' },
{ id: 'ORD-10010', placedAt: '2024-05-26T16:39:00Z', customer: 'Blue Yonder Airlines', product: 'Polaris Reporting', sku: 'RPT-707', region: 'LATAM', channel: 'partner', rep: 'Mateo Russo', quantity: 9, unitPrice: 290, status: 'paid' }
]
export async function main({
from,
to,
region,
status
}: {
from: string
to: string
region: string
status: string
}): Promise<{ orders: Order[]; total: number }> {
let filtered = orders.filter((order) => {
const day = order.placedAt.slice(0, 10)
return day >= from && day <= to
})
if (region && region !== 'all') {
filtered = filtered.filter((order) => order.region === region)
}
if (status && status !== 'all') {
filtered = filtered.filter((order) => order.status === status)
}
return { orders: filtered, total: filtered.length }
}
ai_evals/fixtures/frontend/global/initial/analytics_dashboard/backend/loadOrders/meta.json
+4
View File
@@ -0,0 +1,4 @@
{
"name": "Load Orders",
"language": "bun"
}
ai_evals/fixtures/frontend/global/initial/analytics_dashboard/frontend/components/DateRangePicker.tsx
+45
View File
@@ -0,0 +1,45 @@
import React from 'react'
import type { DateRange } from '../lib/api'
import { rangeForPreset } from '../lib/api'
import { formatDateShort } from '../lib/format'
interface DateRangePickerProps {
preset: string
range: DateRange
onPresetChange: (preset: string, range: DateRange) => void
}
const PRESETS: { id: string; label: string }[] = [
{ id: '7d', label: 'Last 7 days' },
{ id: '14d', label: 'Last 14 days' },
{ id: '30d', label: 'Last 30 days' },
{ id: 'qtd', label: 'Quarter to date' }
]
export const DateRangePicker: React.FC<DateRangePickerProps> = ({
preset,
range,
onPresetChange
}) => {
return (
<div className="flex items-center gap-2">
<select
className="rounded-lg border border-gray-300 bg-white px-3 py-2 text-sm text-gray-700"
value={preset}
onChange={(event) => {
const next = event.target.value
onPresetChange(next, rangeForPreset(next))
}}
>
{PRESETS.map((item) => (
<option key={item.id} value={item.id}>
{item.label}
</option>
))}
</select>
<span className="text-xs text-gray-400">
{formatDateShort(range.from)} – {formatDateShort(range.to)}
</span>
</div>
)
}
ai_evals/fixtures/frontend/global/initial/analytics_dashboard/frontend/components/EmptyState.tsx
+28
View File
@@ -0,0 +1,28 @@
import React from 'react'
interface EmptyStateProps {
title: string
description?: string
icon?: string
action?: React.ReactNode
}
export const EmptyState: React.FC<EmptyStateProps> = ({
title,
description,
icon = '📊',
action
}) => {
return (
<div className="flex flex-col items-center justify-center rounded-lg border border-dashed border-gray-300 bg-white py-12 text-center">
<div className="text-3xl" aria-hidden>
{icon}
</div>
<h3 className="mt-3 text-sm font-semibold text-gray-700">{title}</h3>
{description ? (
<p className="mt-1 max-w-sm text-sm text-gray-500">{description}</p>
) : null}
{action ? <div className="mt-4">{action}</div> : null}
</div>
)
}
ai_evals/fixtures/frontend/global/initial/analytics_dashboard/frontend/components/ExportButton.tsx
+51
View File
@@ -0,0 +1,51 @@
import React, { useState } from 'react'
import { requestExport } from '../lib/api'
import type { DateRange } from '../lib/api'
interface ExportButtonProps {
range: DateRange
region: string
}
export const ExportButton: React.FC<ExportButtonProps> = ({ range, region }) => {
const [busy, setBusy] = useState(false)
const [error, setError] = useState<string | null>(null)
const handleExport = async (format: 'csv' | 'json') => {
setBusy(true)
setError(null)
try {
const result = await requestExport(range, region, format)
const anchor = document.createElement('a')
anchor.href = result.url
anchor.download = `revenue-report.${format}`
anchor.click()
} catch (err) {
setError(err instanceof Error ? err.message : 'Export failed')
} finally {
setBusy(false)
}
}
return (
<div className="flex items-center gap-2">
<button
type="button"
disabled={busy}
onClick={() => handleExport('csv')}
className="rounded-lg border border-gray-300 bg-white px-3 py-2 text-sm font-medium text-gray-700 hover:bg-gray-50 disabled:opacity-50"
>
{busy ? 'Exporting…' : 'Export CSV'}
</button>
<button
type="button"
disabled={busy}
onClick={() => handleExport('json')}
className="rounded-lg border border-gray-300 bg-white px-3 py-2 text-sm font-medium text-gray-700 hover:bg-gray-50 disabled:opacity-50"
>
Export JSON
</button>
{error ? <span className="text-xs text-rose-600">{error}</span> : null}
</div>
)
}
ai_evals/fixtures/frontend/global/initial/analytics_dashboard/frontend/components/FilterBar.tsx
+59
View File
@@ -0,0 +1,59 @@
import React from 'react'
import type { DateRange } from '../lib/api'
import type { OrderStatus } from '../data/seedData'
import { REGIONS, ORDER_STATUSES, STATUS_LABELS } from '../data/seedData'
import { DateRangePicker } from './DateRangePicker'
import { ExportButton } from './ExportButton'
interface FilterBarProps {
region: string
status: string
preset: string
range: DateRange
onRegionChange: (region: string) => void
onStatusChange: (status: string) => void
onPresetChange: (preset: string, range: DateRange) => void
}
export const FilterBar: React.FC<FilterBarProps> = ({
region,
status,
preset,
range,
onRegionChange,
onStatusChange,
onPresetChange
}) => {
return (
<div className="flex flex-wrap items-center justify-between gap-3 border-b border-gray-200 bg-white px-6 py-4">
<div className="flex flex-wrap items-center gap-3">
<DateRangePicker preset={preset} range={range} onPresetChange={onPresetChange} />
<select
className="rounded-lg border border-gray-300 bg-white px-3 py-2 text-sm text-gray-700"
value={region}
onChange={(event) => onRegionChange(event.target.value)}
>
<option value="all">All regions</option>
{REGIONS.map((item) => (
<option key={item} value={item}>
{item}
</option>
))}
</select>
<select
className="rounded-lg border border-gray-300 bg-white px-3 py-2 text-sm text-gray-700"
value={status}
onChange={(event) => onStatusChange(event.target.value)}
>
<option value="all">All statuses</option>
{ORDER_STATUSES.map((item) => (
<option key={item} value={item}>
{STATUS_LABELS[item as OrderStatus]}
</option>
))}
</select>
</div>
<ExportButton range={range} region={region} />
</div>
)
}
ai_evals/fixtures/frontend/global/initial/analytics_dashboard/frontend/components/MetricCard.tsx
+40
View File
@@ -0,0 +1,40 @@
import React from 'react'
import type { MetricCardData } from '../data/seedData'
import { formatCurrency, formatNumber, formatPercent, formatSignedPercent } from '../lib/format'
interface MetricCardProps {
metric: MetricCardData
loading?: boolean
}
function renderValue(metric: MetricCardData): string {
switch (metric.unit) {
case 'currency':
return formatCurrency(metric.value)
case 'percent':
return formatPercent(metric.value)
case 'count':
default:
return formatNumber(metric.value)
}
}
export const MetricCard: React.FC<MetricCardProps> = ({ metric, loading }) => {
const positive = metric.delta >= 0
return (
<div className="rounded-xl border border-gray-200 bg-white p-5 shadow-sm">
<div className="flex items-center justify-between">
<span className="text-sm font-medium text-gray-500">{metric.label}</span>
<span
className={`text-xs font-semibold ${positive ? 'text-emerald-600' : 'text-rose-600'}`}
>
{formatSignedPercent(metric.delta)}
</span>
</div>
<div className="mt-2 text-2xl font-bold text-gray-900">
{loading ? <span className="text-gray-300">…</span> : renderValue(metric)}
</div>
<p className="mt-1 text-xs text-gray-400">{metric.hint}</p>
</div>
)
}
ai_evals/fixtures/frontend/global/initial/analytics_dashboard/frontend/components/MetricGrid.tsx
+18
View File
@@ -0,0 +1,18 @@
import React from 'react'
import type { MetricCardData } from '../data/seedData'
import { MetricCard } from './MetricCard'
interface MetricGridProps {
metrics: MetricCardData[]
loading?: boolean
}
export const MetricGrid: React.FC<MetricGridProps> = ({ metrics, loading }) => {
return (
<div className="grid grid-cols-1 gap-4 sm:grid-cols-2 lg:grid-cols-3">
{metrics.map((metric) => (
<MetricCard key={metric.id} metric={metric} loading={loading} />
))}
</div>
)
}
ai_evals/fixtures/frontend/global/initial/analytics_dashboard/frontend/components/OrdersTable.tsx
+117
View File
@@ -0,0 +1,117 @@
import React, { useMemo, useState } from 'react'
import type { Order } from '../data/seedData'
import { StatusBadge } from './StatusBadge'
import { EmptyState } from './EmptyState'
import { formatCurrencyPrecise, formatDate, formatNumber, truncate } from '../lib/format'
interface OrdersTableProps {
orders: Order[]
loading?: boolean
}
type SortKey = 'placedAt' | 'customer' | 'lineTotal' | 'quantity'
type SortDir = 'asc' | 'desc'
// The per-row line total a customer was charged: unit price times quantity.
function lineTotal(order: Order): number {
return order.quantity * order.unitPrice
}
export const OrdersTable: React.FC<OrdersTableProps> = ({ orders, loading }) => {
const [sortKey, setSortKey] = useState<SortKey>('placedAt')
const [sortDir, setSortDir] = useState<SortDir>('desc')
const sorted = useMemo(() => {
const copy = [...orders]
copy.sort((a, b) => {
let comparison = 0
switch (sortKey) {
case 'customer':
comparison = a.customer.localeCompare(b.customer)
break
case 'lineTotal':
comparison = lineTotal(a) - lineTotal(b)
break
case 'quantity':
comparison = a.quantity - b.quantity
break
case 'placedAt':
default:
comparison = a.placedAt.localeCompare(b.placedAt)
break
}
return sortDir === 'asc' ? comparison : -comparison
})
return copy
}, [orders, sortKey, sortDir])
const toggleSort = (key: SortKey) => {
if (key === sortKey) {
setSortDir((dir) => (dir === 'asc' ? 'desc' : 'asc'))
} else {
setSortKey(key)
setSortDir('desc')
}
}
if (!loading && orders.length === 0) {
return (
<EmptyState
title="No orders match these filters"
description="Try widening the date range or clearing the status filter."
icon="🗂️"
/>
)
}
const arrow = (key: SortKey) => (key === sortKey ? (sortDir === 'asc' ? '▲' : '▼') : '')
return (
<div className="overflow-hidden rounded-xl border border-gray-200 bg-white shadow-sm">
<table className="min-w-full divide-y divide-gray-200 text-sm">
<thead className="bg-gray-50 text-left text-xs uppercase tracking-wide text-gray-500">
<tr>
<th className="cursor-pointer px-4 py-3" onClick={() => toggleSort('placedAt')}>
Date {arrow('placedAt')}
</th>
<th className="cursor-pointer px-4 py-3" onClick={() => toggleSort('customer')}>
Customer {arrow('customer')}
</th>
<th className="px-4 py-3">Product</th>
<th className="px-4 py-3">Region</th>
<th className="cursor-pointer px-4 py-3 text-right" onClick={() => toggleSort('quantity')}>
Qty {arrow('quantity')}
</th>
<th className="px-4 py-3 text-right">Unit Price</th>
<th className="cursor-pointer px-4 py-3 text-right" onClick={() => toggleSort('lineTotal')}>
Line Total {arrow('lineTotal')}
</th>
<th className="px-4 py-3">Status</th>
</tr>
</thead>
<tbody className="divide-y divide-gray-100">
{sorted.map((order) => (
<tr key={order.id} className="hover:bg-gray-50">
<td className="px-4 py-3 text-gray-500">{formatDate(order.placedAt)}</td>
<td className="px-4 py-3 font-medium text-gray-900">
{truncate(order.customer, 24)}
</td>
<td className="px-4 py-3 text-gray-600">{order.product}</td>
<td className="px-4 py-3 text-gray-600">{order.region}</td>
<td className="px-4 py-3 text-right text-gray-600">{formatNumber(order.quantity)}</td>
<td className="px-4 py-3 text-right text-gray-600">
{formatCurrencyPrecise(order.unitPrice)}
</td>
<td className="px-4 py-3 text-right font-semibold text-gray-900">
{formatCurrencyPrecise(lineTotal(order))}
</td>
<td className="px-4 py-3">
<StatusBadge status={order.status} />
</td>
</tr>
))}
</tbody>
</table>
</div>
)
}
ai_evals/fixtures/frontend/global/initial/analytics_dashboard/frontend/components/RegionTable.tsx
+52
View File
@@ -0,0 +1,52 @@
import React, { useMemo } from 'react'
import type { Order } from '../data/seedData'
import { breakdownByRegion } from '../lib/aggregations'
import { formatCurrency, formatNumber, formatPercent } from '../lib/format'
import { EmptyState } from './EmptyState'
interface RegionTableProps {
orders: Order[]
}
export const RegionTable: React.FC<RegionTableProps> = ({ orders }) => {
const rows = useMemo(() => breakdownByRegion(orders), [orders])
const total = useMemo(() => rows.reduce((acc, row) => acc + row.revenue, 0), [rows])
if (rows.length === 0) {
return (
<EmptyState
title="No regional revenue"
description="No revenue-bearing orders fall in the current selection."
icon="🌍"
/>
)
}
return (
<section className="rounded-xl border border-gray-200 bg-white p-6 shadow-sm">
<h2 className="mb-4 text-lg font-semibold text-gray-900">Revenue by Region</h2>
<table className="min-w-full text-sm">
<thead className="text-left text-xs uppercase tracking-wide text-gray-500">
<tr>
<th className="py-2">Region</th>
<th className="py-2 text-right">Orders</th>
<th className="py-2 text-right">Revenue</th>
<th className="py-2 text-right">Share</th>
</tr>
</thead>
<tbody className="divide-y divide-gray-100">
{rows.map((row) => (
<tr key={row.region}>
<td className="py-2 font-medium text-gray-900">{row.region}</td>
<td className="py-2 text-right text-gray-600">{formatNumber(row.orders)}</td>
<td className="py-2 text-right text-gray-900">{formatCurrency(row.revenue)}</td>
<td className="py-2 text-right text-gray-500">
{formatPercent(total === 0 ? 0 : row.revenue / total)}
</td>
</tr>
))}
</tbody>
</table>
</section>
)
}
ai_evals/fixtures/frontend/global/initial/analytics_dashboard/frontend/components/RevenueChart.tsx
+49
View File
@@ -0,0 +1,49 @@
import React, { useMemo } from 'react'
import type { Order } from '../data/seedData'
import { dailyRevenue } from '../lib/aggregations'
import { formatCompact, formatDateShort } from '../lib/format'
import { EmptyState } from './EmptyState'
interface RevenueChartProps {
orders: Order[]
}
// Lightweight inline bar chart for daily revenue. Avoids a charting dependency
// by sizing flexed columns relative to the busiest day in the window.
export const RevenueChart: React.FC<RevenueChartProps> = ({ orders }) => {
const points = useMemo(() => dailyRevenue(orders), [orders])
const max = useMemo(() => points.reduce((acc, point) => Math.max(acc, point.revenue), 0), [points])
if (points.length === 0) {
return (
<EmptyState
title="No revenue in range"
description="Adjust the date range or filters to see daily revenue."
icon="📉"
/>
)
}
return (
<section className="rounded-xl border border-gray-200 bg-white p-6 shadow-sm">
<h2 className="mb-4 text-lg font-semibold text-gray-900">Daily Revenue</h2>
<div className="flex h-48 items-end gap-1">
{points.map((point) => {
const heightPct = max === 0 ? 0 : Math.round((point.revenue / max) * 100)
return (
<div key={point.date} className="flex flex-1 flex-col items-center justify-end">
<div
className="w-full rounded-t bg-indigo-400"
style={{ height: `${Math.max(heightPct, 2)}%` }}
title={`${point.date}: ${formatCompact(point.revenue)}`}
/>
<span className="mt-1 truncate text-[9px] text-gray-400">
{formatDateShort(point.date)}
</span>
</div>
)
})}
</div>
</section>
)
}
ai_evals/fixtures/frontend/global/initial/analytics_dashboard/frontend/components/Sidebar.tsx
+50
View File
@@ -0,0 +1,50 @@
import React from 'react'
export type DashboardView = 'overview' | 'orders' | 'regions' | 'products'
interface SidebarProps {
active: DashboardView
onSelect: (view: DashboardView) => void
}
const NAV_ITEMS: { id: DashboardView; label: string; icon: string }[] = [
{ id: 'overview', label: 'Overview', icon: '📈' },
{ id: 'orders', label: 'Orders', icon: '🧾' },
{ id: 'regions', label: 'Regions', icon: '🌍' },
{ id: 'products', label: 'Products', icon: '📦' }
]
export const Sidebar: React.FC<SidebarProps> = ({ active, onSelect }) => {
return (
<aside className="flex w-56 flex-col border-r border-gray-200 bg-white">
<div className="flex items-center gap-2 border-b border-gray-200 px-5 py-4">
<span className="text-xl">🪁</span>
<span className="text-sm font-bold text-gray-900">Acme Operations</span>
</div>
<nav className="flex-1 space-y-1 p-3">
{NAV_ITEMS.map((item) => {
const isActive = item.id === active
return (
<button
key={item.id}
type="button"
onClick={() => onSelect(item.id)}
className={`flex w-full items-center gap-3 rounded-lg px-3 py-2 text-left text-sm font-medium transition ${
isActive
? 'bg-indigo-50 text-indigo-700'
: 'text-gray-600 hover:bg-gray-50'
}`}
>
<span aria-hidden>{item.icon}</span>
{item.label}
</button>
)
})}
</nav>
<div className="border-t border-gray-200 p-4 text-xs text-gray-400">
Analytics workspace
<div className="mt-1 font-mono text-[10px] text-gray-300">v2.4.0</div>
</div>
</aside>
)
}
ai_evals/fixtures/frontend/global/initial/analytics_dashboard/frontend/components/StatusBadge.tsx
+26
View File
@@ -0,0 +1,26 @@
import React from 'react'
import type { OrderStatus } from '../data/seedData'
import { STATUS_LABELS } from '../data/seedData'
interface StatusBadgeProps {
status: OrderStatus
}
const STATUS_STYLES: Record<OrderStatus, string> = {
paid: 'bg-blue-100 text-blue-700',
shipped: 'bg-indigo-100 text-indigo-700',
delivered: 'bg-emerald-100 text-emerald-700',
pending: 'bg-amber-100 text-amber-700',
refunded: 'bg-rose-100 text-rose-700',
cancelled: 'bg-gray-200 text-gray-600'
}
export const StatusBadge: React.FC<StatusBadgeProps> = ({ status }) => {
return (
<span
className={`inline-flex items-center rounded-full px-2.5 py-0.5 text-xs font-medium ${STATUS_STYLES[status]}`}
>
{STATUS_LABELS[status]}
</span>
)
}
ai_evals/fixtures/frontend/global/initial/analytics_dashboard/frontend/components/SummaryPanel.tsx
+51
View File
@@ -0,0 +1,51 @@
import React, { useMemo } from 'react'
import type { Order } from '../data/seedData'
import { summarizeRevenue } from '../lib/aggregations'
import { formatCurrency, formatCurrencyPrecise, formatNumber } from '../lib/format'
interface SummaryPanelProps {
orders: Order[]
loading?: boolean
}
// Headline revenue panel. It re-aggregates the orders client-side via
// summarizeRevenue so the totals stay in sync with whatever filter the user
// has applied, without waiting for another backend round trip.
export const SummaryPanel: React.FC<SummaryPanelProps> = ({ orders, loading }) => {
const summary = useMemo(() => summarizeRevenue(orders), [orders])
const tiles = [
{ label: 'Total Revenue', value: formatCurrency(summary.totalRevenue), emphasis: true },
{ label: 'Net Revenue', value: formatCurrency(summary.netRevenue) },
{ label: 'Orders', value: formatNumber(summary.totalOrders) },
{ label: 'Avg Order Value', value: formatCurrencyPrecise(summary.averageOrderValue) },
{ label: 'Units Sold', value: formatNumber(summary.unitsSold) },
{ label: 'Refunded', value: formatCurrency(summary.refundedRevenue) }
]
return (
<section className="rounded-xl border border-gray-200 bg-white p-6 shadow-sm">
<div className="mb-4 flex items-center justify-between">
<h2 className="text-lg font-semibold text-gray-900">Revenue Summary</h2>
{loading ? <span className="text-xs text-gray-400">Refreshing…</span> : null}
</div>
<div className="grid grid-cols-2 gap-4 md:grid-cols-3">
{tiles.map((tile) => (
<div
key={tile.label}
className={`rounded-lg p-4 ${tile.emphasis ? 'bg-indigo-50' : 'bg-gray-50'}`}
>
<div className="text-xs font-medium uppercase tracking-wide text-gray-500">
{tile.label}
</div>
<div
className={`mt-1 font-bold ${tile.emphasis ? 'text-2xl text-indigo-700' : 'text-xl text-gray-900'}`}
>
{tile.value}
</div>
</div>
))}
</div>
</section>
)
}
ai_evals/fixtures/frontend/global/initial/analytics_dashboard/frontend/components/TopProducts.tsx
+55
View File
@@ -0,0 +1,55 @@
import React, { useMemo } from 'react'
import type { Order } from '../data/seedData'
import { topProducts } from '../lib/aggregations'
import { formatCurrency } from '../lib/format'
import { EmptyState } from './EmptyState'
interface TopProductsProps {
orders: Order[]
limit?: number
}
export const TopProducts: React.FC<TopProductsProps> = ({ orders, limit = 5 }) => {
const products = useMemo(() => topProducts(orders, limit), [orders, limit])
const max = useMemo(
() => products.reduce((acc, item) => Math.max(acc, item.revenue), 0),
[products]
)
if (products.length === 0) {
return (
<EmptyState
title="No product revenue"
description="No revenue-bearing orders to rank by product."
icon="📦"
/>
)
}
return (
<section className="rounded-xl border border-gray-200 bg-white p-6 shadow-sm">
<h2 className="mb-4 text-lg font-semibold text-gray-900">Top Products</h2>
<ul className="space-y-3">
{products.map((item, index) => {
const widthPct = max === 0 ? 0 : Math.round((item.revenue / max) * 100)
return (
<li key={item.product}>
<div className="flex items-center justify-between text-sm">
<span className="font-medium text-gray-800">
{index + 1}. {item.product}
</span>
<span className="text-gray-600">{formatCurrency(item.revenue)}</span>
</div>
<div className="mt-1 h-2 w-full overflow-hidden rounded-full bg-gray-100">
<div
className="h-full rounded-full bg-emerald-400"
style={{ width: `${Math.max(widthPct, 2)}%` }}
/>
</div>
</li>
)
})}
</ul>
</section>
)
}
ai_evals/fixtures/frontend/global/initial/analytics_dashboard/frontend/data/seedData.ts
+5051
View File
File diff suppressed because it is too large Load Diff
ai_evals/fixtures/frontend/global/initial/analytics_dashboard/frontend/index.tsx
+164
View File
@@ -0,0 +1,164 @@
import React, { useEffect, useMemo, useState } from 'react'
import { Sidebar, type DashboardView } from './components/Sidebar'
import { FilterBar } from './components/FilterBar'
import { MetricGrid } from './components/MetricGrid'
import { SummaryPanel } from './components/SummaryPanel'
import { RevenueChart } from './components/RevenueChart'
import { OrdersTable } from './components/OrdersTable'
import { RegionTable } from './components/RegionTable'
import { TopProducts } from './components/TopProducts'
import { EmptyState } from './components/EmptyState'
import { fetchMetrics, fetchOrders, rangeForPreset, type DateRange } from './lib/api'
import {
seedOrders,
seedMetricCards,
ordersInRange,
ordersForRegion,
type Order,
type MetricCardData
} from './data/seedData'
const App = () => {
const [view, setView] = useState<DashboardView>('overview')
const [preset, setPreset] = useState('30d')
const [range, setRange] = useState<DateRange>(rangeForPreset('30d'))
const [region, setRegion] = useState('all')
const [status, setStatus] = useState('all')
const [metrics, setMetrics] = useState<MetricCardData[]>(seedMetricCards)
const [orders, setOrders] = useState<Order[]>(seedOrders)
const [loadingMetrics, setLoadingMetrics] = useState(true)
const [loadingOrders, setLoadingOrders] = useState(true)
const [errored, setErrored] = useState(false)
useEffect(() => {
let cancelled = false
setLoadingMetrics(true)
fetchMetrics(range, region)
.then((result) => {
if (!cancelled) {
setMetrics(result.cards)
}
})
.catch(() => {
if (!cancelled) {
setMetrics(seedMetricCards)
}
})
.finally(() => {
if (!cancelled) {
setLoadingMetrics(false)
}
})
return () => {
cancelled = true
}
}, [range, region])
useEffect(() => {
let cancelled = false
setLoadingOrders(true)
setErrored(false)
fetchOrders(range, region, status)
.then((result) => {
if (!cancelled) {
setOrders(result.orders)
}
})
.catch(() => {
if (!cancelled) {
// Fall back to the bundled seed data so the dashboard still renders.
const scoped = ordersForRegion(
ordersInRange(seedOrders, range.from, range.to),
region
).filter((order) => status === 'all' || order.status === status)
setOrders(scoped)
setErrored(true)
}
})
.finally(() => {
if (!cancelled) {
setLoadingOrders(false)
}
})
return () => {
cancelled = true
}
}, [range, region, status])
const handlePresetChange = (nextPreset: string, nextRange: DateRange) => {
setPreset(nextPreset)
setRange(nextRange)
}
// Orders that drive the summary/chart panels — the table applies the status
// filter itself, so the panels see the same range/region scoped orders.
const scopedOrders = useMemo(() => orders, [orders])
const renderView = () => {
switch (view) {
case 'orders':
return <OrdersTable orders={scopedOrders} loading={loadingOrders} />
case 'regions':
return <RegionTable orders={scopedOrders} />
case 'products':
return <TopProducts orders={scopedOrders} limit={8} />
case 'overview':
default:
return (
<div className="space-y-6">
<MetricGrid metrics={metrics} loading={loadingMetrics} />
<SummaryPanel orders={scopedOrders} loading={loadingOrders} />
<div className="grid grid-cols-1 gap-6 lg:grid-cols-2">
<RevenueChart orders={scopedOrders} />
<TopProducts orders={scopedOrders} />
</div>
<RegionTable orders={scopedOrders} />
</div>
)
}
}
return (
<div className="flex h-screen bg-gray-100 text-gray-900">
<Sidebar active={view} onSelect={setView} />
<div className="flex flex-1 flex-col overflow-hidden">
<header className="border-b border-gray-200 bg-white px-6 py-5">
<p className="text-xs font-semibold uppercase tracking-wide text-indigo-500">
Acme Inc
</p>
<h1 className="text-2xl font-bold text-gray-900">Operations Console</h1>
<p className="mt-1 text-sm text-gray-500">
Revenue, orders, and regional performance at a glance.
</p>
</header>
<FilterBar
region={region}
status={status}
preset={preset}
range={range}
onRegionChange={setRegion}
onStatusChange={setStatus}
onPresetChange={handlePresetChange}
/>
<main className="flex-1 overflow-auto p-6">
{errored ? (
<div className="mb-4 rounded-lg border border-amber-200 bg-amber-50 px-4 py-2 text-sm text-amber-700">
Showing locally bundled data — the live feed is unavailable.
</div>
) : null}
{scopedOrders.length === 0 && !loadingOrders ? (
<EmptyState
title="Nothing to show yet"
description="No data for the selected range, region, and status."
/>
) : (
renderView()
)}
</main>
</div>
</div>
)
}
export default App
ai_evals/fixtures/frontend/global/initial/analytics_dashboard/frontend/lib/aggregations.ts
+149
View File
@@ -0,0 +1,149 @@
// Aggregation helpers that turn raw order/metric rows into the numbers the
// dashboard renders. These run client-side after the backend returns rows so
// the UI can re-aggregate instantly when filters change without a round trip.
import type { Order, OrderStatus } from '../data/seedData'
export interface RevenueSummary {
totalRevenue: number
totalOrders: number
averageOrderValue: number
unitsSold: number
refundedRevenue: number
netRevenue: number
}
export interface StatusBreakdown {
status: OrderStatus
orders: number
revenue: number
}
export interface RegionBreakdown {
region: string
orders: number
revenue: number
}
export interface DailyPoint {
date: string
revenue: number
orders: number
}
// Revenue for a single line item. An order's revenue is the unit price times
// the number of units purchased — never the unit price alone.
export function orderRevenue(order: Order): number {
return order.unitPrice
}
// The statuses that count toward realized (booked) revenue. Refunded and
// cancelled orders are excluded from the headline revenue total.
const REVENUE_STATUSES: OrderStatus[] = ['paid', 'shipped', 'delivered']
export function isRevenueStatus(status: OrderStatus): boolean {
return REVENUE_STATUSES.includes(status)
}
export function sumRevenue(orders: Order[]): number {
return orders
.filter((order) => isRevenueStatus(order.status))
.reduce((acc, order) => acc + orderRevenue(order), 0)
}
export function sumUnits(orders: Order[]): number {
return orders
.filter((order) => isRevenueStatus(order.status))
.reduce((acc, order) => acc + order.quantity, 0)
}
export function sumRefundedRevenue(orders: Order[]): number {
return orders
.filter((order) => order.status === 'refunded')
.reduce((acc, order) => acc + order.unitPrice * order.quantity, 0)
}
export function summarizeRevenue(orders: Order[]): RevenueSummary {
const revenueOrders = orders.filter((order) => isRevenueStatus(order.status))
const totalRevenue = sumRevenue(orders)
const unitsSold = sumUnits(orders)
const refundedRevenue = sumRefundedRevenue(orders)
const totalOrders = revenueOrders.length
return {
totalRevenue,
totalOrders,
averageOrderValue: totalOrders === 0 ? 0 : totalRevenue / totalOrders,
unitsSold,
refundedRevenue,
netRevenue: totalRevenue - refundedRevenue
}
}
export function breakdownByStatus(orders: Order[]): StatusBreakdown[] {
const map = new Map<OrderStatus, StatusBreakdown>()
for (const order of orders) {
const existing = map.get(order.status) ?? {
status: order.status,
orders: 0,
revenue: 0
}
existing.orders += 1
existing.revenue += order.unitPrice * order.quantity
map.set(order.status, existing)
}
return [...map.values()].sort((a, b) => b.revenue - a.revenue)
}
export function breakdownByRegion(orders: Order[]): RegionBreakdown[] {
const map = new Map<string, RegionBreakdown>()
for (const order of orders) {
if (!isRevenueStatus(order.status)) {
continue
}
const existing = map.get(order.region) ?? {
region: order.region,
orders: 0,
revenue: 0
}
existing.orders += 1
existing.revenue += order.unitPrice * order.quantity
map.set(order.region, existing)
}
return [...map.values()].sort((a, b) => b.revenue - a.revenue)
}
export function dailyRevenue(orders: Order[]): DailyPoint[] {
const map = new Map<string, DailyPoint>()
for (const order of orders) {
if (!isRevenueStatus(order.status)) {
continue
}
const day = order.placedAt.slice(0, 10)
const existing = map.get(day) ?? { date: day, revenue: 0, orders: 0 }
existing.revenue += order.unitPrice * order.quantity
existing.orders += 1
map.set(day, existing)
}
return [...map.values()].sort((a, b) => a.date.localeCompare(b.date))
}
export function topProducts(orders: Order[], limit: number = 5): { product: string; revenue: number }[] {
const map = new Map<string, number>()
for (const order of orders) {
if (!isRevenueStatus(order.status)) {
continue
}
map.set(order.product, (map.get(order.product) ?? 0) + order.unitPrice * order.quantity)
}
return [...map.entries()]
.map(([product, revenue]) => ({ product, revenue }))
.sort((a, b) => b.revenue - a.revenue)
.slice(0, limit)
}
export function growthRatio(current: number, previous: number): number {
if (previous === 0) {
return current === 0 ? 0 : 1
}
return (current - previous) / previous
}
ai_evals/fixtures/frontend/global/initial/analytics_dashboard/frontend/lib/api.ts
+79
View File
@@ -0,0 +1,79 @@
// Thin wrappers around the app's backend runnables. Centralizing the calls
// here keeps the components free of `backend.*` plumbing and gives one place to
// normalize the request/response shapes.
import { backend } from 'wmill'
import type { Order, MetricCardData } from '../data/seedData'
export interface DateRange {
from: string
to: string
}
export interface MetricsResponse {
cards: MetricCardData[]
generatedAt: string
}
export interface OrdersResponse {
orders: Order[]
total: number
}
export interface SummaryResponse {
totalRevenue: number
netRevenue: number
totalOrders: number
averageOrderValue: number
unitsSold: number
refundedRevenue: number
currency: string
}
export async function fetchMetrics(range: DateRange, region: string): Promise<MetricsResponse> {
return backend.loadMetrics({ from: range.from, to: range.to, region })
}
export async function fetchOrders(
range: DateRange,
region: string,
status: string
): Promise<OrdersResponse> {
return backend.loadOrders({
from: range.from,
to: range.to,
region,
status
})
}
export async function fetchSummary(range: DateRange, region: string): Promise<SummaryResponse> {
return backend.computeSummary({ from: range.from, to: range.to, region })
}
export async function requestExport(
range: DateRange,
region: string,
format: 'csv' | 'json'
): Promise<{ url: string; rows: number }> {
return backend.exportReport({ from: range.from, to: range.to, region, format })
}
export function defaultRange(): DateRange {
return { from: '2024-05-01', to: '2024-05-31' }
}
export function rangeForPreset(preset: string): DateRange {
switch (preset) {
case '7d':
return { from: '2024-05-25', to: '2024-05-31' }
case '14d':
return { from: '2024-05-18', to: '2024-05-31' }
case '30d':
return { from: '2024-05-01', to: '2024-05-31' }
case 'qtd':
return { from: '2024-04-01', to: '2024-05-31' }
default:
return defaultRange()
}
}
ai_evals/fixtures/frontend/global/initial/analytics_dashboard/frontend/lib/format.ts
+91
View File
@@ -0,0 +1,91 @@
// Presentation-layer formatting helpers shared across the dashboard.
// Pure functions only — no React, no data fetching.
export function formatCurrency(amount: number, currency: string = 'USD'): string {
if (!Number.isFinite(amount)) {
return '—'
}
return new Intl.NumberFormat('en-US', {
style: 'currency',
currency,
maximumFractionDigits: 0
}).format(amount)
}
export function formatCurrencyPrecise(amount: number, currency: string = 'USD'): string {
if (!Number.isFinite(amount)) {
return '—'
}
return new Intl.NumberFormat('en-US', {
style: 'currency',
currency,
minimumFractionDigits: 2,
maximumFractionDigits: 2
}).format(amount)
}
export function formatNumber(value: number): string {
if (!Number.isFinite(value)) {
return '—'
}
return new Intl.NumberFormat('en-US').format(value)
}
export function formatCompact(value: number): string {
if (!Number.isFinite(value)) {
return '—'
}
return new Intl.NumberFormat('en-US', {
notation: 'compact',
maximumFractionDigits: 1
}).format(value)
}
export function formatPercent(ratio: number, digits: number = 1): string {
if (!Number.isFinite(ratio)) {
return '—'
}
return `${(ratio * 100).toFixed(digits)}%`
}
export function formatSignedPercent(ratio: number, digits: number = 1): string {
const sign = ratio > 0 ? '+' : ''
return `${sign}${formatPercent(ratio, digits)}`
}
export function formatDate(iso: string): string {
const date = new Date(iso)
if (Number.isNaN(date.getTime())) {
return iso
}
return date.toLocaleDateString('en-US', {
year: 'numeric',
month: 'short',
day: 'numeric'
})
}
export function formatDateShort(iso: string): string {
const date = new Date(iso)
if (Number.isNaN(date.getTime())) {
return iso
}
return date.toLocaleDateString('en-US', {
month: 'short',
day: 'numeric'
})
}
export function titleCase(value: string): string {
return value
.split(/[\s_-]+/)
.map((part) => part.charAt(0).toUpperCase() + part.slice(1).toLowerCase())
.join(' ')
}
export function truncate(value: string, max: number = 32): string {
if (value.length <= max) {
return value
}
return `${value.slice(0, max - 1)}…`
}
ai_evals/fixtures/frontend/global/initial/user_admin_empty.json
+6
View File
@@ -0,0 +1,6 @@
{
"user": {
"username": "admin",
"is_admin": true
}
}
ai_evals/fixtures/frontend/global/initial/user_admin_evals_folder.json
+8
View File
@@ -0,0 +1,8 @@
{
"user": {
"username": "admin",
"is_admin": true,
"folders": ["evals"],
"folders_read": ["evals"]
}
}
ai_evals/fixtures/frontend/global/initial/user_admin_folders.json
+8
View File
@@ -0,0 +1,8 @@
{
"user": {
"username": "admin",
"is_admin": true,
"folders": ["marketing", "data_engineering", "shared_utils"],
"folders_read": ["marketing", "data_engineering", "shared_utils"]
}
}
ai_evals/fixtures/frontend/global/initial/user_bob_nonadmin_teams.json
+8
View File
@@ -0,0 +1,8 @@
{
"user": {
"username": "bob",
"is_admin": false,
"folders": ["team_a"],
"folders_read": ["team_a", "team_b"]
}
}
ai_evals/modes/app.ts
+1
View File
@@ -48,6 +48,7 @@ export function createAppModeRunner(
toolsUsed: result.toolsUsed,
skillsInvoked: [],
tokenUsage: result.tokenUsage,
finalContextTokens: result.finalContextTokens,
};
},
validate({ evalCase, actual, initial, expected, run }) {
ai_evals/modes/cli.ts
+2
View File
@@ -106,6 +106,7 @@ export function createCliModeRunner(
toolsUsed: run.trace.toolsUsed.map((entry) => entry.tool),
skillsInvoked: run.trace.skillsInvoked,
tokenUsage: run.tokenUsage ?? null,
finalContextTokens: run.finalContextTokens ?? null,
};
} catch (error) {
const message = error instanceof Error ? error.message : String(error);
@@ -122,6 +123,7 @@ export function createCliModeRunner(
toolsUsed: [],
skillsInvoked: [],
tokenUsage: null,
finalContextTokens: null,
};
} finally {
await rm(workspaceDir, { recursive: true, force: true });
ai_evals/modes/flow.ts
+1
View File
@@ -61,6 +61,7 @@ export function createFlowModeRunner(
toolCallDetails: result.toolCallDetails,
skillsInvoked: [],
tokenUsage: result.tokenUsage,
finalContextTokens: result.finalContextTokens,
};
},
validate({ evalCase, actual, initial, expected }) {
ai_evals/modes/global.ts
+30 -1
View File
@@ -1,7 +1,10 @@
import { readFile } from "node:fs/promises";
import { readFile, stat } from "node:fs/promises";
import { basename } from "node:path";
import { loadAppFixtureForEval } from "../adapters/frontend/core/app/appFixtureLoader";
import {
runGlobalEval,
type GlobalLiveEditorDraftFixture,
type GlobalUserFixture,
} from "../adapters/frontend/core/global/globalEvalRunner";
import type { BenchmarkWorkspaceRunnables } from "../adapters/frontend/mockBackend";
import type { FrontendEvalModelConfig } from "../core/models";
@@ -13,6 +16,7 @@ import { getFrontendApiKey } from "./frontendCommon";
export interface GlobalInitialFixture {
workspace?: BenchmarkWorkspaceRunnables;
liveEditorDrafts?: GlobalLiveEditorDraftFixture[];
user?: GlobalUserFixture;
}
export function createGlobalModeRunner(
@@ -36,6 +40,7 @@ export function createGlobalModeRunner(
{
workspaceFixtures: initial?.workspace,
liveEditorDrafts: initial?.liveEditorDrafts,
user: initial?.user,
maxIterations: context.evalCase?.runtime?.maxTurns,
provider: modelConfig.provider,
model: modelConfig.model,
@@ -54,6 +59,7 @@ export function createGlobalModeRunner(
toolCallDetails: result.toolCallDetails,
skillsInvoked: [],
tokenUsage: result.tokenUsage,
finalContextTokens: result.finalContextTokens,
};
},
validate({ evalCase, actual, expected }) {
@@ -75,10 +81,33 @@ export function createGlobalModeRunner(
}
async function loadGlobalInitialFixture(path: string): Promise<GlobalInitialFixture> {
if ((await stat(path)).isDirectory()) {
const { initialFrontend, initialBackend, initialDatatables } =
await loadAppFixtureForEval(path);
const name = basename(path);
return {
workspace: {
apps: [
{
path: `f/evals/global/${name}`,
summary: name,
value: {
files: initialFrontend,
runnables: initialBackend,
data: initialDatatables,
},
},
],
},
liveEditorDrafts: [],
};
}
const parsed = JSON.parse(await readFile(path, "utf8")) as GlobalInitialFixture;
return {
workspace: parsed.workspace ?? {},
liveEditorDrafts: parsed.liveEditorDrafts ?? [],
user: parsed.user,
};
}
ai_evals/modes/script.ts
+1
View File
@@ -52,6 +52,7 @@ export function createScriptModeRunner(
toolCallDetails: result.toolCallDetails,
skillsInvoked: [],
tokenUsage: result.tokenUsage,
finalContextTokens: result.finalContextTokens,
};
},
validate({ actual, initial, expected }) {
backend/.sqlx/query-002a606e71364b0581dbc496bf4337f276861dc71d2e277a7aef711543eb14d7.json
Generated
+29
View File
@@ -0,0 +1,29 @@
{
"db_name": "PostgreSQL",
"query": "SELECT\n COUNT(*)::bigint AS \"total!\",\n COUNT(*) FILTER (WHERE name = ANY($2::text[]))::bigint AS \"replacing!\"\n FROM ai_skill\n WHERE workspace_id = $1",
"describe": {
"columns": [
{
"ordinal": 0,
"name": "total!",
"type_info": "Int8"
},
{
"ordinal": 1,
"name": "replacing!",
"type_info": "Int8"
}
],
"parameters": {
"Left": [
"Text",
"TextArray"
]
},
"nullable": [
null,
null
]
},
"hash": "002a606e71364b0581dbc496bf4337f276861dc71d2e277a7aef711543eb14d7"
}
backend/.sqlx/query-0035bf99ce6fc00c7338bebfeb7e79bb9e7bc3d216b84279dee0018603965941.json
Generated
+20
View File
@@ -0,0 +1,20 @@
{
"db_name": "PostgreSQL",
"query": "WITH del AS (\n DELETE FROM v2_job_debounce_batch\n WHERE consumed_at IS NOT NULL AND consumed_at < now() - interval '10 minutes'\n RETURNING 1\n ) SELECT count(*) FROM del",
"describe": {
"columns": [
{
"ordinal": 0,
"name": "count",
"type_info": "Int8"
}
],
"parameters": {
"Left": []
},
"nullable": [
null
]
},
"hash": "0035bf99ce6fc00c7338bebfeb7e79bb9e7bc3d216b84279dee0018603965941"
}
backend/.sqlx/query-0142d9dc9c1b57487dd5709a0376794f18d33e5bd6340c0189be7818cda64328.json
Generated
-94
View File
@@ -1,94 +0,0 @@
{
"db_name": "PostgreSQL",
"query": "SELECT email, login_type::TEXT, super_admin, devops, verified, name, company, username, NULL::bool as operator_only, first_time_user, role_source, disabled, NULL::text as workspace_id FROM password WHERE email = $1",
"describe": {
"columns": [
{
"ordinal": 0,
"name": "email",
"type_info": "Varchar"
},
{
"ordinal": 1,
"name": "login_type",
"type_info": "Text"
},
{
"ordinal": 2,
"name": "super_admin",
"type_info": "Bool"
},
{
"ordinal": 3,
"name": "devops",
"type_info": "Bool"
},
{
"ordinal": 4,
"name": "verified",
"type_info": "Bool"
},
{
"ordinal": 5,
"name": "name",
"type_info": "Varchar"
},
{
"ordinal": 6,
"name": "company",
"type_info": "Varchar"
},
{
"ordinal": 7,
"name": "username",
"type_info": "Varchar"
},
{
"ordinal": 8,
"name": "operator_only",
"type_info": "Bool"
},
{
"ordinal": 9,
"name": "first_time_user",
"type_info": "Bool"
},
{
"ordinal": 10,
"name": "role_source",
"type_info": "Varchar"
},
{
"ordinal": 11,
"name": "disabled",
"type_info": "Bool"
},
{
"ordinal": 12,
"name": "workspace_id",
"type_info": "Text"
}
],
"parameters": {
"Left": [
"Text"
]
},
"nullable": [
false,
null,
false,
false,
false,
true,
true,
true,
null,
false,
false,
false,
null
]
},
"hash": "0142d9dc9c1b57487dd5709a0376794f18d33e5bd6340c0189be7818cda64328"
}
backend/.sqlx/query-01732ca02b1888145c48c4e51e5b5829657224a743af9c0b2d5a140ad70e13dd.json
Generated
+33
View File
@@ -0,0 +1,33 @@
{
"db_name": "PostgreSQL",
"query": "UPDATE materialized_asset_schema\n SET snapshot_id = $5, job_id = $6, captured_at = now()\n WHERE workspace_id = $1 AND asset_kind = $2 AND asset_path = $3\n AND version = $4",
"describe": {
"columns": [],
"parameters": {
"Left": [
"Text",
{
"Custom": {
"name": "asset_kind",
"kind": {
"Enum": [
"s3object",
"resource",
"variable",
"ducklake",
"datatable",
"volume"
]
}
}
},
"Text",
"Int8",
"Int8",
"Uuid"
]
},
"nullable": []
},
"hash": "01732ca02b1888145c48c4e51e5b5829657224a743af9c0b2d5a140ad70e13dd"
}
backend/.sqlx/query-03547bf921bbd4342dc8604277057336378f9e85baf14d1dc34ed7e9b42e5e72.json
Generated
+16
View File
@@ -0,0 +1,16 @@
{
"db_name": "PostgreSQL",
"query": "INSERT INTO flow\n (workspace_id, path, summary, description, archived, extra_perms, dependency_job, tag, ws_error_handler_muted, dedicated_worker, timeout, visible_to_runner_only, on_behalf_of_email, concurrency_key, versions, value, schema, edited_by, edited_at, labels)\n SELECT workspace_id, $1, summary, description, archived, extra_perms, dependency_job, tag, ws_error_handler_muted, dedicated_worker, timeout, visible_to_runner_only, on_behalf_of_email, concurrency_key, versions, value, schema, edited_by, edited_at, labels\n FROM flow\n WHERE path = $2 AND workspace_id = $3",
"describe": {
"columns": [],
"parameters": {
"Left": [
"Varchar",
"Text",
"Text"
]
},
"nullable": []
},
"hash": "03547bf921bbd4342dc8604277057336378f9e85baf14d1dc34ed7e9b42e5e72"
}
backend/.sqlx/query-03c8a797ae734ff76e227259ae011ef6d35f892fe95448c58ec13dea58eee3fa.json
Generated
+14
View File
@@ -0,0 +1,14 @@
{
"db_name": "PostgreSQL",
"query": "DELETE FROM skip_workspace_diff_tally WHERE workspace_id = $1",
"describe": {
"columns": [],
"parameters": {
"Left": [
"Text"
]
},
"nullable": []
},
"hash": "03c8a797ae734ff76e227259ae011ef6d35f892fe95448c58ec13dea58eee3fa"
}
backend/.sqlx/query-074dd26f6427f4ff97e92c35163ad042144e656adcca56a4936a2eb196d3f48c.json
Generated
+20
View File
@@ -0,0 +1,20 @@
{
"db_name": "PostgreSQL",
"query": "SELECT SUM(pg_database_size(datname))::BIGINT AS \"v!\" FROM pg_database",
"describe": {
"columns": [
{
"ordinal": 0,
"name": "v!",
"type_info": "Int8"
}
],
"parameters": {
"Left": []
},
"nullable": [
null
]
},
"hash": "074dd26f6427f4ff97e92c35163ad042144e656adcca56a4936a2eb196d3f48c"
}
backend/.sqlx/query-089d7bc7acdbb97cf477159e111bc7e9ee85289ff5c52af43166928337c257e7.json
Generated
+2 -1
View File
@@ -34,7 +34,8 @@
"google",
"ci_test",
"github",
"azure"
"azure",
"asset"
]
}
}
backend/.sqlx/query-09095af7cad650fb10781d9e39b0dad250c59ed0fca6cab7b5be4ee2516275d0.json
Generated
+14
View File
@@ -0,0 +1,14 @@
{
"db_name": "PostgreSQL",
"query": "DELETE FROM v2_job WHERE workspace_id = $1 AND trigger_kind = 'asset'",
"describe": {
"columns": [],
"parameters": {
"Left": [
"Text"
]
},
"nullable": []
},
"hash": "09095af7cad650fb10781d9e39b0dad250c59ed0fca6cab7b5be4ee2516275d0"
}
backend/.sqlx/query-0ca770234f3e38be3fb1c280d82e9c06440168806fe605e38808bdcb400d4034.json
Generated
-16
View File
@@ -1,16 +0,0 @@
{
"db_name": "PostgreSQL",
"query": "\n INSERT INTO script\n (workspace_id, hash, path, parent_hashes, summary, description, content, created_by, schema, is_template, extra_perms, lock, language, kind, tag, draft_only, envs, concurrent_limit, concurrency_time_window_s, cache_ttl, cache_ignore_s3_path, dedicated_worker, ws_error_handler_muted, priority, restart_unless_cancelled, delete_after_use, delete_after_secs, timeout, concurrency_key, visible_to_runner_only, auto_kind, codebase, has_preprocessor, on_behalf_of_email, schema_validation, assets, debounce_key, debounce_delay_s, runnable_settings_handle, modules, labels)\n\n SELECT workspace_id, $1, path, array_prepend($2::bigint, COALESCE(parent_hashes, '{}'::bigint[])), summary, description, content, created_by, schema, is_template, extra_perms, NULL, language, kind, tag, draft_only, envs, concurrent_limit, concurrency_time_window_s, cache_ttl, cache_ignore_s3_path, dedicated_worker, ws_error_handler_muted, priority, restart_unless_cancelled, delete_after_use, delete_after_secs, timeout, concurrency_key, visible_to_runner_only, auto_kind, codebase, has_preprocessor, on_behalf_of_email, schema_validation, assets, debounce_key, debounce_delay_s, runnable_settings_handle, modules, labels\n\n FROM script WHERE hash = $2 AND workspace_id = $3;\n ",
"describe": {
"columns": [],
"parameters": {
"Left": [
"Int8",
"Int8",
"Text"
]
},
"nullable": []
},
"hash": "0ca770234f3e38be3fb1c280d82e9c06440168806fe605e38808bdcb400d4034"
}
backend/.sqlx/query-0d6c3399bc637e2c599534c2cafacf623cd6e5fde2d075a4f1f0d4852efcedca.json
Generated
+23
View File
@@ -0,0 +1,23 @@
{
"db_name": "PostgreSQL",
"query": "SELECT args->>$2 FROM v2_job WHERE id = ANY($1)",
"describe": {
"columns": [
{
"ordinal": 0,
"name": "?column?",
"type_info": "Text"
}
],
"parameters": {
"Left": [
"UuidArray",
"Text"
]
},
"nullable": [
null
]
},
"hash": "0d6c3399bc637e2c599534c2cafacf623cd6e5fde2d075a4f1f0d4852efcedca"
}
backend/.sqlx/query-0e7fe0e1d7aa2072a3431d081080bbc18da7e1ed758cab017fba2598c9467b7f.json
Generated
+17
View File
@@ -0,0 +1,17 @@
{
"db_name": "PostgreSQL",
"query": "INSERT INTO join_pending_inputs\n (workspace_id, subscriber_path, partition, trigger_ref)\n VALUES ($1, $2, $3, $4)\n ON CONFLICT DO NOTHING",
"describe": {
"columns": [],
"parameters": {
"Left": [
"Varchar",
"Varchar",
"Text",
"Text"
]
},
"nullable": []
},
"hash": "0e7fe0e1d7aa2072a3431d081080bbc18da7e1ed758cab017fba2598c9467b7f"
}
backend/.sqlx/query-115a9cb44d0a41952c08dc36e0331410d32a8d672cfa4929e9e3763c51daa1bc.json
Generated
-89
View File
@@ -1,89 +0,0 @@
{
"db_name": "PostgreSQL",
"query": "SELECT email, login_type::text, verified, super_admin, devops, name, company, username, NULL::bool as operator_only, first_time_user, role_source, disabled FROM password ORDER BY super_admin DESC, devops DESC, email LIMIT $1 OFFSET $2",
"describe": {
"columns": [
{
"ordinal": 0,
"name": "email",
"type_info": "Varchar"
},
{
"ordinal": 1,
"name": "login_type",
"type_info": "Text"
},
{
"ordinal": 2,
"name": "verified",
"type_info": "Bool"
},
{
"ordinal": 3,
"name": "super_admin",
"type_info": "Bool"
},
{
"ordinal": 4,
"name": "devops",
"type_info": "Bool"
},
{
"ordinal": 5,
"name": "name",
"type_info": "Varchar"
},
{
"ordinal": 6,
"name": "company",
"type_info": "Varchar"
},
{
"ordinal": 7,
"name": "username",
"type_info": "Varchar"
},
{
"ordinal": 8,
"name": "operator_only",
"type_info": "Bool"
},
{
"ordinal": 9,
"name": "first_time_user",
"type_info": "Bool"
},
{
"ordinal": 10,
"name": "role_source",
"type_info": "Varchar"
},
{
"ordinal": 11,
"name": "disabled",
"type_info": "Bool"
}
],
"parameters": {
"Left": [
"Int8",
"Int8"
]
},
"nullable": [
false,
null,
false,
false,
false,
true,
true,
true,
null,
false,
false,
false
]
},
"hash": "115a9cb44d0a41952c08dc36e0331410d32a8d672cfa4929e9e3763c51daa1bc"
}
backend/.sqlx/query-11813108dbf6b104eba968c3609c74ac5a589542d3e76b1e756a82fb19d49ee8.json
Generated
+58
View File
@@ -0,0 +1,58 @@
{
"db_name": "PostgreSQL",
"query": "\n SELECT language AS \"language!: _\", COUNT(*)::BIGINT AS \"count!\"\n FROM script\n WHERE archived = false AND deleted = false AND kind = 'script'\n AND (auto_kind IS NULL OR auto_kind <> 'wac')\n GROUP BY language\n ",
"describe": {
"columns": [
{
"ordinal": 0,
"name": "language!: _",
"type_info": {
"Custom": {
"name": "script_lang",
"kind": {
"Enum": [
"python3",
"deno",
"go",
"bash",
"postgresql",
"nativets",
"bun",
"mysql",
"bigquery",
"snowflake",
"graphql",
"powershell",
"mssql",
"php",
"bunnative",
"rust",
"ansible",
"csharp",
"oracledb",
"nu",
"java",
"duckdb",
"ruby",
"rlang"
]
}
}
}
},
{
"ordinal": 1,
"name": "count!",
"type_info": "Int8"
}
],
"parameters": {
"Left": []
},
"nullable": [
false,
null
]
},
"hash": "11813108dbf6b104eba968c3609c74ac5a589542d3e76b1e756a82fb19d49ee8"
}
backend/.sqlx/query-126cf6d54f8d2c916cd799f6663892119a94d66637062d1bf5a5fe97d89f8096.json
Generated
+20
View File
@@ -0,0 +1,20 @@
{
"db_name": "PostgreSQL",
"query": "SELECT COUNT(*) FROM workspace_diff\n WHERE source_workspace_id = 'wm-fork-test-workspace'\n OR fork_workspace_id = 'wm-fork-test-workspace'",
"describe": {
"columns": [
{
"ordinal": 0,
"name": "count",
"type_info": "Int8"
}
],
"parameters": {
"Left": []
},
"nullable": [
null
]
},
"hash": "126cf6d54f8d2c916cd799f6663892119a94d66637062d1bf5a5fe97d89f8096"
}
backend/.sqlx/query-16b4496c21d0619dab4521dca22e5fe144c59156a8f06d8592291684c49b2f37.json
Generated
-95
View File
@@ -1,95 +0,0 @@
{
"db_name": "PostgreSQL",
"query": "SELECT email as \"email!\", login_type::text, verified as \"verified!\", super_admin as \"super_admin!\", devops as \"devops!\", name, company, username, NULL::bool as operator_only, first_time_user as \"first_time_user!\", role_source as \"role_source!\", disabled as \"disabled!\", NULL::text as workspace_id FROM password\n UNION ALL\n SELECT email as \"email!\", 'service_account'::text as login_type, true as \"verified!\", false as \"super_admin!\", false as \"devops!\", NULL::text as name, NULL::text as company, username, true as operator_only, false as \"first_time_user!\", 'service_account'::text as \"role_source!\", disabled as \"disabled!\", workspace_id\n FROM usr\n WHERE is_service_account IS true\n ORDER BY \"super_admin!\" DESC, \"devops!\" DESC, \"email!\"\n LIMIT $1 OFFSET $2",
"describe": {
"columns": [
{
"ordinal": 0,
"name": "email!",
"type_info": "Varchar"
},
{
"ordinal": 1,
"name": "login_type",
"type_info": "Text"
},
{
"ordinal": 2,
"name": "verified!",
"type_info": "Bool"
},
{
"ordinal": 3,
"name": "super_admin!",
"type_info": "Bool"
},
{
"ordinal": 4,
"name": "devops!",
"type_info": "Bool"
},
{
"ordinal": 5,
"name": "name",
"type_info": "Varchar"
},
{
"ordinal": 6,
"name": "company",
"type_info": "Varchar"
},
{
"ordinal": 7,
"name": "username",
"type_info": "Varchar"
},
{
"ordinal": 8,
"name": "operator_only",
"type_info": "Bool"
},
{
"ordinal": 9,
"name": "first_time_user!",
"type_info": "Bool"
},
{
"ordinal": 10,
"name": "role_source!",
"type_info": "Varchar"
},
{
"ordinal": 11,
"name": "disabled!",
"type_info": "Bool"
},
{
"ordinal": 12,
"name": "workspace_id",
"type_info": "Text"
}
],
"parameters": {
"Left": [
"Int8",
"Int8"
]
},
"nullable": [
null,
null,
null,
null,
null,
null,
null,
null,
null,
null,
null,
null,
null
]
},
"hash": "16b4496c21d0619dab4521dca22e5fe144c59156a8f06d8592291684c49b2f37"
}
backend/.sqlx/query-1860c102a5309a8e1bb7288d0616f48e4330af7fbe61426e999c6073f630d88f.json
Generated
+41
View File
@@ -0,0 +1,41 @@
{
"db_name": "PostgreSQL",
"query": "SELECT DISTINCT ON (path)\n path,\n value as \"value!: sqlx::types::Json<Box<serde_json::value::RawValue>>\",\n created_at,\n typ::text as \"typ!\"\n FROM draft\n WHERE workspace_id = $1\n AND typ IN ('app', 'raw_app')\n AND (email = $2 OR email IS NULL)\n AND NOT EXISTS (\n SELECT 1 FROM app a\n WHERE a.workspace_id = draft.workspace_id\n AND a.path = draft.path\n )\n ORDER BY path, (email IS NULL), created_at DESC",
"describe": {
"columns": [
{
"ordinal": 0,
"name": "path",
"type_info": "Varchar"
},
{
"ordinal": 1,
"name": "value!: sqlx::types::Json<Box<serde_json::value::RawValue>>",
"type_info": "Json"
},
{
"ordinal": 2,
"name": "created_at",
"type_info": "Timestamptz"
},
{
"ordinal": 3,
"name": "typ!",
"type_info": "Text"
}
],
"parameters": {
"Left": [
"Text",
"Text"
]
},
"nullable": [
false,
false,
false,
null
]
},
"hash": "1860c102a5309a8e1bb7288d0616f48e4330af7fbe61426e999c6073f630d88f"
}
backend/.sqlx/query-19513c4158267cc7fe10d999ad571052c112e6bbb3cf834f16176cbb7e1ac319.json
Generated
+2 -2
View File
@@ -5,7 +5,7 @@
"columns": [
{
"ordinal": 0,
"name": "id",
"name": "id!",
"type_info": "Uuid"
}
],
@@ -16,7 +16,7 @@
]
},
"nullable": [
false
null
]
},
"hash": "19513c4158267cc7fe10d999ad571052c112e6bbb3cf834f16176cbb7e1ac319"
backend/.sqlx/query-19f1cb1c7a1974920549917a6392ff0d56f31ca5662062f4a71fed0ccf859cc4.json
Generated
+35
View File
@@ -0,0 +1,35 @@
{
"db_name": "PostgreSQL",
"query": "SELECT kind::text as \"kind!\", parent_job, runnable_path\n FROM v2_job WHERE id = $1 AND workspace_id = $2",
"describe": {
"columns": [
{
"ordinal": 0,
"name": "kind!",
"type_info": "Text"
},
{
"ordinal": 1,
"name": "parent_job",
"type_info": "Uuid"
},
{
"ordinal": 2,
"name": "runnable_path",
"type_info": "Varchar"
}
],
"parameters": {
"Left": [
"Uuid",
"Text"
]
},
"nullable": [
null,
true,
true
]
},
"hash": "19f1cb1c7a1974920549917a6392ff0d56f31ca5662062f4a71fed0ccf859cc4"
}
backend/.sqlx/query-1acfeed9c7a5b1e3d2da262d338655dba6e43067a9912cc2b775830856390c5d.json
Generated
+27
View File
@@ -0,0 +1,27 @@
{
"db_name": "PostgreSQL",
"query": "WITH del AS (\n DELETE FROM asset WHERE workspace_id = $1 AND usage_path = $2 AND usage_kind = $3\n RETURNING usage_access_type\n )\n INSERT INTO notify_event (channel, payload)\n SELECT 'notify_asset_producer_change', $1\n WHERE $3 = 'script'\n AND EXISTS (SELECT 1 FROM del WHERE usage_access_type IN ('w', 'rw'))",
"describe": {
"columns": [],
"parameters": {
"Left": [
"Text",
"Text",
{
"Custom": {
"name": "asset_usage_kind",
"kind": {
"Enum": [
"script",
"flow",
"job"
]
}
}
}
]
},
"nullable": []
},
"hash": "1acfeed9c7a5b1e3d2da262d338655dba6e43067a9912cc2b775830856390c5d"
}
backend/.sqlx/query-1f375b37ff9f6f01972e284e84a7b2f9d2d323a3da55f20ff6671e8eba510043.json
Generated
+15
View File
@@ -0,0 +1,15 @@
{
"db_name": "PostgreSQL",
"query": "WITH del AS (\n DELETE FROM asset WHERE workspace_id = $1 AND usage_kind = 'script'\n AND usage_path = (SELECT path FROM script WHERE hash = $2 AND workspace_id = $1)\n RETURNING usage_access_type\n )\n INSERT INTO notify_event (channel, payload)\n SELECT 'notify_asset_producer_change', $1\n WHERE EXISTS (SELECT 1 FROM del WHERE usage_access_type IN ('w', 'rw'))",
"describe": {
"columns": [],
"parameters": {
"Left": [
"Text",
"Int8"
]
},
"nullable": []
},
"hash": "1f375b37ff9f6f01972e284e84a7b2f9d2d323a3da55f20ff6671e8eba510043"
}
backend/.sqlx/query-1f486036b2902a0ffa0ce15f665d82c85129e2407819477adf7e87f097276ae2.json
Generated
+16
View File
@@ -0,0 +1,16 @@
{
"db_name": "PostgreSQL",
"query": "INSERT INTO draft (workspace_id, path, typ, value, created_at, email)\n SELECT $2, path, typ, value, created_at, email\n FROM draft\n WHERE workspace_id = $1 AND (email = $3 OR email IS NULL)",
"describe": {
"columns": [],
"parameters": {
"Left": [
"Text",
"Varchar",
"Text"
]
},
"nullable": []
},
"hash": "1f486036b2902a0ffa0ce15f665d82c85129e2407819477adf7e87f097276ae2"
}
backend/.sqlx/query-1fc04d31ae69dbb1df9c63cb69e83e8f8e6770b78f6ed052b99afed6cea28650.json
Generated
+17
View File
@@ -0,0 +1,17 @@
{
"db_name": "PostgreSQL",
"query": "INSERT INTO v2_job (id, workspace_id, kind, runnable_path, args, created_by,\n permissioned_as, permissioned_as_email, tag, script_lang)\n VALUES ($1, $2, 'script'::job_kind, $3, $4, 'test-user',\n 'u/test-user', 'test@windmill.dev', 'deno', 'bash'::script_lang)",
"describe": {
"columns": [],
"parameters": {
"Left": [
"Uuid",
"Varchar",
"Varchar",
"Jsonb"
]
},
"nullable": []
},
"hash": "1fc04d31ae69dbb1df9c63cb69e83e8f8e6770b78f6ed052b99afed6cea28650"
}
backend/.sqlx/query-231475dc825518aa88f562698f8061d2562c3ac4af8d088f9eea9f0bc11d5fe7.json
Generated
+44
View File
@@ -0,0 +1,44 @@
{
"db_name": "PostgreSQL",
"query": "SELECT version, columns AS \"columns: Json<Vec<SchemaColumn>>\"\n FROM materialized_asset_schema\n WHERE workspace_id = $1 AND asset_kind = $2 AND asset_path = $3\n ORDER BY version DESC\n LIMIT 1",
"describe": {
"columns": [
{
"ordinal": 0,
"name": "version",
"type_info": "Int8"
},
{
"ordinal": 1,
"name": "columns: Json<Vec<SchemaColumn>>",
"type_info": "Jsonb"
}
],
"parameters": {
"Left": [
"Text",
{
"Custom": {
"name": "asset_kind",
"kind": {
"Enum": [
"s3object",
"resource",
"variable",
"ducklake",
"datatable",
"volume"
]
}
}
},
"Text"
]
},
"nullable": [
false,
false
]
},
"hash": "231475dc825518aa88f562698f8061d2562c3ac4af8d088f9eea9f0bc11d5fe7"
}

Some files were not shown because too many files have changed in this diff Show More