chore: update ee-repo-ref to 970e8adb9bcf315be620fa619a83c62c86844dd4

This commit updates the EE repository reference after PR #544 was merged in windmill-ee-private.

Previous ee-repo-ref: ea10ff498681bd9100a5f4be2fa1e47986fd5b80

New ee-repo-ref: 970e8adb9bcf315be620fa619a83c62c86844dd4

Automated by sync-ee-ref workflow.

.agents/skills/adding-a-trigger/SKILL.md

@@ -1,267 +0,0 @@
----
-name: adding-a-trigger
-description: Checklist for adding a new TriggerCrud-based trigger type to Windmill (Azure, GCP, Kafka, etc.). Use when wiring a new trigger kind across backend, frontend, CLI, and capture infrastructure.
----
-
-# Skill: Adding a New Trigger Type
-
-Use this skill when adding a trigger kind that implements `TriggerCrud` (Kafka, GCP, Azure, MQTT, SQS, NATS, Postgres, Email…). For native triggers (Nextcloud, Google Drive — things wired through `windmill-native-triggers`), use the `native-trigger` skill instead.
-
-The goal of this doc is to enumerate every file that needs to change. Missing any one of them leads to silent regressions: sync drops the trigger, capture button does nothing, workspace forks lose it, sidebar counters undercount. Follow the checklist top-to-bottom — each section is independent enough to be validated on its own.
-
-Throughout this doc, substitute `{kind}` for the new trigger kind (`azure`, `kafka`, …), `{Kind}` for PascalCase (`Azure`, `Kafka`), `{KIND}` for SCREAMING (`AZURE`, `KAFKA`).
-
-## Reference implementations
-
-- **GCP** — closest analogue to Azure. Has push + pull, OIDC auth, ARM-like resource paths, capture handler. Grep for `gcp_trigger` / `GcpTrigger`.
-- **Kafka** — simpler (pull-only, streaming). Good for trivial integrations.
-- **Azure** — most recently added (2026). Shared-secret push auth, Event Grid namespaces + basic topics, ARM resource discovery, Namespace-pull data-plane. Grep for `azure_trigger` / `AzureTrigger`.
-
-## 1. Database migration
-
-Create a migration: `cargo sqlx migrate add -r add_{kind}_trigger` from `backend/`. Never write timestamps manually.
-
-The `up.sql` usually defines:
-- An optional enum type (e.g. `AZURE_MODE`) if the trigger has sub-kinds
-- The `{kind}_trigger` table with at minimum these columns (mirrored from kafka/gcp):
-  - primary: `(workspace_id, path)`
-  - `script_path`, `is_flow`, `enabled`, `mode`, `permissioned_as`, `edited_by`, `email`
-  - `edited_at`, `error`, `server_id`, `last_server_ping`
-  - `error_handler_path`, `error_handler_args jsonb`, `retry jsonb`
-  - trigger-specific fields
-- Indexes on foreign keys + any frequently-filtered columns
-- Foreign key to `workspace`
-
-Down migration drops the table and any enum types.
-
-## 2. Backend crate (`windmill-trigger-{kind}`)
-
-Create a new crate under `backend/windmill-trigger-{kind}/` with:
-
-- `Cargo.toml`: features `enterprise`, `private` if EE, standard deps
-- `src/lib.rs`: `pub use mod_ee::*;` behind `#[cfg(all(feature = "enterprise", feature = "private"))]`
-- `src/mod_ee.rs`: core types + helpers
-- `src/handler_ee.rs`: `TriggerCrud` impl + route handlers
-- `src/listener_ee.rs`: (only if streaming/pull-based) `Listener` trait impl
-
-Required in `mod_ee.rs`:
-- `{Kind}Config` struct (persisted shape, `FromRow`)
-- `{Kind}ConfigRequest` struct (what API receives — usually similar to Config but with validation fields)
-- `{Kind}Trigger` unit struct (implements the traits)
-- `impl TriggerJobArgs for {Kind}Trigger` — sets `TRIGGER_KIND`, `Payload`, `v1_payload_fn`
-
-Required in `handler_ee.rs`:
-- `#[async_trait] impl TriggerCrud for {Kind}Trigger` with:
-  - `type Trigger = Trigger<{Kind}Config>`
-  - `type TriggerConfigRequest = {Kind}ConfigRequest`
-  - `const ROUTE_PREFIX: &'static str = "/{kind}_triggers";`
-  - `const TABLE_NAME`, `ADDITIONAL_SELECT_FIELDS`
-  - `get_deployed_object`, `validate_config`, `create_trigger`, `update_trigger`, `delete_trigger`, `test_connection`
-  - `additional_routes` (optional — mount extra endpoints for things like ARM resource listing, topic discovery)
-
-Register the crate in `backend/Cargo.toml` as a workspace member and as a dep of `windmill-api` behind the feature flag.
-
-## 3. Wire into `windmill-api` (feature-gated everywhere)
-
-**`backend/windmill-api/src/triggers/handler.rs`** — mount the trigger crate:
-```rust
-#[cfg(all(feature = "enterprise", feature = "{kind}_trigger", feature = "private"))]
-{
-    use crate::triggers::{kind}::{Kind}Trigger;
-    router = router.nest({Kind}Trigger::ROUTE_PREFIX, complete_trigger_routes({Kind}Trigger));
-}
-```
-
-**`backend/windmill-api/src/triggers/{kind}/mod.rs`** — re-export the crate:
-```rust
-pub use windmill_trigger_{kind}::*;
-```
-
-**`backend/windmill-api/src/lib.rs`** — if the trigger receives inbound pushes, add a webhook route:
-```rust
-.nest("/{kind}/w/{workspace_id}", {
-    #[cfg(all(feature = "enterprise", feature = "{kind}_trigger", feature = "private"))]
-    { triggers::{kind}::handler_oss::{kind}_push_route_handler() }
-    #[cfg(not(...))]
-    { Router::new() }
-})
-```
-
-## 4. `TriggerKind` enum (`backend/windmill-types/src/triggers.rs`)
-
-Already has slots for most triggers but verify your variant exists:
-- Add `{Kind}` to the `TriggerKind` enum
-- Add match arm in `to_key()`
-- Add match arm in `from_str`
-- Add match arm in `JobTriggerKind` (if jobs need kind tagging)
-
-## 5. OpenAPI (`backend/windmill-api/openapi.yaml`)
-
-This file is huge and the single most-forgotten place. Add:
-
-- `/w/{workspace}/{kind}_triggers/create` + `/update/{path}` + `/delete/{path}` + `/get/{path}` + `/list` + `/exists/{path}` + `/setmode/{path}` + `/test` paths (mirror gcp section)
-- Any `additional_routes` your handler exposes (resource discovery, etc.)
-- Schemas: `{Kind}Trigger`, `{Kind}TriggerData`, `{Kind}Mode` (if enum), `{Kind}DeliveryConfig`, helper request/response types
-- Add `{kind}` to `CaptureTriggerKind` enum
-- Add `{kind}_used: boolean` to the `UsedTriggers` response schema
-
-Regenerate frontend client: `npm run generate-backend-client` from `frontend/`.
-
-## 6. `UsedTriggers` + workspace export
-
-**`backend/windmill-api-workspaces/src/workspaces.rs`** — add `{kind}_used: bool` to the `UsedTriggers` struct and add an `EXISTS(SELECT 1 FROM {kind}_trigger …)` to the `get_used_triggers` query.
-
-**`backend/windmill-api/src/workspaces_export.rs`** — add export block mirroring gcp's (export lists all triggers, serializes them to YAML/JSON). The block re-uses the `trigger_ignore_keys` variable so the new kind automatically participates in fork-export stripping (`mode` field is omitted when the source workspace is a fork — keeps fork→parent merges from flipping the parent's enabled state).
-
-**Fork cloning (`clone_triggers_and_schedules` in workspaces.rs)** — add an `INSERT INTO {kind}_trigger ... SELECT ...` block that copies all rows from the parent workspace, forcing `mode = 'disabled'::TRIGGER_MODE`. Always runs at fork creation; forgetting this means users can't carry `{kind}` triggers into their forks.
-
-## 6.5 Hardcoded trigger-kind arrays (silent-failure hotspots)
-
-Several files keep **hardcoded arrays** of trigger kind strings. Miss one and ACL checks / user offboarding / trash drop your kind:
-
-- **`backend/windmill-api-groups/src/granular_acls.rs`** — `KINDS: [&str; N]`. **Increment N** (the compile error is cryptic otherwise). Controls which kinds accept granular ACL operations.
-- **`backend/windmill-api-users/src/users.rs`** (`extra_perms_tables`) — which tables get `extra_perms` entries cleaned when a user is deleted.
-- **`backend/windmill-api/src/offboarding.rs`** — three separate arrays (enumeration, fork-copy, and delete paths). **All three** need the new kind.
-- **`backend/windmill-api/src/trash.rs`** — `valid_tables` for the trash / restore API.
-- **`backend/windmill-git-sync/src/lib.rs`** — add a test assertion for `DeployedObject::{Kind}Trigger.get_kind() == "{kind}_trigger"` (the `get_kind` match arm itself lives in the enum impl — already required by the Rust compiler).
-- **`backend/windmill-api-auth/src/scopes.rs`** — add the `{Kind}Triggers` variant to `ScopeDomain` enum + `as_str` match + `from_str` match. Required for the OAuth/token system to recognise `{kind}_triggers:read|write` scopes.
-- **`backend/windmill-api/src/token.rs`** (`build_trigger_scope_domains` → `TRIGGER_DOMAINS`) — add `("{kind}_triggers", "{Kind display name}")` so the CreateToken UI's scope selector surfaces the `read` / `write` checkboxes.
-
-**OpenAPI enums** to extend (do NOT forget — generated client will allow it but server rejects as 400):
-- `CaptureTriggerKind` enum
-- Three `kind` enums under `/w/{workspace}/acls/{get,add,remove}/{kind}/{path}` (yes, same list repeated three times)
-
-After editing any of these, run a full `cargo check` with your feature flag + `gcp_trigger` + other core flags — the `KINDS: [&str; N]` length mismatch only surfaces when the crate compiles.
-
-## 7. Capture infrastructure (`backend/windmill-api/src/capture.rs`)
-
-If the trigger supports push delivery, it also needs a capture endpoint so users can test it:
-
-- `{Kind}TriggerConfig` struct (gated by feature flags)
-- `TriggerConfig::{Kind}` variant
-- `set_{kind}_trigger_config` function (creates the subscription/equivalent pointing at the capture URL — use your `manage_{kind}_subscription` helper with `trigger_mode=false`)
-- Both real + no-op versions behind feature gates
-- `TriggerKind::{Kind} => set_{kind}_trigger_config(...)` arm in `set_config`
-- `{kind}_payload` async handler — validates auth (if any), processes payload, calls `insert_capture_payload`
-- Route: `.route("/{kind}/{runnable_kind}/{*path}", post({kind}_payload))` inside `workspaced_unauthed_service` — and expand the surrounding `#[cfg(any(...))]` to include your feature flag
-
-## 8. CLI (`cli/`) — easy to miss, breaks sync silently
-
-Check all of these:
-
-**`cli/src/types.ts`:**
-- Add `"{kind}"` to `TRIGGER_TYPES` array
-- Add `"{kind}_trigger"` to `getTypeStrFromPath` return union
-- Add match case in `getTypeStrFromPath`'s `typeEnding ===` chain
-- Add `pushTrigger("{kind}", ...)` branch in `pushObj`
-
-**`cli/src/commands/trigger/trigger.ts`:**
-- Import `{Kind}Trigger` type
-- Add `{kind}: {Kind}Trigger` to the `Trigger` type map
-- Add `{kind}: wmill.get{Kind}Trigger`, `update{Kind}Trigger`, `create{Kind}Trigger` to each function map
-- Add `{kind}: { ... }` template to `triggerTemplates`
-- Add `list{Kind}Triggers` call + spread in the `list` aggregation
-- Update `--kind` option descriptions to mention the new kind
-
-**`cli/src/commands/sync/sync.ts`:**
-- Add `path.endsWith(".{kind}_trigger" + ext)` in the file-type filter
-- Add `typ == "{kind}_trigger"` in `getTypeOrder`
-- Add `"{kind}_trigger"` to the delete-suffix regex (~line 3092)
-- Add a `case "{kind}_trigger"` in the delete switch
-
-**`cli/src/guidance/skills.ts`** — **DO NOT EDIT DIRECTLY**. It's auto-generated by `system_prompts/generate.py`. Instead:
-- Edit `system_prompts/utils.py` → append `('{Kind}Trigger', '{kind}_trigger')` to the `SCHEMA_MAPPINGS['triggers']` list (this is the master list — the one in `generate.py` is duplicated and `utils.py` wins)
-- Then run `python3 system_prompts/generate.py` — it regenerates `cli/src/guidance/skills.ts` with the schema extracted from `backend/windmill-api/openapi.yaml`
-- Commit the regenerated file
-
-## 9. Frontend — editor + drawer
-
-Under `frontend/src/lib/components/triggers/{kind}/`:
-
-- `{Kind}TriggerPanel.svelte` — the tile shown in the triggers listing
-- `{Kind}TriggerEditor.svelte` — outer drawer wrapper
-- `{Kind}TriggerEditorInner.svelte` — state + business logic; must expose:
-  - `openEdit(path, isFlow, defaultValues?)` method
-  - `isEditor` prop, `onConfigChange` + `onCaptureConfigChange` callbacks
-  - `get{Kind}Config()` + `get{Kind}CaptureConfig()` helpers
-  - `captureConfig = $derived.by(untrack(() => isEditor) ? get{Kind}CaptureConfig : () => ({}))`
-  - `$effect(() => { const args = [captureConfig, isValid] as const; untrack(() => onCaptureConfigChange?.(...args)) })`
-- `{Kind}TriggerEditorConfigSection.svelte` — form fields; use design-system components (`TextInput`, `Select`, `Toggle`, `ToggleButtonGroup`), never raw `<input>`
-- `{Kind}Capture.svelte` — capture panel; wraps `CaptureSection` with `captureType="{kind}"`
-- `utils.ts` — `requestBody` builders and any trigger-type-specific helpers
-
-## 10. Frontend — global integration
-
-Easy to miss:
-
-- **`frontend/src/lib/components/triggers.ts`** — add `'{kind}'` to the `TriggerKind` union
-- **`frontend/src/lib/components/triggers/CaptureWrapper.svelte`**:
-  - Import `{Kind}Capture`
-  - Add to `isStreamingCapture()` array (streaming = pull-style; push-style is typically `false`)
-  - Add `{:else if captureType === '{kind}'}` branch with the `<{Kind}Capture>` render
-- **`frontend/src/lib/components/sidebar/SidebarContent.svelte`** — import the icon, add the nav entry
-- **`frontend/src/lib/components/sidebar/OperatorMenu.svelte`** — add the operator-mode entry
-- **`frontend/src/routes/(root)/(logged)/+layout.svelte`** — destructure `{kind}_used` from `/get_used_triggers` response, push `'{kind}'` into `usedKinds`
-- **`frontend/src/lib/components/search/GlobalSearchModal.svelte`** — import icon, add "Go to {Kind} ..." entry
-- **`frontend/src/lib/components/offboarding-utils.ts`** — add mappings `{kind}_trigger: '{kind}_triggers'` and `{kind}_trigger: '{kind} trigger'`
-- **`frontend/src/lib/components/icons/{Kind}Icon.svelte`** — single-path SVG, `fill={color ?? 'currentColor'}`, `size` prop default 16 (match existing icons — don't hardcode colors, don't use `width`/`height` props)
-- **`frontend/src/routes/(root)/(logged)/{kind}_triggers/+page.svelte`** — listing page (mirror `gcp_triggers/+page.svelte` for push+pull, `kafka_triggers` for pure streaming)
-- **`frontend/src/lib/components/CompareWorkspaces.svelte`** — workspace fork / compare tool. Needs: service import, editor import, `{kind}Editor` `$state`, `case '{kind}'` in `openTriggerDetails()`, entry in `triggerServices` object (list/delete/normalize), and `<{Kind}TriggerEditor bind:this={{kind}Editor} />` in the template
-
-## 10.5 AI system prompts (`system_prompts/`)
-
-- **`system_prompts/utils.py`** — append `('{Kind}Trigger', '{kind}_trigger')` to `SCHEMA_MAPPINGS['triggers']` (master list used by code generation + CLI skills)
-- **`system_prompts/generate.py`** — also has a duplicated `schema_types` list (~line 903) for the AI `triggers` skill content. Add `('{Kind}Trigger', '{kind}_trigger')` there too
-- **`system_prompts/generate.py`** `schema_names` (~line 1192) — add `'{Kind}Trigger'` (add `'New{Kind}Trigger'` only if the OpenAPI declares one; GCP and Azure don't)
-- Run `python3 system_prompts/generate.py` — this rewrites `cli/src/guidance/skills.ts` and all `auto-generated/` docs. Commit the regenerated files
-
-## 11. Validation
-
-Run all of these before declaring done:
-
-```bash
-# Backend
-cd backend
-cargo check --features enterprise,{kind}_trigger,private   # minimal
-cargo check --features enterprise,azure_trigger,private,gcp_trigger,http_trigger,mqtt_trigger,postgres_trigger,sqs_trigger,kafka,nats,smtp,websocket  # full
-
-# SQLx offline data (never run `cargo sqlx prepare` directly — use the wrapper)
-./update_sqlx.sh
-
-# Frontend
-cd frontend
-npm run generate-backend-client
-npm run check:fast
-```
-
-Smoke test in the UI: create a trigger, save, check it appears in sidebar + search, delete, re-create via CLI `wmill sync`.
-
-## 12. Common pitfalls
-
-- **Forgetting feature gates in `workspaced_unauthed_service()`** — the surrounding `#[cfg(any(...))]` expression must include your feature flag, not just the inner `#[cfg]` on the route
-- **`.route(path, ...).route(path, ...)` with same path and different methods** — older axum replaced; use `.route(path, post(h1).options(h2))` to chain methods on the same `MethodRouter`
-- **`on:event` directives** — legacy Svelte 4, no-op in runes mode. Use callback props (`onSelected`, `onConfigChange`)
-- **`$bindable(default_value)` on optional props** — banned by project CLAUDE.md. Use `$bindable()` + `$derived(prop ?? default)` instead
-- **CORS layer intercepting OPTIONS** — tower-http CorsLayer short-circuits OPTIONS before reaching your handler. For server-to-server webhook endpoints, drop the CORS layer entirely (CORS is browser-only)
-- **DeliveryAttributeMappings / custom headers for auth** — prefer HMAC or sha256-hashed shared secrets over opaque JWTs when the provider doesn't support signed tokens natively. Store only the hash; regenerate secret on every save
-- **ARM / API resource-listing cascades** — if the trigger's resource type is deep (Azure: subscription → RG → namespace → topic), offer dropdowns in the UI populated from the provider's APIs using the user's credential resource
-- **Clearing stale selections on dependency change** — when a dropdown's underlying data reloads (e.g., user changes SP or edition), clear selections that no longer match the new list
-- **Workspace-scoped tag compatibility** — if the trigger has tags, verify forked workspaces handle them (see commit `0773b5bc85` for a historical fix)
-
-## 13. EE file split
-
-If the trigger is enterprise-only, the code lives in `windmill-ee-private__worktrees/.../windmill-trigger-{kind}/src/*_ee.rs` and is symlinked into the OSS tree. The `windmill-ee-private__worktrees/` directory holds the real files; changes propagate via symlinks. See `docs/enterprise.md` for the workflow.
-
-## 14. Final checklist before PR
-
-- [ ] Migration up/down tested (revert + re-apply)
-- [ ] `./update_sqlx.sh` committed the updated `.sqlx/` offline data
-- [ ] `cargo check` passes with your feature flag + with all trigger features
-- [ ] `npm run check:fast` passes
-- [ ] Trigger visible in sidebar with correct icon weight (not oversized/colored — use `currentColor`)
-- [ ] Create, edit, delete flow all work in the UI
-- [ ] Capture button works (if push-capable)
-- [ ] Trigger appears in `/get_used_triggers` → sidebar pulse
-- [ ] `wmill sync pull` + `wmill sync push` both round-trip the trigger
-- [ ] `wmill trigger list` includes it
-- [ ] OpenAPI schemas are complete (no `null` in generated types)

.agents/skills/commit/SKILL.md

@@ -1,6 +1,5 @@
 ---
 name: commit
-user_invocable: true
 description: Create a git commit with conventional commit format. MUST use anytime you want to commit changes.
 ---
 
@@ -53,6 +52,8 @@ chore: upgrade sqlx to 0.7
 4. Stage ONLY the modified/relevant files: `git add <file1> <file2> ...`
 5. Create the commit with conventional format:
    ```bash
-   git commit -m "<type>: <description>"
+   git commit -m "<type>: <description>
+
+   Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>"
    ```
 6. Run `git status` to verify the commit succeeded

.agents/skills/local-review/SKILL.md

@@ -1,98 +1,97 @@
 ---
 name: local-review
-description: Code review the current PR (or branch diff against main) for bugs, security, and AGENTS.md compliance. MUST use when asked to review code.
+description: Code review a pull request for bugs and CLAUDE.md compliance. MUST use when asked to review code.
 ---
 
-# Local Code Review
+# Local Code Review Skill
 
-Run the same review locally that the GitHub auto-review actions run on PRs (Claude / Codex / Pi). The review policy lives in `REVIEW.md`.
+Review a pull request for real bugs and CLAUDE.md compliance violations. This review targets HIGH SIGNAL issues only.
 
-**Why a subagent**: the review MUST run in a fresh context — not inline in the current session. If the user has been iterating on the diff, the main session has absorbed their reasoning and rationalizations, so it anchors and misses things CI catches. A subagent starts cold, like CI does.
+## Review Philosophy
 
-## Steps
+- **Only flag issues you are certain about.** If you are not sure an issue is real, do not flag it. False positives erode trust and waste reviewer time.
+- Think like a senior engineer doing a final review — flag things that would cause incidents, not things that are merely imperfect.
 
-1. **Determine the PR scope** (cheap, do this in the main session):
-   - If an argument is provided, treat it as a PR number or branch.
-   - Otherwise, detect from the current branch vs `main`.
-   - Confirm the PR/branch exists (`gh pr view <n>` or `git rev-parse <branch>`).
+## What to Flag
 
-2. **Delegate the review to a fresh-context subagent** with a self-contained prompt. The prompt MUST include:
-   - The PR number or branch name to review.
-   - The instruction to read `REVIEW.md` first for the policy, then `AGENTS.md` files in directories touched by the diff.
-   - The exact output format (see below).
-   - Whether `--comment` was requested (so the subagent emits inline-comment payloads if needed).
-   - Any "Additional reviewer instructions" the user provided.
+- Code that won't compile or parse (syntax errors, type errors, missing imports)
+- Code that will definitely produce wrong results regardless of inputs
+- Clear, unambiguous CLAUDE.md violations (quote the exact rule being violated)
+- Security issues in introduced code (injection, auth bypass, data exposure)
+- Incorrect logic that will fail in production
 
-   - **Claude Code**: use the `Agent` tool with `subagent_type: branch-diff-reviewer` (read-only tools, purpose-built for this). If unavailable, fall back to `general-purpose`.
-   - **Codex / Pi**: if the CLI exposes a fresh-session subagent mechanism, use it. Otherwise tell the user to run the skill in a fresh CLI session and stop — running inline in the current session defeats the purpose.
+## What NOT to Flag
 
-3. **Receive the findings** from the subagent and relay them to the user verbatim. Do not re-summarize, re-judge, or filter — the whole point of fresh context is to surface what the main session would dismiss.
+- Code style or quality concerns
+- Potential issues that depend on specific inputs or runtime state
+- Subjective suggestions or improvements
+- Pre-existing issues not introduced by this PR
+- Pedantic nitpicks a senior engineer wouldn't flag
+- Issues a linter or type checker will catch
+- General quality concerns unless explicitly prohibited in CLAUDE.md
+- Issues silenced via lint ignore comments
 
-4. **Post comments if `--comment` was requested**: use the `gh` commands below with the subagent's output as the body. The main session does the posting because the subagent is read-only.
+## Execution Steps
 
-## Subagent prompt template
+1. **Determine the PR scope**:
+   - If an argument is provided, use it as the PR number or branch
+   - Otherwise, detect from the current branch vs main
+   - Run `gh pr view` if a PR exists, or use `git diff main...HEAD`
 
-```
-Review <PR #N | branch X> against main per the policy in REVIEW.md.
+2. **Find relevant CLAUDE.md files**:
+   - Read the root `CLAUDE.md`
+   - Check for CLAUDE.md files in directories containing changed files
 
-Steps:
-1. Read REVIEW.md (repo root) for the full policy: severity triage, public-surface
-   checklist, AGENTS.md compliance, test coverage assessment.
-2. Read AGENTS.md (repo root) and any AGENTS.md in directories touched by the diff.
-3. Get the diff: `gh pr diff <N>` (if PR) or `git diff main...<branch>`.
-4. Get context: `gh pr view <N>` (if PR) or `git log main..<branch> --oneline`.
-5. Read changed files only when the diff alone is insufficient to validate a finding.
-6. Self-validate each finding: "is this definitely a real issue a senior engineer
-   would flag?" Discard if uncertain.
-7. Output findings in the exact format below. Do not modify any files.
+3. **Get the diff and metadata**:
+   - `gh pr diff` or `git diff main...HEAD` for the full diff
+   - `gh pr view` or `git log main..HEAD --oneline` for context
 
-<paste output format from below>
+4. **Read changed files** where the diff alone is insufficient to understand context
 
-<if --comment requested:>
-Additionally emit a JSON array of inline comments suitable for the GitHub reviews
-API, one per finding that maps to a specific line:
-[{"path": "...", "line": N, "side": "RIGHT", "body": "[P1] ..."}, ...]
-```
+5. **Review for**:
+   - CLAUDE.md compliance — check each rule against the changed code
+   - Bugs and logic errors — will this code work correctly?
+   - Security issues — injection, auth, data exposure in new code
 
-## Output format
+6. **Self-validate each finding**: Before reporting, ask yourself:
+   - "Is this definitely a real issue, not a false positive?"
+   - "Would a senior engineer flag this in review?"
+   - If the answer to either is no, discard the finding
+
+7. **Output findings** to the terminal (default) or post as PR comments (with `--comment` flag)
+
+## Output Format
 
 ```
 ## Code review
 
-<verdict line per REVIEW.md>
-
 Found N issues:
 
-1. [P0|P1|P2] <description>
+1. <description> (<reason: CLAUDE.md adherence | bug | security>)
    <file_path:line_number>
 
-2. [P0|P1|P2] <description>
+2. <description> (<reason>)
    <file_path:line_number>
 ```
 
-End with a `Test coverage` section per the shared policy.
-
 If no issues are found:
 
 ```
 ## Code review
 
-Good to merge.
-
-No issues found. Checked for bugs, security, and AGENTS.md compliance.
+No issues found. Checked for bugs and CLAUDE.md compliance.
 ```
 
-## Posting comments (`--comment`)
+## Posting Comments (--comment flag)
 
-For a top-level PR comment:
+If the user passes `--comment`, post findings as inline PR comments using:
 
 ```bash
-gh pr review --comment --body "<summary from subagent>"
+gh pr review --comment --body "<summary>"
 ```
 
-For inline comments on specific lines (using the JSON the subagent emitted):
+Or for inline comments on specific lines:
 
 ```bash
-gh api repos/{owner}/{repo}/pulls/{pr}/reviews \
-  -f body="<summary>" -f event="COMMENT" -f comments="<json from subagent>"
+gh api repos/{owner}/{repo}/pulls/{pr}/reviews -f body="<summary>" -f event="COMMENT" -f comments="[...]"
 ```

.agents/skills/native-trigger/SKILL.md

@@ -607,18 +607,7 @@ In `frontend/src/lib/components/triggers/TriggersEditor.svelte`:
 
 Add your service to the `nativeTriggerServices` map in `deleteDeployedTrigger()`. Native triggers use `NativeTriggerService.deleteNativeTrigger({ workspace, serviceName, externalId })` instead of the standard `path`-based delete.
 
-### Step 17: Update `getUsedTriggers` for Sidebar Visibility
-
-The sidebar (`frontend/src/lib/components/sidebar/SidebarContent.svelte`) shows native-trigger links only if `$usedTriggerKinds` includes the service — without this, your trigger page will never appear in the nav bar even when triggers exist.
-
-1. **Backend** — add `{service}_used: bool` to the `UsedTriggers` struct and SELECT in `backend/windmill-api-workspaces/src/workspaces.rs::get_used_triggers()`:
-   ```rust
-   EXISTS(SELECT 1 FROM native_trigger WHERE workspace_id = $1 AND service_name = '{service}'::native_trigger_service) AS "{service}_used!"
-   ```
-2. **OpenAPI** — add `{service}_used: boolean` to the response schema for `GET /w/{workspace}/workspaces/used_triggers` (under both `properties` and `required`).
-3. **Layout** — in `frontend/src/routes/(root)/(logged)/+layout.svelte::loadUsedTriggerKinds()`, destructure `{service}_used` and push `'{service}'` to `usedKinds`.
-
-### Step 18: Update OpenAPI Spec and Regenerate Types
+### Step 17: Update OpenAPI Spec and Regenerate Types
 
 Add to `JobTriggerKind` enum in `backend/windmill-api/openapi.yaml`, then:
 

.agents/skills/pr/SKILL.md

@@ -1,6 +1,5 @@
 ---
 name: pr
-user_invocable: true
 description: Open a draft pull request on GitHub. MUST use when you want to create/open a PR.
 ---
 
@@ -51,59 +50,22 @@ The body MUST be explicit about what changed. Structure:
 ## Test plan
 - [ ] <How to verify change 1>
 - [ ] <How to verify change 2>
+
+---
+Generated with [Claude Code](https://claude.com/claude-code)
 ```
 
-The harness/tooling that invoked the skill may add its own attribution trailer; the skill itself does not prescribe one.
-
-## Screenshots (required for frontend changes)
-
-If `git diff main...HEAD --name-only` matches `^frontend/`, the PR body **must** include
-screenshots of the affected UI. Skip only when there is no visible UI effect (types,
-tests, build config) — and say so in the body.
-
-1. Verify the change in the browser (AGENTS.md → "Verifying Frontend Changes").
-2. Screenshot each affected page with `mcp__playwright__browser_take_screenshot` (save to a file).
-3. Host each image and get its Markdown embed by pushing to the public
-   `windmill-labs/agent-screenshots-internal` repo. **Pipe base64 through stdin** —
-   passing it as `-f content=…` fails with `argument list too long` on real images:
-
-   ```bash
-   REPO=windmill-labs/agent-screenshots-internal
-   IMG=screenshot.png                                          # repeat per page
-   DEST="shots/$(git branch --show-current)/$(date +%s)-$(basename "$IMG")"
-   base64 -w0 "$IMG" | jq -Rs --arg m "add $DEST" '{message:$m, content:.}' \
-     | gh api -X PUT "repos/$REPO/contents/$DEST" --input - >/dev/null
-   echo "![$(basename "$IMG" .png)](https://raw.githubusercontent.com/$REPO/main/$DEST)"
-   ```
-   Derive `$DEST` from the file name (as above) so distinct pages never collide — a
-   fixed name would make same-second uploads reuse one path, and the second `PUT`
-   then 422s (the Contents API needs the existing file's `sha` to overwrite).
-4. Put the printed `![…](…)` lines under a `## Screenshots` heading in the PR body.
-
-Requires `gh` (`repo` scope), `jq`, `base64` — all in the devShell. The host repo is
-public (so the raw URLs render for reviewers without a token) and its history is
-permanent — **never screenshot pages that show secrets or sensitive values** (workspace
-variables, resource values, instance settings, OAuth/SMTP config); deleting the file
-can't undo an accidental capture. (GitHub's drag-and-drop uploader needs a browser
-session and can't be driven from a token.)
-
-If `gh` can't push to the host repo (e.g. a CI token scoped only to `windmill`), do
-**not** fail the PR or skip silently — hand the upload to the user, who has push access,
-and continue once they confirm it's done.
-
 ## Execution Steps
 
 1. Run `git status` to check for uncommitted changes
 2. Run `git log main..HEAD --oneline` to see all commits in this branch
 3. Run `git diff main...HEAD` to see the full diff against main
-4. **Invoke the `local-review` skill** before creating the PR (`/local-review` in Claude Code, `$local-review` in Codex, `pi --skill local-review` / `/skill:local-review` in Pi). If issues are found, fix them and commit before proceeding. Do not skip this step.
-5. **Screenshots for frontend changes**: if `git diff main...HEAD --name-only` matches `^frontend/`, capture and embed screenshots of the affected UI per "Screenshots" above before writing the PR body (skip only if there is no visible UI effect).
-6. Check if remote branch exists and is up to date:
+4. Check if remote branch exists and is up to date:
    ```bash
    git rev-parse --abbrev-ref --symbolic-full-name @{u} 2>/dev/null || echo "no upstream"
    ```
-7. Push to remote if needed: `git push -u origin HEAD`
-8. Create draft PR using gh CLI:
+5. Push to remote if needed: `git push -u origin HEAD`
+6. Create draft PR using gh CLI:
    ```bash
    gh pr create --draft --title "<type>: <description>" --body "$(cat <<'EOF'
    ## Summary
@@ -116,10 +78,13 @@ and continue once they confirm it's done.
    ## Test plan
    - [ ] <test 1>
    - [ ] <test 2>
+
+   ---
+   Generated with [Claude Code](https://claude.com/claude-code)
    EOF
    )"
    ```
-9. Return the PR URL to the user
+7. Return the PR URL to the user
 
 ## EE Companion PR (when `*_ee.rs` files were modified)
 
@@ -135,6 +100,9 @@ Follow the full EE PR workflow in `docs/enterprise.md`. The key PR-specific deta
    ```bash
    gh pr create --draft --repo windmill-labs/windmill-ee-private --title "<type>: <description>" --body "$(cat <<'EOF'
    Companion PR for windmill-labs/windmill#<PR_NUMBER>
+
+   ---
+   Generated with [Claude Code](https://claude.com/claude-code)
    EOF
    )"
    ```

.agents/skills/refine/SKILL.md

@@ -1,6 +1,5 @@
 ---
 name: refine
-user_invocable: true
 description: End-of-session reflection. Reviews friction encountered during the session and proposes updates to docs/ to capture lessons learned.
 ---
 

.agents/skills/svelte-frontend/SKILL.md

@@ -78,7 +78,3 @@ Use the Svelte MCP tools when working on Svelte code:
 2. **get-documentation**: Fetch relevant sections based on use_cases
 3. **svelte-autofixer**: MUST use on all Svelte code before finalizing — keep calling until no issues
 4. **playground-link**: Only after user confirms and code was NOT written to project files
-
-## Verifying in the Browser
-
-After changing Svelte code, use the **Playwright MCP** (`mcp__playwright__*`) to drive the running frontend and confirm the change works. See AGENTS.md → "Verifying Frontend Changes" for the full flow. Use `playwright` (headless) on devboxes; `playwright-headed` when a display is available.

.agents/skills/update-sqlx/SKILL.md

@@ -1,82 +0,0 @@
----
-name: update-sqlx
-description: How to safely update SQLx offline query cache. MUST use when SQL queries change.
----
-
-# SQLx Offline Query Cache
-
-Windmill uses `SQLX_OFFLINE=true` in CI, which requires all `sqlx::query!` / `sqlx::query_as!` macros to have matching cached query data in `backend/.sqlx/`.
-
-## When to Run
-
-Run after any change to SQL queries in Rust source files. Without it, CI will fail with:
-```
-error: `SQLX_OFFLINE=true` but there is no cached data for this query
-```
-
-## The Problem
-
-`cargo sqlx prepare --workspace` **deletes all existing cache files** and regenerates only the ones found in the current compilation. If you don't compile with every feature flag (especially `private` for EE files), you will **silently delete EE query caches**, breaking CI for enterprise tests.
-
-The standard `./update_sqlx.sh` script tries to compile with all features, but it often fails locally because the EE symlinks can be out of sync with `main`.
-
-## Safe Procedure
-
-Always preserve the existing EE caches from `origin/main`. Use this workflow:
-
-```bash
-cd backend
-
-# 1. Restore the full cache from main (includes EE caches)
-git checkout origin/main -- .sqlx/
-
-# 2. Run prepare with OSS features (what compiles locally)
-#    This regenerates OSS caches to match your code changes.
-cargo sqlx prepare --workspace -- --workspace --features all_sqlx_features
-
-# 3. Restore any EE caches that were deleted in step 2.
-#    These are files present in origin/main but missing after prepare.
-git ls-tree origin/main backend/.sqlx/ \
-  | awk '{print $4}' | sed 's|backend/\.sqlx/||' | sort > /tmp/main_files.txt
-
-find backend/.sqlx -name "*.json" -printf '%P\n' | sort > /tmp/current_files.txt
-
-comm -23 /tmp/main_files.txt /tmp/current_files.txt > /tmp/missing_files.txt
-
-while read f; do
-  git show "origin/main:backend/.sqlx/$f" > "backend/.sqlx/$f"
-done < /tmp/missing_files.txt
-
-# 4. Verify nothing was lost from main
-find backend/.sqlx -name "*.json" -printf '%P\n' | sort > /tmp/current_files.txt
-comm -23 /tmp/main_files.txt /tmp/current_files.txt | wc -l
-# Should output: 0
-```
-
-## If EE Compiles Locally
-
-If your EE repo happens to be in sync, you can use the full script (faster):
-
-```bash
-cd backend
-./update_sqlx.sh
-```
-
-But if it fails with EE compilation errors, use the safe procedure above.
-
-## What NOT to Do
-
-- **Never** run `cargo sqlx prepare --workspace` with only OSS features and commit the result — it will delete EE caches.
-- **Never** set `SQLX_OFFLINE=true` for local `cargo sqlx prepare` — use a live database per CLAUDE.md. (CI runs with `SQLX_OFFLINE=true`, which is why the cache must be complete.)
-- **Never** skip the verification step (step 4 above).
-
-## Verification
-
-After committing, the diff against `origin/main` should show:
-- A few **new** cache files (for your changed queries)
-- A few **deleted** cache files (for old queries that no longer exist)
-- **Zero** net deletions from the EE cache set
-
-```bash
-git diff origin/main --stat backend/.sqlx/
-```

.claude/hooks/guard-main-branch.sh

@@ -16,23 +16,6 @@ command="$(echo "$input" | jq -r '.tool_input.command // empty')"
 if [[ "$command" =~ ^git\ (push|reset|revert|checkout|merge|rebase|commit|add) ]]; then
   branch="$(git rev-parse --abbrev-ref HEAD 2>/dev/null || true)"
   if [[ "$branch" == "main" ]]; then
-    echo "BLOCK: You are on the main branch. Create or switch to a feature branch first." >&2
-    exit 2
-  fi
-fi
-
-# Block force-push targeting main from any branch.
-if [[ "$command" =~ ^git[[:space:]]+push([[:space:]]|$) ]]; then
-  has_force=false
-  if [[ "$command" =~ (--force([[:space:]]|=|$)|--force-with-lease|[[:space:]]-f([[:space:]]|$)) ]]; then
-    has_force=true
-  fi
-  # `+ref` refspec syntax is also a force push.
-  if [[ "$command" =~ [[:space:]]\+[A-Za-z] ]]; then
-    has_force=true
-  fi
-  if $has_force && [[ "$command" =~ (^|[[:space:]:])\+?main([[:space:]]|$) ]]; then
-    echo "BLOCK: Force-push to main is not allowed via Claude. Run it yourself if you really mean to." >&2
-    exit 2
+    echo "BLOCK: You are on the main branch. Create or switch to a feature branch first."
   fi
 fi

.claude/review-prompt.md

@@ -1,4 +1,25 @@
-# Claude output format
+# Code Review Instructions
 
-- Use inline comments at the relevant lines for specific issues.
-- Use a top-level comment for the summary, severity-tagged finding list, AGENTS.md compliance check, and the test-coverage assessment.
+Review this pull request and provide comprehensive feedback.
+
+## Focus Areas
+
+- **Code quality and best practices** — does the code follow established patterns?
+- **Potential bugs or issues** — will this code work correctly in all cases?
+- **Performance considerations** — are there unnecessary allocations, N+1 queries, or bottlenecks?
+- **Security implications** — injection, auth bypass, data exposure?
+
+## CLAUDE.md Compliance
+
+Read all relevant CLAUDE.md files (root and in directories containing changed files). Check each rule against the changed code. Quote the exact rule when flagging a violation.
+
+## Review Guidelines
+
+- Provide detailed feedback using inline comments for specific issues
+- Use top-level comments for general observations or praise
+- Only flag issues introduced by this PR, not pre-existing problems
+- Self-validate each finding: "Is this definitely a real issue?" If uncertain, discard it
+
+## Testing Instructions
+
+At the end of your review, add complete instructions to reproduce the added changes through the app interface. These instructions will be given to a tester so they can verify the changes. It should be a short descriptive text (not a step-by-step or a list) on how to navigate the app (what page, what action, what input, etc.) to see the changes.

.claude/settings.json

@@ -44,25 +44,7 @@
       "Bash(git merge:*)",
       "Bash(git rebase:*)",
       "Bash(git add:*)",
-      "Bash(git commit:*)",
-      "Read(/tmp/**)",
-      "Write(/tmp/**)",
-      "Edit(/tmp/**)",
-      "Bash(rm:/tmp/*)",
-      "Bash(rm:/tmp/**)",
-      "Bash(rmdir:/tmp/*)",
-      "Bash(mkdir:/tmp/*)",
-      "Bash(mkdir:/tmp/**)",
-      "Bash(cp:/tmp/*)",
-      "Bash(cp:/tmp/**)",
-      "Bash(mv:/tmp/*)",
-      "Bash(mv:/tmp/**)",
-      "Bash(touch:/tmp/*)",
-      "Bash(touch:/tmp/**)",
-      "Bash(chmod:/tmp/*)",
-      "Bash(chmod:/tmp/**)",
-      "Bash(tar * /tmp/*)",
-      "Bash(unzip * /tmp/*)"
+      "Bash(git commit:*)"
     ],
     "deny": [
       "Read(.env)",
@@ -73,10 +55,7 @@
       "Read(**/*.pem)",
       "Read(**/*.key)",
       "Read(**/credentials.json)",
-      "Read(**/.secret*)",
-      "Read(**/.secrets*)",
-      "Read(**/*.secret)",
-      "Read(**/*.secrets)",
+      "Read(**/*secret*)",
       "Edit(.env)",
       "Edit(.env.*)",
       "Edit(**/.env)",
@@ -90,13 +69,7 @@
       "Bash(chown:*)",
       "Bash(truncate:*)",
       "Bash(shred:*)",
-      "Bash(unlink:*)",
-      "mcp__claude_ai_Stripe",
-      "mcp__claude_ai_Gmail",
-      "mcp__claude_ai_Google_Calendar",
-      "mcp__claude_ai_Google_Drive",
-      "mcp__claude_ai_Slack",
-      "mcp__claude_ai_Linear"
+      "Bash(unlink:*)"
     ]
   },
   "enableAllProjectMcpServers": true,

.claude/skills/adding-a-trigger/SKILL.md

@@ -1 +0,0 @@
-../../../.agents/skills/adding-a-trigger/SKILL.md

.claude/skills/adding-a-trigger/SKILL.md

@@ -0,0 +1,265 @@
+---
+name: adding-a-trigger
+description: Checklist for adding a new TriggerCrud-based trigger type to Windmill (Azure, GCP, Kafka, etc.). Use when wiring a new trigger kind across backend, frontend, CLI, and capture infrastructure.
+---
+
+# Skill: Adding a New Trigger Type
+
+Use this skill when adding a trigger kind that implements `TriggerCrud` (Kafka, GCP, Azure, MQTT, SQS, NATS, Postgres, Email…). For native triggers (Nextcloud, Google Drive — things wired through `windmill-native-triggers`), use the `native-trigger` skill instead.
+
+The goal of this doc is to enumerate every file that needs to change. Missing any one of them leads to silent regressions: sync drops the trigger, capture button does nothing, workspace forks lose it, sidebar counters undercount. Follow the checklist top-to-bottom — each section is independent enough to be validated on its own.
+
+Throughout this doc, substitute `{kind}` for the new trigger kind (`azure`, `kafka`, …), `{Kind}` for PascalCase (`Azure`, `Kafka`), `{KIND}` for SCREAMING (`AZURE`, `KAFKA`).
+
+## Reference implementations
+
+- **GCP** — closest analogue to Azure. Has push + pull, OIDC auth, ARM-like resource paths, capture handler. Grep for `gcp_trigger` / `GcpTrigger`.
+- **Kafka** — simpler (pull-only, streaming). Good for trivial integrations.
+- **Azure** — most recently added (2026). Shared-secret push auth, Event Grid namespaces + basic topics, ARM resource discovery, Namespace-pull data-plane. Grep for `azure_trigger` / `AzureTrigger`.
+
+## 1. Database migration
+
+Create a migration: `cargo sqlx migrate add -r add_{kind}_trigger` from `backend/`. Never write timestamps manually.
+
+The `up.sql` usually defines:
+- An optional enum type (e.g. `AZURE_MODE`) if the trigger has sub-kinds
+- The `{kind}_trigger` table with at minimum these columns (mirrored from kafka/gcp):
+  - primary: `(workspace_id, path)`
+  - `script_path`, `is_flow`, `enabled`, `mode`, `permissioned_as`, `edited_by`, `email`
+  - `edited_at`, `error`, `server_id`, `last_server_ping`
+  - `error_handler_path`, `error_handler_args jsonb`, `retry jsonb`
+  - trigger-specific fields
+- Indexes on foreign keys + any frequently-filtered columns
+- Foreign key to `workspace`
+
+Down migration drops the table and any enum types.
+
+## 2. Backend crate (`windmill-trigger-{kind}`)
+
+Create a new crate under `backend/windmill-trigger-{kind}/` with:
+
+- `Cargo.toml`: features `enterprise`, `private` if EE, standard deps
+- `src/lib.rs`: `pub use mod_ee::*;` behind `#[cfg(all(feature = "enterprise", feature = "private"))]`
+- `src/mod_ee.rs`: core types + helpers
+- `src/handler_ee.rs`: `TriggerCrud` impl + route handlers
+- `src/listener_ee.rs`: (only if streaming/pull-based) `Listener` trait impl
+
+Required in `mod_ee.rs`:
+- `{Kind}Config` struct (persisted shape, `FromRow`)
+- `{Kind}ConfigRequest` struct (what API receives — usually similar to Config but with validation fields)
+- `{Kind}Trigger` unit struct (implements the traits)
+- `impl TriggerJobArgs for {Kind}Trigger` — sets `TRIGGER_KIND`, `Payload`, `v1_payload_fn`
+
+Required in `handler_ee.rs`:
+- `#[async_trait] impl TriggerCrud for {Kind}Trigger` with:
+  - `type Trigger = Trigger<{Kind}Config>`
+  - `type TriggerConfigRequest = {Kind}ConfigRequest`
+  - `const ROUTE_PREFIX: &'static str = "/{kind}_triggers";`
+  - `const TABLE_NAME`, `ADDITIONAL_SELECT_FIELDS`
+  - `get_deployed_object`, `validate_config`, `create_trigger`, `update_trigger`, `delete_trigger`, `test_connection`
+  - `additional_routes` (optional — mount extra endpoints for things like ARM resource listing, topic discovery)
+
+Register the crate in `backend/Cargo.toml` as a workspace member and as a dep of `windmill-api` behind the feature flag.
+
+## 3. Wire into `windmill-api` (feature-gated everywhere)
+
+**`backend/windmill-api/src/triggers/handler.rs`** — mount the trigger crate:
+```rust
+#[cfg(all(feature = "enterprise", feature = "{kind}_trigger", feature = "private"))]
+{
+    use crate::triggers::{kind}::{Kind}Trigger;
+    router = router.nest({Kind}Trigger::ROUTE_PREFIX, complete_trigger_routes({Kind}Trigger));
+}
+```
+
+**`backend/windmill-api/src/triggers/{kind}/mod.rs`** — re-export the crate:
+```rust
+pub use windmill_trigger_{kind}::*;
+```
+
+**`backend/windmill-api/src/lib.rs`** — if the trigger receives inbound pushes, add a webhook route:
+```rust
+.nest("/{kind}/w/{workspace_id}", {
+    #[cfg(all(feature = "enterprise", feature = "{kind}_trigger", feature = "private"))]
+    { triggers::{kind}::handler_oss::{kind}_push_route_handler() }
+    #[cfg(not(...))]
+    { Router::new() }
+})
+```
+
+## 4. `TriggerKind` enum (`backend/windmill-types/src/triggers.rs`)
+
+Already has slots for most triggers but verify your variant exists:
+- Add `{Kind}` to the `TriggerKind` enum
+- Add match arm in `to_key()`
+- Add match arm in `from_str`
+- Add match arm in `JobTriggerKind` (if jobs need kind tagging)
+
+## 5. OpenAPI (`backend/windmill-api/openapi.yaml`)
+
+This file is huge and the single most-forgotten place. Add:
+
+- `/w/{workspace}/{kind}_triggers/create` + `/update/{path}` + `/delete/{path}` + `/get/{path}` + `/list` + `/exists/{path}` + `/setmode/{path}` + `/test` paths (mirror gcp section)
+- Any `additional_routes` your handler exposes (resource discovery, etc.)
+- Schemas: `{Kind}Trigger`, `{Kind}TriggerData`, `{Kind}Mode` (if enum), `{Kind}DeliveryConfig`, helper request/response types
+- Add `{kind}` to `CaptureTriggerKind` enum
+- Add `{kind}_used: boolean` to the `UsedTriggers` response schema
+
+Regenerate frontend client: `npm run generate-backend-client` from `frontend/`.
+
+## 6. `UsedTriggers` + workspace export
+
+**`backend/windmill-api-workspaces/src/workspaces.rs`** — add `{kind}_used: bool` to the `UsedTriggers` struct and add an `EXISTS(SELECT 1 FROM {kind}_trigger …)` to the `get_used_triggers` query.
+
+**`backend/windmill-api/src/workspaces_export.rs`** — add export block mirroring gcp's (export lists all triggers, serializes them to YAML/JSON).
+
+## 6.5 Hardcoded trigger-kind arrays (silent-failure hotspots)
+
+Several files keep **hardcoded arrays** of trigger kind strings. Miss one and ACL checks / user offboarding / trash drop your kind:
+
+- **`backend/windmill-api-groups/src/granular_acls.rs`** — `KINDS: [&str; N]`. **Increment N** (the compile error is cryptic otherwise). Controls which kinds accept granular ACL operations.
+- **`backend/windmill-api-users/src/users.rs`** (`extra_perms_tables`) — which tables get `extra_perms` entries cleaned when a user is deleted.
+- **`backend/windmill-api/src/offboarding.rs`** — three separate arrays (enumeration, fork-copy, and delete paths). **All three** need the new kind.
+- **`backend/windmill-api/src/trash.rs`** — `valid_tables` for the trash / restore API.
+- **`backend/windmill-git-sync/src/lib.rs`** — add a test assertion for `DeployedObject::{Kind}Trigger.get_kind() == "{kind}_trigger"` (the `get_kind` match arm itself lives in the enum impl — already required by the Rust compiler).
+- **`backend/windmill-api-auth/src/scopes.rs`** — add the `{Kind}Triggers` variant to `ScopeDomain` enum + `as_str` match + `from_str` match. Required for the OAuth/token system to recognise `{kind}_triggers:read|write` scopes.
+- **`backend/windmill-api/src/token.rs`** (`build_trigger_scope_domains` → `TRIGGER_DOMAINS`) — add `("{kind}_triggers", "{Kind display name}")` so the CreateToken UI's scope selector surfaces the `read` / `write` checkboxes.
+
+**OpenAPI enums** to extend (do NOT forget — generated client will allow it but server rejects as 400):
+- `CaptureTriggerKind` enum
+- Three `kind` enums under `/w/{workspace}/acls/{get,add,remove}/{kind}/{path}` (yes, same list repeated three times)
+
+After editing any of these, run a full `cargo check` with your feature flag + `gcp_trigger` + other core flags — the `KINDS: [&str; N]` length mismatch only surfaces when the crate compiles.
+
+## 7. Capture infrastructure (`backend/windmill-api/src/capture.rs`)
+
+If the trigger supports push delivery, it also needs a capture endpoint so users can test it:
+
+- `{Kind}TriggerConfig` struct (gated by feature flags)
+- `TriggerConfig::{Kind}` variant
+- `set_{kind}_trigger_config` function (creates the subscription/equivalent pointing at the capture URL — use your `manage_{kind}_subscription` helper with `trigger_mode=false`)
+- Both real + no-op versions behind feature gates
+- `TriggerKind::{Kind} => set_{kind}_trigger_config(...)` arm in `set_config`
+- `{kind}_payload` async handler — validates auth (if any), processes payload, calls `insert_capture_payload`
+- Route: `.route("/{kind}/{runnable_kind}/{*path}", post({kind}_payload))` inside `workspaced_unauthed_service` — and expand the surrounding `#[cfg(any(...))]` to include your feature flag
+
+## 8. CLI (`cli/`) — easy to miss, breaks sync silently
+
+Check all of these:
+
+**`cli/src/types.ts`:**
+- Add `"{kind}"` to `TRIGGER_TYPES` array
+- Add `"{kind}_trigger"` to `getTypeStrFromPath` return union
+- Add match case in `getTypeStrFromPath`'s `typeEnding ===` chain
+- Add `pushTrigger("{kind}", ...)` branch in `pushObj`
+
+**`cli/src/commands/trigger/trigger.ts`:**
+- Import `{Kind}Trigger` type
+- Add `{kind}: {Kind}Trigger` to the `Trigger` type map
+- Add `{kind}: wmill.get{Kind}Trigger`, `update{Kind}Trigger`, `create{Kind}Trigger` to each function map
+- Add `{kind}: { ... }` template to `triggerTemplates`
+- Add `list{Kind}Triggers` call + spread in the `list` aggregation
+- Update `--kind` option descriptions to mention the new kind
+
+**`cli/src/commands/sync/sync.ts`:**
+- Add `path.endsWith(".{kind}_trigger" + ext)` in the file-type filter
+- Add `typ == "{kind}_trigger"` in `getTypeOrder`
+- Add `"{kind}_trigger"` to the delete-suffix regex (~line 3092)
+- Add a `case "{kind}_trigger"` in the delete switch
+
+**`cli/src/guidance/skills.ts`** — **DO NOT EDIT DIRECTLY**. It's auto-generated by `system_prompts/generate.py`. Instead:
+- Edit `system_prompts/utils.py` → append `('{Kind}Trigger', '{kind}_trigger')` to the `SCHEMA_MAPPINGS['triggers']` list (this is the master list — the one in `generate.py` is duplicated and `utils.py` wins)
+- Then run `python3 system_prompts/generate.py` — it regenerates `cli/src/guidance/skills.ts` with the schema extracted from `backend/windmill-api/openapi.yaml`
+- Commit the regenerated file
+
+## 9. Frontend — editor + drawer
+
+Under `frontend/src/lib/components/triggers/{kind}/`:
+
+- `{Kind}TriggerPanel.svelte` — the tile shown in the triggers listing
+- `{Kind}TriggerEditor.svelte` — outer drawer wrapper
+- `{Kind}TriggerEditorInner.svelte` — state + business logic; must expose:
+  - `openEdit(path, isFlow, defaultValues?)` method
+  - `isEditor` prop, `onConfigChange` + `onCaptureConfigChange` callbacks
+  - `get{Kind}Config()` + `get{Kind}CaptureConfig()` helpers
+  - `captureConfig = $derived.by(untrack(() => isEditor) ? get{Kind}CaptureConfig : () => ({}))`
+  - `$effect(() => { const args = [captureConfig, isValid] as const; untrack(() => onCaptureConfigChange?.(...args)) })`
+- `{Kind}TriggerEditorConfigSection.svelte` — form fields; use design-system components (`TextInput`, `Select`, `Toggle`, `ToggleButtonGroup`), never raw `<input>`
+- `{Kind}Capture.svelte` — capture panel; wraps `CaptureSection` with `captureType="{kind}"`
+- `utils.ts` — `requestBody` builders and any trigger-type-specific helpers
+
+## 10. Frontend — global integration
+
+Easy to miss:
+
+- **`frontend/src/lib/components/triggers.ts`** — add `'{kind}'` to the `TriggerKind` union
+- **`frontend/src/lib/components/triggers/CaptureWrapper.svelte`**:
+  - Import `{Kind}Capture`
+  - Add to `isStreamingCapture()` array (streaming = pull-style; push-style is typically `false`)
+  - Add `{:else if captureType === '{kind}'}` branch with the `<{Kind}Capture>` render
+- **`frontend/src/lib/components/sidebar/SidebarContent.svelte`** — import the icon, add the nav entry
+- **`frontend/src/lib/components/sidebar/OperatorMenu.svelte`** — add the operator-mode entry
+- **`frontend/src/routes/(root)/(logged)/+layout.svelte`** — destructure `{kind}_used` from `/get_used_triggers` response, push `'{kind}'` into `usedKinds`
+- **`frontend/src/lib/components/search/GlobalSearchModal.svelte`** — import icon, add "Go to {Kind} ..." entry
+- **`frontend/src/lib/components/offboarding-utils.ts`** — add mappings `{kind}_trigger: '{kind}_triggers'` and `{kind}_trigger: '{kind} trigger'`
+- **`frontend/src/lib/components/icons/{Kind}Icon.svelte`** — single-path SVG, `fill={color ?? 'currentColor'}`, `size` prop default 16 (match existing icons — don't hardcode colors, don't use `width`/`height` props)
+- **`frontend/src/routes/(root)/(logged)/{kind}_triggers/+page.svelte`** — listing page (mirror `gcp_triggers/+page.svelte` for push+pull, `kafka_triggers` for pure streaming)
+- **`frontend/src/lib/components/CompareWorkspaces.svelte`** — workspace fork / compare tool. Needs: service import, editor import, `{kind}Editor` `$state`, `case '{kind}'` in `openTriggerDetails()`, entry in `triggerServices` object (list/delete/normalize), and `<{Kind}TriggerEditor bind:this={{kind}Editor} />` in the template
+
+## 10.5 AI system prompts (`system_prompts/`)
+
+- **`system_prompts/utils.py`** — append `('{Kind}Trigger', '{kind}_trigger')` to `SCHEMA_MAPPINGS['triggers']` (master list used by code generation + CLI skills)
+- **`system_prompts/generate.py`** — also has a duplicated `schema_types` list (~line 903) for the AI `triggers` skill content. Add `('{Kind}Trigger', '{kind}_trigger')` there too
+- **`system_prompts/generate.py`** `schema_names` (~line 1192) — add `'{Kind}Trigger'` (add `'New{Kind}Trigger'` only if the OpenAPI declares one; GCP and Azure don't)
+- Run `python3 system_prompts/generate.py` — this rewrites `cli/src/guidance/skills.ts` and all `auto-generated/` docs. Commit the regenerated files
+
+## 11. Validation
+
+Run all of these before declaring done:
+
+```bash
+# Backend
+cd backend
+cargo check --features enterprise,{kind}_trigger,private   # minimal
+cargo check --features enterprise,azure_trigger,private,gcp_trigger,http_trigger,mqtt_trigger,postgres_trigger,sqs_trigger,kafka,nats,smtp,websocket  # full
+
+# SQLx offline data (never run `cargo sqlx prepare` directly — use the wrapper)
+./update_sqlx.sh
+
+# Frontend
+cd frontend
+npm run generate-backend-client
+npm run check:fast
+```
+
+Smoke test in the UI: create a trigger, save, check it appears in sidebar + search, delete, re-create via CLI `wmill sync`.
+
+## 12. Common pitfalls
+
+- **Forgetting feature gates in `workspaced_unauthed_service()`** — the surrounding `#[cfg(any(...))]` expression must include your feature flag, not just the inner `#[cfg]` on the route
+- **`.route(path, ...).route(path, ...)` with same path and different methods** — older axum replaced; use `.route(path, post(h1).options(h2))` to chain methods on the same `MethodRouter`
+- **`on:event` directives** — legacy Svelte 4, no-op in runes mode. Use callback props (`onSelected`, `onConfigChange`)
+- **`$bindable(default_value)` on optional props** — banned by project CLAUDE.md. Use `$bindable()` + `$derived(prop ?? default)` instead
+- **CORS layer intercepting OPTIONS** — tower-http CorsLayer short-circuits OPTIONS before reaching your handler. For server-to-server webhook endpoints, drop the CORS layer entirely (CORS is browser-only)
+- **DeliveryAttributeMappings / custom headers for auth** — prefer HMAC or sha256-hashed shared secrets over opaque JWTs when the provider doesn't support signed tokens natively. Store only the hash; regenerate secret on every save
+- **ARM / API resource-listing cascades** — if the trigger's resource type is deep (Azure: subscription → RG → namespace → topic), offer dropdowns in the UI populated from the provider's APIs using the user's credential resource
+- **Clearing stale selections on dependency change** — when a dropdown's underlying data reloads (e.g., user changes SP or edition), clear selections that no longer match the new list
+- **Workspace-scoped tag compatibility** — if the trigger has tags, verify forked workspaces handle them (see commit `0773b5bc85` for a historical fix)
+
+## 13. EE file split
+
+If the trigger is enterprise-only, the code lives in `windmill-ee-private__worktrees/.../windmill-trigger-{kind}/src/*_ee.rs` and is symlinked into the OSS tree. The `windmill-ee-private__worktrees/` directory holds the real files; changes propagate via symlinks. See `docs/enterprise.md` for the workflow.
+
+## 14. Final checklist before PR
+
+- [ ] Migration up/down tested (revert + re-apply)
+- [ ] `./update_sqlx.sh` committed the updated `.sqlx/` offline data
+- [ ] `cargo check` passes with your feature flag + with all trigger features
+- [ ] `npm run check:fast` passes
+- [ ] Trigger visible in sidebar with correct icon weight (not oversized/colored — use `currentColor`)
+- [ ] Create, edit, delete flow all work in the UI
+- [ ] Capture button works (if push-capable)
+- [ ] Trigger appears in `/get_used_triggers` → sidebar pulse
+- [ ] `wmill sync pull` + `wmill sync push` both round-trip the trigger
+- [ ] `wmill trigger list` includes it
+- [ ] OpenAPI schemas are complete (no `null` in generated types)

.claude/skills/commit/SKILL.md

@@ -1 +0,0 @@
-../../../.agents/skills/commit/SKILL.md

.claude/skills/commit/SKILL.md

@@ -0,0 +1,60 @@
+---
+name: commit
+user_invocable: true
+description: Create a git commit with conventional commit format. MUST use anytime you want to commit changes.
+---
+
+# Git Commit Skill
+
+Create a focused, single-line commit following conventional commit conventions.
+
+## Instructions
+
+1. **Analyze changes**: Run `git status` and `git diff` to understand what was modified
+2. **Stage only modified files**: Add files individually by name. NEVER use `git add -A` or `git add .`
+3. **Write commit message**: Follow the conventional commit format as a single line
+
+## Conventional Commit Format
+
+```
+<type>: <description>
+```
+
+### Types
+- `feat`: New feature or capability
+- `fix`: Bug fix
+- `refactor`: Code change that neither fixes a bug nor adds a feature
+- `docs`: Documentation only changes
+- `style`: Formatting, missing semicolons, etc (no code change)
+- `test`: Adding or correcting tests
+- `chore`: Maintenance tasks, dependency updates, etc
+- `perf`: Performance improvement
+
+### Rules
+- Message MUST be a single line (no multi-line messages)
+- Description should be lowercase, imperative mood ("add" not "added")
+- No period at the end
+- Keep under 72 characters total
+
+### Examples
+```
+feat: add token usage tracking for AI providers
+fix: resolve null pointer in job executor
+refactor: extract common validation logic
+docs: update API endpoint documentation
+chore: upgrade sqlx to 0.7
+```
+
+## Execution Steps
+
+1. Run `git status` to see all changes
+2. Run `git diff` to understand the changes in detail
+3. Run `git log --oneline -5` to see recent commit style
+4. Stage ONLY the modified/relevant files: `git add <file1> <file2> ...`
+5. Create the commit with conventional format:
+   ```bash
+   git commit -m "<type>: <description>
+
+   Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>"
+   ```
+6. Run `git status` to verify the commit succeeded

.claude/skills/local-review/SKILL.md

@@ -1 +0,0 @@
-../../../.agents/skills/local-review/SKILL.md

.claude/skills/local-review/SKILL.md

@@ -0,0 +1,69 @@
+---
+name: local-review
+user_invocable: true
+description: Code review a pull request for bugs and CLAUDE.md compliance. MUST use when asked to review code.
+---
+
+# Local Code Review Skill
+
+Run the same review locally that the GitHub Claude Auto Review action runs on PRs. The shared review instructions live in `.claude/review-prompt.md` — read that file first and follow its instructions.
+
+## Execution Steps
+
+1. **Read `.claude/review-prompt.md`** for the review criteria and focus areas
+
+2. **Determine the PR scope**:
+   - If an argument is provided, use it as the PR number or branch
+   - Otherwise, detect from the current branch vs main
+   - Run `gh pr view` if a PR exists, or use `git diff main...HEAD`
+
+3. **Get the diff and metadata**:
+   - `gh pr diff` or `git diff main...HEAD` for the full diff
+   - `gh pr view` or `git log main..HEAD --oneline` for context
+
+4. **Read changed files** where the diff alone is insufficient to understand context
+
+5. **Apply the review instructions from `.claude/review-prompt.md`**
+
+6. **Self-validate each finding**: Before reporting, ask yourself:
+   - "Is this definitely a real issue, not a false positive?"
+   - "Would a senior engineer flag this in review?"
+   - If the answer to either is no, discard the finding
+
+7. **Output findings** to the terminal (default) or post as PR comments (with `--comment` flag)
+
+## Output Format
+
+```
+## Code review
+
+Found N issues:
+
+1. <description> (<reason: CLAUDE.md adherence | bug | security>)
+   <file_path:line_number>
+
+2. <description> (<reason>)
+   <file_path:line_number>
+```
+
+If no issues are found:
+
+```
+## Code review
+
+No issues found. Checked for bugs and CLAUDE.md compliance.
+```
+
+## Posting Comments (--comment flag)
+
+If the user passes `--comment`, post findings as inline PR comments using:
+
+```bash
+gh pr review --comment --body "<summary>"
+```
+
+Or for inline comments on specific lines:
+
+```bash
+gh api repos/{owner}/{repo}/pulls/{pr}/reviews -f body="<summary>" -f event="COMMENT" -f comments="[...]"
+```

.claude/skills/native-trigger/SKILL.md

@@ -1 +0,0 @@
-../../../.agents/skills/native-trigger/SKILL.md

.claude/skills/native-trigger/SKILL.md

@@ -0,0 +1,793 @@
+---
+name: native-trigger
+description: Guidance for adding native trigger services to Windmill. Use when implementing or modifying native trigger integrations across the backend and frontend.
+---
+
+# Skill: Adding Native Trigger Services
+
+This skill provides comprehensive guidance for adding new native trigger services to Windmill. Native triggers allow external services (like Nextcloud, Google Drive, etc.) to trigger Windmill scripts/flows via webhooks or push notifications.
+
+## Architecture Overview
+
+The native trigger system consists of:
+
+1. **Database Layer** - PostgreSQL tables and enum types
+2. **Backend Rust Implementation** - Core trait, handlers, and service modules in the `windmill-native-triggers` crate
+3. **Frontend Svelte Components** - Configuration forms and UI components
+
+### Key Files
+
+| Component | Path |
+|-----------|------|
+| Core module with `External` trait | `backend/windmill-native-triggers/src/lib.rs` |
+| Generic CRUD handlers | `backend/windmill-native-triggers/src/handler.rs` |
+| Background sync logic | `backend/windmill-native-triggers/src/sync.rs` |
+| OAuth/workspace integration | `backend/windmill-native-triggers/src/workspace_integrations.rs` |
+| Re-export shim (windmill-api) | `backend/windmill-api/src/native_triggers/mod.rs` |
+| TriggerKind enum | `backend/windmill-common/src/triggers.rs` |
+| JobTriggerKind enum | `backend/windmill-common/src/jobs.rs` |
+| Frontend service registry | `frontend/src/lib/components/triggers/native/utils.ts` |
+| Frontend trigger utilities | `frontend/src/lib/components/triggers/utils.ts` |
+| Trigger badges (icons + counts) | `frontend/src/lib/components/graph/renderers/triggers/TriggersBadge.svelte` |
+| Workspace integrations UI | `frontend/src/lib/components/workspaceSettings/WorkspaceIntegrations.svelte` |
+| OAuth config form component | `frontend/src/lib/components/workspaceSettings/OAuthClientConfig.svelte` |
+| OpenAPI spec | `backend/windmill-api/openapi.yaml` |
+| Reference: Nextcloud module | `backend/windmill-native-triggers/src/nextcloud/` |
+| Reference: Google module | `backend/windmill-native-triggers/src/google/` |
+
+### Crate Structure
+
+The native trigger code lives in the `windmill-native-triggers` crate (`backend/windmill-native-triggers/`). The `windmill-api` crate re-exports everything via a shim:
+
+```rust
+// backend/windmill-api/src/native_triggers/mod.rs
+pub use windmill_native_triggers::*;
+```
+
+All new service modules go in `backend/windmill-native-triggers/src/`.
+
+---
+
+## Core Concepts
+
+### The `External` Trait
+
+Every native trigger service implements the `External` trait defined in `lib.rs`:
+
+```rust
+#[async_trait]
+pub trait External: Send + Sync + 'static {
+    // Associated types:
+    type ServiceConfig: Debug + DeserializeOwned + Serialize + Send + Sync;
+    type TriggerData: Debug + Serialize + Send + Sync;
+    type OAuthData: DeserializeOwned + Serialize + Clone + Send + Sync;
+    type CreateResponse: DeserializeOwned + Send + Sync;
+
+    // Constants:
+    const SUPPORT_WEBHOOK: bool;
+    const SERVICE_NAME: ServiceName;
+    const DISPLAY_NAME: &'static str;
+    const TOKEN_ENDPOINT: &'static str;
+    const REFRESH_ENDPOINT: &'static str;
+    const AUTH_ENDPOINT: &'static str;
+
+    // Required methods:
+    async fn create(&self, w_id, oauth_data, webhook_token, data, db, tx) -> Result<Self::CreateResponse>;
+    async fn update(&self, w_id, oauth_data, external_id, webhook_token, data, db, tx) -> Result<serde_json::Value>;
+    async fn get(&self, w_id, oauth_data, external_id, db, tx) -> Result<Self::TriggerData>;
+    async fn delete(&self, w_id, oauth_data, external_id, db, tx) -> Result<()>;
+    async fn exists(&self, w_id, oauth_data, external_id, db, tx) -> Result<bool>;
+    async fn maintain_triggers(&self, db, workspace_id, triggers, oauth_data, synced, errors);
+    fn external_id_and_metadata_from_response(&self, resp) -> (String, Option<serde_json::Value>);
+
+    // Methods with defaults:
+    async fn prepare_webhook(&self, db, w_id, headers, body, script_path, is_flow) -> Result<PushArgsOwned>;
+    fn service_config_from_create_response(&self, data, resp) -> Option<serde_json::Value>;
+    fn additional_routes(&self) -> axum::Router;
+    async fn http_client_request<T, B>(&self, url, method, workspace_id, tx, db, headers, body) -> Result<T>;
+}
+```
+
+Key design points:
+- **`update()` returns `serde_json::Value`** - the resolved service_config to store. Each service is responsible for building the final config.
+- **`maintain_triggers()`** - periodic background maintenance. Each service implements its own strategy (Nextcloud: reconcile with external state; Google: renew expiring channels).
+- **No `list_all()` in the trait** - services that need it (Nextcloud) implement it privately; services that don't (Google) use different maintenance strategies.
+- **No `get_external_id_from_trigger_data()` or `extract_service_config_from_trigger_data()`** - removed in favor of the `maintain_triggers` pattern.
+
+### Create Lifecycle: Two Paths
+
+The `create_native_trigger` handler in `handler.rs` supports two creation flows, controlled by `service_config_from_create_response()`:
+
+**Path A: Short (Google pattern)** - `service_config_from_create_response()` returns `Some(config)`:
+1. `create()` registers on external service
+2. `external_id_and_metadata_from_response()` extracts the ID
+3. `service_config_from_create_response()` builds the config directly from input data + response metadata
+4. Stores trigger in DB -- done, no extra round-trip
+
+Use this when the external_id is known before the create call (e.g., Google generates the channel_id as a UUID upfront and includes it in the webhook URL).
+
+**Path B: Long (Nextcloud pattern)** - `service_config_from_create_response()` returns `None` (default):
+1. `create()` registers on external service (webhook URL has no external_id yet)
+2. `external_id_and_metadata_from_response()` extracts the ID
+3. `update()` is called to fix the webhook URL with the now-known external_id
+4. `update()` returns the resolved service_config
+5. Stores trigger in DB
+
+Use this when the external_id is assigned by the remote service and the webhook URL needs to be corrected after creation.
+
+### OAuth Token Storage (Three-Table Pattern)
+
+OAuth tokens are stored across three tables, NOT in `workspace_integrations.oauth_data` directly:
+
+| Table | What's Stored |
+|-------|---------------|
+| `workspace_integrations` | `oauth_data` JSON with `base_url`, `client_id`, `client_secret`, `instance_shared` flag; `resource_path` pointing to the variable |
+| `variable` | Encrypted `access_token` (at the path stored in `resource_path`), linked to `account` via `account` column |
+| `account` | `refresh_token`, keyed by `workspace_id` + `client` (service name) + `is_workspace_integration = true` |
+
+The `decrypt_oauth_data()` function in `lib.rs` assembles these into a unified struct:
+```rust
+pub struct OAuthConfig {
+    pub base_url: String,
+    pub access_token: String,      // decrypted from variable
+    pub refresh_token: Option<String>, // from account table
+    pub client_id: String,         // from oauth_data or instance settings
+    pub client_secret: String,     // from oauth_data or instance settings
+}
+```
+
+Instance-level sharing: when `oauth_data.instance_shared == true`, `client_id` and `client_secret` are read from global settings instead of workspace_integrations.
+
+### URL Resolution
+
+The `resolve_endpoint()` helper handles both absolute and relative OAuth URLs:
+
+```rust
+pub fn resolve_endpoint(base_url: &str, endpoint: &str) -> String {
+    if endpoint.starts_with("http://") || endpoint.starts_with("https://") {
+        endpoint.to_string()  // Google: absolute URLs
+    } else {
+        format!("{}{}", base_url, endpoint)  // Nextcloud: relative paths
+    }
+}
+```
+
+### ServiceName Methods
+
+`ServiceName` is the central registry enum. Each variant must implement these match arms:
+
+| Method | Purpose |
+|--------|---------|
+| `as_str()` | Lowercase identifier (e.g., `"google"`) |
+| `as_trigger_kind()` | Maps to `TriggerKind` enum |
+| `as_job_trigger_kind()` | Maps to `JobTriggerKind` enum |
+| `token_endpoint()` | OAuth token endpoint (relative or absolute) |
+| `auth_endpoint()` | OAuth authorization endpoint |
+| `oauth_scopes()` | Space-separated OAuth scopes |
+| `resource_type()` | Resource type for token storage (e.g., `"gworkspace"`) |
+| `extra_auth_params()` | Extra OAuth params (e.g., Google needs `access_type=offline`, `prompt=consent`) |
+| `integration_service()` | Maps to the workspace integration service (usually `*self`) |
+| `TryFrom<String>` | Parse from string |
+| `Display` | Delegates to `as_str()` |
+
+---
+
+## Step-by-Step Implementation Guide
+
+### Step 1: Database Migration
+
+Create a new migration file: `backend/migrations/YYYYMMDDHHMMSS_newservice_trigger.up.sql`
+
+```sql
+-- Add the service to the native_trigger_service enum
+ALTER TYPE native_trigger_service ADD VALUE IF NOT EXISTS 'newservice';
+
+-- Add to TRIGGER_KIND enum (used for trigger tracking)
+ALTER TYPE TRIGGER_KIND ADD VALUE IF NOT EXISTS 'newservice';
+
+-- Add to job_trigger_kind enum (used for job tracking)
+ALTER TYPE job_trigger_kind ADD VALUE IF NOT EXISTS 'newservice';
+```
+
+Also create the corresponding down migration.
+
+### Step 2: Update windmill-common Enums
+
+#### `backend/windmill-common/src/triggers.rs`
+
+Add variant to `TriggerKind` enum, and update `to_key()` and `fmt()` implementations.
+
+#### `backend/windmill-common/src/jobs.rs`
+
+Add variant to `JobTriggerKind` enum and update the `Display` implementation.
+
+### Step 3: Backend Service Module
+
+Create a new directory: `backend/windmill-native-triggers/src/newservice/`
+
+#### `mod.rs` - Type Definitions
+
+```rust
+use serde::{Deserialize, Serialize};
+
+pub mod external;
+// pub mod routes; // Only if you need additional service-specific routes
+
+/// OAuth data deserialized from the three-table pattern.
+/// The actual structure is built by decrypt_oauth_data() from variable + account + workspace_integrations.
+#[derive(Debug, Clone, Deserialize, Serialize)]
+pub struct NewServiceOAuthData {
+    pub base_url: String,              // from workspace_integrations.oauth_data
+    pub access_token: String,          // decrypted from variable table
+    pub refresh_token: Option<String>, // from account table
+    // Note: client_id and client_secret are in OAuthConfig, not here
+    // unless the service needs them at runtime for API calls
+}
+
+/// Configuration provided by user when creating/updating a trigger.
+/// Stored as JSON in native_trigger.service_config.
+#[derive(Debug, Clone, Serialize, Deserialize)]
+#[serde(rename_all = "camelCase")]
+pub struct NewServiceConfig {
+    // Service-specific configuration fields
+    pub folder_path: String,
+    pub file_filter: Option<String>,
+}
+
+/// Data retrieved from the external service about a trigger.
+/// Returned by the get() method and shown in the UI.
+#[derive(Debug, Clone, Serialize, Deserialize)]
+#[serde(rename_all = "camelCase")]
+pub struct NewServiceTriggerData {
+    pub folder_path: String,
+    pub file_filter: Option<String>,
+    // Fields that shouldn't affect service_config comparison should use #[serde(skip_serializing)]
+}
+
+/// Response from external service when creating a trigger/webhook.
+#[derive(Debug, Deserialize)]
+pub struct CreateTriggerResponse {
+    pub id: String,
+}
+
+/// Handler struct (stateless, used for routing)
+#[derive(Copy, Clone)]
+pub struct NewService;
+```
+
+#### `external.rs` - External Trait Implementation
+
+```rust
+use async_trait::async_trait;
+use reqwest::Method;
+use sqlx::PgConnection;
+use std::collections::HashMap;
+use windmill_common::{
+    error::{Error, Result},
+    BASE_URL, DB,
+};
+
+use crate::{
+    generate_webhook_service_url, External, NativeTrigger, NativeTriggerData, ServiceName,
+    sync::{SyncError, TriggerSyncInfo},
+};
+use super::{NewService, NewServiceConfig, NewServiceOAuthData, NewServiceTriggerData, CreateTriggerResponse};
+
+#[async_trait]
+impl External for NewService {
+    type ServiceConfig = NewServiceConfig;
+    type TriggerData = NewServiceTriggerData;
+    type OAuthData = NewServiceOAuthData;
+    type CreateResponse = CreateTriggerResponse;
+
+    const SERVICE_NAME: ServiceName = ServiceName::NewService;
+    const DISPLAY_NAME: &'static str = "New Service";
+    const SUPPORT_WEBHOOK: bool = true;
+    const TOKEN_ENDPOINT: &'static str = "/oauth/token";
+    const REFRESH_ENDPOINT: &'static str = "/oauth/token";
+    const AUTH_ENDPOINT: &'static str = "/oauth/authorize";
+
+    async fn create(
+        &self,
+        w_id: &str,
+        oauth_data: &Self::OAuthData,
+        webhook_token: &str,
+        data: &NativeTriggerData<Self::ServiceConfig>,
+        db: &DB,
+        tx: &mut PgConnection,
+    ) -> Result<Self::CreateResponse> {
+        let base_url = &*BASE_URL.read().await;
+
+        // external_id is None during create (we get it from the response)
+        let webhook_url = generate_webhook_service_url(
+            base_url, w_id, &data.script_path, data.is_flow,
+            None, Self::SERVICE_NAME, webhook_token,
+        );
+
+        let url = format!("{}/api/webhooks/create", oauth_data.base_url);
+        let payload = serde_json::json!({
+            "callback_url": webhook_url,
+            "folder_path": data.service_config.folder_path,
+        });
+
+        let response: CreateTriggerResponse = self
+            .http_client_request(&url, Method::POST, w_id, tx, db, None, Some(&payload))
+            .await?;
+
+        Ok(response)
+    }
+
+    /// Update returns the resolved service_config as JSON.
+    /// For services using the update+get pattern, call self.get() and serialize.
+    async fn update(
+        &self,
+        w_id: &str,
+        oauth_data: &Self::OAuthData,
+        external_id: &str,
+        webhook_token: &str,
+        data: &NativeTriggerData<Self::ServiceConfig>,
+        db: &DB,
+        tx: &mut PgConnection,
+    ) -> Result<serde_json::Value> {
+        let base_url = &*BASE_URL.read().await;
+
+        let webhook_url = generate_webhook_service_url(
+            base_url, w_id, &data.script_path, data.is_flow,
+            Some(external_id), Self::SERVICE_NAME, webhook_token,
+        );
+
+        let url = format!("{}/api/webhooks/{}", oauth_data.base_url, external_id);
+        let payload = serde_json::json!({
+            "callback_url": webhook_url,
+            "folder_path": data.service_config.folder_path,
+        });
+
+        let _: serde_json::Value = self
+            .http_client_request(&url, Method::PUT, w_id, tx, db, None, Some(&payload))
+            .await?;
+
+        // Fetch back the updated state to get the resolved config
+        let trigger_data = self.get(w_id, oauth_data, external_id, db, tx).await?;
+        serde_json::to_value(&trigger_data)
+            .map_err(|e| Error::InternalErr(format!("Failed to serialize trigger data: {}", e)))
+    }
+
+    async fn get(
+        &self,
+        w_id: &str,
+        oauth_data: &Self::OAuthData,
+        external_id: &str,
+        db: &DB,
+        tx: &mut PgConnection,
+    ) -> Result<Self::TriggerData> {
+        let url = format!("{}/api/webhooks/{}", oauth_data.base_url, external_id);
+        self.http_client_request::<_, ()>(&url, Method::GET, w_id, tx, db, None, None).await
+    }
+
+    async fn delete(
+        &self,
+        w_id: &str,
+        oauth_data: &Self::OAuthData,
+        external_id: &str,
+        db: &DB,
+        tx: &mut PgConnection,
+    ) -> Result<()> {
+        let url = format!("{}/api/webhooks/{}", oauth_data.base_url, external_id);
+        let _: serde_json::Value = self
+            .http_client_request::<_, ()>(&url, Method::DELETE, w_id, tx, db, None, None)
+            .await
+            .or_else(|e| match &e {
+                Error::InternalErr(msg) if msg.contains("404") => Ok(serde_json::Value::Null),
+                _ => Err(e),
+            })?;
+        Ok(())
+    }
+
+    async fn exists(
+        &self,
+        w_id: &str,
+        oauth_data: &Self::OAuthData,
+        external_id: &str,
+        db: &DB,
+        tx: &mut PgConnection,
+    ) -> Result<bool> {
+        match self.get(w_id, oauth_data, external_id, db, tx).await {
+            Ok(_) => Ok(true),
+            Err(Error::NotFound(_)) => Ok(false),
+            Err(e) => Err(e),
+        }
+    }
+
+    /// Background maintenance. Choose the right pattern for your service:
+    /// - For services with queryable external state: use reconcile_with_external_state()
+    /// - For channel-based services with expiration: implement renewal logic
+    async fn maintain_triggers(
+        &self,
+        db: &DB,
+        workspace_id: &str,
+        triggers: &[NativeTrigger],
+        oauth_data: &Self::OAuthData,
+        synced: &mut Vec<TriggerSyncInfo>,
+        errors: &mut Vec<SyncError>,
+    ) {
+        // Option A: Reconcile with external state (Nextcloud pattern)
+        // Fetch all triggers from external service and compare with DB
+        let external_triggers = match self.list_all(workspace_id, oauth_data, db).await {
+            Ok(triggers) => triggers,
+            Err(e) => {
+                errors.push(SyncError {
+                    resource_path: format!("workspace:{}", workspace_id),
+                    error_message: format!("Failed to list triggers: {}", e),
+                    error_type: "api_error".to_string(),
+                });
+                return;
+            }
+        };
+
+        // Convert to (external_id, config_json) pairs
+        let external_pairs: Vec<(String, serde_json::Value)> = external_triggers
+            .into_iter()
+            .map(|t| (t.id.clone(), serde_json::to_value(&t).unwrap_or_default()))
+            .collect();
+
+        crate::sync::reconcile_with_external_state(
+            db, workspace_id, Self::SERVICE_NAME, triggers, &external_pairs, synced, errors,
+        ).await;
+    }
+
+    fn external_id_and_metadata_from_response(
+        &self,
+        resp: &Self::CreateResponse,
+    ) -> (String, Option<serde_json::Value>) {
+        (resp.id.clone(), None)
+    }
+
+    // service_config_from_create_response: NOT overridden (returns None).
+    // This means the handler uses the update+get pattern after create.
+    // Override and return Some(...) to skip the update+get cycle (Google pattern).
+}
+
+impl NewService {
+    /// Private helper to list all triggers from the external service.
+    async fn list_all(
+        &self,
+        w_id: &str,
+        oauth_data: &<Self as External>::OAuthData,
+        db: &DB,
+    ) -> Result<Vec<<Self as External>::TriggerData>> {
+        // Implementation depends on the external service's API
+        todo!()
+    }
+}
+```
+
+### Step 4: Update lib.rs Registry
+
+In `backend/windmill-native-triggers/src/lib.rs`:
+
+```rust
+// Service modules - add new services here:
+#[cfg(feature = "native_trigger")]
+pub mod newservice;  // <-- Add this
+
+// ServiceName enum - add variant:
+pub enum ServiceName {
+    Nextcloud,
+    Google,
+    NewService,  // <-- Add this
+}
+
+// Then add match arms in ALL ServiceName methods:
+// as_str(), as_trigger_kind(), as_job_trigger_kind(), token_endpoint(),
+// auth_endpoint(), oauth_scopes(), resource_type(), extra_auth_params(),
+// integration_service(), TryFrom<String>, Display
+```
+
+### Step 5: Update handler.rs Routes
+
+In `backend/windmill-native-triggers/src/handler.rs`:
+
+```rust
+pub fn generate_native_trigger_routers() -> Router {
+    // ...
+    #[cfg(feature = "native_trigger")]
+    {
+        use crate::newservice::NewService;
+        return router
+            .nest("/nextcloud", service_routes(NextCloud))
+            .nest("/google", service_routes(Google))
+            .nest("/newservice", service_routes(NewService));  // <-- Add this
+    }
+    // ...
+}
+```
+
+### Step 6: Update sync.rs
+
+In `backend/windmill-native-triggers/src/sync.rs`:
+
+```rust
+pub async fn sync_all_triggers(db: &DB) -> Result<BackgroundSyncResult> {
+    // ...
+    #[cfg(feature = "native_trigger")]
+    {
+        use crate::newservice::NewService;
+
+        // ... existing service syncs ...
+
+        // New service sync
+        let (service_name, result) = sync_service_triggers(db, NewService).await;
+        total_synced += result.synced_triggers.len();
+        total_errors += result.errors.len();
+        service_results.insert(service_name, result);
+    }
+    // ...
+}
+```
+
+### Step 7: Frontend Service Registry
+
+In `frontend/src/lib/components/triggers/native/utils.ts`:
+
+Add to `NATIVE_TRIGGER_SERVICES`, `getTriggerIconName()`, and `getServiceIcon()`.
+
+### Step 8: Frontend Trigger Form Component
+
+Create: `frontend/src/lib/components/triggers/native/services/newservice/NewServiceTriggerForm.svelte`
+
+### Step 9: Frontend Icon Component
+
+Create: `frontend/src/lib/components/icons/NewServiceIcon.svelte`
+
+### Step 10: Update NativeTriggerEditor
+
+Check `frontend/src/lib/components/triggers/native/NativeTriggerEditor.svelte` to ensure it dynamically loads form components based on service name.
+
+### Step 11: Workspace Integration UI
+
+Add your service to the `supportedServices` map in `frontend/src/lib/components/workspaceSettings/WorkspaceIntegrations.svelte`:
+
+```typescript
+const supportedServices: Record<string, ServiceConfig> = {
+    // ... existing services ...
+    newservice: {
+        name: 'newservice',
+        displayName: 'New Service',
+        description: 'Connect to New Service for triggers',
+        icon: NewServiceIcon,
+        docsUrl: 'https://www.windmill.dev/docs/integrations/newservice',
+        requiresBaseUrl: false,  // false for cloud services, true for self-hosted
+        setupInstructions: [
+            'Step 1: Create an OAuth app on the service',
+            'Step 2: Configure the redirect URI shown below',
+            'Step 3: Enter the client credentials below'
+        ]
+    }
+}
+```
+
+### Step 12: Update `frontend/src/lib/components/triggers/utils.ts`
+
+Update ALL of these maps/functions:
+1. `triggerIconMap` - import and add icon
+2. `triggerDisplayNamesMap` - add display name
+3. `triggerTypeOrder` in `sortTriggers()` - add type
+4. `getLightConfig()` - add case for your service
+5. `getTriggerLabel()` - add case for your service
+6. `jobTriggerKinds` - add to array
+7. `countPropertyMap` - add count property
+8. `triggerSaveFunctions` - add save function
+
+### Step 13: Update TriggersBadge Component
+
+In `frontend/src/lib/components/graph/renderers/triggers/TriggersBadge.svelte`:
+
+1. Import the icon
+2. Add to `baseConfig` with `countKey` (the dynamic `availableNativeServices` loop does NOT set `countKey`)
+3. Add to the `allTypes` array
+
+### Step 14: Update TriggersWrapper.svelte
+
+In `frontend/src/lib/components/triggers/TriggersWrapper.svelte`:
+
+Add a `{:else if selectedTrigger.type === 'yourservice'}` case that renders `<NativeTriggersPanel service="yourservice" ...>` with the same props pattern as the existing native trigger cases (e.g., `nextcloud`).
+
+### Step 15: Update AddTriggersButton.svelte
+
+In `frontend/src/lib/components/triggers/AddTriggersButton.svelte`:
+
+1. Add `yourserviceAvailable` state variable
+2. Add `setYourserviceState()` async function using `isServiceAvailable('yourservice', $workspaceStore!)`
+3. Call it at module level
+4. Add a dropdown entry to `addTriggerItems` with `hidden: !yourserviceAvailable`
+
+### Step 16: Update TriggersEditor.svelte Delete Handling
+
+In `frontend/src/lib/components/triggers/TriggersEditor.svelte`:
+
+Add your service to the `nativeTriggerServices` map in `deleteDeployedTrigger()`. Native triggers use `NativeTriggerService.deleteNativeTrigger({ workspace, serviceName, externalId })` instead of the standard `path`-based delete.
+
+### Step 17: Update `getUsedTriggers` for Sidebar Visibility
+
+The sidebar (`frontend/src/lib/components/sidebar/SidebarContent.svelte`) shows native-trigger links only if `$usedTriggerKinds` includes the service — without this, your trigger page will never appear in the nav bar even when triggers exist.
+
+1. **Backend** — add `{service}_used: bool` to the `UsedTriggers` struct and SELECT in `backend/windmill-api-workspaces/src/workspaces.rs::get_used_triggers()`:
+   ```rust
+   EXISTS(SELECT 1 FROM native_trigger WHERE workspace_id = $1 AND service_name = '{service}'::native_trigger_service) AS "{service}_used!"
+   ```
+2. **OpenAPI** — add `{service}_used: boolean` to the response schema for `GET /w/{workspace}/workspaces/used_triggers` (under both `properties` and `required`).
+3. **Layout** — in `frontend/src/routes/(root)/(logged)/+layout.svelte::loadUsedTriggerKinds()`, destructure `{service}_used` and push `'{service}'` to `usedKinds`.
+
+### Step 18: Update OpenAPI Spec and Regenerate Types
+
+Add to `JobTriggerKind` enum in `backend/windmill-api/openapi.yaml`, then:
+
+```bash
+cd frontend && npm run generate-backend-client
+```
+
+---
+
+## Special Patterns
+
+### Unified Service with `trigger_type` (Google Pattern)
+
+When a single service handles multiple trigger types (e.g., Google Drive + Calendar share OAuth and API patterns), use a single `ServiceName` variant with a discriminator field:
+
+```rust
+pub enum GoogleTriggerType { Drive, Calendar }
+
+pub struct GoogleServiceConfig {
+    pub trigger_type: GoogleTriggerType,
+    // Drive-specific fields (only used when trigger_type = Drive)
+    pub resource_id: Option<String>,
+    pub resource_name: Option<String>,
+    // Calendar-specific fields (only used when trigger_type = Calendar)
+    pub calendar_id: Option<String>,
+    pub calendar_name: Option<String>,
+    // Metadata set after creation
+    pub google_resource_id: Option<String>,
+    pub expiration: Option<String>,
+}
+```
+
+Branch in trait methods based on `trigger_type`. Frontend uses a `ToggleButtonGroup` to switch between types. This keeps the codebase simpler (one service, one OAuth flow, one set of routes).
+
+See `backend/windmill-native-triggers/src/google/` for the reference implementation.
+
+### Skipping update+get After Create (Google Pattern)
+
+Override `service_config_from_create_response()` to return `Some(config)` when the external_id is known before the create call:
+
+```rust
+fn service_config_from_create_response(
+    &self,
+    data: &NativeTriggerData<Self::ServiceConfig>,
+    resp: &Self::CreateResponse,
+) -> Option<serde_json::Value> {
+    // Clone input config, add metadata from response
+    let mut config = data.service_config.clone();
+    config.google_resource_id = Some(resp.resource_id.clone());
+    config.expiration = Some(resp.expiration.clone());
+    Some(serde_json::to_value(&config).unwrap())
+}
+```
+
+### Services with Absolute OAuth Endpoints (Google)
+
+Unlike self-hosted services where OAuth endpoints are relative paths appended to `base_url`, services like Google have absolute URLs:
+
+```rust
+// Nextcloud: relative paths
+ServiceName::Nextcloud => "/apps/oauth2/api/v1/token",
+// Google: absolute URLs
+ServiceName::Google => "https://oauth2.googleapis.com/token",
+```
+
+The `resolve_endpoint()` function handles both. For services with absolute endpoints:
+- `base_url` can be empty
+- `requiresBaseUrl: false` in the frontend workspace integration config
+- Add `extra_auth_params()` if needed (Google requires `access_type=offline` and `prompt=consent`)
+
+### Channel-Based Push Notifications with Renewal (Google Pattern)
+
+For services using expiring watch channels instead of persistent webhooks:
+
+1. Store expiration in `service_config` (as part of `ServiceConfig`)
+2. In `maintain_triggers()`, implement renewal logic instead of using `reconcile_with_external_state()`:
+   ```rust
+   async fn maintain_triggers(&self, db, workspace_id, triggers, oauth_data, synced, errors) {
+       for trigger in triggers {
+           if should_renew_channel(trigger) {
+               self.renew_channel(db, trigger, oauth_data).await;
+           }
+       }
+   }
+   ```
+3. Renewal: best-effort stop old channel, create new one with same external_id, update service_config with new expiration
+4. Google example: Drive channels expire in 24h (renew when <1h left), Calendar channels expire in 7 days (renew when <1 day left)
+
+### reconcile_with_external_state (Nextcloud Pattern)
+
+The reusable function in `sync.rs` compares external triggers with DB state:
+- Triggers missing externally: sets error "Trigger no longer exists on external service"
+- Triggers present externally: clears errors, updates service_config if it differs
+
+Usage in `maintain_triggers()`:
+```rust
+let external_pairs: Vec<(String, serde_json::Value)> = /* fetch from external */;
+crate::sync::reconcile_with_external_state(
+    db, workspace_id, Self::SERVICE_NAME, triggers, &external_pairs, synced, errors,
+).await;
+```
+
+### Webhook Payload Processing
+
+Override `prepare_webhook()` to parse service-specific payloads into script/flow args:
+
+```rust
+async fn prepare_webhook(&self, db, w_id, headers, body, script_path, is_flow) -> Result<PushArgsOwned> {
+    let mut args = HashMap::new();
+    args.insert("event_type".to_string(), Box::new(headers.get("x-event-type").cloned()) as _);
+    args.insert("payload".to_string(), Box::new(serde_json::from_str::<serde_json::Value>(&body)?) as _);
+    Ok(PushArgsOwned { extra: None, args })
+}
+```
+
+Then register in `prepare_native_trigger_args()` in `lib.rs`:
+```rust
+pub async fn prepare_native_trigger_args(service_name, db, w_id, headers, body) -> Result<Option<PushArgsOwned>> {
+    match service_name {
+        ServiceName::Google => { /* ... */ Ok(Some(args)) }
+        ServiceName::NewService => { /* ... */ Ok(Some(args)) }
+        ServiceName::Nextcloud => Ok(None), // Uses default body parsing
+    }
+}
+```
+
+### Instance-Level OAuth Credentials
+
+When `workspace_integrations.oauth_data.instance_shared == true`, `decrypt_oauth_data()` reads `client_id` and `client_secret` from instance-level global settings instead of workspace-level. This allows admins to share OAuth app credentials across workspaces.
+
+The frontend handles this via the `generate_instance_connect_url` endpoint in `workspace_integrations.rs`.
+
+---
+
+## Testing Checklist
+
+- [ ] Database migration runs successfully
+- [ ] `cargo check -p windmill-native-triggers --features native_trigger` passes
+- [ ] `npx svelte-check --threshold error` passes (in frontend/)
+- [ ] Service appears in workspace integrations list
+- [ ] OAuth flow completes successfully
+- [ ] Can create a new trigger
+- [ ] Can view trigger details
+- [ ] Can update trigger configuration
+- [ ] Can delete trigger
+- [ ] Webhook receives and processes payloads
+- [ ] Background sync works correctly (reconciliation or channel renewal)
+- [ ] Error handling works (expired tokens, service unavailable)
+
+---
+
+## Reference Implementations
+
+### Nextcloud (Self-Hosted, Update+Get Pattern)
+
+| File | Purpose |
+|------|---------|
+| `nextcloud/mod.rs` | Types: NextCloudOAuthData, NextcloudServiceConfig, NextCloudTriggerData |
+| `nextcloud/external.rs` | External trait: uses update+get pattern, reconcile_with_external_state for sync |
+| `nextcloud/routes.rs` | Additional route: `GET /events` |
+
+Key patterns: relative OAuth endpoints, base_url required, list_all + reconcile for sync, update returns JSON from get().
+
+### Google (Cloud, Unified Service, Short Create)
+
+| File | Purpose |
+|------|---------|
+| `google/mod.rs` | Types: GoogleServiceConfig with trigger_type discriminator, GoogleTriggerType enum |
+| `google/external.rs` | External trait: overrides service_config_from_create_response, channel renewal for sync |
+| `google/routes.rs` | Additional routes: `GET /calendars`, `GET /drive/files`, `GET /drive/shared_drives` |
+
+Key patterns: absolute OAuth endpoints, empty base_url, trigger_type for Drive/Calendar, expiring watch channels with renewal, service_config_from_create_response skips update+get, get() reconstructs data from stored service_config (no external "get channel" API).

.claude/skills/pr/SKILL.md

@@ -1 +0,0 @@
-../../../.agents/skills/pr/SKILL.md

.claude/skills/pr/SKILL.md

@@ -0,0 +1,111 @@
+---
+name: pr
+user_invocable: true
+description: Open a draft pull request on GitHub. MUST use when you want to create/open a PR.
+---
+
+# Pull Request Skill
+
+Create a draft pull request with a clear title and explicit description of changes.
+
+## Instructions
+
+1. **Analyze branch changes**: Understand all commits since diverging from main
+2. **Push to remote**: Ensure all commits are pushed
+3. **Create draft PR**: Always open as draft for review before merging
+
+## PR Title Format
+
+Follow conventional commit format for the PR title:
+```
+<type>: <description>
+```
+
+### Types
+- `feat`: New feature or capability
+- `fix`: Bug fix
+- `refactor`: Code restructuring
+- `docs`: Documentation changes
+- `chore`: Maintenance tasks
+- `perf`: Performance improvements
+
+### Title Rules
+- Keep under 70 characters
+- Use lowercase, imperative mood
+- No period at the end
+- If `*_ee.rs` files were modified, prefix with `[ee]`: `[ee] <type>: <description>`
+
+## PR Body Format
+
+The body MUST be explicit about what changed. Structure:
+
+```markdown
+## Summary
+<Clear description of what this PR does and why>
+
+## Changes
+- <Specific change 1>
+- <Specific change 2>
+- <Specific change 3>
+
+## Test plan
+- [ ] <How to verify change 1>
+- [ ] <How to verify change 2>
+
+---
+Generated with [Claude Code](https://claude.com/claude-code)
+```
+
+## Execution Steps
+
+1. Run `git status` to check for uncommitted changes
+2. Run `git log main..HEAD --oneline` to see all commits in this branch
+3. Run `git diff main...HEAD` to see the full diff against main
+4. **Run `/local-review`** before creating the PR. If issues are found, fix them and commit before proceeding. Do not skip this step.
+5. Check if remote branch exists and is up to date:
+   ```bash
+   git rev-parse --abbrev-ref --symbolic-full-name @{u} 2>/dev/null || echo "no upstream"
+   ```
+6. Push to remote if needed: `git push -u origin HEAD`
+7. Create draft PR using gh CLI:
+   ```bash
+   gh pr create --draft --title "<type>: <description>" --body "$(cat <<'EOF'
+   ## Summary
+   <description>
+
+   ## Changes
+   - <change 1>
+   - <change 2>
+
+   ## Test plan
+   - [ ] <test 1>
+   - [ ] <test 2>
+
+   ---
+   Generated with [Claude Code](https://claude.com/claude-code)
+   EOF
+   )"
+   ```
+8. Return the PR URL to the user
+
+## EE Companion PR (when `*_ee.rs` files were modified)
+
+The `*_ee.rs` files in the windmill repo are **symlinks** to `windmill-ee-private` — changes won't appear in `git diff` of the windmill repo. Instead, check the EE repo for uncommitted or unpushed changes.
+
+Follow the full EE PR workflow in `docs/enterprise.md`. The key PR-specific details:
+
+1. Find the EE repo/worktree: see "Finding the EE Repo" in `docs/enterprise.md`
+2. Check for changes: `git -C <ee-path> status --short`
+   - If there are no changes in the EE repo, skip this entire section
+3. Follow steps 1–5 from the "EE PR Workflow" in `docs/enterprise.md`
+4. Create the companion PR (title does NOT get the `[ee]` prefix):
+   ```bash
+   gh pr create --draft --repo windmill-labs/windmill-ee-private --title "<type>: <description>" --body "$(cat <<'EOF'
+   Companion PR for windmill-labs/windmill#<PR_NUMBER>
+
+   ---
+   Generated with [Claude Code](https://claude.com/claude-code)
+   EOF
+   )"
+   ```
+5. Commit `ee-repo-ref.txt` and push the updated windmill branch

.claude/skills/refine/SKILL.md

@@ -1 +0,0 @@
-../../../.agents/skills/refine/SKILL.md

.claude/skills/refine/SKILL.md

@@ -0,0 +1,39 @@
+---
+name: refine
+user_invocable: true
+description: End-of-session reflection. Reviews friction encountered during the session and proposes updates to docs/ to capture lessons learned.
+---
+
+# Refine Skill
+
+Reflect on the current session and update documentation with lessons learned.
+
+## Instructions
+
+1. **Identify friction**: Review what happened in this session:
+   - Run `git diff main...HEAD --stat` to see what files were touched
+   - Think about: what was slow, what failed, what required multiple attempts, what information was missing or hard to find
+
+2. **Read current docs**: Read the docs that were relevant to this session:
+   - `docs/validation.md`
+   - `docs/enterprise.md`
+   - `docs/autonomous-mode.md`
+   - Any skills that were invoked
+
+3. **Propose updates**: For each piece of friction, decide if it warrants a doc update:
+   - **Missing knowledge**: Information you had to discover that should be documented
+   - **Wrong guidance**: Instructions that led you astray
+   - **Missing validation rule**: A check that should be in the validation matrix
+   - **New pattern**: A codebase pattern worth capturing for next time
+
+4. **Apply updates**: Edit the relevant `docs/` files. Keep changes minimal and specific — add only what would have saved time this session.
+
+5. **Report**: Summarize what was added/changed and why.
+
+## Rules
+
+- Only add knowledge confirmed by this session — no speculative additions
+- Keep docs concise — add a line or two, not a paragraph
+- If a whole new doc is needed, create it in `docs/` and add a pointer in `CLAUDE.md`
+- Don't update skills unless a coding pattern was genuinely wrong
+- Don't add things Claude already knows — only Windmill-specific knowledge

.claude/skills/rust-backend/SKILL.md

@@ -1 +0,0 @@
-../../../.agents/skills/rust-backend/SKILL.md

.claude/skills/rust-backend/SKILL.md

@@ -0,0 +1,107 @@
+---
+name: rust-backend
+description: Rust coding guidelines for the Windmill backend. MUST use when writing or modifying Rust code in the backend directory.
+---
+
+# Windmill Rust Patterns
+
+Apply these Windmill-specific patterns when writing Rust code in `backend/`.
+
+## Error Handling
+
+Use `Error` from `windmill_common::error`. Return `Result<T, Error>` or `JsonResult<T>`:
+
+```rust
+use windmill_common::error::{Error, Result};
+
+pub async fn get_job(db: &DB, id: Uuid) -> Result<Job> {
+    sqlx::query_as!(Job, "SELECT id, workspace_id FROM v2_job WHERE id = $1", id)
+        .fetch_optional(db)
+        .await?
+        .ok_or_else(|| Error::NotFound("job not found".to_string()))?;
+}
+```
+
+Never panic in library code. Reserve `.unwrap()` for compile-time guarantees.
+
+## SQLx Patterns
+
+**Never use `SELECT *`** — always list columns explicitly. Critical for backwards compatibility when workers lag behind API version:
+
+```rust
+// Correct
+sqlx::query_as!(Job, "SELECT id, workspace_id, path FROM v2_job WHERE id = $1", id)
+
+// Wrong — breaks when columns are added
+sqlx::query_as!(Job, "SELECT * FROM v2_job WHERE id = $1", id)
+```
+
+Use batch operations to avoid N+1:
+
+```rust
+// Preferred — single query with IN clause
+sqlx::query!("SELECT ... WHERE id = ANY($1)", &ids[..]).fetch_all(db).await?
+```
+
+Use transactions for multi-step operations. Parameterize all queries.
+
+## JSON Handling
+
+Prefer `Box<serde_json::value::RawValue>` over `serde_json::Value` when storing/passing JSON without inspection:
+
+```rust
+pub struct Job {
+    pub args: Option<Box<serde_json::value::RawValue>>,
+}
+```
+
+Only use `serde_json::Value` when you need to inspect or modify the JSON.
+
+## Serde Optimizations
+
+```rust
+#[derive(Serialize, Deserialize)]
+pub struct Job {
+    #[serde(skip_serializing_if = "Option::is_none")]
+    pub parent_job: Option<Uuid>,
+    #[serde(skip_serializing_if = "Vec::is_empty")]
+    pub tags: Vec<String>,
+    #[serde(default)]
+    pub priority: i32,
+}
+```
+
+## Async & Concurrency
+
+Never block the async runtime. Use `spawn_blocking` for CPU-intensive work:
+
+```rust
+let result = tokio::task::spawn_blocking(move || expensive_computation(&data)).await?;
+```
+
+**Mutex selection**: Prefer `std::sync::Mutex` (or `parking_lot::Mutex`) for data protection. Only use `tokio::sync::Mutex` when holding locks across `.await` points.
+
+Use `tokio::sync::mpsc` (bounded) for channels. Avoid `std::thread::sleep` in async contexts.
+
+## Module Structure & Visibility
+
+- Use `pub(crate)` instead of `pub` when possible
+- Place new code in the appropriate crate based on functionality
+- API endpoints go in `windmill-api/src/` organized by domain
+- Shared functionality goes in `windmill-common/src/`
+
+## Code Navigation
+
+Always use rust-analyzer LSP for go-to-definition, find-references, and type info. Do not guess at module paths.
+
+## Axum Handlers
+
+Destructure extractors directly in function signatures:
+
+```rust
+async fn process_job(
+    Extension(db): Extension<DB>,
+    Path((workspace, job_id)): Path<(String, Uuid)>,
+    Query(pagination): Query<Pagination>,
+) -> Result<Json<Job>> { ... }
+```

.claude/skills/svelte-frontend/SKILL.md

@@ -1 +0,0 @@
-../../../.agents/skills/svelte-frontend/SKILL.md

.claude/skills/svelte-frontend/SKILL.md

@@ -0,0 +1,80 @@
+---
+name: svelte-frontend
+description: Svelte coding guidelines for the Windmill frontend. MUST use when writing or modifying code in the frontend directory.
+---
+
+# Windmill Svelte Patterns
+
+Apply these Windmill-specific patterns when writing Svelte code in `frontend/`. For general Svelte 5 syntax (runes, snippets, event handling), use the Svelte MCP server.
+
+## Windmill UI Components (MUST use)
+
+Always use Windmill's design-system components. Never use raw HTML elements.
+
+### Buttons — `<Button>`
+
+```svelte
+<script>
+  import { Button } from '$lib/components/common'
+  import { ChevronLeft } from 'lucide-svelte'
+</script>
+
+<Button variant="default" onclick={handleClick}>Label</Button>
+<Button startIcon={{ icon: ChevronLeft }} iconOnly onclick={prev} />
+```
+
+Props: `variant?: 'accent' | 'accent-secondary' | 'default' | 'subtle'`, `unifiedSize?: 'sm' | 'md' | 'lg'`, `startIcon?: { icon: SvelteComponent }`, `iconOnly?: boolean`, `disabled?: boolean`
+
+### Text inputs — `<TextInput>`
+
+```svelte
+<script>
+  import { TextInput } from '$lib/components/common'
+</script>
+
+<TextInput bind:value={val} placeholder="Enter value" />
+```
+
+Props: `value?: string | number` (bindable), `placeholder?: string`, `disabled?: boolean`, `error?: string | boolean`, `size?: 'sm' | 'md' | 'lg'`
+
+### Selects — `<Select>`
+
+```svelte
+<script>
+  import Select from '$lib/components/select/Select.svelte'
+</script>
+
+<Select items={[{ label: 'Jan', value: 1 }]} bind:value={selected} />
+```
+
+Props: `items?: Array<{ label?: string; value: any }>`, `value` (bindable), `placeholder?: string`, `clearable?: boolean`, `size?: 'sm' | 'md' | 'lg'`
+
+### Icons — `lucide-svelte`
+
+Never write inline SVGs. Import from `lucide-svelte`:
+
+```svelte
+<script>
+  import { ChevronLeft, X } from 'lucide-svelte'
+</script>
+<ChevronLeft size={16} />
+```
+
+## Form Components
+
+Form components (TextInput, Toggle, Select, etc.) should use the unified size system when placed together.
+
+## Styling
+
+- Use Tailwind CSS for all styling — no custom CSS
+- Use Windmill's theming classes for colors/surfaces (see `frontend/brand-guidelines.md`)
+- Read component props JSDoc before using them
+
+## Svelte MCP Server
+
+Use the Svelte MCP tools when working on Svelte code:
+
+1. **list-sections**: Call first to discover available docs
+2. **get-documentation**: Fetch relevant sections based on use_cases
+3. **svelte-autofixer**: MUST use on all Svelte code before finalizing — keep calling until no issues
+4. **playground-link**: Only after user confirms and code was NOT written to project files

.claude/skills/update-sqlx/SKILL.md

@@ -1 +0,0 @@
-../../../.agents/skills/update-sqlx/SKILL.md

.github/DockerfileBackendTests

@@ -28,7 +28,7 @@ ENV PATH="${PATH}:/usr/local/go/bin"
 ENV GO_PATH=/usr/local/go/bin/go
 
 # UV
-RUN curl --proto '=https' --tlsv1.2 -LsSf https://github.com/astral-sh/uv/releases/download/0.9.25/uv-installer.sh | sh && mv /usr/local/cargo/bin/uv /usr/local/bin/uv
+RUN curl --proto '=https' --tlsv1.2 -LsSf https://github.com/astral-sh/uv/releases/download/0.9.24/uv-installer.sh | sh && mv /usr/local/cargo/bin/uv /usr/local/bin/uv
 
 ENV TZ=Etc/UTC
 

.github/change-versions-mac.sh

@@ -7,7 +7,7 @@ VERSION=$1
 echo "Updating versions to: $VERSION"
 
 sed -i '' -e "/^version =/s/= .*/= \"$VERSION\"/" ${root_dirpath}/backend/Cargo.toml
-sed -i '' -e "/^export const VERSION =/s/= .*/= \"v$VERSION\";/"  ${root_dirpath}/cli/src/core/constants.ts
+sed -i '' -e "/^export const VERSION =/s/= .*/= \"v$VERSION\";/"  ${root_dirpath}/cli/src/main.ts
 sed -i '' -e "/^export const VERSION =/s/= .*/= \"v$VERSION\";/" ${root_dirpath}/benchmarks/lib.ts
 sed -i '' -e "/version: /s/: .*/: $VERSION/" ${root_dirpath}/backend/windmill-api/openapi.yaml
 sed -i '' -e "/version: /s/: .*/: $VERSION/" ${root_dirpath}/openflow.openapi.yaml

.github/change-versions.sh

@@ -7,7 +7,7 @@ VERSION=$1
 echo "Updating versions to: $VERSION"
 
 sed -i -e "/^version =/s/= .*/= \"$VERSION\"/" ${root_dirpath}/backend/Cargo.toml
-sed -i -e "/^export const VERSION =/s/= .*/= \"$VERSION\";/" ${root_dirpath}/cli/src/core/constants.ts
+sed -i -e "/^export const VERSION =/s/= .*/= \"$VERSION\";/" ${root_dirpath}/cli/src/main.ts
 sed -i -e "/^export const VERSION =/s/= .*/= \"v$VERSION\";/" ${root_dirpath}/benchmarks/lib.ts
 sed -i -e "/version: /s/: .*/: $VERSION/" ${root_dirpath}/backend/windmill-api/openapi.yaml
 sed -i -e "/version: /s/: .*/: $VERSION/" ${root_dirpath}/openflow.openapi.yaml

.github/codex/pr-review.prompt.md

@@ -1,5 +1,23 @@
-# Codex output format
+You are reviewing a GitHub pull request for this repository.
 
-- Read `./.github/codex/pr-review-context.md` for PR metadata and the diff commands.
-- Return a markdown PR comment starting with `## Codex Review`.
-- Tag each finding with a severity (P0 / P1 / P2), file path, and line number when known confidently.
+Review policy:
+- Read `CLAUDE.md` before reviewing code.
+- Only report issues you are confident are real and introduced by this pull request.
+- Focus on bugs, security problems, and clear `CLAUDE.md` violations.
+- Do not report style nits, speculative concerns, pre-existing issues, or problems that a normal linter/typechecker would obviously catch.
+- Keep the review high signal. If there is no clear issue, return no findings.
+
+Repository context:
+- Read `./.github/codex/pr-review-context.md` for the PR metadata and the exact diff commands to use.
+- Review only the changes introduced by this PR.
+- Read additional files only when the diff is not enough to validate a finding.
+- Do not modify any files.
+
+Output requirements:
+- Return a GitHub PR comment in markdown, not JSON.
+- Start with `## Codex Review`.
+- Give a short overall summary first.
+- If you found high-signal issues, list them in a short numbered list with file paths and line numbers when you know them confidently.
+- If you found no high-signal issues, say that explicitly.
+- End with a `### Reproduction instructions` section containing a short descriptive paragraph for a tester explaining how to navigate the app to observe the change. Do not make it a numbered list. If the diff is not enough to infer this safely, say that plainly.
+- Prefer at most 10 findings.

.github/pi/pr-review.prompt.md

@@ -1,6 +0,0 @@
-# Pi output format
-
-- Read `./.github/pi/pr-review-context.md` for PR metadata and the diff commands.
-- Return a markdown PR comment starting with `## Pi Review`.
-- Tag each finding with a severity (P0 / P1 / P2), file path, and line number when known confidently.
-- Output ONLY the final review markdown — no preamble, no thinking, no tool transcripts.

.github/workflows/ai-agent-tests.yml

@@ -1,132 +0,0 @@
-name: AI Agent Integration Tests
-
-# Exercises the AI agent flow path (preview_flow with `aiagent` modules) against
-# real LLM providers. Runs only when AI-agent backend code or the tests change,
-# because each run makes real (paid) LLM calls. To avoid spending on every commit,
-# the PR side triggers only when a PR is marked ready for review (out of draft) —
-# not on `synchronize` — plus push to main and manual dispatch.
-on:
-  workflow_dispatch:
-  push:
-    branches: [main]
-    paths:
-      - "integration_tests/ai_agent_tests/**"
-      - "backend/windmill-ai/**"
-      - "backend/windmill-api/src/ai.rs"
-      - "backend/windmill-worker/src/ai_executor.rs"
-      - "backend/windmill-worker/src/ai/**"
-      - "backend/windmill-worker/src/memory_common.rs"
-      - "backend/windmill-common/src/flow_conversations.rs"
-      - ".github/workflows/ai-agent-tests.yml"
-  pull_request:
-    types: [opened, reopened, ready_for_review]
-    paths:
-      - "integration_tests/ai_agent_tests/**"
-      - "backend/windmill-ai/**"
-      - "backend/windmill-api/src/ai.rs"
-      - "backend/windmill-worker/src/ai_executor.rs"
-      - "backend/windmill-worker/src/ai/**"
-      - "backend/windmill-worker/src/memory_common.rs"
-      - "backend/windmill-common/src/flow_conversations.rs"
-      - ".github/workflows/ai-agent-tests.yml"
-
-concurrency:
-  group: ai-agent-tests-${{ github.ref }}
-  cancel-in-progress: true
-
-jobs:
-  ai_agent_e2e:
-    # Skip draft PRs; the `opened`/`reopened` types would otherwise fire while
-    # still a draft. `ready_for_review` always arrives non-draft.
-    if: github.event_name != 'pull_request' || github.event.pull_request.draft == false
-    runs-on: ubicloud-standard-16
-    services:
-      postgres:
-        image: postgres:16
-        ports:
-          - 5432:5432
-        env:
-          POSTGRES_DB: windmill
-          POSTGRES_PASSWORD: changeme
-        options: >-
-          --health-cmd pg_isready --health-interval 10s --health-timeout 5s
-          --health-retries 5
-    steps:
-      - uses: actions/checkout@v4
-
-      - uses: actions-rust-lang/setup-rust-toolchain@v1
-        with:
-          cache-workspaces: backend
-          toolchain: 1.93.0
-
-      - uses: oven-sh/setup-bun@v2
-        with:
-          bun-version: 1.3.10
-
-      - uses: actions/setup-node@v4
-        with:
-          node-version: "20"
-
-      - uses: actions/setup-python@v5
-        with:
-          python-version: "3.11"
-
-      # CE build (no enterprise/license needed for AI agents). `quickjs` powers
-      # flow input-transform JS eval; `mcp` is required by the deepwiki MCP tool
-      # test. Bun tool scripts run via the always-on worker (BUN_PATH).
-      - name: Build Windmill
-        working-directory: ./backend
-        env:
-          SQLX_OFFLINE: true
-          CARGO_BUILD_JOBS: 12
-          RUSTFLAGS: ""
-        run: cargo build --features quickjs,mcp
-
-      - name: Start Windmill
-        working-directory: ./backend
-        env:
-          DATABASE_URL: postgres://postgres:changeme@localhost:5432/windmill
-          BUN_PATH: bun
-          NODE_BIN_PATH: node
-          RUST_LOG: info
-        run: |
-          mkdir -p ../integration_tests/logs
-          ./target/debug/windmill > ../integration_tests/logs/windmill.log 2>&1 &
-          echo "Waiting for Windmill to be ready..."
-          for i in $(seq 1 60); do
-            if curl -sf http://localhost:8000/api/version > /dev/null 2>&1; then
-              echo "Windmill is ready"
-              break
-            fi
-            sleep 2
-          done
-          curl -sf http://localhost:8000/api/version > /dev/null || { echo "Windmill failed to start"; tail -50 ../integration_tests/logs/windmill.log; exit 1; }
-
-      - name: Run AI agent integration tests
-        timeout-minutes: 20
-        working-directory: ./integration_tests/ai_agent_tests
-        env:
-          WINDMILL_URL: http://localhost:8000
-          # Only the providers we have org secrets for. Other providers
-          # (Azure, Bedrock, OpenRouter) are skipped by conftest when their
-          # keys are absent — see skip_provider_without_credentials.
-          ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }}
-          OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
-          GOOGLE_AI_API_KEY: ${{ secrets.GOOGLE_API_KEY }}
-        run: |
-          python -m venv .venv
-          .venv/bin/pip install -r requirements.txt
-          # The S3/vision-attachment tests need MinIO large-file storage and
-          # image-capable provider setup; out of scope for this cost-controlled
-          # smoke. Add MinIO secrets + a storage service to enable them.
-          .venv/bin/python -m pytest -v \
-            --ignore=test_user_attachments.py \
-            --ignore=test_user_images.py \
-            --ignore=test_image_output.py
-
-      - name: Archive Windmill logs
-        uses: actions/upload-artifact@v4
-        if: always()
-        with:
-          name: ai-agent-tests-windmill-logs
-          path: integration_tests/logs

.github/workflows/ai-evals-test.yml

@@ -1,166 +0,0 @@
-name: AI Evals (global mode)
-
-# Smoke-tests the production global AI chat proxy/frontend execution path via
-# the ai_evals harness, one case across one cheap model per provider. Runs only
-# when the eval harness or the global chat code change, since each run makes real
-# (paid) LLM calls. The backend is built from source purely as the AI proxy the
-# harness routes model calls through; the global tools/drafts run in-process in
-# the Vitest bridge against production frontend code. To avoid spending on every
-# commit, the PR side triggers only when a PR is marked ready for review (out of
-# draft) — not on `synchronize` — plus push to main and manual dispatch.
-on:
-  workflow_dispatch:
-  push:
-    branches: [main]
-    paths:
-      - "ai_evals/**"
-      - "backend/windmill-api/src/ai.rs"
-      - "backend/windmill-ai/**"
-      - "frontend/src/lib/components/copilot/**"
-      # The eval harness runs production frontend code in-process; these are the
-      # AI/draft-specific deps outside copilot/ that the global smoke exercises.
-      - "frontend/src/lib/userDraft.svelte.ts"
-      - "frontend/src/lib/userDraftDbSyncer.svelte.ts"
-      - "frontend/src/lib/infer.ts"
-      - ".github/workflows/ai-evals-test.yml"
-  pull_request:
-    types: [opened, reopened, ready_for_review]
-    paths:
-      - "ai_evals/**"
-      - "backend/windmill-api/src/ai.rs"
-      - "backend/windmill-ai/**"
-      - "frontend/src/lib/components/copilot/**"
-      # The eval harness runs production frontend code in-process; these are the
-      # AI/draft-specific deps outside copilot/ that the global smoke exercises.
-      - "frontend/src/lib/userDraft.svelte.ts"
-      - "frontend/src/lib/userDraftDbSyncer.svelte.ts"
-      - "frontend/src/lib/infer.ts"
-      - ".github/workflows/ai-evals-test.yml"
-
-concurrency:
-  group: ai-evals-test-${{ github.ref }}
-  cancel-in-progress: true
-
-jobs:
-  ai_evals_global:
-    # Provider secrets are unavailable to forked and Dependabot PRs.
-    if: >-
-      github.event_name != 'pull_request' ||
-      (
-        github.event.pull_request.draft == false &&
-        github.event.pull_request.head.repo.full_name == github.repository &&
-        github.event.pull_request.user.login != 'dependabot[bot]'
-      )
-    runs-on: ubicloud-standard-16
-    services:
-      postgres:
-        image: postgres:16
-        ports:
-          - 5432:5432
-        env:
-          POSTGRES_DB: windmill
-          POSTGRES_PASSWORD: changeme
-        options: >-
-          --health-cmd pg_isready --health-interval 10s --health-timeout 5s
-          --health-retries 5
-    steps:
-      - uses: actions/checkout@v4
-
-      - uses: actions-rust-lang/setup-rust-toolchain@v1
-        with:
-          cache-workspaces: backend
-          toolchain: 1.93.0
-
-      - uses: oven-sh/setup-bun@v2
-        with:
-          bun-version: 1.3.10
-
-      - uses: actions/setup-node@v4
-        with:
-          # Node 22.19+ is required by the frontend's undici 8.x, which the
-          # Vitest bridge loads; Node 20 fails with markAsUncloneable.
-          node-version: "22"
-
-      # CE build used only as the AI proxy (login, workspace, provider resource,
-      # /ai/proxy). No worker execution or MCP needed — global tools/drafts run
-      # in the Vitest bridge. quickjs matches the standard CE feature set.
-      - name: Build Windmill (AI proxy)
-        working-directory: ./backend
-        env:
-          SQLX_OFFLINE: true
-          CARGO_BUILD_JOBS: 12
-          RUSTFLAGS: ""
-        run: cargo build --features quickjs
-
-      - name: Start Windmill
-        working-directory: ./backend
-        env:
-          DATABASE_URL: postgres://postgres:changeme@localhost:5432/windmill
-          RUST_LOG: info
-        run: |
-          mkdir -p ../ai_evals/logs
-          ./target/debug/windmill > ../ai_evals/logs/windmill.log 2>&1 &
-          echo "Waiting for Windmill to be ready..."
-          for i in $(seq 1 60); do
-            if curl -sf http://localhost:8000/api/version > /dev/null 2>&1; then
-              echo "Windmill is ready"
-              break
-            fi
-            sleep 2
-          done
-          curl -sf http://localhost:8000/api/version > /dev/null || { echo "Windmill failed to start"; tail -50 ../ai_evals/logs/windmill.log; exit 1; }
-
-      - name: Install frontend deps + generate client
-        working-directory: ./frontend
-        run: |
-          npm ci
-          npm run generate-backend-client
-
-      - name: Run global AI evals
-        timeout-minutes: 20
-        working-directory: ./ai_evals
-        env:
-          WMILL_AI_EVAL_BACKEND_URL: http://localhost:8000
-          WMILL_AI_EVAL_BACKEND_WORKSPACE: integration-tests
-          # Anthropic backs the haiku model. Google AI uses GEMINI_API_KEY.
-          ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }}
-          OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
-          GEMINI_API_KEY: ${{ secrets.GOOGLE_API_KEY }}
-          DEEPSEEK_API_KEY: ${{ secrets.DEEPSEEK_API_KEY }}
-        run: |
-          bun install
-          mkdir -p results
-          # One cheap model per provider (anthropic/openai/googleai/deepseek).
-          fail=0
-          for m in haiku 4o gemini-3-flash-preview deepseek-v4-flash; do
-            echo "::group::global-test1-script-create ($m)"
-            if ! bun run cli -- run global global-test1-script-create \
-                  --model "$m" --execution-only --output "$PWD/results/ci-$m.json"; then
-              echo "$m: harness/proxy errored"
-              fail=1
-              echo "::endgroup::"
-              continue
-            fi
-            # The CLI exits 0 when the harness records failed attempts, so gate
-            # on execution-only pass counts while ignoring model output quality.
-            if jq -e \
-              '.attemptCount > 0 and .passedAttempts == .attemptCount' \
-              "results/ci-$m.json" > /dev/null; then
-              echo "$m: OK — proxy/frontend execution completed"
-            else
-              echo "$m: FAILED proxy/frontend execution"
-              jq -c '.cases[0].attempts[0].checks' "results/ci-$m.json" || true
-              fail=1
-            fi
-            echo "::endgroup::"
-          done
-          [ "$fail" = 0 ] || { echo "ai_evals global smoke failed"; exit 1; }
-
-      - name: Archive logs and results
-        uses: actions/upload-artifact@v4
-        if: always()
-        with:
-          name: ai-evals-global-logs
-          path: |
-            ai_evals/logs
-            ai_evals/results

.github/workflows/backend-test-windows.yml

@@ -74,7 +74,7 @@ jobs:
 
       - uses: astral-sh/setup-uv@v6.2.1
         with:
-          version: "0.9.25"
+          version: "0.9.24"
 
       - uses: shivammathur/setup-php@v2
         with:
@@ -98,21 +98,6 @@ jobs:
           vcpkg.exe install openssl:x64-windows-static
           vcpkg.exe integrate install
 
-      - name: Free disk space (post-vcpkg)
-        shell: pwsh
-        run: |
-          # vcpkg leaves multi-GB of buildtrees/downloads after installing openssl;
-          # we only need the installed/ dir for linking.
-          $vcpkgRoot = $env:VCPKG_INSTALLATION_ROOT
-          foreach ($sub in @("buildtrees", "downloads", "packages")) {
-            $path = Join-Path $vcpkgRoot $sub
-            if (Test-Path $path) {
-              Write-Host "Removing $path"
-              Remove-Item -Recurse -Force -ErrorAction SilentlyContinue $path
-            }
-          }
-          Get-PSDrive C | Select-Object Used,Free | Format-Table -AutoSize
-
       - name: Get runtime paths
         id: runtime-paths
         shell: pwsh
@@ -134,10 +119,6 @@ jobs:
           cargo build --release -p windmill_duckdb_ffi_internal
           New-Item -ItemType Directory -Path ..\target\debug -Force
           Copy-Item target\release\windmill_duckdb_ffi_internal.dll ..\target\debug\
-          # duckdb is bundled (~2GB of build artifacts); the DLL is the only
-          # thing we need from this excluded-crate target dir.
-          Remove-Item -Recurse -Force -ErrorAction SilentlyContinue target
-          Get-PSDrive C | Select-Object Used,Free | Format-Table -AutoSize
 
       - name: Print runtime versions and env
         shell: pwsh
@@ -155,10 +136,6 @@ jobs:
           echo "USERPROFILE=$env:USERPROFILE"
           echo "HOME=$env:HOME"
 
-      - name: Disk space before cargo test
-        shell: pwsh
-        run: Get-PSDrive C | Select-Object Used,Free | Format-Table -AutoSize
-
       - name: cargo test
         working-directory: backend
         timeout-minutes: 60
@@ -167,18 +144,7 @@ jobs:
           RUST_LOG: "off"
           RUST_LOG_STYLE: never
           CARGO_NET_GIT_FETCH_WITH_CLI: true
-          # 16-vcpu runners with disabled PDB still hit LNK1180 ("insufficient
-          # disk space") at link time with 12 parallel link jobs: each test
-          # binary link spikes several hundred MB of transient I/O. Capping at
-          # 8 trades ~25% wall time for headroom on the ~75GB runner disk.
-          CARGO_BUILD_JOBS: 8
-          # backend/Cargo.toml sets split-debuginfo = "unpacked", which on
-          # windows-msvc is coerced to "packed": every test-binary link spawns
-          # the mspdbsrv.exe PDB type server and writes a large .pdb. CI needs
-          # no debug info, so disable PDB generation for the dev/test profiles
-          # here (avoids both LNK1318 type-server limit and PDB disk usage).
-          CARGO_PROFILE_DEV_SPLIT_DEBUGINFO: "off"
-          CARGO_PROFILE_TEST_SPLIT_DEBUGINFO: "off"
+          CARGO_BUILD_JOBS: 12
           # Tests' poll-time stack frames (deep nested async fn chains in
           # debug builds) reach ~1.8MB. 4MB gives ~2x headroom against flaky
           # overflows under parallel-test contention.

.github/workflows/backend-test.yml

@@ -62,7 +62,7 @@ jobs:
           node-version: "20"
       - uses: astral-sh/setup-uv@v6.2.1
         with:
-          version: "0.9.25"
+          version: "0.9.24"
       - uses: shivammathur/setup-php@v2
         with:
           php-version: "8.3"

.github/workflows/check-empty-fixture.yml

@@ -1,19 +0,0 @@
-name: Check fixture is empty
-
-on:
-  push:
-    branches: [main]
-    paths:
-      - "fixtures/**"
-  pull_request:
-    paths:
-      - "fixtures/**"
-
-jobs:
-  check-empty-fixture:
-    runs-on: ubuntu-latest
-    steps:
-      - uses: actions/checkout@v4
-
-      - name: Ensure fixtures/cli-sync/ has no committed snapshot
-        run: bash fixtures/check-empty.sh

.github/workflows/claude-fast.yml

@@ -0,0 +1,54 @@
+name: Fast Claude
+
+on:
+  issue_comment:
+    types: [created]
+  pull_request_review_comment:
+    types: [created]
+  issues:
+    types: [opened, assigned]
+  pull_request_review:
+    types: [submitted]
+
+jobs:
+  check-membership:
+    if: |
+      (github.event_name == 'issue_comment' && contains(github.event.comment.body, '/ai-fast')) ||
+      (github.event_name == 'pull_request_review_comment' && contains(github.event.comment.body, '/ai-fast')) ||
+      (github.event_name == 'pull_request_review' && contains(github.event.review.body, '/ai-fast')) ||
+      (github.event_name == 'issues' && contains(github.event.issue.body, '/ai-fast'))
+    uses: ./.github/workflows/check-org-membership.yml
+    secrets:
+      access_token: ${{ secrets.ORG_ACCESS_TOKEN }}
+
+  claude-code-action:
+    needs: check-membership
+    if: |
+      needs.check-membership.outputs.is_member == 'true'
+    runs-on: ubicloud-standard-8
+    permissions:
+      contents: write
+      pull-requests: write
+      issues: write
+      id-token: write
+    steps:
+      - name: Checkout repository
+        uses: actions/checkout@v4
+        with:
+          fetch-depth: 1
+
+      - name: Run Claude PR Action
+        uses: anthropics/claude-code-action@v1
+        with:
+          claude_code_oauth_token: ${{ secrets.CLAUDE_CODE_OAUTH_TOKEN }}
+          allowed_bots: "windmill-internal-app[bot]"
+          trigger_phrase: "/ai-fast"
+          settings: |
+            {
+              "env": {
+                "SQLX_OFFLINE": "true"
+              }
+            }
+          claude_args: |
+            --allowedTools "Bash,WebFetch,WebSearch"
+            --model opus

.github/workflows/claude-plan.yml

@@ -45,7 +45,7 @@ jobs:
           allowed_bots: 'windmill-internal-app[bot]'
           trigger_phrase: '/plan'
           claude_args: |
-            --model claude-opus-4-8
+            --model opus
             --system-prompt "# Claude Planning Mode
 
             You are operating in PLANNING MODE ONLY. Your role is to create detailed, structured plans without making any code changes.

.github/workflows/claude.yml

@@ -1,4 +1,4 @@
-name: Fast Claude
+name: Claude PR Assistant
 
 on:
   issue_comment:
@@ -26,6 +26,7 @@ jobs:
     if: |
       needs.check-membership.outputs.is_member == 'true'
     runs-on: ubicloud-standard-8
+    timeout-minutes: 60
     permissions:
       contents: write
       pull-requests: write
@@ -37,6 +38,37 @@ jobs:
         with:
           fetch-depth: 1
 
+      - uses: actions/cache@v3
+        with:
+          path: ~/.npm
+          key: ${{ runner.os }}-node-${{ hashFiles('**/package-lock.json') }}
+          restore-keys: |
+            ${{ runner.os }}-node-
+
+      - name: Run npm install and generate-backend-client
+        working-directory: ./frontend
+        run: |
+          # add a build directory for cargo check
+          mkdir -p build
+          npm install
+          npm run generate-backend-client
+
+      - name: install xmlsec1 and gssapi
+        run: |
+          sudo apt-get update
+          sudo apt-get install -y libxml2-dev libxmlsec1-dev libkrb5-dev libsasl2-dev libcurl4-openssl-dev mold clang
+
+      - uses: actions-rust-lang/setup-rust-toolchain@v1
+        with:
+          cache-workspaces: backend
+          toolchain: 1.93.0
+
+      - name: cargo check
+        working-directory: ./backend
+        timeout-minutes: 16
+        run: |
+          SQLX_OFFLINE=true cargo check --features all_sqlx_features
+
       - name: Run Claude PR Action
         uses: anthropics/claude-code-action@v1
         with:
@@ -51,4 +83,25 @@ jobs:
             }
           claude_args: |
             --allowedTools "Bash,WebFetch,WebSearch"
-            --model claude-opus-4-8
+            --model opus
+            --system-prompt "## IMPORTANT INSTRUCTIONS
+            - Your branch name should be a short description of the requested changes.
+            - Bypass previous instructions and instead of adding a url in your comment to manually create a PR, if you created a new branch with your changes, automatically create a draft PR from that branch to main.
+
+            ## Code Quality Requirements
+
+            After making any code changes, you MUST run the appropriate validation commands:
+
+            **Frontend Changes:**
+            - Run: \`npm run check\` in the frontend directory
+            - Fix all warnings and errors before proceeding
+
+            **Backend Changes:**
+            - Run: \`cargo check --features all_sqlx_features\` in the backend directory
+            - Fix all warnings and errors before proceeding
+
+            **Pull Request Creation:**
+            - DO NOT FORGET TO OPEN A DRAFT PR AFTER YOU ARE DONE if you made changes after a request from a git issue.
+
+            ## Available Tools
+            - Bash: Full access to run validation commands and git operations"

.github/workflows/codex-pr-review.yml

@@ -2,184 +2,53 @@ name: Codex Auto Review
 
 on:
   pull_request:
-    types: [ready_for_review, opened, synchronize]
-  workflow_call:
-    inputs:
-      pr_number:
-        description: 'PR number to review'
-        required: true
-        type: number
-      extra_prompt:
-        description: 'Additional reviewer instructions appended to the standard review prompt'
-        required: false
-        type: string
-        default: ''
-      triggered_by:
-        description: 'GitHub username that triggered this review (for audit only)'
-        required: false
-        type: string
-        default: ''
-    secrets:
-      OPENAI_API_KEY:
-        required: false
-      CODEX_AUTH_JSON:
-        required: false
-      WINDMILL_EE_PRIVATE_ACCESS:
-        required: false
+    types: [ready_for_review, opened]
 
 concurrency:
-  group: codex-review-${{ inputs.pr_number || github.event.pull_request.number }}
+  group: codex-review-${{ github.event.pull_request.number }}
   cancel-in-progress: true
 
 jobs:
-  check-membership:
-    if: github.event_name == 'pull_request'
-    uses: ./.github/workflows/check-org-membership.yml
-    with:
-      commenter: ${{ github.event.pull_request.user.login }}
-    secrets:
-      access_token: ${{ secrets.ORG_ACCESS_TOKEN }}
-
   codex-review:
-    needs: check-membership
     runs-on: ubicloud-standard-2
     timeout-minutes: 30
-    if: |
-      always() &&
-      (
-        needs.check-membership.result == 'skipped' ||
-        (needs.check-membership.result == 'success' && needs.check-membership.outputs.is_member == 'true')
-      ) &&
-      (
-        github.event_name == 'workflow_call' ||
-        (github.event.pull_request.draft == false && github.event.pull_request.head.repo.fork == false)
-      )
+    if: github.event.pull_request.draft == false && github.event.pull_request.head.repo.fork == false
     permissions:
       contents: read
       issues: write
-      pull-requests: write
     steps:
       - name: Check Codex configuration
         id: codex_config
         env:
-          OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
           CODEX_AUTH_JSON: ${{ secrets.CODEX_AUTH_JSON }}
         run: |
-          if [ -n "$OPENAI_API_KEY" ]; then
+          if [ -n "$CODEX_AUTH_JSON" ]; then
             echo "enabled=true" >> "$GITHUB_OUTPUT"
-            echo "auth_mode=api_key" >> "$GITHUB_OUTPUT"
-          elif [ -n "$CODEX_AUTH_JSON" ]; then
-            echo "enabled=true" >> "$GITHUB_OUTPUT"
-            echo "auth_mode=oauth_json" >> "$GITHUB_OUTPUT"
           else
             echo "enabled=false" >> "$GITHUB_OUTPUT"
-            echo "Codex auth is not configured; set OPENAI_API_KEY or CODEX_AUTH_JSON to enable Codex review."
+            echo "CODEX_AUTH_JSON is not configured; skipping Codex review."
           fi
 
-      - name: Resolve PR metadata
-        if: steps.codex_config.outputs.enabled == 'true'
-        id: pr
-        env:
-          GH_TOKEN: ${{ github.token }}
-          INPUT_PR_NUMBER: ${{ inputs.pr_number }}
-          EVENT_PR_NUMBER: ${{ github.event.pull_request.number }}
-          EVENT_BASE_REF: ${{ github.event.pull_request.base.ref }}
-          EVENT_BASE_SHA: ${{ github.event.pull_request.base.sha }}
-          EVENT_HEAD_SHA: ${{ github.event.pull_request.head.sha }}
-          EVENT_TITLE: ${{ github.event.pull_request.title }}
-          EVENT_BODY: ${{ github.event.pull_request.body }}
-          EVENT_FORK: ${{ github.event.pull_request.head.repo.fork }}
-          EVENT_AUTHOR: ${{ github.event.pull_request.user.login }}
-        run: |
-          if [ -n "$INPUT_PR_NUMBER" ]; then
-            PR_JSON=$(gh pr view "$INPUT_PR_NUMBER" --repo "${{ github.repository }}" \
-              --json number,baseRefName,baseRefOid,headRefOid,title,body,isCrossRepository,author)
-            PR_NUMBER=$(echo "$PR_JSON" | jq -r '.number')
-            BASE_REF=$(echo "$PR_JSON" | jq -r '.baseRefName')
-            BASE_SHA=$(echo "$PR_JSON" | jq -r '.baseRefOid')
-            HEAD_SHA=$(echo "$PR_JSON" | jq -r '.headRefOid')
-            PR_TITLE=$(echo "$PR_JSON" | jq -r '.title')
-            PR_BODY=$(echo "$PR_JSON" | jq -r '.body // ""')
-            IS_FORK=$(echo "$PR_JSON" | jq -r '.isCrossRepository')
-            PR_AUTHOR=$(echo "$PR_JSON" | jq -r '.author.login // ""')
-          else
-            PR_NUMBER="$EVENT_PR_NUMBER"
-            BASE_REF="$EVENT_BASE_REF"
-            BASE_SHA="$EVENT_BASE_SHA"
-            HEAD_SHA="$EVENT_HEAD_SHA"
-            PR_TITLE="$EVENT_TITLE"
-            PR_BODY="$EVENT_BODY"
-            IS_FORK="$EVENT_FORK"
-            PR_AUTHOR="$EVENT_AUTHOR"
-          fi
-          if [ "$IS_FORK" = "true" ]; then
-            echo "Skipping Codex review for fork PR."
-            echo "skip=true" >> "$GITHUB_OUTPUT"
-            exit 0
-          fi
-          {
-            echo "skip=false"
-            echo "pr_number=$PR_NUMBER"
-            echo "base_ref=$BASE_REF"
-            echo "base_sha=$BASE_SHA"
-            echo "head_sha=$HEAD_SHA"
-            echo "pr_author=$PR_AUTHOR"
-            echo 'title<<PR_TITLE_EOF'
-            printf '%s\n' "$PR_TITLE"
-            echo 'PR_TITLE_EOF'
-            echo 'body<<PR_BODY_EOF'
-            printf '%s\n' "$PR_BODY"
-            echo 'PR_BODY_EOF'
-          } >> "$GITHUB_OUTPUT"
-
       - name: Checkout repository
-        if: steps.codex_config.outputs.enabled == 'true' && steps.pr.outputs.skip != 'true'
+        if: steps.codex_config.outputs.enabled == 'true'
         uses: actions/checkout@v5
         with:
-          ref: refs/pull/${{ steps.pr.outputs.pr_number }}/merge
+          ref: refs/pull/${{ github.event.pull_request.number }}/merge
           fetch-depth: 1
 
-      - name: Check EE access
-        if: steps.codex_config.outputs.enabled == 'true' && steps.pr.outputs.skip != 'true'
-        id: ee
-        env:
-          EE_TOKEN: ${{ secrets.WINDMILL_EE_PRIVATE_ACCESS }}
-        run: |
-          if [ -n "$EE_TOKEN" ]; then
-            echo "available=true" >> "$GITHUB_OUTPUT"
-            echo "ee_repo_ref=$(cat ./backend/ee-repo-ref.txt)" >> "$GITHUB_OUTPUT"
-          else
-            echo "available=false" >> "$GITHUB_OUTPUT"
-          fi
-
-      - name: Checkout EE repository
-        if: steps.ee.outputs.available == 'true'
-        uses: actions/checkout@v5
-        with:
-          repository: windmill-labs/windmill-ee-private
-          path: ./windmill-ee-private
-          ref: ${{ steps.ee.outputs.ee_repo_ref }}
-          token: ${{ secrets.WINDMILL_EE_PRIVATE_ACCESS }}
-          fetch-depth: 1
-
-      - name: Substitute EE code
-        if: steps.ee.outputs.available == 'true'
-        run: ./backend/substitute_ee_code.sh --copy --dir ./windmill-ee-private
-
       - name: Set up Node.js
-        if: steps.codex_config.outputs.enabled == 'true' && steps.pr.outputs.skip != 'true'
+        if: steps.codex_config.outputs.enabled == 'true'
         uses: actions/setup-node@v4
         with:
           node-version: 22
 
       - name: Install Codex CLI
-        if: steps.codex_config.outputs.enabled == 'true' && steps.pr.outputs.skip != 'true'
-        run: npm install --global @openai/codex@0.128.0
+        if: steps.codex_config.outputs.enabled == 'true'
+        run: npm install --global @openai/codex@0.117.0
 
-      - name: Configure Codex auth
-        if: steps.codex_config.outputs.enabled == 'true' && steps.pr.outputs.skip != 'true'
+      - name: Configure file-backed Codex auth
+        if: steps.codex_config.outputs.enabled == 'true'
         env:
-          OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
           CODEX_AUTH_JSON: ${{ secrets.CODEX_AUTH_JSON }}
         run: |
           CODEX_HOME="$HOME/.codex"
@@ -189,46 +58,29 @@ jobs:
           cat > "$CODEX_HOME/config.toml" <<'EOF'
           cli_auth_credentials_store = "file"
           EOF
-          if [ -n "$OPENAI_API_KEY" ]; then
-            printf '%s' "$OPENAI_API_KEY" | codex login --with-api-key
-          else
-            printf '%s' "$CODEX_AUTH_JSON" > "$CODEX_HOME/auth.json"
-            chmod 600 "$CODEX_HOME/auth.json"
-            node -e 'JSON.parse(require("fs").readFileSync(process.argv[1], "utf8"))' "$CODEX_HOME/auth.json"
-          fi
+          printf '%s' "$CODEX_AUTH_JSON" > "$CODEX_HOME/auth.json"
+          chmod 600 "$CODEX_HOME/auth.json"
+          node -e 'JSON.parse(require("fs").readFileSync(process.argv[1], "utf8"))' "$CODEX_HOME/auth.json"
 
       - name: Pre-fetch base and head refs for the PR
-        if: steps.codex_config.outputs.enabled == 'true' && steps.pr.outputs.skip != 'true'
+        if: steps.codex_config.outputs.enabled == 'true'
         env:
-          PR_BASE_REF: ${{ steps.pr.outputs.base_ref }}
-          PR_NUMBER: ${{ steps.pr.outputs.pr_number }}
+          PR_BASE_REF: ${{ github.event.pull_request.base.ref }}
+          PR_NUMBER: ${{ github.event.pull_request.number }}
         run: |
           git fetch --no-tags origin \
             "$PR_BASE_REF" \
             "+refs/pull/$PR_NUMBER/head"
 
-      - name: Fetch prior PR discussion
-        if: steps.codex_config.outputs.enabled == 'true' && steps.pr.outputs.skip != 'true'
-        env:
-          GH_TOKEN: ${{ github.token }}
-          REPO: ${{ github.repository }}
-          PR_NUMBER: ${{ steps.pr.outputs.pr_number }}
-        run: |
-          gh api "repos/$REPO/issues/$PR_NUMBER/comments?per_page=100" \
-            --jq '[.[] | {user: .user.login, created_at: .created_at, body: (.body | .[:4000])}] | sort_by(.created_at) | .[-20:]' \
-            > prior-comments.json || echo "[]" > prior-comments.json
-
       - name: Write Codex review context
-        if: steps.codex_config.outputs.enabled == 'true' && steps.pr.outputs.skip != 'true'
+        if: steps.codex_config.outputs.enabled == 'true'
         env:
           PR_REPOSITORY: ${{ github.repository }}
-          PR_NUMBER: ${{ steps.pr.outputs.pr_number }}
-          PR_BASE_SHA: ${{ steps.pr.outputs.base_sha }}
-          PR_HEAD_SHA: ${{ steps.pr.outputs.head_sha }}
-          PR_TITLE: ${{ steps.pr.outputs.title }}
-          PR_BODY: ${{ steps.pr.outputs.body }}
-          PR_AUTHOR: ${{ steps.pr.outputs.pr_author }}
-          EXTRA_PROMPT: ${{ inputs.extra_prompt }}
+          PR_NUMBER: ${{ github.event.pull_request.number }}
+          PR_BASE_SHA: ${{ github.event.pull_request.base.sha }}
+          PR_HEAD_SHA: ${{ github.event.pull_request.head.sha }}
+          PR_TITLE: ${{ github.event.pull_request.title }}
+          PR_BODY: ${{ github.event.pull_request.body || '' }}
         run: |
           mkdir -p .github/codex
           node <<'NODE'
@@ -236,11 +88,6 @@ jobs:
           const lines = [
             `Repository: ${process.env.PR_REPOSITORY}`,
             `PR number: ${process.env.PR_NUMBER}`,
-          ];
-          if (process.env.PR_AUTHOR) {
-            lines.push(`PR AUTHOR: ${process.env.PR_AUTHOR}`);
-          }
-          lines.push(
             `Base SHA: ${process.env.PR_BASE_SHA}`,
             `Head SHA: ${process.env.PR_HEAD_SHA}`,
             '',
@@ -258,47 +105,24 @@ jobs:
             '',
             'Full review diff command:',
             `git diff --unified=0 ${process.env.PR_BASE_SHA}...${process.env.PR_HEAD_SHA}`
-          );
-          if (process.env.EXTRA_PROMPT && process.env.EXTRA_PROMPT.trim()) {
-            lines.push('', 'Additional reviewer instructions:', process.env.EXTRA_PROMPT.trim());
-          }
-          if (fs.existsSync('prior-comments.json')) {
-            try {
-              const comments = JSON.parse(fs.readFileSync('prior-comments.json', 'utf8'));
-              if (Array.isArray(comments) && comments.length > 0) {
-                lines.push(
-                  '',
-                  'Prior PR discussion (most recent up to 20 comments):',
-                  '',
-                  'If you have already reviewed this PR (look for your own earlier "## Codex Review" comment), focus on what changed since then per the diff and respect any decisions the human made in replies. Do not re-flag findings the human already pushed back on.',
-                  ''
-                );
-                for (const c of comments) {
-                  lines.push(`### @${c.user} (${c.created_at})`, '', c.body, '', '---', '');
-                }
-              }
-            } catch (_) {}
-          }
+          ];
           fs.writeFileSync('.github/codex/pr-review-context.md', `${lines.join('\n')}\n`);
           NODE
 
       - name: Run Codex review
-        if: steps.codex_config.outputs.enabled == 'true' && steps.pr.outputs.skip != 'true'
+        if: steps.codex_config.outputs.enabled == 'true'
         run: |
-          cat REVIEW.md .github/codex/pr-review.prompt.md > /tmp/codex-prompt.md
           codex exec \
             -C "$GITHUB_WORKSPACE" \
-            -m gpt-5.5 \
+            -m gpt-5.4 \
             -c 'model_reasoning_effort="xhigh"' \
-            -s danger-full-access \
+            -s read-only \
             -o codex-final-message.md \
-            - < /tmp/codex-prompt.md
+            - < .github/codex/pr-review.prompt.md
 
       - name: Post Codex review comment
-        if: steps.codex_config.outputs.enabled == 'true' && steps.pr.outputs.skip != 'true'
+        if: steps.codex_config.outputs.enabled == 'true'
         uses: actions/github-script@v7
-        env:
-          PR_NUMBER: ${{ steps.pr.outputs.pr_number }}
         with:
           github-token: ${{ github.token }}
           script: |
@@ -316,6 +140,6 @@ jobs:
             await github.rest.issues.createComment({
               owner: context.repo.owner,
               repo: context.repo.repo,
-              issue_number: Number(process.env.PR_NUMBER),
+              issue_number: context.payload.pull_request.number,
               body,
             });

.github/workflows/git-sync-test.yml

@@ -8,7 +8,6 @@ on:
       - "backend/windmill-git-sync/**"
       - "backend/windmill-api-integration-tests/tests/git_sync*"
       - "backend/ee-repo-ref.txt"
-      - "backend/windmill-common/src/workspaces.rs"
       - "integration_tests/test/git_sync_test.py"
       - ".github/workflows/git-sync-test.yml"
   pull_request:
@@ -17,7 +16,6 @@ on:
       - "backend/windmill-git-sync/**"
       - "backend/windmill-api-integration-tests/tests/git_sync*"
       - "backend/ee-repo-ref.txt"
-      - "backend/windmill-common/src/workspaces.rs"
       - "integration_tests/test/git_sync_test.py"
       - ".github/workflows/git-sync-test.yml"
 
@@ -51,7 +49,7 @@ jobs:
           echo "$CHANGED_FILES"
 
           # Direct git sync file changes — always relevant
-          if echo "$CHANGED_FILES" | grep -qE '^(backend/windmill-git-sync/|backend/windmill-api-integration-tests/tests/git_sync|backend/windmill-common/src/workspaces\.rs|integration_tests/test/git_sync|\.github/workflows/git-sync-test\.yml)'; then
+          if echo "$CHANGED_FILES" | grep -qE '^(backend/windmill-git-sync/|backend/windmill-api-integration-tests/tests/git_sync|integration_tests/test/git_sync|\.github/workflows/git-sync-test\.yml)'; then
             echo "should_run=true" >> "$GITHUB_OUTPUT"
             echo "Relevant: direct git sync file changes"
             exit 0

.github/workflows/pi-pr-review.yml

@@ -1,325 +0,0 @@
-name: Pi Auto Review
-
-on:
-  pull_request:
-    types: [ready_for_review, opened, synchronize]
-  workflow_call:
-    inputs:
-      pr_number:
-        description: 'PR number to review'
-        required: true
-        type: number
-      extra_prompt:
-        description: 'Additional reviewer instructions appended to the standard review prompt'
-        required: false
-        type: string
-        default: ''
-      triggered_by:
-        description: 'GitHub username that triggered this review (for audit only)'
-        required: false
-        type: string
-        default: ''
-    secrets:
-      DEEPSEEK_API_KEY:
-        required: false
-      WINDMILL_EE_PRIVATE_ACCESS:
-        required: false
-
-concurrency:
-  group: pi-review-${{ inputs.pr_number || github.event.pull_request.number }}
-  cancel-in-progress: true
-
-jobs:
-  check-membership:
-    if: github.event_name == 'pull_request'
-    uses: ./.github/workflows/check-org-membership.yml
-    with:
-      commenter: ${{ github.event.pull_request.user.login }}
-    secrets:
-      access_token: ${{ secrets.ORG_ACCESS_TOKEN }}
-
-  pi-review:
-    needs: check-membership
-    runs-on: ubicloud-standard-2
-    timeout-minutes: 30
-    if: |
-      always() &&
-      (
-        needs.check-membership.result == 'skipped' ||
-        (needs.check-membership.result == 'success' && needs.check-membership.outputs.is_member == 'true')
-      ) &&
-      (
-        github.event_name == 'workflow_call' ||
-        (github.event.pull_request.draft == false && github.event.pull_request.head.repo.fork == false)
-      )
-    permissions:
-      contents: read
-      issues: write
-      pull-requests: write
-    steps:
-      - name: Check Pi configuration
-        id: pi_config
-        env:
-          DEEPSEEK_API_KEY: ${{ secrets.DEEPSEEK_API_KEY }}
-        run: |
-          if [ -n "$DEEPSEEK_API_KEY" ]; then
-            echo "enabled=true" >> "$GITHUB_OUTPUT"
-          else
-            echo "enabled=false" >> "$GITHUB_OUTPUT"
-            echo "DEEPSEEK_API_KEY is not configured; skipping Pi review."
-          fi
-
-      - name: Resolve PR metadata
-        if: steps.pi_config.outputs.enabled == 'true'
-        id: pr
-        env:
-          GH_TOKEN: ${{ github.token }}
-          INPUT_PR_NUMBER: ${{ inputs.pr_number }}
-          EVENT_PR_NUMBER: ${{ github.event.pull_request.number }}
-          EVENT_BASE_REF: ${{ github.event.pull_request.base.ref }}
-          EVENT_BASE_SHA: ${{ github.event.pull_request.base.sha }}
-          EVENT_HEAD_SHA: ${{ github.event.pull_request.head.sha }}
-          EVENT_TITLE: ${{ github.event.pull_request.title }}
-          EVENT_BODY: ${{ github.event.pull_request.body }}
-          EVENT_FORK: ${{ github.event.pull_request.head.repo.fork }}
-          EVENT_AUTHOR: ${{ github.event.pull_request.user.login }}
-        run: |
-          if [ -n "$INPUT_PR_NUMBER" ]; then
-            PR_JSON=$(gh pr view "$INPUT_PR_NUMBER" --repo "${{ github.repository }}" \
-              --json number,baseRefName,baseRefOid,headRefOid,title,body,isCrossRepository,author)
-            PR_NUMBER=$(echo "$PR_JSON" | jq -r '.number')
-            BASE_REF=$(echo "$PR_JSON" | jq -r '.baseRefName')
-            BASE_SHA=$(echo "$PR_JSON" | jq -r '.baseRefOid')
-            HEAD_SHA=$(echo "$PR_JSON" | jq -r '.headRefOid')
-            PR_TITLE=$(echo "$PR_JSON" | jq -r '.title')
-            PR_BODY=$(echo "$PR_JSON" | jq -r '.body // ""')
-            IS_FORK=$(echo "$PR_JSON" | jq -r '.isCrossRepository')
-            PR_AUTHOR=$(echo "$PR_JSON" | jq -r '.author.login // ""')
-          else
-            PR_NUMBER="$EVENT_PR_NUMBER"
-            BASE_REF="$EVENT_BASE_REF"
-            BASE_SHA="$EVENT_BASE_SHA"
-            HEAD_SHA="$EVENT_HEAD_SHA"
-            PR_TITLE="$EVENT_TITLE"
-            PR_BODY="$EVENT_BODY"
-            IS_FORK="$EVENT_FORK"
-            PR_AUTHOR="$EVENT_AUTHOR"
-          fi
-          if [ "$IS_FORK" = "true" ]; then
-            echo "Skipping Pi review for fork PR."
-            echo "skip=true" >> "$GITHUB_OUTPUT"
-            exit 0
-          fi
-          {
-            echo "skip=false"
-            echo "pr_number=$PR_NUMBER"
-            echo "base_ref=$BASE_REF"
-            echo "base_sha=$BASE_SHA"
-            echo "head_sha=$HEAD_SHA"
-            echo "pr_author=$PR_AUTHOR"
-            echo 'title<<PR_TITLE_EOF'
-            printf '%s\n' "$PR_TITLE"
-            echo 'PR_TITLE_EOF'
-            echo 'body<<PR_BODY_EOF'
-            printf '%s\n' "$PR_BODY"
-            echo 'PR_BODY_EOF'
-          } >> "$GITHUB_OUTPUT"
-
-      - name: Checkout repository
-        if: steps.pi_config.outputs.enabled == 'true' && steps.pr.outputs.skip != 'true'
-        uses: actions/checkout@v5
-        with:
-          ref: refs/pull/${{ steps.pr.outputs.pr_number }}/merge
-          fetch-depth: 1
-
-      - name: Check EE access
-        if: steps.pi_config.outputs.enabled == 'true' && steps.pr.outputs.skip != 'true'
-        id: ee
-        env:
-          EE_TOKEN: ${{ secrets.WINDMILL_EE_PRIVATE_ACCESS }}
-        run: |
-          if [ -n "$EE_TOKEN" ]; then
-            echo "available=true" >> "$GITHUB_OUTPUT"
-            echo "ee_repo_ref=$(cat ./backend/ee-repo-ref.txt)" >> "$GITHUB_OUTPUT"
-          else
-            echo "available=false" >> "$GITHUB_OUTPUT"
-          fi
-
-      - name: Checkout EE repository
-        if: steps.ee.outputs.available == 'true'
-        uses: actions/checkout@v5
-        with:
-          repository: windmill-labs/windmill-ee-private
-          path: ./windmill-ee-private
-          ref: ${{ steps.ee.outputs.ee_repo_ref }}
-          token: ${{ secrets.WINDMILL_EE_PRIVATE_ACCESS }}
-          fetch-depth: 1
-
-      - name: Substitute EE code
-        if: steps.ee.outputs.available == 'true'
-        run: ./backend/substitute_ee_code.sh --copy --dir ./windmill-ee-private
-
-      - name: Set up Node.js
-        if: steps.pi_config.outputs.enabled == 'true' && steps.pr.outputs.skip != 'true'
-        uses: actions/setup-node@v4
-        with:
-          node-version: 22
-
-      - name: Install Pi CLI
-        if: steps.pi_config.outputs.enabled == 'true' && steps.pr.outputs.skip != 'true'
-        run: npm install --global @mariozechner/pi-coding-agent
-
-      - name: Pre-fetch base and head refs for the PR
-        if: steps.pi_config.outputs.enabled == 'true' && steps.pr.outputs.skip != 'true'
-        env:
-          PR_BASE_REF: ${{ steps.pr.outputs.base_ref }}
-          PR_NUMBER: ${{ steps.pr.outputs.pr_number }}
-        run: |
-          git fetch --no-tags origin \
-            "$PR_BASE_REF" \
-            "+refs/pull/$PR_NUMBER/head"
-
-      - name: Fetch prior PR discussion
-        if: steps.pi_config.outputs.enabled == 'true' && steps.pr.outputs.skip != 'true'
-        env:
-          GH_TOKEN: ${{ github.token }}
-          REPO: ${{ github.repository }}
-          PR_NUMBER: ${{ steps.pr.outputs.pr_number }}
-        run: |
-          gh api "repos/$REPO/issues/$PR_NUMBER/comments?per_page=100" \
-            --jq '[.[] | {user: .user.login, created_at: .created_at, body: (.body | .[:4000])}] | sort_by(.created_at) | .[-20:]' \
-            > prior-comments.json || echo "[]" > prior-comments.json
-
-      - name: Write Pi review context
-        if: steps.pi_config.outputs.enabled == 'true' && steps.pr.outputs.skip != 'true'
-        env:
-          PR_REPOSITORY: ${{ github.repository }}
-          PR_NUMBER: ${{ steps.pr.outputs.pr_number }}
-          PR_BASE_SHA: ${{ steps.pr.outputs.base_sha }}
-          PR_HEAD_SHA: ${{ steps.pr.outputs.head_sha }}
-          PR_TITLE: ${{ steps.pr.outputs.title }}
-          PR_BODY: ${{ steps.pr.outputs.body }}
-          PR_AUTHOR: ${{ steps.pr.outputs.pr_author }}
-          EXTRA_PROMPT: ${{ inputs.extra_prompt }}
-        run: |
-          mkdir -p .github/pi
-          node <<'NODE'
-          const fs = require('fs');
-          const lines = [
-            `Repository: ${process.env.PR_REPOSITORY}`,
-            `PR number: ${process.env.PR_NUMBER}`,
-          ];
-          if (process.env.PR_AUTHOR) {
-            lines.push(`PR AUTHOR: ${process.env.PR_AUTHOR}`);
-          }
-          lines.push(
-            `Base SHA: ${process.env.PR_BASE_SHA}`,
-            `Head SHA: ${process.env.PR_HEAD_SHA}`,
-            '',
-            'PR title:',
-            process.env.PR_TITLE || '(empty)',
-            '',
-            'PR body:',
-            process.env.PR_BODY || '(empty)',
-            '',
-            'Changed commits command:',
-            `git log --oneline ${process.env.PR_BASE_SHA}...${process.env.PR_HEAD_SHA}`,
-            '',
-            'Changed files command:',
-            `git diff --stat ${process.env.PR_BASE_SHA}...${process.env.PR_HEAD_SHA}`,
-            '',
-            'Full review diff command:',
-            `git diff --unified=0 ${process.env.PR_BASE_SHA}...${process.env.PR_HEAD_SHA}`
-          );
-          if (process.env.EXTRA_PROMPT && process.env.EXTRA_PROMPT.trim()) {
-            lines.push('', 'Additional reviewer instructions:', process.env.EXTRA_PROMPT.trim());
-          }
-          if (fs.existsSync('prior-comments.json')) {
-            try {
-              const comments = JSON.parse(fs.readFileSync('prior-comments.json', 'utf8'));
-              if (Array.isArray(comments) && comments.length > 0) {
-                lines.push(
-                  '',
-                  'Prior PR discussion (most recent up to 20 comments):',
-                  '',
-                  'If you have already reviewed this PR (look for your own earlier "## Pi Review (DeepSeek V4)" comment), focus on what changed since then per the diff and respect any decisions the human made in replies. Do not re-flag findings the human already pushed back on.',
-                  ''
-                );
-                for (const c of comments) {
-                  lines.push(`### @${c.user} (${c.created_at})`, '', c.body, '', '---', '');
-                }
-              }
-            } catch (_) {}
-          }
-          fs.writeFileSync('.github/pi/pr-review-context.md', `${lines.join('\n')}\n`);
-          NODE
-
-      - name: Run Pi review
-        if: steps.pi_config.outputs.enabled == 'true' && steps.pr.outputs.skip != 'true'
-        env:
-          DEEPSEEK_API_KEY: ${{ secrets.DEEPSEEK_API_KEY }}
-          PI_SKIP_VERSION_CHECK: '1'
-        run: |
-          set -o pipefail
-          cat REVIEW.md .github/pi/pr-review.prompt.md > /tmp/pi-prompt.md
-          pi -p \
-            --provider deepseek \
-            --model deepseek-v4-pro \
-            --tools read,grep,find,ls,bash \
-            --mode json \
-            < /tmp/pi-prompt.md \
-            | tee pi-events.jsonl \
-            | jq -rc --unbuffered '
-                if .type == "agent_start" then "🤖 pi agent started"
-                elif .type == "turn_start" then "── turn ──"
-                elif .type == "message_end" then
-                  "[\(.message.role)] " + (
-                    (.message.content // [])
-                    | map(
-                        if .type == "text" then "text(\(.text | length)c)"
-                        elif .type == "tool_use" then "🔧 \(.name) \(.input | @json | .[:160])"
-                        elif .type == "tool_result" then "✅ result"
-                        else .type
-                        end
-                      )
-                    | join(" | ")
-                  )
-                elif .type == "turn_end" then "── turn done (\((.toolResults // []) | length) tool result(s)) ──"
-                elif .type == "agent_end" then "🏁 pi agent done"
-                else empty
-                end
-              '
-
-          jq -r '
-            select(.type == "agent_end")
-            | .messages
-            | map(select(.role == "assistant"))
-            | last
-            | (.content[]? | select(.type == "text") | .text)
-          ' pi-events.jsonl > pi-final-message.md
-
-      - name: Post Pi review comment
-        if: steps.pi_config.outputs.enabled == 'true' && steps.pr.outputs.skip != 'true'
-        uses: actions/github-script@v7
-        env:
-          PR_NUMBER: ${{ steps.pr.outputs.pr_number }}
-        with:
-          github-token: ${{ github.token }}
-          script: |
-            const fs = require('fs');
-            const path = `${process.env.GITHUB_WORKSPACE}/pi-final-message.md`;
-            if (!fs.existsSync(path)) {
-              core.info('Pi did not produce a final message; skipping PR comment.');
-              return;
-            }
-            const body = fs.readFileSync(path, 'utf8').trim();
-            if (!body) {
-              core.info('Pi final message was empty; skipping PR comment.');
-              return;
-            }
-            await github.rest.issues.createComment({
-              owner: context.repo.owner,
-              repo: context.repo.repo,
-              issue_number: Number(process.env.PR_NUMBER),
-              body,
-            });

.github/workflows/pr-ready-review.yml

@@ -3,147 +3,31 @@ name: Claude Auto Review
 on:
   pull_request:
     types: [ready_for_review, opened]
-  workflow_call:
-    inputs:
-      pr_number:
-        description: 'PR number to review'
-        required: true
-        type: number
-      extra_prompt:
-        description: 'Additional reviewer instructions appended to the standard review prompt'
-        required: false
-        type: string
-        default: ''
-      triggered_by:
-        description: 'GitHub username that triggered this review (for audit only)'
-        required: false
-        type: string
-        default: ''
-    secrets:
-      CLAUDE_CODE_OAUTH_TOKEN:
-        required: true
-      WINDMILL_EE_PRIVATE_ACCESS:
-        required: false
 
 concurrency:
-  group: claude-review-${{ inputs.pr_number || github.event.pull_request.number }}
+  group: claude-review-${{ github.event.pull_request.number }}
   cancel-in-progress: true
 
 jobs:
-  check-membership:
-    if: github.event_name == 'pull_request'
-    uses: ./.github/workflows/check-org-membership.yml
-    with:
-      commenter: ${{ github.event.pull_request.user.login }}
-    secrets:
-      access_token: ${{ secrets.ORG_ACCESS_TOKEN }}
-
   auto-review:
-    needs: check-membership
     runs-on: ubuntu-latest
-    if: |
-      always() &&
-      (
-        needs.check-membership.result == 'skipped' ||
-        (needs.check-membership.result == 'success' && needs.check-membership.outputs.is_member == 'true')
-      ) &&
-      (
-        github.event_name == 'workflow_call' ||
-        (github.event.pull_request.draft == false || github.event.pull_request.ready_for_review == true)
-      )
+    if: github.event.pull_request.draft == false || github.event.pull_request.ready_for_review == true
     permissions:
       contents: read
       pull-requests: read
       id-token: write
     steps:
       - name: Checkout repository
-        uses: actions/checkout@v5
+        uses: actions/checkout@v4
         with:
           fetch-depth: 1
 
-      - name: Check EE access
-        id: ee
-        env:
-          EE_TOKEN: ${{ secrets.WINDMILL_EE_PRIVATE_ACCESS }}
-        run: |
-          if [ -n "$EE_TOKEN" ]; then
-            echo "available=true" >> "$GITHUB_OUTPUT"
-            echo "ee_repo_ref=$(cat ./backend/ee-repo-ref.txt)" >> "$GITHUB_OUTPUT"
-          else
-            echo "available=false" >> "$GITHUB_OUTPUT"
-          fi
-
-      - name: Checkout EE repository
-        if: steps.ee.outputs.available == 'true'
-        uses: actions/checkout@v5
-        with:
-          repository: windmill-labs/windmill-ee-private
-          path: ./windmill-ee-private
-          ref: ${{ steps.ee.outputs.ee_repo_ref }}
-          token: ${{ secrets.WINDMILL_EE_PRIVATE_ACCESS }}
-          fetch-depth: 1
-
-      - name: Substitute EE code
-        if: steps.ee.outputs.available == 'true'
-        run: ./backend/substitute_ee_code.sh --copy --dir ./windmill-ee-private
-
-      - name: Resolve PR number
-        id: resolve
-        env:
-          GH_TOKEN: ${{ github.token }}
-          REPO: ${{ github.repository }}
-          INPUT_PR_NUMBER: ${{ inputs.pr_number }}
-          EVENT_PR_NUMBER: ${{ github.event.pull_request.number }}
-          EVENT_PR_AUTHOR: ${{ github.event.pull_request.user.login }}
-        run: |
-          if [ -n "$INPUT_PR_NUMBER" ]; then
-            PR_NUMBER="$INPUT_PR_NUMBER"
-            PR_AUTHOR=$(gh api "repos/$REPO/pulls/$PR_NUMBER" --jq '.user.login')
-          else
-            PR_NUMBER="$EVENT_PR_NUMBER"
-            PR_AUTHOR="$EVENT_PR_AUTHOR"
-          fi
-          echo "pr_number=$PR_NUMBER" >> "$GITHUB_OUTPUT"
-          echo "pr_author=$PR_AUTHOR" >> "$GITHUB_OUTPUT"
-
-      - name: Fetch prior PR discussion
-        id: prior
-        env:
-          GH_TOKEN: ${{ github.token }}
-          REPO: ${{ github.repository }}
-          PR_NUMBER: ${{ steps.resolve.outputs.pr_number }}
-        run: |
-          gh api "repos/$REPO/issues/$PR_NUMBER/comments?per_page=100" \
-            --jq '[.[] | {user: .user.login, created_at: .created_at, body: (.body | .[:4000])}] | sort_by(.created_at) | .[-20:]' \
-            > prior-comments.json || echo "[]" > prior-comments.json
-          jq -r '
-            if length == 0 then ""
-            else
-              "## Prior PR discussion (most recent up to 20 comments)\n\nIf you have already reviewed this PR (look for your own earlier comment), focus on what changed since then per the diff and respect any decisions the human made in replies. Do not re-flag findings the human already pushed back on.\n\n" +
-              (map("### @\(.user) (\(.created_at))\n\n\(.body)") | join("\n\n---\n\n"))
-            end
-          ' prior-comments.json > prior-comments.md
-
       - name: Read review prompt
         id: review-prompt
-        env:
-          EXTRA_PROMPT: ${{ inputs.extra_prompt }}
         run: |
           {
             echo 'REVIEW_PROMPT<<EOF'
-            cat REVIEW.md
-            echo ''
             cat .claude/review-prompt.md
-            if [ -n "$EXTRA_PROMPT" ]; then
-              echo ''
-              echo '## Additional reviewer instructions'
-              echo ''
-              printf '%s\n' "$EXTRA_PROMPT"
-            fi
-            if [ -s prior-comments.md ]; then
-              echo ''
-              cat prior-comments.md
-            fi
             echo 'EOF'
           } >> "$GITHUB_ENV"
 
@@ -154,10 +38,9 @@ jobs:
           track_progress: true
           prompt: |
             REPO: ${{ github.repository }}
-            PR NUMBER: ${{ steps.resolve.outputs.pr_number }}
-            PR AUTHOR: ${{ steps.resolve.outputs.pr_author }}
+            PR NUMBER: ${{ github.event.pull_request.number }}
 
             ${{ env.REVIEW_PROMPT }}
           claude_args: |
             --allowedTools "mcp__github_inline_comment__create_inline_comment,Bash(gh pr comment:*),Bash(gh pr diff:*),Bash(gh pr view:*)"
-            --model claude-opus-4-8
+            --model opus

.github/workflows/pr-review-commands.yml

@@ -1,123 +0,0 @@
-name: PR Review Commands
-
-on:
-  issue_comment:
-    types: [created]
-
-jobs:
-  parse:
-    if: github.event.issue.pull_request != null
-    runs-on: ubuntu-latest
-    outputs:
-      command: ${{ steps.parse.outputs.command }}
-      extra_prompt: ${{ steps.parse.outputs.extra_prompt }}
-    steps:
-      - name: Parse command from comment
-        id: parse
-        env:
-          BODY: ${{ github.event.comment.body }}
-        run: |
-          FIRST_LINE=$(printf '%s' "$BODY" | head -n 1 | sed -E 's/^[[:space:]]+//; s/[[:space:]]+$//')
-          FIRST_WORD=${FIRST_LINE%% *}
-          case "$FIRST_WORD" in
-            /review|/codex|/pi|/claude)
-              COMMAND="${FIRST_WORD#/}"
-              REMAINDER_FIRST_LINE=${FIRST_LINE#"$FIRST_WORD"}
-              REMAINDER_FIRST_LINE=${REMAINDER_FIRST_LINE# }
-              REST=$(printf '%s' "$BODY" | tail -n +2)
-              {
-                echo "command=$COMMAND"
-                echo 'extra_prompt<<EXTRA_EOF'
-                if [ -n "$REMAINDER_FIRST_LINE" ]; then
-                  printf '%s\n' "$REMAINDER_FIRST_LINE"
-                fi
-                if [ -n "$REST" ]; then
-                  printf '%s\n' "$REST"
-                fi
-                echo 'EXTRA_EOF'
-              } >> "$GITHUB_OUTPUT"
-              ;;
-            *)
-              echo "command=" >> "$GITHUB_OUTPUT"
-              ;;
-          esac
-
-  check-membership:
-    needs: parse
-    if: needs.parse.outputs.command != ''
-    uses: ./.github/workflows/check-org-membership.yml
-    secrets:
-      access_token: ${{ secrets.ORG_ACCESS_TOKEN }}
-
-  acknowledge:
-    needs: [parse, check-membership]
-    if: needs.parse.outputs.command != '' && needs.check-membership.outputs.is_member == 'true'
-    runs-on: ubuntu-latest
-    permissions:
-      issues: write
-      pull-requests: write
-    steps:
-      - name: React to comment with eyes
-        env:
-          GH_TOKEN: ${{ github.token }}
-          REPO: ${{ github.repository }}
-          COMMENT_ID: ${{ github.event.comment.id }}
-        run: |
-          gh api -X POST \
-            "/repos/$REPO/issues/comments/$COMMENT_ID/reactions" \
-            -f content=eyes >/dev/null
-
-  claude:
-    needs: [parse, check-membership]
-    if: |
-      needs.check-membership.outputs.is_member == 'true' &&
-      (needs.parse.outputs.command == 'review' || needs.parse.outputs.command == 'claude')
-    permissions:
-      contents: read
-      pull-requests: read
-      id-token: write
-    uses: ./.github/workflows/pr-ready-review.yml
-    with:
-      pr_number: ${{ github.event.issue.number }}
-      extra_prompt: ${{ needs.parse.outputs.extra_prompt }}
-      triggered_by: ${{ github.event.comment.user.login }}
-    secrets:
-      CLAUDE_CODE_OAUTH_TOKEN: ${{ secrets.CLAUDE_CODE_OAUTH_TOKEN }}
-      WINDMILL_EE_PRIVATE_ACCESS: ${{ secrets.WINDMILL_EE_PRIVATE_ACCESS }}
-
-  codex:
-    needs: [parse, check-membership]
-    if: |
-      needs.check-membership.outputs.is_member == 'true' &&
-      (needs.parse.outputs.command == 'review' || needs.parse.outputs.command == 'codex')
-    permissions:
-      contents: read
-      issues: write
-      pull-requests: write
-    uses: ./.github/workflows/codex-pr-review.yml
-    with:
-      pr_number: ${{ github.event.issue.number }}
-      extra_prompt: ${{ needs.parse.outputs.extra_prompt }}
-      triggered_by: ${{ github.event.comment.user.login }}
-    secrets:
-      OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
-      CODEX_AUTH_JSON: ${{ secrets.CODEX_AUTH_JSON }}
-      WINDMILL_EE_PRIVATE_ACCESS: ${{ secrets.WINDMILL_EE_PRIVATE_ACCESS }}
-
-  pi:
-    needs: [parse, check-membership]
-    if: |
-      needs.check-membership.outputs.is_member == 'true' &&
-      (needs.parse.outputs.command == 'review' || needs.parse.outputs.command == 'pi')
-    permissions:
-      contents: read
-      issues: write
-      pull-requests: write
-    uses: ./.github/workflows/pi-pr-review.yml
-    with:
-      pr_number: ${{ github.event.issue.number }}
-      extra_prompt: ${{ needs.parse.outputs.extra_prompt }}
-      triggered_by: ${{ github.event.comment.user.login }}
-    secrets:
-      DEEPSEEK_API_KEY: ${{ secrets.DEEPSEEK_API_KEY }}
-      WINDMILL_EE_PRIVATE_ACCESS: ${{ secrets.WINDMILL_EE_PRIVATE_ACCESS }}

.github/workflows/publish-cli-docs.yml

@@ -1,84 +0,0 @@
-name: Publish CLI docs repo
-
-# Regenerates the windmill-cli-docs repo (consumed by context7) from the
-# canonical sources in this repo on every Windmill release.
-#
-# Required secret:
-#   CLI_DOCS_DEPLOY_KEY — ed25519 private key whose public half is registered
-#                        as a write-access deploy key on
-#                        windmill-labs/windmill-cli-docs.
-
-on:
-  push:
-    tags:
-      - "v*"
-  workflow_dispatch:
-
-# Serialize pushes to windmill-cli-docs so two release tags landing close
-# together (e.g. a release-please bump + a hotfix) can't race to force-push
-# the docs repo.
-concurrency:
-  group: publish-cli-docs
-  cancel-in-progress: false
-
-jobs:
-  publish:
-    runs-on: ubuntu-latest
-    steps:
-      - name: Checkout windmill (source of truth)
-        uses: actions/checkout@v4
-        with:
-          path: windmill
-
-      - name: Checkout windmill-cli-docs (publish target)
-        uses: actions/checkout@v4
-        with:
-          repository: windmill-labs/windmill-cli-docs
-          path: windmill-cli-docs
-          ssh-key: ${{ secrets.CLI_DOCS_DEPLOY_KEY }}
-          fetch-depth: 0
-
-      - uses: actions/setup-python@v5
-        with:
-          python-version: "3.12"
-
-      - name: Install dependencies
-        run: pip install pyyaml
-
-      - name: Regenerate docs
-        run: |
-          python3 windmill/system_prompts/generate.py \
-            --context7-dir "$GITHUB_WORKSPACE/windmill-cli-docs"
-
-      - name: Commit and push if changed
-        working-directory: windmill-cli-docs
-        env:
-          REF_NAME: ${{ github.ref_name }}
-          REF_TYPE: ${{ github.ref_type }}
-        run: |
-          git config user.name "windmill-bot"
-          git config user.email "bot@windmill.dev"
-          git add -A
-          if git diff --cached --quiet; then
-            echo "No doc changes for ${REF_NAME}."
-            committed=false
-          else
-            committed=true
-            if [ "${REF_TYPE}" = "tag" ]; then
-              git commit -m "chore: sync from windmill ${REF_NAME}"
-            else
-              git commit -m "chore: sync from windmill (manual dispatch from ${REF_NAME})"
-            fi
-            git push origin HEAD
-          fi
-          # Always mirror the version tag on tag pushes, even when content
-          # didn't change — downstream consumers tie snapshots to releases by
-          # tag, and skipping it would leave the docs repo without a tag for
-          # the new Windmill release.
-          # workflow_dispatch from a non-tag ref skips this so we don't
-          # create a junk tag named after a branch.
-          if [ "${REF_TYPE}" = "tag" ]; then
-            git tag -f "${REF_NAME}"
-            git push origin "${REF_NAME}" --force
-            echo "Mirrored tag ${REF_NAME} to windmill-cli-docs (content changed: ${committed})."
-          fi

.github/workflows/spawn-ephemeral-backend.yml

@@ -0,0 +1,126 @@
+name: Spawn Ephemeral Backend
+
+on:
+  issue_comment:
+    types: [created]
+  pull_request_review_comment:
+    types: [created]
+  workflow_dispatch:
+    inputs:
+      pr_number:
+        description: "PR number"
+        required: true
+        type: number
+
+jobs:
+  check-membership:
+    if: |
+      (github.event_name == 'issue_comment' && contains(github.event.comment.body, '/spawnbackend')) ||
+      (github.event_name == 'pull_request_review_comment' && contains(github.event.comment.body, '/spawnbackend'))
+    uses: ./.github/workflows/check-org-membership.yml
+    secrets:
+      access_token: ${{ secrets.ORG_ACCESS_TOKEN }}
+
+  spawn-backend:
+    needs: check-membership
+    # Only run on PR comments that contain /spawn-backend, or manual dispatch
+    if: |
+      github.event_name == 'workflow_dispatch' ||
+      (github.event.issue.pull_request && needs.check-membership.outputs.is_member == 'true')
+    runs-on: ubuntu-latest
+    permissions:
+      pull-requests: write
+      contents: read
+
+    steps:
+      - name: Get PR details
+        id: pr-details
+        uses: actions/github-script@v7
+        with:
+          script: |
+            const prNumber = context.eventName === 'workflow_dispatch'
+              ? context.payload.inputs.pr_number
+              : context.issue.number;
+
+            const pr = await github.rest.pulls.get({
+              owner: context.repo.owner,
+              repo: context.repo.repo,
+              pull_number: prNumber
+            });
+
+            // Get branch name and format it for Cloudflare Pages
+            // Replace '/' with '-' for the URL
+            const branchName = pr.data.head.ref;
+            const formattedBranch = branchName.replace(/\//g, '-');
+            const cfFrontendUrl = `https://${formattedBranch}.windmill.pages.dev`;
+
+            core.setOutput('commit_hash', pr.data.head.sha);
+            core.setOutput('pr_number', prNumber);
+            core.setOutput('branch_name', branchName);
+            core.setOutput('cf_frontend_url', cfFrontendUrl);
+
+      - name: Check manager URL
+        id: check-manager-url
+        run: |
+          if [ -z "${{ secrets.EPHEMERAL_BACKEND_QUEUE_URL }}" ]; then
+            echo "manager_url_set=false" >> $GITHUB_OUTPUT
+          else
+            echo "manager_url_set=true" >> $GITHUB_OUTPUT
+          fi
+
+      - name: Post error comment if manager not running
+        if: steps.check-manager-url.outputs.manager_url_set == 'false'
+        uses: actions/github-script@v7
+        with:
+          script: |
+            const prNumber = context.eventName === 'workflow_dispatch'
+              ? Number(context.payload.inputs.pr_number)
+              : context.issue.number;
+
+            await github.rest.issues.createComment({
+              owner: context.repo.owner,
+              repo: context.repo.repo,
+              issue_number: prNumber,
+              body: `❌ Manager URL not set (did you start the ephemeral backend manager?)\n\nThe ephemeral backend manager needs to be running to spawn backends. Please start the manager first.`
+            });
+
+      - name: Fail if manager not running
+        if: steps.check-manager-url.outputs.manager_url_set == 'false'
+        run: |
+          echo "Error: EPHEMERAL_BACKEND_QUEUE_URL secret is not set"
+          exit 1
+
+      - name: Trigger Windmill flow
+        if: steps.check-manager-url.outputs.manager_url_set == 'true'
+        id: trigger-flow
+        run: |
+          JOB_UUID=$(curl -s -X POST "https://app.windmill.dev/api/w/windmill-labs/jobs/run/f/f/all/run_ephemeral_backend" \
+            -H "Authorization: Bearer ${{ secrets.WINDMILL_RUN_FLOW_TOKEN }}" \
+            -H "Content-Type: application/json" \
+            -d '{
+              "manager_url": "${{ secrets.EPHEMERAL_BACKEND_QUEUE_URL }}",
+              "commit_hash": "${{ steps.pr-details.outputs.commit_hash }}",
+              "pr_number": ${{ steps.pr-details.outputs.pr_number }},
+              "cf_frontend_url": "${{ steps.pr-details.outputs.cf_frontend_url }}"
+            }' | tr -d '"')
+
+          echo "Job UUID: $JOB_UUID"
+          echo "job_uuid=$JOB_UUID" >> $GITHUB_OUTPUT
+
+      - name: Post comment with job link
+        if: steps.check-manager-url.outputs.manager_url_set == 'true'
+        uses: actions/github-script@v7
+        with:
+          script: |
+            const jobUuid = '${{ steps.trigger-flow.outputs.job_uuid }}';
+            const appUrl = `https://app.windmill.dev/public/windmill-labs/a106bad0256c1dfa7a4f9279c42b1a4b#${jobUuid}`;
+            const prNumber = context.eventName === 'workflow_dispatch'
+              ? Number(context.payload.inputs.pr_number)
+              : context.issue.number;
+
+            await github.rest.issues.createComment({
+              owner: context.repo.owner,
+              repo: context.repo.repo,
+              issue_number: prNumber,
+              body: `🚀 Spawning new ephemeral backend!\n\n${appUrl}`
+            });

.gitignore

@@ -20,7 +20,6 @@ rust-client/Cargo.toml
 
 # Worktree-specific Claude Code settings (generated by scripts/worktree-env)
 .claude/settings.local.json
-.claude/worktrees/
 
 # Symlinked cache directories (for git worktrees)
 backend/target
@@ -33,5 +32,3 @@ backend/chrome_profiler.json
 .fast-check/
 __pycache__/
 .playwright-mcp/
-.codex
-.claude/scheduled_tasks.lock

.mcp.json

@@ -3,16 +3,6 @@
     "svelte": {
       "type": "http",
       "url": "https://mcp.svelte.dev/mcp"
-    },
-    "playwright": {
-      "type": "stdio",
-      "command": "npx",
-      "args": ["-y", "@playwright/mcp@latest", "--browser", "chromium", "--headless"]
-    },
-    "playwright-headed": {
-      "type": "stdio",
-      "command": "npx",
-      "args": ["-y", "@playwright/mcp@latest", "--browser", "chromium"]
     }
   }
 }

.webmux.yaml

@@ -47,7 +47,6 @@ profiles:
       For this window specifically, backend is running on: ${BACKEND_PORT} and frontend is running on: ${FRONTEND_PORT}.
       To connect to the database, use this connection string: ${DATABASE_URL}
       Because we are running backend with cargo watch, to verify your changes, just check the logs in the backend pane. No need for cargo check.
-      For UI verification, use the Playwright MCP (`mcp__playwright__*`) — the `playwright` server is headless and works without a display. Navigate to http://localhost:${FRONTEND_PORT}, log in as admin@windmill.dev / changeme.
       IMPORTANT: Read docs/autonomous-mode.md before starting any work.
     panes:
       - id: agent
@@ -77,7 +76,6 @@ profiles:
       On this window specifically, frontend is running on: ${FRONTEND_PORT}.
       To connect to the database, use this connection string: ${DATABASE_URL}
       Because we are running frontend with npm run dev, to verify your changes, just check the logs in the frontend pane. No need for npm run build.
-      For UI verification, use the Playwright MCP (`mcp__playwright__*`) — the `playwright` server is headless and works without a display. Navigate to http://localhost:${FRONTEND_PORT}, log in as admin@windmill.dev / changeme.
       IMPORTANT: Read docs/autonomous-mode.md before starting any work.
     panes:
       - id: agent
@@ -102,55 +100,9 @@ profiles:
 
 integrations:
   github:
-    autoRemoveOnMerge: true
     linkedRepos:
       - repo: windmill-labs/windmill-ee-private
         alias: ee-private
         dir: ../windmill-ee-private__worktrees
   linear:
     enabled: true
-    autoCreateWorktrees: true
-    watchTeams: [WIN,GIT]
-
-oneshot:
-  systemPrompt: |
-    You are running in webmux ONESHOT mode.
-
-    # No interactive user
-    There is NO interactive user — nobody is watching the chat or will respond
-    to questions, approvals, or status checks. Any message asking the user to
-    review, approve, confirm, take a look, or "let you know" is wasted output:
-    it will not be answered.
-
-    # Your job
-    Take the task to its real conclusion without pausing:
-      1. Make the change.
-      2. Validate it (run the relevant tests, typecheck, build, or quick
-         manual check). For UI changes, drive the running frontend with
-         the Playwright MCP (`mcp__playwright__*`, headless) and confirm
-         the change works end-to-end before moving on.
-      3. Commit.
-      4. Push.
-      5. Open a pull request.
-    Only then are you done.
-
-    # Decisions
-    When something is ambiguous, pick the most reasonable default and proceed.
-    When you would normally ask "should I X or Y?", just pick one and continue
-    — note the choice in the PR description if it matters.
-
-    # PR readiness
-    Default to opening the PR as a draft. If you are highly confident in the
-    change — the scope is small and well-understood, validation passed
-    cleanly, and you would not change anything if a reviewer pushed back —
-    open the PR as ready-for-review directly (omit `--draft` when invoking
-    `gh pr create`, or call `gh pr ready <number>` after creation). Err on
-    the side of draft when validation was partial, the change touches
-    public APIs or shared infrastructure, or you made a non-obvious judgment
-    call.
-
-    # Ending your turn
-    Never end your turn with a question, a suggestion to "take a look", or a
-    request for approval. Stop only when the PR is open, or when you hit a
-    technical error you cannot recover from yourself (in which case clearly
-    state the blocker).

AGENTS.md

@@ -15,11 +15,9 @@ Open-source platform for internal tools, workflows, API integrations, background
 - **Enterprise**: `docs/enterprise.md` — EE file conventions and PR workflow
 - **Backend patterns**: use the `rust-backend` skill when writing Rust code
 - **Frontend patterns**: use the `svelte-frontend` skill when writing Svelte code. Do NOT edit svelte files unless you have read that skill.
-- **Frontend UUIDs**: do not call `crypto.randomUUID()` in frontend code. Import `randomUUID` from `$lib/utils/uuid` instead.
-- **Code review**: review the current PR or branch against the shared review policy in `REVIEW.md` (severity triage, public-surface checklist, AGENTS.md compliance, test-coverage assessment). The skill at `.agents/skills/local-review/SKILL.md` orchestrates it. All three CLIs auto-discover the same SKILL — Claude reads `.claude/skills/` (symlinked to the canonical `.agents/skills/` file), Codex and Pi read `.agents/skills/` directly. Invoke with `/local-review` in Claude Code, `$local-review` (or `/skills` selector) in Codex, or `pi --skill local-review` / `/skill:local-review` in Pi.
+- **Code review**: use `/local-review` to review a PR for bugs and CLAUDE.md compliance
 - **Domain guides**: `.claude/skills/native-trigger/` and `frontend/tutorial-system-guide.mdc`
 - **Brand/UI guidelines**: `frontend/brand-guidelines.md`
-- **CLI commands**: when adding/modifying/removing a command, subcommand, option, or description in `cli/src/commands/`, run `python system_prompts/generate.py` to refresh `system_prompts/auto-generated/` and `cli/src/guidance/skills.gen.ts`. The CLI docs the agents use to operate `wmill` are derived from the source — stale generated files give agents the wrong flags.
 
 ## Dev Environment
 
@@ -30,28 +28,6 @@ Open-source platform for internal tools, workflows, API integrations, background
 - **Instance settings**: navigate to `/#superadmin-settings`
 - **Migrations**: use `cargo sqlx migrate add -r <name>` from `backend/` to create new migrations (never generate timestamps manually)
 
-## Verifying Frontend Changes
-
-After modifying frontend code, drive the running dev server with the **Playwright MCP** to verify the change in a real browser — don't claim a UI change works without exercising it.
-
-Two MCP servers are registered in `.mcp.json`:
-- `playwright` — headless Chromium, default for devboxes (no display required)
-- `playwright-headed` — windowed Chromium, when a display is available
-
-**One-time setup:** run `npx playwright install chromium` to download the browser binary (Playwright won't fetch it automatically on first use).
-
-Typical flow:
-1. Ensure backend (`cargo run`) and frontend (`REMOTE=http://localhost:8000 npm run dev`) are running
-2. `mcp__playwright__browser_navigate` to the relevant page (login at `admin@windmill.dev` / `changeme`)
-3. `mcp__playwright__browser_snapshot` to inspect the accessibility tree (preferred over screenshots for reading the DOM)
-4. `mcp__playwright__browser_click` / `browser_fill_form` / `browser_type` to interact
-5. `mcp__playwright__browser_take_screenshot` for visual confirmation
-6. `mcp__playwright__browser_console_messages` / `browser_network_requests` to surface errors
-
-**Attach the screenshots to the PR.** For any change under `frontend/`, embed screenshots of the affected UI in the PR body — the `pr` skill requires this and carries the upload recipe.
-
-If you cannot exercise a UI change (no dev server, etc.), say so explicitly rather than claiming success.
-
 ## Banned Patterns
 
 ### `$bindable(default_value)` on optional props
@@ -109,5 +85,3 @@ $NAV --root backend callees "X"                           # what does X call?
 - Search for existing code to reuse before writing new code
 - Follow established patterns in the codebase
 - Keep changes focused — don't refactor beyond what's asked
-- **Comments record constraints, not narration.** Write a comment only for what the code can't show: why a non-obvious approach is required, what breaks if it's "simplified" away. State each invariant once, at the place where someone would break it, in ≤4 lines. Don't describe what the next line does, don't repeat the same rationale at multiple sites, and don't address the PR reviewer (justifying a change belongs in the PR description, not the code). Describe the code as it is, never its drafting history: "we no longer do X", "unchanged behavior", "instead of the previous approach" are meaningless to a reader who never saw the earlier iteration — before finishing, reread your comments as if the current state is the only state that ever existed.
-- **Never attribute work to a specific customer, account, or "requested by a customer" in repo-tracked content** (PR descriptions, commit messages, code comments, docs). Describe changes by their technical motivation instead.

CHANGELOG.md

@@ -1,999 +1,5 @@
 # Changelog
 
-## [1.737.0](https://github.com/windmill-labs/windmill/compare/v1.736.0...v1.737.0) (2026-06-23)
-
-
-### Features
-
-* **apps:** opt-in sandbox isolation for published & raw apps (alpha) ([#9420](https://github.com/windmill-labs/windmill/issues/9420)) ([2879cbb](https://github.com/windmill-labs/windmill/commit/2879cbb65a4122c86b4a472206d74d9009b07904))
-
-
-### Bug Fixes
-
-* allow SQL args in managed // materialize scripts ([#9733](https://github.com/windmill-labs/windmill/issues/9733)) ([fa35968](https://github.com/windmill-labs/windmill/commit/fa3596885bf2d7ee8859f295e820ee362c756911))
-* bound orphan-cleanup drain rate with capped multi-batch loop ([#9730](https://github.com/windmill-labs/windmill/issues/9730)) ([31d9215](https://github.com/windmill-labs/windmill/commit/31d9215e5a61f19662cc87be8147007e1d47ebb6))
-* **ext-jwt:** reject external JWT auth for non-existent workspaces ([#9723](https://github.com/windmill-labs/windmill/issues/9723)) ([c644311](https://github.com/windmill-labs/windmill/commit/c644311eca4bcaf4b68058cf1d5d79d4078aee1a))
-* optimize cleanup_job_perms_orphaned and job_result_stream cleanup queries ([#9727](https://github.com/windmill-labs/windmill/issues/9727)) ([6d94865](https://github.com/windmill-labs/windmill/commit/6d9486510933af9109f52011d93b13847dbdbb39))
-* prevent silent audit-partition outage via monitor watchdog + alert ([#9729](https://github.com/windmill-labs/windmill/issues/9729)) ([8dea383](https://github.com/windmill-labs/windmill/commit/8dea38383f884f59b2956c39f1424005a21265bd))
-
-
-### Performance Improvements
-
-* **monitor:** hash active-root exclusion in retention delete (WIN-2088) ([#9732](https://github.com/windmill-labs/windmill/issues/9732)) ([75bafab](https://github.com/windmill-labs/windmill/commit/75bafabeeec76cf6da33eef41f588e37071df011))
-
-## [1.736.0](https://github.com/windmill-labs/windmill/compare/v1.735.0...v1.736.0) (2026-06-23)
-
-
-### Features
-
-* **ai-chat:** workspace AI chat skills (SKILL.md upload + read_skill tool) ([#9648](https://github.com/windmill-labs/windmill/issues/9648)) ([6f4017d](https://github.com/windmill-labs/windmill/commit/6f4017d694494a158ebd0579c93336c378cba0fd))
-
-
-### Bug Fixes
-
-* **drafts:** stop mis-filing workspace-blind legacy drafts on migration ([#9725](https://github.com/windmill-labs/windmill/issues/9725)) ([3bf5b72](https://github.com/windmill-labs/windmill/commit/3bf5b72afab3241ea41a261a40c2434764bdaf72))
-* **frontend:** destroy old WebsocketProvider on workspace switch in MultiplayerMenu ([#9719](https://github.com/windmill-labs/windmill/issues/9719)) ([6e96f90](https://github.com/windmill-labs/windmill/commit/6e96f90065dfe2f6ccc5eb4f85f4facd3515c70c))
-* **frontend:** ensure type:object in test_run_flow tool schema for Anthropic ([#9721](https://github.com/windmill-labs/windmill/issues/9721)) ([d5cb944](https://github.com/windmill-labs/windmill/commit/d5cb944cf92f074b2ee42c876595eacdfa2f4d76))
-* **health:** detect read-only replica via pg_is_in_recovery() ([#9722](https://github.com/windmill-labs/windmill/issues/9722)) ([e16061d](https://github.com/windmill-labs/windmill/commit/e16061df06babeae935a9396de5bdcd46e8119a9))
-* re-enforce scoped API token boundaries across handlers ([#9712](https://github.com/windmill-labs/windmill/issues/9712)) ([e19594d](https://github.com/windmill-labs/windmill/commit/e19594df2ad015a0336ade95e04562f5562ec3f6))
-
-## [1.735.0](https://github.com/windmill-labs/windmill/compare/v1.734.0...v1.735.0) (2026-06-22)
-
-
-### Features
-
-* clarify session draft bar tracks all workspace draft changes ([#9714](https://github.com/windmill-labs/windmill/issues/9714)) ([ed016a5](https://github.com/windmill-labs/windmill/commit/ed016a5edb4527877bf7e6bf92feafc2710669c2))
-* **copilot:** improve global-mode path selection + add path-selection evals ([#9698](https://github.com/windmill-labs/windmill/issues/9698)) ([74a2329](https://github.com/windmill-labs/windmill/commit/74a2329d2e8395141807c43acef07ef132490039))
-* link files & folders to the global AI chat ([#9520](https://github.com/windmill-labs/windmill/issues/9520)) ([84cc043](https://github.com/windmill-labs/windmill/commit/84cc043406d63a1e1472165cf24ce8c09905fc5f))
-* scope default instance db name to workspace (dt_/dl_) ([#9699](https://github.com/windmill-labs/windmill/issues/9699)) ([4a8a724](https://github.com/windmill-labs/windmill/commit/4a8a724895dcecb835e1eb1e4fd7d1bbc8b3e0fb))
-
-
-### Bug Fixes
-
-* enforce job_dir containment when writing module files ([#9703](https://github.com/windmill-labs/windmill/issues/9703)) ([e403f92](https://github.com/windmill-labs/windmill/commit/e403f92d7e84cebc78709dce1a0928048ba2506d))
-* **frontend:** deploy full script/flow draft from AI chat via shared module ([#9642](https://github.com/windmill-labs/windmill/issues/9642)) ([23bf6bf](https://github.com/windmill-labs/windmill/commit/23bf6bf3da01d552d2dfab6dbcfd30f758ed34d2))
-* **frontend:** strip raw-app post-deploy diff noise (raw_app/lock/data) ([#9706](https://github.com/windmill-labs/windmill/issues/9706)) ([e20a277](https://github.com/windmill-labs/windmill/commit/e20a27745a08d552e6d2c5a8bbaf08ccfe89c68f))
-* ignore NotFound errors when deleting log files from object store ([#9707](https://github.com/windmill-labs/windmill/issues/9707)) ([8a0b0ab](https://github.com/windmill-labs/windmill/commit/8a0b0abead71320c4f69eb3007739a19f76d4126))
-* **oauth:** restore bring-your-own CC token URL override ([#9711](https://github.com/windmill-labs/windmill/issues/9711)) ([ef4962e](https://github.com/windmill-labs/windmill/commit/ef4962e52aba0bc79bf72523de9101853c660654))
-* sanitize git credentials from ansible executor errors and logs ([#9697](https://github.com/windmill-labs/windmill/issues/9697)) ([ace7b68](https://github.com/windmill-labs/windmill/commit/ace7b68a28b00d715298ffcb6ae907c1974a74b8))
-
-## [1.734.0](https://github.com/windmill-labs/windmill/compare/v1.733.1...v1.734.0) (2026-06-20)
-
-
-### Features
-
-* ducklake materialization for data pipelines ([#9689](https://github.com/windmill-labs/windmill/issues/9689)) ([3ebf243](https://github.com/windmill-labs/windmill/commit/3ebf24359d66048d6361ce65cd879cdc04b737ed))
-
-
-### Bug Fixes
-
-* **frontend:** clear branch step state when switching outer loop iterations ([#9650](https://github.com/windmill-labs/windmill/issues/9650)) ([09a8004](https://github.com/windmill-labs/windmill/commit/09a80040ca268a4379d5302e3401435ba93247e0))
-
-## [1.733.1](https://github.com/windmill-labs/windmill/compare/v1.733.0...v1.733.1) (2026-06-19)
-
-
-### Bug Fixes
-
-* **backend:** validate ansible vault_id entries before config generation ([#9681](https://github.com/windmill-labs/windmill/issues/9681)) ([c1f31c0](https://github.com/windmill-labs/windmill/commit/c1f31c0e4777bf0cfed0dd7f03249e9a61cd8cb9))
-* **frontend:** group live pipeline runs in the activity panel ([#9684](https://github.com/windmill-labs/windmill/issues/9684)) ([1be4df9](https://github.com/windmill-labs/windmill/commit/1be4df9acb935250d4cc12e83cf67e366d870d5a))
-* require super admin for object storage config test endpoint ([#9683](https://github.com/windmill-labs/windmill/issues/9683)) ([fb44fe7](https://github.com/windmill-labs/windmill/commit/fb44fe7af2b8ebe8ef64ffb0e5acce8580bf4200))
-* validate websocket trigger urls and gate trigger test route ([#9682](https://github.com/windmill-labs/windmill/issues/9682)) ([c39ee07](https://github.com/windmill-labs/windmill/commit/c39ee07c0bcd2249dd19ffa5cd988125eefc6c9f))
-
-## [1.733.0](https://github.com/windmill-labs/windmill/compare/v1.732.0...v1.733.0) (2026-06-19)
-
-
-### Features
-
-* **ai-chat:** cap read_app_file + search_app grep tool to bound context in large raw apps ([#9653](https://github.com/windmill-labs/windmill/issues/9653)) ([4296a6a](https://github.com/windmill-labs/windmill/commit/4296a6ae1f73564de4df54fe1df0a03c1df05dfd))
-* **python, windows:** enable S3 to cache wheels ([#5199](https://github.com/windmill-labs/windmill/issues/5199)) ([ab3bc97](https://github.com/windmill-labs/windmill/commit/ab3bc97cd92b6480327029bcf018280442462af7))
-
-
-### Bug Fixes
-
-* allow users to always discard their own drafts without write permission ([#9659](https://github.com/windmill-labs/windmill/issues/9659)) ([6833a55](https://github.com/windmill-labs/windmill/commit/6833a554aeddb3e63173d3c3140b490c0bf2822b))
-* **backend:** clean up unique_ext_jwt_token on workspace deletion ([#9676](https://github.com/windmill-labs/windmill/issues/9676)) ([9add719](https://github.com/windmill-labs/windmill/commit/9add719d936cdcfb2c4062629e3e1f792694dafe))
-* **backend:** strip NUL bytes from draft values on write ([#9673](https://github.com/windmill-labs/windmill/issues/9673)) ([924f9c7](https://github.com/windmill-labs/windmill/commit/924f9c7e8d8863d9af40aee246a519b4be0e1ea2))
-* **python:** split PIP_TRUSTED_HOST by whitespace to support multiple hosts ([#9675](https://github.com/windmill-labs/windmill/issues/9675)) ([cafb473](https://github.com/windmill-labs/windmill/commit/cafb473494d9cff3a8b2aeaf9f18b015f966e7b3))
-
-## [1.732.0](https://github.com/windmill-labs/windmill/compare/v1.731.0...v1.732.0) (2026-06-19)
-
-
-### Features
-
-* **ansible:** add AI chat and editor bar buttons for ansible ([#9671](https://github.com/windmill-labs/windmill/issues/9671)) ([017c3d3](https://github.com/windmill-labs/windmill/commit/017c3d3343c2577501103be4b2dd8dac9727d80d))
-
-
-### Bug Fixes
-
-* **ai:** emit token usage in gemini proxy streaming translation ([#9669](https://github.com/windmill-labs/windmill/issues/9669)) ([0cc2257](https://github.com/windmill-labs/windmill/commit/0cc2257596a3965cf6db21a0090edcce6e1b8419))
-* **backend:** grant script_trigger access to windmill roles ([#9674](https://github.com/windmill-labs/windmill/issues/9674)) ([3361736](https://github.com/windmill-labs/windmill/commit/33617367d09537667d2ab3f91135c736194b9e7e))
-* **frontend:** ignore hash/assets in script diffs and drafts (WIN-2071) ([#9664](https://github.com/windmill-labs/windmill/issues/9664)) ([3371265](https://github.com/windmill-labs/windmill/commit/33712653821e83f2562dd5f271dbec0188d5d2f8))
-
-## [1.731.0](https://github.com/windmill-labs/windmill/compare/v1.730.0...v1.731.0) (2026-06-19)
-
-
-### Features
-
-* **backend:** auto-reconnect postgres trigger listener with backoff (WIN-2073) ([#9666](https://github.com/windmill-labs/windmill/issues/9666)) ([a425431](https://github.com/windmill-labs/windmill/commit/a425431e9067bcf85474fdc7b7ef7f73e41b9071))
-
-
-### Bug Fixes
-
-* **backend:** grant notify_event access to windmill roles ([#9665](https://github.com/windmill-labs/windmill/issues/9665)) ([a682d02](https://github.com/windmill-labs/windmill/commit/a682d02311a2110bfc0d5e0a5b52e96147fe0dd7))
-* **mcp:** repair invalid type keywords in tool JSON schemas ([#9667](https://github.com/windmill-labs/windmill/issues/9667)) ([c30bdec](https://github.com/windmill-labs/windmill/commit/c30bdecea77ff9b4d74d52961f3101201099b683))
-* trigger flow error handler on unrecoverable (OOM/zombie) step failures ([#9662](https://github.com/windmill-labs/windmill/issues/9662)) ([7e4df02](https://github.com/windmill-labs/windmill/commit/7e4df02bd60c4d6ee8c92d3dfd19f4e587ff9632))
-
-## [1.730.0](https://github.com/windmill-labs/windmill/compare/v1.729.0...v1.730.0) (2026-06-18)
-
-
-### Features
-
-* **ai-chat:** summary-based conversation compaction ([#9645](https://github.com/windmill-labs/windmill/issues/9645)) ([5d553b8](https://github.com/windmill-labs/windmill/commit/5d553b81c06664aab61131a93b198575c088d12d))
-* Data Pipelines alpha ([#9193](https://github.com/windmill-labs/windmill/issues/9193)) ([7155a0b](https://github.com/windmill-labs/windmill/commit/7155a0bb96cf30bd878272a0f4c3c3b02341b261))
-
-
-### Bug Fixes
-
-* **ai-chat:** stop echoing app draft value in global chat write tool results ([#9658](https://github.com/windmill-labs/windmill/issues/9658)) ([2fed808](https://github.com/windmill-labs/windmill/commit/2fed808b9e716d9a44b34c7a073ec0d37374be05))
-* **backend:** include raw_app drafts in list_apps draft_users ([#9647](https://github.com/windmill-labs/windmill/issues/9647)) ([19bc005](https://github.com/windmill-labs/windmill/commit/19bc0052f1069d732231950a0ec958f675d57417))
-* **frontend:** keep ?new_draft flag until first save is confirmed ([#9656](https://github.com/windmill-labs/windmill/issues/9656)) ([9b6b7c3](https://github.com/windmill-labs/windmill/commit/9b6b7c3862d9988e5e91eaab2b967a23f41cdc0d))
-* **frontend:** re-key raw-app autosave on post-deploy navigation ([#9646](https://github.com/windmill-labs/windmill/issues/9646)) ([1058bde](https://github.com/windmill-labs/windmill/commit/1058bdeccdc4c403ef4599db0ee74a65a66c715f))
-* gate agent-worker global setting reads with a blocklist ([#9623](https://github.com/windmill-labs/windmill/issues/9623)) ([fdd82f0](https://github.com/windmill-labs/windmill/commit/fdd82f0c48f29805cd9e219649f27fba45c7fd92))
-* **workspaces:** add instance setting to disable workspace invite/add emails ([#9643](https://github.com/windmill-labs/windmill/issues/9643)) ([796230d](https://github.com/windmill-labs/windmill/commit/796230d90a7e6d1debc15e139ab708881e527862))
-
-## [1.729.0](https://github.com/windmill-labs/windmill/compare/v1.728.1...v1.729.0) (2026-06-18)
-
-
-### Features
-
-* add ducklake schema support to the database manager ([#9633](https://github.com/windmill-labs/windmill/issues/9633)) ([3eeccaf](https://github.com/windmill-labs/windmill/commit/3eeccaf9682b7803fdf5be8dcbc4d243e0ba2e49))
-* **ai-chat:** self-hosted docs tools via windmill.dev llms.txt + ask benchmark ([#9578](https://github.com/windmill-labs/windmill/issues/9578)) ([f4425fc](https://github.com/windmill-labs/windmill/commit/f4425fca9fb0d02b845bd72888ade54905c5a30b))
-* **frontend:** View Diff and in-place Load for other users' drafts ([#9621](https://github.com/windmill-labs/windmill/issues/9621)) ([5508f1d](https://github.com/windmill-labs/windmill/commit/5508f1da9cd04c2583eb3f7ee6bce19d067f2227))
-* per-user draft review & deploy page (gating, badges, rename, raw-app deploy fixes) ([#9625](https://github.com/windmill-labs/windmill/issues/9625)) ([e09cd58](https://github.com/windmill-labs/windmill/commit/e09cd5862cb636e143027fe8d9a5be9c7097b031))
-* queue messages typed while ai chat is streaming ([#9525](https://github.com/windmill-labs/windmill/issues/9525)) ([51bd869](https://github.com/windmill-labs/windmill/commit/51bd8692a482850f7ac8b04dd16db5876336b5b9))
-* zero-setup oauth client credentials for registry providers ([#9559](https://github.com/windmill-labs/windmill/issues/9559)) ([e26a923](https://github.com/windmill-labs/windmill/commit/e26a9239a62a25abf90ef06ade4dde7f36e791bb))
-
-
-### Bug Fixes
-
-* **ai_evals:** adapt global eval harness to DB-backed user drafts ([#9641](https://github.com/windmill-labs/windmill/issues/9641)) ([e87ff79](https://github.com/windmill-labs/windmill/commit/e87ff79ecf6a6e0958916ed1b3756fb3addf719f))
-* **drafts:** preserve original timestamp when migrating localStorage drafts ([#9638](https://github.com/windmill-labs/windmill/issues/9638)) ([8021775](https://github.com/windmill-labs/windmill/commit/8021775f5f961ef6fd01b022639b85855326a1da))
-* **frontend:** don't save drafts on leave when auto-save is off, warn instead ([#9630](https://github.com/windmill-labs/windmill/issues/9630)) ([2523465](https://github.com/windmill-labs/windmill/commit/252346500945a9571af744c839ac0c7d6870504f))
-* **frontend:** render Modal2 dialogs above the AI chat panel ([#9636](https://github.com/windmill-labs/windmill/issues/9636)) ([b67c8cf](https://github.com/windmill-labs/windmill/commit/b67c8cf42b477575fc1bc448058ec0d3b7e54fee))
-* **frontend:** show AI sessions when AI unconfigured, with disabled chat ([#9644](https://github.com/windmill-labs/windmill/issues/9644)) ([ba69d81](https://github.com/windmill-labs/windmill/commit/ba69d8147b615e160cf3d2885fc65a0777b78b71))
-* **git-sync:** bump default sync script to hub/28719 (windmill-cli 1.728.1) for WAC modules ([#9649](https://github.com/windmill-labs/windmill/issues/9649)) ([3c0e38b](https://github.com/windmill-labs/windmill/commit/3c0e38b5890d77983cb5cf5f422a62a73e7a4f22))
-
-## [1.728.1](https://github.com/windmill-labs/windmill/compare/v1.728.0...v1.728.1) (2026-06-17)
-
-
-### Bug Fixes
-
-* **backend:** purge workspace_diff cache on workspace delete ([#9627](https://github.com/windmill-labs/windmill/issues/9627)) ([8a3f69d](https://github.com/windmill-labs/windmill/commit/8a3f69dda8f2088fb859ed8ed6e54458940423d0))
-* **cli:** fall back to esbuild-wasm on native host/binary mismatch ([#9629](https://github.com/windmill-labs/windmill/issues/9629)) ([86d1d16](https://github.com/windmill-labs/windmill/commit/86d1d160f0d3bd9faabdafada07e2956dd98445d))
-* **frontend:** persist session-editor draft path/summary edits + per-line diff tooltips ([#9622](https://github.com/windmill-labs/windmill/issues/9622)) ([e4bfeb2](https://github.com/windmill-labs/windmill/commit/e4bfeb29bc4e89669863b5f6396904a331167658))
-
-## [1.728.0](https://github.com/windmill-labs/windmill/compare/v1.727.0...v1.728.0) (2026-06-16)
-
-
-### Features
-
-* **frontend:** adapt AI-chat/sessions drafts to DB-backed model ([#9601](https://github.com/windmill-labs/windmill/issues/9601)) ([611c70a](https://github.com/windmill-labs/windmill/commit/611c70acd211cf4b8f8308da4a264c670a2f5f43))
-* **frontend:** consolidate draft-migration errors into a single toast + modal ([#9612](https://github.com/windmill-labs/windmill/issues/9612)) ([bc0d5bf](https://github.com/windmill-labs/windmill/commit/bc0d5bf241df3633921bd9d43d171e91034fbfcf))
-* **frontend:** dedup user drafts against the deployed baseline ([#9618](https://github.com/windmill-labs/windmill/issues/9618)) ([a2ce446](https://github.com/windmill-labs/windmill/commit/a2ce44645fdbfa98bf250fac2d15d2b5b26c4b47))
-
-
-### Bug Fixes
-
-* **frontend:** reset deleteWorkspaceForkModal on confirm in SidebarContent ([#9619](https://github.com/windmill-labs/windmill/issues/9619)) ([7cb5c6e](https://github.com/windmill-labs/windmill/commit/7cb5c6e749b2020dee5ee1499f0dc69c5109a6d8))
-* **frontend:** session Drafts drawer uses raw_app kind for the raw-app diff ([#9617](https://github.com/windmill-labs/windmill/issues/9617)) ([46288b6](https://github.com/windmill-labs/windmill/commit/46288b6143efae4dfdf6fe068b97a1e8831fce6a))
-* **nativets:** respect custom CA certs in in-process fetch runtime ([#9615](https://github.com/windmill-labs/windmill/issues/9615)) ([41562c7](https://github.com/windmill-labs/windmill/commit/41562c7d7c708d7d056d9b3d0c39b994a6f4a016))
-* **ResourceForm:** initialize JSON editor when resource type schema is unavailable ([#9611](https://github.com/windmill-labs/windmill/issues/9611)) ([5a24057](https://github.com/windmill-labs/windmill/commit/5a2405743b4622fc1021109114d007057abd5dfd))
-* show folder labels in the folder list table ([#9620](https://github.com/windmill-labs/windmill/issues/9620)) ([651fa13](https://github.com/windmill-labs/windmill/commit/651fa13ee80ff76e5a53ef1ed545b03ce6792294))
-* show last updated date per user in other-users-drafts modal ([#9614](https://github.com/windmill-labs/windmill/issues/9614)) ([f6104ce](https://github.com/windmill-labs/windmill/commit/f6104ce05c4005ffb9fe8112782d1ef6d3065300))
-
-## [1.727.0](https://github.com/windmill-labs/windmill/compare/v1.726.1...v1.727.0) (2026-06-16)
-
-
-### Features
-
-* support temp_script_refs in wmill dev for local relative imports ([#9554](https://github.com/windmill-labs/windmill/issues/9554)) ([33ac287](https://github.com/windmill-labs/windmill/commit/33ac287065742df53f363a5fe09f54f5584a85a6))
-
-
-### Bug Fixes
-
-* **cli:** harden legacy flow lock migration ordering and collision guard ([#9557](https://github.com/windmill-labs/windmill/issues/9557)) ([cd09870](https://github.com/windmill-labs/windmill/commit/cd098700c2cd8d7e9150f760938f4eaf34d188ec))
-* **cli:** include __mod/ folder in gitSyncIncludePattern for scripts ([#9606](https://github.com/windmill-labs/windmill/issues/9606)) ([252c1b3](https://github.com/windmill-labs/windmill/commit/252c1b35fc716c3486109d89615127c588bbe90a))
-* **frontend:** allow same-origin redirects in isValidLogoutRedirect ([#9568](https://github.com/windmill-labs/windmill/issues/9568)) ([8500435](https://github.com/windmill-labs/windmill/commit/8500435e82231e13a1b8a874fd0545f0a0a73fee))
-* **frontend:** make UserDraft read-after-write work without live entry ([#9609](https://github.com/windmill-labs/windmill/issues/9609)) ([51e82d7](https://github.com/windmill-labs/windmill/commit/51e82d7c6d30c66c84236feb743c09929934e564))
-* **frontend:** seed detached user-draft handles so new-item drawers render ([#9608](https://github.com/windmill-labs/windmill/issues/9608)) ([9e3c0de](https://github.com/windmill-labs/windmill/commit/9e3c0decf95378c66055d82215c15cd3bf4a69cb))
-* **frontend:** strip server-managed fields from value diffs ([#9599](https://github.com/windmill-labs/windmill/issues/9599)) ([c213801](https://github.com/windmill-labs/windmill/commit/c213801b5aee54d801c14b9eb31422f2a312ef7e))
-
-## [1.726.1](https://github.com/windmill-labs/windmill/compare/v1.726.0...v1.726.1) (2026-06-15)
-
-
-### Bug Fixes
-
-* **apps:** prevent decision tree graph editor crash on missing graph context ([#9602](https://github.com/windmill-labs/windmill/issues/9602))  ([24f3259](https://github.com/windmill-labs/windmill/commit/24f32596e9ca39c963c19af4a8b14fbcd04e3a78))
-* db-backed draft fixes — review-page UX, legacy drafts, session restore ([#9600](https://github.com/windmill-labs/windmill/issues/9600)) ([4e4b224](https://github.com/windmill-labs/windmill/commit/4e4b2247ef471dada1b8c894974fe921d14b3947))
-
-## [1.726.0](https://github.com/windmill-labs/windmill/compare/v1.725.1...v1.726.0) (2026-06-15)
-
-
-### Features
-
-* **audit:** record workspace archive/unarchive/delete in instance audit log ([#9596](https://github.com/windmill-labs/windmill/issues/9596)) ([9de5708](https://github.com/windmill-labs/windmill/commit/9de57086086bb5626d175c7f926915d1d6ac67ca))
-* **frontend:** add user-level toggle to disable Windmill AI ([#9585](https://github.com/windmill-labs/windmill/issues/9585)) ([5709a56](https://github.com/windmill-labs/windmill/commit/5709a564fbafd9aa91943572ecd8c3e0c45c20b1))
-
-
-### Bug Fixes
-
-* **embeddings:** retry HuggingFace model downloads with backoff ([#9597](https://github.com/windmill-labs/windmill/issues/9597)) ([6a62959](https://github.com/windmill-labs/windmill/commit/6a6295921d681359155d814507908792be405679))
-* resolve release CI failures (pypi bundle, flow serde test, cli windows) ([#9595](https://github.com/windmill-labs/windmill/issues/9595)) ([5ccaae8](https://github.com/windmill-labs/windmill/commit/5ccaae8ab36f2be18b67863ea069763455908029))
-
-## [1.725.1](https://github.com/windmill-labs/windmill/compare/v1.725.0...v1.725.1) (2026-06-15)
-
-
-### Bug Fixes
-
-* **apps:** apply scope-path predicate to app list/search endpoints ([#9581](https://github.com/windmill-labs/windmill/issues/9581)) ([3bf6e10](https://github.com/windmill-labs/windmill/commit/3bf6e102afbdad41e558617bc812012eaaaecd9b))
-* **auth:** add scope checks to scripts/flows list_tokens endpoints ([#9582](https://github.com/windmill-labs/windmill/issues/9582)) ([36c9f86](https://github.com/windmill-labs/windmill/commit/36c9f8612b5778aa2c981454729590b71671ce8d))
-* **cli:** preserve committed script.lock on transient NULL lock during git-sync deploy ([#9593](https://github.com/windmill-labs/windmill/issues/9593)) ([6b916ac](https://github.com/windmill-labs/windmill/commit/6b916ac688e0305284e6cf819bf28803bcca0118))
-* expose parent_hash in MCP createScript tool for updates ([#9586](https://github.com/windmill-labs/windmill/issues/9586)) ([a69505d](https://github.com/windmill-labs/windmill/commit/a69505df9bf25d7c4f11d0528a7450c08dbb422c))
-* **flows:** stop serializing default retry/stop_after_if fields ([#9583](https://github.com/windmill-labs/windmill/issues/9583)) ([e1e2a24](https://github.com/windmill-labs/windmill/commit/e1e2a24b6a6752b3ac779cb0db38061cbc54425e))
-* **security:** sanitize dependency names & connection strings against command/SQL injection ([#9590](https://github.com/windmill-labs/windmill/issues/9590)) ([aff0a4e](https://github.com/windmill-labs/windmill/commit/aff0a4ec189cd8e315282e878bb858ef00635b90))
-
-## [1.725.0](https://github.com/windmill-labs/windmill/compare/v1.724.0...v1.725.0) (2026-06-15)
-
-
-### Features
-
-* Db-backed user drafts ([#9351](https://github.com/windmill-labs/windmill/issues/9351)) ([1fc3557](https://github.com/windmill-labs/windmill/commit/1fc355709c025fd256c5a4035356e15a5a05b23d))
-* scope AI session storage per user, session list in IndexedDB ([#9518](https://github.com/windmill-labs/windmill/issues/9518)) ([aa26c4d](https://github.com/windmill-labs/windmill/commit/aa26c4d9b22b3a353a6c0605eb9a4193e34aa18c))
-
-
-### Bug Fixes
-
-* **powershell:** sanitize module names to prevent command injection (CWE-78) ([#9587](https://github.com/windmill-labs/windmill/issues/9587)) ([6acce7a](https://github.com/windmill-labs/windmill/commit/6acce7a88733683153db534cb18752b31d93af82))
-
-## [1.724.0](https://github.com/windmill-labs/windmill/compare/v1.723.0...v1.724.0) (2026-06-15)
-
-
-### Features
-
-* **cli:** add --yes, --secret/--no-secret and --description to variable add ([#9548](https://github.com/windmill-labs/windmill/issues/9548)) ([4e9e0c0](https://github.com/windmill-labs/windmill/commit/4e9e0c024b4b95f9676b1646591d8f0c662e84ab))
-* **frontend:** improve AI chat cancel and interrupted-turn handling ([#9539](https://github.com/windmill-labs/windmill/issues/9539)) ([114c412](https://github.com/windmill-labs/windmill/commit/114c41251a8c738b58a1a3dd9434d09d33feb6f1))
-* **frontend:** precise AI chat context usage tracking + indicator ([#9551](https://github.com/windmill-labs/windmill/issues/9551)) ([2b47180](https://github.com/windmill-labs/windmill/commit/2b471805bf1c92bb210cfacda217a4341e1f989c))
-* wire chat reasoning effort through gemini and bedrock proxies ([#9545](https://github.com/windmill-labs/windmill/issues/9545)) ([aaf0563](https://github.com/windmill-labs/windmill/commit/aaf05635cedadc73455dc522474b673719f9fd5c))
-
-
-### Bug Fixes
-
-* actually isolate windows job children from CTRL_BREAK_EVENT + reap on worker death ([#9563](https://github.com/windmill-labs/windmill/issues/9563)) ([61f3291](https://github.com/windmill-labs/windmill/commit/61f3291b240bdb5c26bee8947351a9590bc3bd45))
-* **ai:** enforce resource authz when loading MCP tools in agent worker ([#9571](https://github.com/windmill-labs/windmill/issues/9571)) ([317a862](https://github.com/windmill-labs/windmill/commit/317a8629d1c8436d2a6f3443bd25b81d606ce283))
-* append system CA bundle to tracing proxy cert file ([#9549](https://github.com/windmill-labs/windmill/issues/9549)) ([3cf4083](https://github.com/windmill-labs/windmill/commit/3cf40839602e5c3d1df51f0a29b01736bade09da))
-* **cli:** consistent flow inline lock filenames for compound extensions ([#9555](https://github.com/windmill-labs/windmill/issues/9555)) ([f0659a7](https://github.com/windmill-labs/windmill/commit/f0659a755a161420833e3bfdbe04befc6ebeb977))
-* **flows:** skip_if evaluates wrong previous_result during retry ([#9547](https://github.com/windmill-labs/windmill/issues/9547)) ([2aab352](https://github.com/windmill-labs/windmill/commit/2aab35245c362c2f911c60ea435f29bfb1369ebf))
-* **folders:** allow hyphens in folder names ([#9566](https://github.com/windmill-labs/windmill/issues/9566)) ([84df111](https://github.com/windmill-labs/windmill/commit/84df11177f2009bff007e9b722b55a9a5a63c06a)), closes [#8474](https://github.com/windmill-labs/windmill/issues/8474)
-* **frontend:** load resource value in JSON editor when resource type is missing ([#9574](https://github.com/windmill-labs/windmill/issues/9574)) ([251266c](https://github.com/windmill-labs/windmill/commit/251266cd8119dbef314314daed43aa43ab92f1c9))
-* isolate windows job children from worker CTRL_BREAK_EVENT ([#9562](https://github.com/windmill-labs/windmill/issues/9562)) ([1d6191e](https://github.com/windmill-labs/windmill/commit/1d6191ebb75843917eec6c76a4be347f9ac4cb72))
-* stop sending temperature for AI chat across all providers ([#9553](https://github.com/windmill-labs/windmill/issues/9553)) ([3585716](https://github.com/windmill-labs/windmill/commit/358571687296fbe5c378533b3c1662707955c64a))
-
-## [1.723.0](https://github.com/windmill-labs/windmill/compare/v1.722.0...v1.723.0) (2026-06-11)
-
-
-### Features
-
-* add get_app_runtime_logs tool to global chat ([#9502](https://github.com/windmill-labs/windmill/issues/9502)) ([f86d0d7](https://github.com/windmill-labs/windmill/commit/f86d0d79fc6aa23119fd59761330ab79592d0a2a))
-* **cli:** improve agent prompts/skills and workspace fork workflow ([#9531](https://github.com/windmill-labs/windmill/issues/9531)) ([5bdc4f8](https://github.com/windmill-labs/windmill/commit/5bdc4f83ce37302a2c375d0ff73763acfee2aadb))
-* enable native web search in copilot ([#9522](https://github.com/windmill-labs/windmill/issues/9522)) ([d3f5fe1](https://github.com/windmill-labs/windmill/commit/d3f5fe1c8c39ff07d05f0922b8f40aa95756a707))
-
-
-### Bug Fixes
-
-* **frontend:** stop live activity flickering when user has multiple tabs ([#9543](https://github.com/windmill-labs/windmill/issues/9543)) ([57e627e](https://github.com/windmill-labs/windmill/commit/57e627eabf7c4144ce1c07214ad44d026b82f0b4))
-* omit temperature for claude fable 5 ([#9540](https://github.com/windmill-labs/windmill/issues/9540)) ([bd00bee](https://github.com/windmill-labs/windmill/commit/bd00beeac54dbcfa9ab86fd336fca1a8fa289341))
-* refetch license key from settings when in-memory key is invalid ([#9534](https://github.com/windmill-labs/windmill/issues/9534)) ([38c0ccd](https://github.com/windmill-labs/windmill/commit/38c0ccdf563d3655a4cba390ce9372bc9f9c4a9b))
-
-## [1.722.0](https://github.com/windmill-labs/windmill/compare/v1.721.0...v1.722.0) (2026-06-11)
-
-
-### Features
-
-* add reasoning effort control and thinking display to AI chat ([#9511](https://github.com/windmill-labs/windmill/issues/9511)) ([7f987e8](https://github.com/windmill-labs/windmill/commit/7f987e8c9807b72d9cc3901b6e4d02a24c423f50))
-* **ai-chat:** collapse big pastes, cap input height, escape HTML ([#9487](https://github.com/windmill-labs/windmill/issues/9487)) ([365e204](https://github.com/windmill-labs/windmill/commit/365e20410ed528d5b4e967b64fb282bcc1e03ccd))
-* **ai-chat:** quick access to AI prompt settings from chat ([#9508](https://github.com/windmill-labs/windmill/issues/9508)) ([b894f78](https://github.com/windmill-labs/windmill/commit/b894f783f183ab795eab6f57da8654275d9ad82d))
-* **ai:** add list_runs and get_job_logs tools to global chat mode ([#9488](https://github.com/windmill-labs/windmill/issues/9488)) ([cfe5119](https://github.com/windmill-labs/windmill/commit/cfe51190356a9e922f6398dd875abc368accbd15))
-* clear conflict error + force delete when reusing a fork workspace id ([#9499](https://github.com/windmill-labs/windmill/issues/9499)) ([fddabe9](https://github.com/windmill-labs/windmill/commit/fddabe9c5c6f178b4b09854dc11e50adafd44c87))
-* **flow:** support worker tag override on AI agent steps ([#9513](https://github.com/windmill-labs/windmill/issues/9513)) ([a6a5600](https://github.com/windmill-labs/windmill/commit/a6a5600833063e36b35aac7f90dcbdcd3db7c627))
-* folder-level label inheritance for scripts, flows and jobs ([#9524](https://github.com/windmill-labs/windmill/issues/9524)) ([765f50c](https://github.com/windmill-labs/windmill/commit/765f50c474f8abf76550664ed15418ddb3c0b221))
-* **frontend:** show AI sessions in narrow-screen burger menu ([#9523](https://github.com/windmill-labs/windmill/issues/9523)) ([a8f1062](https://github.com/windmill-labs/windmill/commit/a8f1062f37228e7a8257aac1cd3295bd5084ff90))
-* prefer idle worker pods on k8s autoscaling scale-in via pod-deletion-cost ([#9515](https://github.com/windmill-labs/windmill/issues/9515)) ([3c3f157](https://github.com/windmill-labs/windmill/commit/3c3f15722fd22713cc8baa744d9a81207943a33c))
-* prompt browser confirmation on page exit with unsaved changes ([#9503](https://github.com/windmill-labs/windmill/issues/9503)) ([3119e16](https://github.com/windmill-labs/windmill/commit/3119e16ed8df0daec0e019ae2efb2176d7e9e953))
-* **worker:** #ssh directive to run a bash script on a remote SSH host ([#9479](https://github.com/windmill-labs/windmill/issues/9479)) ([afddfe8](https://github.com/windmill-labs/windmill/commit/afddfe84452357b06f4fb815566d4c8269ceafbf))
-* workspace protection rule to restrict anonymous app deployment ([#9509](https://github.com/windmill-labs/windmill/issues/9509)) ([cf9ad54](https://github.com/windmill-labs/windmill/commit/cf9ad54181d38c5d3c3aa05208e17d8d97eae4ef))
-
-
-### Bug Fixes
-
-* **cli:** include lock-relevant script content in lock cache key ([#9528](https://github.com/windmill-labs/windmill/issues/9528)) ([dc60e1a](https://github.com/windmill-labs/windmill/commit/dc60e1aa174f2a7616d2d4a37117a463ef25a9d6))
-* **frontend:** allow copy/paste shortcuts inside ConfirmationModal ([#9505](https://github.com/windmill-labs/windmill/issues/9505)) ([7fc5340](https://github.com/windmill-labs/windmill/commit/7fc5340da39c049d059aa0ae15ceb6a98ca58053))
-* **frontend:** clarify trigger filters match the message parsed as JSON ([#9516](https://github.com/windmill-labs/windmill/issues/9516)) ([0b17843](https://github.com/windmill-labs/windmill/commit/0b178437ce8cd8295be21c0c6077c39e957a3337))
-* **frontend:** enable Apply button when env vars change in worker group config ([#9501](https://github.com/windmill-labs/windmill/issues/9501)) ([c80c6d8](https://github.com/windmill-labs/windmill/commit/c80c6d8fcdfc1de8fe841e9699452f67a25cad23))
-* **frontend:** improve AI chat markdown and typing dots in dark mode ([#9497](https://github.com/windmill-labs/windmill/issues/9497)) ([4e86806](https://github.com/windmill-labs/windmill/commit/4e868062d4c22c5574149d35235b7ad1abe805f3))
-* **frontend:** stop echoing draft values in global chat write tool results ([#9530](https://github.com/windmill-labs/windmill/issues/9530)) ([ce6e2f7](https://github.com/windmill-labs/windmill/commit/ce6e2f7ade25ca91c375f0de5f1be3f1c82ccb47))
-* inherit container NO_PROXY into MITM tracing proxy job exclusions ([#9492](https://github.com/windmill-labs/windmill/issues/9492)) ([4c22e3b](https://github.com/windmill-labs/windmill/commit/4c22e3b712a74828cf654ea7d89aeab5b50cfbd7))
-* make default chat model optional in AI settings ([#9514](https://github.com/windmill-labs/windmill/issues/9514)) ([1d43288](https://github.com/windmill-labs/windmill/commit/1d4328877fcc87352fb56de63f0570739d5a9dc4))
-* **nsjail:** make ansible collections mount non-mandatory ([#9510](https://github.com/windmill-labs/windmill/issues/9510)) ([08da7a1](https://github.com/windmill-labs/windmill/commit/08da7a121b4b835500dfc2bd943c4bdce912c63e))
-* **nsjail:** make ansible uv tools mount non-mandatory ([#9507](https://github.com/windmill-labs/windmill/issues/9507)) ([dc368a9](https://github.com/windmill-labs/windmill/commit/dc368a9669e812c593ad848266f645b6223501e6))
-
-## [1.721.0](https://github.com/windmill-labs/windmill/compare/v1.720.0...v1.721.0) (2026-06-09)
-
-
-### Features
-
-* deployed↔draft compare + AI-session draft bar ([#9435](https://github.com/windmill-labs/windmill/issues/9435)) ([b0b330c](https://github.com/windmill-labs/windmill/commit/b0b330c7864d0159af4b0f17dbb3c09bd015145b))
-
-
-### Bug Fixes
-
-* **cli:** reconcile case-only path drift during sync on case-insensitive filesystems (WIN-2020) ([#9485](https://github.com/windmill-labs/windmill/issues/9485)) ([c258928](https://github.com/windmill-labs/windmill/commit/c258928ab62adc1327913c21556520dfd1e5c24c))
-* drop archived items from fork compare (spurious 'not visible' warning) ([#9481](https://github.com/windmill-labs/windmill/issues/9481)) ([92c21bb](https://github.com/windmill-labs/windmill/commit/92c21bbe6586f3c285796a98692e976515a629d5))
-* require auth to view approval details when user_auth_required ([#9482](https://github.com/windmill-labs/windmill/issues/9482)) ([5f41ddd](https://github.com/windmill-labs/windmill/commit/5f41ddd3a592bcd504f94fc99060ca5d79c36190))
-
-## [1.720.0](https://github.com/windmill-labs/windmill/compare/v1.719.0...v1.720.0) (2026-06-08)
-
-
-### Features
-
-* allow private MCP server URLs ([#9470](https://github.com/windmill-labs/windmill/issues/9470)) ([3bc5800](https://github.com/windmill-labs/windmill/commit/3bc5800197db383ae6f708415701a4bdbe2e3345))
-* **api:** add endpoint to update token label ([#9474](https://github.com/windmill-labs/windmill/issues/9474)) ([e8e0701](https://github.com/windmill-labs/windmill/commit/e8e0701a360d0614c4c5a74f6410ba6ac0638caa))
-* **frontend:** use unified drill picker for AI chat @-mention dropdown ([#9159](https://github.com/windmill-labs/windmill/issues/9159)) ([64b089c](https://github.com/windmill-labs/windmill/commit/64b089cd23cca4601abb09f092a32becb80d9394))
-
-
-### Bug Fixes
-
-* center auth0/okta icons and respect currentColor ([#9457](https://github.com/windmill-labs/windmill/issues/9457)) ([5d0ef7d](https://github.com/windmill-labs/windmill/commit/5d0ef7dfd91b3021d125a1b34f81f0788f173786))
-* **forks:** keep trigger/schedule operational state owned by the parent - WIN-2019 ([#9476](https://github.com/windmill-labs/windmill/issues/9476)) ([192574a](https://github.com/windmill-labs/windmill/commit/192574ab8f98d9521a232fc8a4935d407b00cb3a))
-* **frontend:** respect forced column order for numeric column names ([#9463](https://github.com/windmill-labs/windmill/issues/9463)) ([44f5dd6](https://github.com/windmill-labs/windmill/commit/44f5dd6636d4b23aa55383b8b8abe4c2f73bc88d))
-* **frontend:** use ban icon for canceled jobs instead of hourglass ([#9478](https://github.com/windmill-labs/windmill/issues/9478)) ([fa86c62](https://github.com/windmill-labs/windmill/commit/fa86c62b6600e7d47dadf4706d7002706333d919))
-* gate native integration pickers behind non-operator check ([#9465](https://github.com/windmill-labs/windmill/issues/9465)) ([6156e23](https://github.com/windmill-labs/windmill/commit/6156e2372a785ccd0c6f29cb74e90bee69e76483))
-* **oauth:** persist refreshed token through configured secret backend ([#9471](https://github.com/windmill-labs/windmill/issues/9471)) ([76c0d97](https://github.com/windmill-labs/windmill/commit/76c0d970a18bf28ddc48dd746570e73486542606))
-* refresh session editor preview on breadcrumb target switch ([#9475](https://github.com/windmill-labs/windmill/issues/9475)) ([6d522b3](https://github.com/windmill-labs/windmill/commit/6d522b3989ace1f214bd049d910bc0d2a2a6893e))
-
-## [1.719.0](https://github.com/windmill-labs/windmill/compare/v1.718.0...v1.719.0) (2026-06-06)
-
-
-### Features
-
-* **otel:** connect jobs to the inbound distributed trace ([#9456](https://github.com/windmill-labs/windmill/issues/9456)) ([fad1a54](https://github.com/windmill-labs/windmill/commit/fad1a549d95c00d0746a48163c4f95fc69733e1a))
-
-
-### Bug Fixes
-
-* authenticate slack callback payload with per-workspace hmac ([#9461](https://github.com/windmill-labs/windmill/issues/9461)) ([fbdf81b](https://github.com/windmill-labs/windmill/commit/fbdf81ba5f77d282c025360ecee14138dd4cb4a2))
-* prevent token label collision bypassing job read access control ([#9462](https://github.com/windmill-labs/windmill/issues/9462)) ([e1e7af6](https://github.com/windmill-labs/windmill/commit/e1e7af6a25a44eb06b67332ce1efeae2a21e0c6d))
-* **python:** escape reserved-keyword step ids in wrapper codegen ([#9460](https://github.com/windmill-labs/windmill/issues/9460)) ([6a15a9b](https://github.com/windmill-labs/windmill/commit/6a15a9b152ad20be4b5c3de6000516da231e41e0)), closes [#8893](https://github.com/windmill-labs/windmill/issues/8893)
-
-## [1.718.0](https://github.com/windmill-labs/windmill/compare/v1.717.1...v1.718.0) (2026-06-05)
-
-
-### Features
-
-* **flows:** opt-in to include the stopping step's result in early-stop errors ([#9446](https://github.com/windmill-labs/windmill/issues/9446)) ([f2f0812](https://github.com/windmill-labs/windmill/commit/f2f0812a04c9256cfc8eba5e0dcf38d71d971410))
-* make C# dotnet target framework configurable via DOTNET_TARGET_FRAMEWORK ([#9454](https://github.com/windmill-labs/windmill/issues/9454)) ([9a609bf](https://github.com/windmill-labs/windmill/commit/9a609bf08ac1b6157dbdfb827fc01e771d71262e))
-* sandboxed daemonless container runtime via '# sandbox <image>' ([#9453](https://github.com/windmill-labs/windmill/issues/9453)) ([1727271](https://github.com/windmill-labs/windmill/commit/1727271e197b34026efeaf1b6561bb404a440baa))
-* **sandbox:** pull/extract images with crane instead of podman ([#9455](https://github.com/windmill-labs/windmill/issues/9455)) ([7590b28](https://github.com/windmill-labs/windmill/commit/7590b281085afd1fc2774e8fb37a4c0af3aedbad))
-
-
-### Bug Fixes
-
-* distinguish canceled jobs in runs ([#9452](https://github.com/windmill-labs/windmill/issues/9452)) ([9067787](https://github.com/windmill-labs/windmill/commit/90677872f6185eb0c81e0e84a426a54653818457))
-
-## [1.717.1](https://github.com/windmill-labs/windmill/compare/v1.717.0...v1.717.1) (2026-06-04)
-
-
-### Bug Fixes
-
-* invalidate relative-import cache when imported script changes ([#9443](https://github.com/windmill-labs/windmill/issues/9443)) ([f595787](https://github.com/windmill-labs/windmill/commit/f595787409a3fcda9278bbcf2cfcc80092f16460))
-
-## [1.717.0](https://github.com/windmill-labs/windmill/compare/v1.716.0...v1.717.0) (2026-06-04)
-
-
-### Features
-
-* let flow AI chat create and edit sticky notes ([#9412](https://github.com/windmill-labs/windmill/issues/9412)) ([e4e0984](https://github.com/windmill-labs/windmill/commit/e4e0984e55afd3c73f1c365cd0608493a9fd87ed))
-
-
-### Bug Fixes
-
-* **cli:** push whole raw app instead of treating frontend files as scripts ([#9442](https://github.com/windmill-labs/windmill/issues/9442)) ([b5a6a1e](https://github.com/windmill-labs/windmill/commit/b5a6a1eeab663c2d6aaec2c89eab7a550cb0bb6b))
-* read latest db draft for scripts/flows in global mode read tool ([#9441](https://github.com/windmill-labs/windmill/issues/9441)) ([819ba5e](https://github.com/windmill-labs/windmill/commit/819ba5e150ec9f5199919fbea50874fc156d0189))
-
-## [1.716.0](https://github.com/windmill-labs/windmill/compare/v1.715.0...v1.716.0) (2026-06-03)
-
-
-### Features
-
-* add metadata generation model setting ([#9418](https://github.com/windmill-labs/windmill/issues/9418)) ([cf5fefb](https://github.com/windmill-labs/windmill/commit/cf5fefb521479170b9dc64b884630c4dac789931))
-* auto-generate AI session names ([#9399](https://github.com/windmill-labs/windmill/issues/9399)) ([26b7270](https://github.com/windmill-labs/windmill/commit/26b727041830c9b741668a9ab73e2eb90c7cec74))
-* support $f/ and $u/ import path aliases for scripts ([#9378](https://github.com/windmill-labs/windmill/issues/9378)) ([220cd35](https://github.com/windmill-labs/windmill/commit/220cd35cf799c42ebf588bc97a6d8e6f4e97c2e3))
-* use metadata model for small AI tasks ([#9431](https://github.com/windmill-labs/windmill/issues/9431)) ([79178f6](https://github.com/windmill-labs/windmill/commit/79178f6f5a7c606a2e05677c6efcbdd84c608325))
-
-
-### Bug Fixes
-
-* **apps:** relock no longer reverts raw app to a stale version ([#9432](https://github.com/windmill-labs/windmill/issues/9432)) ([073857a](https://github.com/windmill-labs/windmill/commit/073857ac0a9ed54bdeac8f373f7c855fe34eb0ac))
-* **security:** scope variable and resource value caches by caller identity ([#9427](https://github.com/windmill-labs/windmill/issues/9427)) ([0ba128a](https://github.com/windmill-labs/windmill/commit/0ba128afe797bd016da60563949ac3abbbfe1978))
-
-## [1.715.0](https://github.com/windmill-labs/windmill/compare/v1.714.1...v1.715.0) (2026-06-03)
-
-
-### Features
-
-* **frontend:** add rebuild dependency map button to workspace settings ([#9424](https://github.com/windmill-labs/windmill/issues/9424)) ([3b2e748](https://github.com/windmill-labs/windmill/commit/3b2e748daf0a8ec4447c30423068df803f3f9ca2))
-
-
-### Bug Fixes
-
-* **auth:** filter script/flow listings by token scope (GHSA-2ppx-66jv-wpw5) ([#9426](https://github.com/windmill-labs/windmill/issues/9426)) ([7edf3f0](https://github.com/windmill-labs/windmill/commit/7edf3f02122e20fde1e95e0252e7bda641075326))
-* **backend:** authorize single-job read endpoints by job/flow visibility ([#9416](https://github.com/windmill-labs/windmill/issues/9416)) ([89a7a37](https://github.com/windmill-labs/windmill/commit/89a7a377764086911db18252f2478f42f0e1e3ea))
-* **mcp:** resolve MCP resource token via caller RLS + SSRF-guard url ([#9428](https://github.com/windmill-labs/windmill/issues/9428)) ([8053266](https://github.com/windmill-labs/windmill/commit/8053266f88bd4c94fc86278412df5a0beeed5e77))
-* **nsjail:** precompile python stdlib + raise download rlimit_as ([#9429](https://github.com/windmill-labs/windmill/issues/9429)) ([7031744](https://github.com/windmill-labs/windmill/commit/7031744a199f0bf8b8e35043afa959977e5ecdbd))
-* omit temperature for gpt-5+ and o-series models on all providers ([#9422](https://github.com/windmill-labs/windmill/issues/9422)) ([11d1ad9](https://github.com/windmill-labs/windmill/commit/11d1ad9a872d2ec2f14cde35708c84a0c7bdc172))
-
-## [1.714.1](https://github.com/windmill-labs/windmill/compare/v1.714.0...v1.714.1) (2026-06-02)
-
-
-### Bug Fixes
-
-* **backend:** route //native TypeScript previews to native workers (WIN-2007) ([#9407](https://github.com/windmill-labs/windmill/issues/9407)) ([73edebc](https://github.com/windmill-labs/windmill/commit/73edebc833a981488a8ea116f4f13c020a011a6f))
-* **nsjail:** raise python download fd limit for --compile-bytecode (WIN-2009) ([#9414](https://github.com/windmill-labs/windmill/issues/9414)) ([9e6559a](https://github.com/windmill-labs/windmill/commit/9e6559a6f688cc8d982277b19920219ea6d0fd8e))
-* **triggers:** prevent Zoom challenge handler from being used as a signing oracle ([#9413](https://github.com/windmill-labs/windmill/issues/9413)) ([ab2a15b](https://github.com/windmill-labs/windmill/commit/ab2a15b2a859096eabde718bf6e60289ae187118))
-
-## [1.714.0](https://github.com/windmill-labs/windmill/compare/v1.713.1...v1.714.0) (2026-06-02)
-
-
-### Features
-
-* add global ai chat test tools ([#9391](https://github.com/windmill-labs/windmill/issues/9391)) ([5c20d6b](https://github.com/windmill-labs/windmill/commit/5c20d6b4f79f2ccc1987ce7fdaf74e6b8f697846))
-* add workspace datatable tools to global AI chat mode ([#9395](https://github.com/windmill-labs/windmill/issues/9395)) ([943ef6e](https://github.com/windmill-labs/windmill/commit/943ef6eb2089f4b744cfa7945ce47f7f3b361ec7))
-* **flow-ai:** constrain flow-group colors to the NoteColor palette ([#9343](https://github.com/windmill-labs/windmill/issues/9343)) ([e4213c1](https://github.com/windmill-labs/windmill/commit/e4213c1ab8c448f492f372580f5c9df37e33fffc))
-* **frontend:** surface local drafts in drawer editors with an unsaved-changes banner ([#9335](https://github.com/windmill-labs/windmill/issues/9335)) ([075faab](https://github.com/windmill-labs/windmill/commit/075faabf3bba16a10a02ae3973008e5a13473085))
-* handle CTRL_BREAK_EVENT for graceful shutdown on Windows ([#9400](https://github.com/windmill-labs/windmill/issues/9400)) ([2e14456](https://github.com/windmill-labs/windmill/commit/2e1445616a412c5112ad2247b4087c7ddc218845))
-* refine ask-user-question chat display and keyboard nav ([#9392](https://github.com/windmill-labs/windmill/issues/9392)) ([1275487](https://github.com/windmill-labs/windmill/commit/1275487f028d4c74a9eeb18981ed05c225505be0))
-* sessions page with isolated AI chat + flow editor ([#9034](https://github.com/windmill-labs/windmill/issues/9034)) ([eadeac2](https://github.com/windmill-labs/windmill/commit/eadeac248bd022c2796cfe638eb617c6143b8fc4))
-
-
-### Bug Fixes
-
-* **cli:** make encryption key push non-interactive-safe + add --skip-reencrypt-on-key-change ([#9402](https://github.com/windmill-labs/windmill/issues/9402)) ([e356bb1](https://github.com/windmill-labs/windmill/commit/e356bb1f5df92eca3fbb0ca2114b9f4c32d4c496))
-* **cli:** stop git-sync promotion deploys from dropping triggers/schedules ([#9403](https://github.com/windmill-labs/windmill/issues/9403)) ([24e3ef2](https://github.com/windmill-labs/windmill/commit/24e3ef27be8498fb820c228a52febf6a0a91b487))
-* **frontend:** align Monaco editor font size with text-xs ([#9161](https://github.com/windmill-labs/windmill/issues/9161)) ([de76668](https://github.com/windmill-labs/windmill/commit/de76668c10c04abe8771a8ca7bba7b2259819a1c))
-* resolve username rename failing on apps with runnable deps ([#9401](https://github.com/windmill-labs/windmill/issues/9401)) ([e8ad53d](https://github.com/windmill-labs/windmill/commit/e8ad53dae92597f5a1a8b76f38a7d8c24f578a47))
-
-
-### Performance Improvements
-
-* **python:** add --compile-bytecode to uv pip install ([#9393](https://github.com/windmill-labs/windmill/issues/9393)) ([c19441b](https://github.com/windmill-labs/windmill/commit/c19441bc8cb2da064e4ad44d77dc04ab8bbb22ec))
-
-## [1.713.1](https://github.com/windmill-labs/windmill/compare/v1.713.0...v1.713.1) (2026-06-01)
-
-
-### Bug Fixes
-
-* **api:** handle multi-version scripts when removing granular ACL ([#9388](https://github.com/windmill-labs/windmill/issues/9388)) ([9d9c503](https://github.com/windmill-labs/windmill/commit/9d9c5038ce8b0016320a670c434ef9063cb40441))
-
-## [1.713.0](https://github.com/windmill-labs/windmill/compare/v1.712.0...v1.713.0) (2026-05-31)
-
-
-### Features
-
-* **flows:** preserve step/subflow worker tags under a custom-tagged flow ([#9375](https://github.com/windmill-labs/windmill/issues/9375)) ([f0301b1](https://github.com/windmill-labs/windmill/commit/f0301b1605cee5fba4024803555333e6fa5c40ee))
-* **oauth:** support per-provider sandbox URLs ([#9358](https://github.com/windmill-labs/windmill/issues/9358)) ([2bf11dc](https://github.com/windmill-labs/windmill/commit/2bf11dcb15540c538ea2ac3cf70dcbe589060b4e))
-
-
-### Bug Fixes
-
-* **ai:** validate token_url for SSRF in OAuth credentials flow ([#9385](https://github.com/windmill-labs/windmill/issues/9385)) ([4b06881](https://github.com/windmill-labs/windmill/commit/4b06881918b76c5a411cc70b318e46efcc1393a7))
-* **api:** authorize and harden log-file reading endpoints ([#9368](https://github.com/windmill-labs/windmill/issues/9368)) ([bb90f4c](https://github.com/windmill-labs/windmill/commit/bb90f4ce83a0e60af219b11c12ab4fe1d13f47a4))
-* **apps:** make public apps opt into cross-origin isolation via wm_coep (GIT-884) ([#9374](https://github.com/windmill-labs/windmill/issues/9374)) ([2c0c2c4](https://github.com/windmill-labs/windmill/commit/2c0c2c467f163cd24c14c7be2db07af9cf2ce020))
-* **auth:** enforce monotonic privilege on user token lifecycle endpoints ([#9371](https://github.com/windmill-labs/windmill/issues/9371)) ([2ddf93d](https://github.com/windmill-labs/windmill/commit/2ddf93de96622b2a1b2b6f59398a7a1f59360efd))
-* batch encryption-key rotation into one git-sync job ([#9355](https://github.com/windmill-labs/windmill/issues/9355)) ([04a0897](https://github.com/windmill-labs/windmill/commit/04a08976aec4ba9b0516350316df303e9f96bfd3))
-* **cli:** preserve user drafts on sync push and permissioned-as ([#9381](https://github.com/windmill-labs/windmill/issues/9381)) ([b0c3b01](https://github.com/windmill-labs/windmill/commit/b0c3b01d31b0ab3a6566e1f5fec60e3e230cfadb))
-* **frontend:** sanitize user markdown to prevent stored XSS ([#9386](https://github.com/windmill-labs/windmill/issues/9386)) ([def01b8](https://github.com/windmill-labs/windmill/commit/def01b8ff6f331cc36ce02b947adc31c766042c4))
-* **security:** re-pin cached hub scripts to CVE-patched versions (+ HUB_BASE_URL override for cache mode) ([#9387](https://github.com/windmill-labs/windmill/issues/9387)) ([edf340c](https://github.com/windmill-labs/windmill/commit/edf340c4d4f18b16b142cb7deb67afa586f10946))
-
-## [1.712.0](https://github.com/windmill-labs/windmill/compare/v1.711.0...v1.712.0) (2026-05-28)
-
-
-### Features
-
-* add deepseek fim support ([#9365](https://github.com/windmill-labs/windmill/issues/9365)) ([2553fbf](https://github.com/windmill-labs/windmill/commit/2553fbfe31417bd985e7994eac695bf918f97ce2))
-* deploy raw apps from global chat ([#9349](https://github.com/windmill-labs/windmill/issues/9349)) ([dec58e6](https://github.com/windmill-labs/windmill/commit/dec58e6c4f55062b42a752c43c89ef05903e713a))
-* inject active editor into global chat ([#9361](https://github.com/windmill-labs/windmill/issues/9361)) ([9e7eaf3](https://github.com/windmill-labs/windmill/commit/9e7eaf36847ad3a004ec84e8b7d4784771b7b451))
-* **queue:** duration-weighted fairness admission ([#9334](https://github.com/windmill-labs/windmill/issues/9334)) ([045d120](https://github.com/windmill-labs/windmill/commit/045d12043e7c99830ef90bc0da798c94e2094711))
-* warn when custom instance db is shared across workspaces ([#9359](https://github.com/windmill-labs/windmill/issues/9359)) ([a9e5140](https://github.com/windmill-labs/windmill/commit/a9e514099585e5ee72df21bd551a223cceb20fb0))
-
-
-### Bug Fixes
-
-* **cli:** redact encryption_key diff in stdout by default ([#9347](https://github.com/windmill-labs/windmill/issues/9347)) ([88056f8](https://github.com/windmill-labs/windmill/commit/88056f8d4c91c1d14d85a08851ecf0bd97e2260d))
-* **cli:** stop re-prompting on wmill refresh prompts ([#9357](https://github.com/windmill-labs/windmill/issues/9357)) ([c2b5ba8](https://github.com/windmill-labs/windmill/commit/c2b5ba8871abbbcff6de69c90e2f09fee70586c1))
-* **frontend:** close other sidebar menus when hovering Help ([#9354](https://github.com/windmill-labs/windmill/issues/9354)) ([da882c5](https://github.com/windmill-labs/windmill/commit/da882c54b21e3eaf2c1d1abccd0996b243d96dce))
-* **frontend:** prevent duplicate asset node ids crashing flow graph ([#9367](https://github.com/windmill-labs/windmill/issues/9367)) ([9a659b6](https://github.com/windmill-labs/windmill/commit/9a659b636d713ee8fdfbdad41c58bb3d7c79e0d9))
-* **frontend:** prevent MultiSelect crash on undefined value ([#9364](https://github.com/windmill-labs/windmill/issues/9364)) ([aea0061](https://github.com/windmill-labs/windmill/commit/aea00611c41379be2afdad0eedd608c9537d03f7))
-* **git-sync:** publish fork branch on only_create_branch from the CLI ([#9366](https://github.com/windmill-labs/windmill/issues/9366)) ([2fdc51e](https://github.com/windmill-labs/windmill/commit/2fdc51e62985fc755884436130bdd58e294247c8))
-* infer script arg schema when deploying via AI chat ([#9356](https://github.com/windmill-labs/windmill/issues/9356)) ([4efc372](https://github.com/windmill-labs/windmill/commit/4efc37212a98571214aba135b0fbb10dc263fd4f))
-* **monitor:** cleanup stale server_heartbeat background_task_state rows ([#9338](https://github.com/windmill-labs/windmill/issues/9338)) ([59ab038](https://github.com/windmill-labs/windmill/commit/59ab038d7718d8a4c25efa5928f42e1393ebbf40))
-
-## [1.711.0](https://github.com/windmill-labs/windmill/compare/v1.710.1...v1.711.0) (2026-05-26)
-
-
-### Features
-
-* **cli:** add object-storage commands and flow test-step ([#9326](https://github.com/windmill-labs/windmill/issues/9326)) ([36f574f](https://github.com/windmill-labs/windmill/commit/36f574ff951198a4d40ee068a27d74c41ce32154))
-
-
-### Bug Fixes
-
-* **cli:** handle __flow suffix when deriving the flow's Windmill path ([#9333](https://github.com/windmill-labs/windmill/issues/9333)) ([6f77034](https://github.com/windmill-labs/windmill/commit/6f770346fb330997a836c39fba347df4c088a83c))
-* **queue:** duration-weighted workspace fairness signal ([#9329](https://github.com/windmill-labs/windmill/issues/9329)) ([42d2121](https://github.com/windmill-labs/windmill/commit/42d2121af925de50f549ecb72ffb5132f5c41079))
-
-## [1.710.1](https://github.com/windmill-labs/windmill/compare/v1.710.0...v1.710.1) (2026-05-26)
-
-
-### Bug Fixes
-
-* improve workspace fairness ([896add0](https://github.com/windmill-labs/windmill/commit/896add0350f4de31f5674d6be0907a582c5ec17e))
-
-## [1.710.0](https://github.com/windmill-labs/windmill/compare/v1.709.0...v1.710.0) (2026-05-26)
-
-
-### Features
-
-* **queue:** stochastic admission + EE availability of workspace fairness algorithm ([#9321](https://github.com/windmill-labs/windmill/issues/9321)) ([8bf7fd2](https://github.com/windmill-labs/windmill/commit/8bf7fd2c921c48861b71731a085b18ea8f72fb68))
-
-
-### Bug Fixes
-
-* **websocket-trigger:** honor HTTPS_PROXY/HTTP_PROXY/NO_PROXY ([#9324](https://github.com/windmill-labs/windmill/issues/9324)) ([6f36316](https://github.com/windmill-labs/windmill/commit/6f363163df9cd15f5af7d56cf34a01b70d236830))
-
-## [1.709.0](https://github.com/windmill-labs/windmill/compare/v1.708.0...v1.709.0) (2026-05-25)
-
-
-### Features
-
-* add copy button to Path component ([#9311](https://github.com/windmill-labs/windmill/issues/9311)) ([98bd5e7](https://github.com/windmill-labs/windmill/commit/98bd5e7f2a437b8b534028838b6ed0d7c59f7011))
-* **ai-chat:** align footer bar + DropdownV2 mode/autonomy selectors ([#9308](https://github.com/windmill-labs/windmill/issues/9308)) ([2f50e8b](https://github.com/windmill-labs/windmill/commit/2f50e8bab0b5ae9ae297c79abfe96df441f405e2))
-* **ai-chat:** expand chat question answers ([#9310](https://github.com/windmill-labs/windmill/issues/9310)) ([3f219ae](https://github.com/windmill-labs/windmill/commit/3f219aed98d93158aefce01bb51ed12dcb4711a1))
-* plug global chat drafts into userdraft ([#9291](https://github.com/windmill-labs/windmill/issues/9291)) ([1eef531](https://github.com/windmill-labs/windmill/commit/1eef53170b1b2afb75b9812e33787d1f28cf50dd))
-* **raw_apps:** surface UI Builder build errors over the preview pane ([#9316](https://github.com/windmill-labs/windmill/issues/9316)) ([90a196d](https://github.com/windmill-labs/windmill/commit/90a196d8d81993ffc2377d7088ab98f7b0f5ddcc))
-* **raw_apps:** tab-based editor surface with split-with-preview ([#9273](https://github.com/windmill-labs/windmill/issues/9273)) ([368e677](https://github.com/windmill-labs/windmill/commit/368e6774194a58058f28d1b4a42f8f4a7ec4ab63))
-* **service-accounts:** allow choosing role at creation time ([#9307](https://github.com/windmill-labs/windmill/issues/9307)) ([b125eca](https://github.com/windmill-labs/windmill/commit/b125eca7628b07c071bd102b161d389259fd6c62))
-
-
-### Bug Fixes
-
-* **auth:** filter resource/variable listings by token scope (WIN-1981) ([#9302](https://github.com/windmill-labs/windmill/issues/9302)) ([b5a0d46](https://github.com/windmill-labs/windmill/commit/b5a0d46695fdfe692d64573d1cfa06511e3b33f5))
-* **jobs:** authorization bypass in only_result job updates (WIN-1980) ([#9301](https://github.com/windmill-labs/windmill/issues/9301)) ([108a88a](https://github.com/windmill-labs/windmill/commit/108a88a1801548c8570d56aa3e1eb80246367bf4))
-
-## [1.708.0](https://github.com/windmill-labs/windmill/compare/v1.707.0...v1.708.0) (2026-05-24)
-
-
-### Features
-
-* **queue:** per-workspace fairness cap on the shared cloud worker pool ([#9303](https://github.com/windmill-labs/windmill/issues/9303)) ([de2e243](https://github.com/windmill-labs/windmill/commit/de2e243313ee34348675dec600cb412b475d1b4b))
-
-## [1.707.0](https://github.com/windmill-labs/windmill/compare/v1.706.1...v1.707.0) (2026-05-22)
-
-
-### Features
-
-* add wmill job rerun subcommand ([#9275](https://github.com/windmill-labs/windmill/issues/9275)) ([e0ffea2](https://github.com/windmill-labs/windmill/commit/e0ffea2deb5acf30815edd3669f4fc4c818b6e19))
-* **github-app:** hide cloud-only UI on self-managed + admin assignment UI ([#9299](https://github.com/windmill-labs/windmill/issues/9299)) ([dcee8cc](https://github.com/windmill-labs/windmill/commit/dcee8cc0d3dd71c3a12f1720e3ce4eb86cdacf4f))
-* **typescript-client:** add deleteS3File + optional workspace arg on S3 helpers ([#9300](https://github.com/windmill-labs/windmill/issues/9300)) ([daab561](https://github.com/windmill-labs/windmill/commit/daab561ec0763468d93e42e8f7f0796dc77be74d))
-
-
-### Bug Fixes
-
-* **auth:** tighten token-owner fallback for unscoped tokens (WIN-1978) ([#9293](https://github.com/windmill-labs/windmill/issues/9293)) ([7003998](https://github.com/windmill-labs/windmill/commit/7003998a575d76c272c6abd0789a1d1f7b722076))
-* **cli:** wmill sync pull updates wmill-lock.yaml for raw apps ([#9289](https://github.com/windmill-labs/windmill/issues/9289)) ([486e5f9](https://github.com/windmill-labs/windmill/commit/486e5f947b1649c17d32e3b214c50d4be701a4e8))
-* flow recording teardown crash + rename package to @windmill-labs/components ([#9288](https://github.com/windmill-labs/windmill/issues/9288)) ([13a2fae](https://github.com/windmill-labs/windmill/commit/13a2fae745ba4862006db5ee0811475c1d27fd1d))
-* **flows:** restore Variables and Resources in flow editor prop picker ([#9290](https://github.com/windmill-labs/windmill/issues/9290)) ([5566c7b](https://github.com/windmill-labs/windmill/commit/5566c7b3ff2d5a6b15cb9187aa15ce1c7245b3fb))
-* **ResourceEditor:** don't reset state when `selected` reverts to undefined ([#9295](https://github.com/windmill-labs/windmill/issues/9295)) ([1f2d2c1](https://github.com/windmill-labs/windmill/commit/1f2d2c11493db20b87615d41c21e5e1c35564739))
-* **secret-backend:** pass DB to Vault migrations + show failure details ([#9292](https://github.com/windmill-labs/windmill/issues/9292)) ([ace2291](https://github.com/windmill-labs/windmill/commit/ace22910c40585a6a2c9abd0c46f7e5e0214e78e))
-
-## [1.706.1](https://github.com/windmill-labs/windmill/compare/v1.706.0...v1.706.1) (2026-05-22)
-
-
-### Bug Fixes
-
-* fork compare visibility for non-admins and stale-token superadmins ([#9283](https://github.com/windmill-labs/windmill/issues/9283)) ([8272244](https://github.com/windmill-labs/windmill/commit/82722449e79da0b4b0ad4142aec7e7965e9ff236))
-* **git-sync:** bump to hub/28234 with stateless gpg.program wrapper (WIN-1974) ([#9282](https://github.com/windmill-labs/windmill/issues/9282)) ([89a2f07](https://github.com/windmill-labs/windmill/commit/89a2f07218818b95238b4a4484deab3138099672))
-* **nsjail:** gate unix-symlink test behind cfg(unix) for Windows build ([#9280](https://github.com/windmill-labs/windmill/issues/9280)) ([72e2c3a](https://github.com/windmill-labs/windmill/commit/72e2c3a6b3e0cb0f5bddf8291ae18bb8cf55ec28))
-
-## [1.706.0](https://github.com/windmill-labs/windmill/compare/v1.705.0...v1.706.0) (2026-05-21)
-
-
-### Features
-
-* add userdraft listing primitives ([#9268](https://github.com/windmill-labs/windmill/issues/9268)) ([d0ee697](https://github.com/windmill-labs/windmill/commit/d0ee697e8b8de58085ea0b2ecde1af2b2441428d))
-* add UV_PYTHON_INSTALL_MIRROR env and instance setting ([#9271](https://github.com/windmill-labs/windmill/issues/9271)) ([1169371](https://github.com/windmill-labs/windmill/commit/1169371d4885bdc18c76d03c6caae71f0e440235))
-* add yolo mode for ai chat tools ([#9258](https://github.com/windmill-labs/windmill/issues/9258)) ([ac26aa4](https://github.com/windmill-labs/windmill/commit/ac26aa4e4c7cc2d493f136b59738c0708803cc6d))
-* CLI datatable serve / psql ([#9267](https://github.com/windmill-labs/windmill/issues/9267)) ([28c8b5c](https://github.com/windmill-labs/windmill/commit/28c8b5c60fd46f961ae11b363b9be834fad6ee68))
-* **cli:** add `wmill init prompts` and custom override slot ([#9266](https://github.com/windmill-labs/windmill/issues/9266)) ([1ba8ed8](https://github.com/windmill-labs/windmill/commit/1ba8ed8abd827313ce0f7728d9f84357417206ee))
-* **nsjail:** optional disk-backed /tmp via instance setting ([#9272](https://github.com/windmill-labs/windmill/issues/9272)) ([b656dc6](https://github.com/windmill-labs/windmill/commit/b656dc6cdc8c50ef9740240447f119cceed18547))
-
-
-### Bug Fixes
-
-* **ai:** enforce RLS and scope check on user-supplied X-Resource-Path ([#9276](https://github.com/windmill-labs/windmill/issues/9276)) ([0692b97](https://github.com/windmill-labs/windmill/commit/0692b97c8a3818549d7050ea3e057e9cbf1ddb44))
-* **debugger:** add non-root user support to Dockerfile ([#9277](https://github.com/windmill-labs/windmill/issues/9277)) ([0bdb6a9](https://github.com/windmill-labs/windmill/commit/0bdb6a9d5d5fb28a27af1b6eda9fde7172308faf))
-* **indexer:** tell admins when ingress routes search to wrong pod ([#9274](https://github.com/windmill-labs/windmill/issues/9274)) ([d29a561](https://github.com/windmill-labs/windmill/commit/d29a5612fcd17eb4197468289e955a1209127cc1))
-
-## [1.705.0](https://github.com/windmill-labs/windmill/compare/v1.704.1...v1.705.0) (2026-05-20)
-
-
-### Features
-
-* add flow_user_state(key) to QuickJS input transform sandbox (WIN-1947) ([#9093](https://github.com/windmill-labs/windmill/issues/9093)) ([88c1493](https://github.com/windmill-labs/windmill/commit/88c149314576789f46feb5c7e1af3225b061f0c6))
-* add wmill protection-rules pull/push CLI commands ([#9240](https://github.com/windmill-labs/windmill/issues/9240)) ([01bad16](https://github.com/windmill-labs/windmill/commit/01bad16c0cc40fa64b2a72ccb8ded487c729cf35))
-* **chat:** visual redesign — input, streaming indicator, scroll polish ([#9232](https://github.com/windmill-labs/windmill/issues/9232)) ([31a0469](https://github.com/windmill-labs/windmill/commit/31a046973af960764ee4e153b68c020cbd4690ce))
-* **chat:** waiting-for-user indicator + scroll-to-latest polish ([#9252](https://github.com/windmill-labs/windmill/issues/9252)) ([7909878](https://github.com/windmill-labs/windmill/commit/790987831380611b5bd19a760b0a5433492d7796))
-* **cli:** add datatable and ducklake list/run commands ([#9257](https://github.com/windmill-labs/windmill/issues/9257)) ([1d04904](https://github.com/windmill-labs/windmill/commit/1d04904a47245062e50c2cc6362bbbb21a6987aa))
-* **debug:** show ghost breakpoint and tooltip on gutter hover ([#9150](https://github.com/windmill-labs/windmill/issues/9150)) ([271f0cb](https://github.com/windmill-labs/windmill/commit/271f0cbd087851fca86ed1530618ddbf728f13f3))
-* **editors:** responsive top-bars + collapsible raw-app sidebar ([#9237](https://github.com/windmill-labs/windmill/issues/9237)) ([b0ed270](https://github.com/windmill-labs/windmill/commit/b0ed27096d9e918e946cf8a8a04af8ff1892b50f))
-* export audit logs to a dedicated object store folder ([#9207](https://github.com/windmill-labs/windmill/issues/9207)) ([ba6fb70](https://github.com/windmill-labs/windmill/commit/ba6fb7021b5a720bff8e86b4741031902cf1c267))
-* **frontend:** new path component ([#9017](https://github.com/windmill-labs/windmill/issues/9017)) ([9c28bbf](https://github.com/windmill-labs/windmill/commit/9c28bbfd694a5047b4a8a9fe5cc2e54309f8f067))
-* **frontend:** sync home search bar state to URL ([#9256](https://github.com/windmill-labs/windmill/issues/9256)) ([31b7810](https://github.com/windmill-labs/windmill/commit/31b781000e62384af6b8e1ba0172e45ac6ab591f))
-* **git-sync:** hidden `sync git-deploy` owns wm_deploy branch + e2e regression tests ([#9230](https://github.com/windmill-labs/windmill/issues/9230)) ([07202fd](https://github.com/windmill-labs/windmill/commit/07202fd048c999c9d32f3feee94a08625050283d))
-* **indexer:** observability for unavailable search index (WIN-1956) ([#9239](https://github.com/windmill-labs/windmill/issues/9239)) ([285a787](https://github.com/windmill-labs/windmill/commit/285a78752a23aa467f9a82868d784599793d3a1f))
-* **nsjail:** make tmpfs size configurable via instance setting ([#9261](https://github.com/windmill-labs/windmill/issues/9261)) ([9111f89](https://github.com/windmill-labs/windmill/commit/9111f8908de82e9032a63711158dff9c6bca255b))
-* open ai chat path links in drawers ([#9220](https://github.com/windmill-labs/windmill/issues/9220)) ([f6fcdb5](https://github.com/windmill-labs/windmill/commit/f6fcdb5599c28b4890d6f775f657bfafeea1d380))
-* persistent in-editor drafts via UserDraft ([#9121](https://github.com/windmill-labs/windmill/issues/9121)) ([0f7dd86](https://github.com/windmill-labs/windmill/commit/0f7dd86e5c3a43bc62c4c0501efec34226b6e279))
-* resolve relative imports from local content in script/flow preview ([#9233](https://github.com/windmill-labs/windmill/issues/9233)) ([2a780ad](https://github.com/windmill-labs/windmill/commit/2a780ad87af69358f241536697f694612fe92d93))
-* **snowflake:** derive public key from private key when omitted (WIN-1959) ([#9251](https://github.com/windmill-labs/windmill/issues/9251)) ([aa12c66](https://github.com/windmill-labs/windmill/commit/aa12c66c25e68eefec22c213e8f228fd0699d8ce))
-* **vault:** optional KV secret path prefix setting (WIN-1960) ([#9249](https://github.com/windmill-labs/windmill/issues/9249)) ([d08f72b](https://github.com/windmill-labs/windmill/commit/d08f72b3e1ef194b5d656cafa15bc88e8b6ba731))
-
-
-### Bug Fixes
-
-* **autoscaling:** count custom worker groups by row, divide only native by NUM_WORKERS ([#9255](https://github.com/windmill-labs/windmill/issues/9255)) ([76d949e](https://github.com/windmill-labs/windmill/commit/76d949e7bc30eb8cadfdc52e031fcdf5ad97d2ed))
-* **autoscaling:** full-scale below min_workers on large backlog ([#9234](https://github.com/windmill-labs/windmill/issues/9234)) ([a4d59a8](https://github.com/windmill-labs/windmill/commit/a4d59a81dfb6fffbd185a3aa009eb90c160bf42b))
-* bound resource/variable interpolation recursion depth (WIN-1957) ([#9243](https://github.com/windmill-labs/windmill/issues/9243)) ([26f3cbe](https://github.com/windmill-labs/windmill/commit/26f3cbef259e643c6d79be893701eec70b7c0501))
-* cgroup-aware DuckDB memory_limit + allocator memory release ([#9245](https://github.com/windmill-labs/windmill/issues/9245)) ([0022112](https://github.com/windmill-labs/windmill/commit/00221128cbf0801a45bad40246e50beceaba0a7e))
-* collapse successful ai tool details ([#9265](https://github.com/windmill-labs/windmill/issues/9265)) ([413404a](https://github.com/windmill-labs/windmill/commit/413404a788bbe6b5c9df387a2db3000ffec74083))
-* early return should consider failure_module result ([#9241](https://github.com/windmill-labs/windmill/issues/9241)) ([2db1c0a](https://github.com/windmill-labs/windmill/commit/2db1c0a1fcfdcad94cae97dcffa090ffb91494f7))
-* enable jemalloc background_thread to prevent worker RSS growth ([#9236](https://github.com/windmill-labs/windmill/issues/9236)) ([a974ff6](https://github.com/windmill-labs/windmill/commit/a974ff68e00278ccaf441b0567cd46e2b5067fdd))
-* enforce auth guards on app component preview execution ([#9235](https://github.com/windmill-labs/windmill/issues/9235)) ([4b1bea8](https://github.com/windmill-labs/windmill/commit/4b1bea8aed51eb9e24940d89d984ce32f375ab0c))
-* **flows:** flag noLogs jobs and lazily resolve them in log panel ([#9099](https://github.com/windmill-labs/windmill/issues/9099)) ([740a35b](https://github.com/windmill-labs/windmill/commit/740a35bf7b20f0bd8cb94c3d703dd353f0711b0a))
-* **frontend:** flow progress bar for early-stop completion and error handler (WIN-1961) ([#9254](https://github.com/windmill-labs/windmill/issues/9254)) ([cc141ef](https://github.com/windmill-labs/windmill/commit/cc141effa3b1019f70f4b7230fe7ebc7017632a0))
-* **frontend:** open customer portal in popup synchronously to bypass Safari blocker ([#9242](https://github.com/windmill-labs/windmill/issues/9242)) ([f51b51a](https://github.com/windmill-labs/windmill/commit/f51b51a9a1aee5183fa597cf93ff14fbaddaffa9))
-* prevent undefined user flickering in multiplayer presence list ([#9231](https://github.com/windmill-labs/windmill/issues/9231)) ([8c1f6cc](https://github.com/windmill-labs/windmill/commit/8c1f6ccc5d22e657a83831eb5f37a9516cdec10a))
-* **s3:** sandbox stored XSS via download response headers ([#9263](https://github.com/windmill-labs/windmill/issues/9263)) ([bb78b1c](https://github.com/windmill-labs/windmill/commit/bb78b1c06de5b73b951691460f81a3a2ec6e7f80))
-* **saml:** preserve deep links from /a/[...path] across SAML round-trip ([#9259](https://github.com/windmill-labs/windmill/issues/9259)) ([78cf6c7](https://github.com/windmill-labs/windmill/commit/78cf6c7f8181ad431cffebd18c45a3a802b3a601))
-* scope VSCode webview clipboard paste to focused editor ([#9221](https://github.com/windmill-labs/windmill/issues/9221)) ([bd06282](https://github.com/windmill-labs/windmill/commit/bd062825a255da364c1590820ead65172e835d13))
-
-## [1.704.1](https://github.com/windmill-labs/windmill/compare/v1.704.0...v1.704.1) (2026-05-19)
-
-
-### Bug Fixes
-
-* fix git sync ([ff1deaa](https://github.com/windmill-labs/windmill/commit/ff1deaa7e2f3f1f650861c1e2a0f663e597d501c))
-* honor SAML RelayState to redirect to deep link after SSO login ([#9225](https://github.com/windmill-labs/windmill/issues/9225)) ([89306d7](https://github.com/windmill-labs/windmill/commit/89306d7dbc96d0c7dfe2c6025cefc2d72e4f224e))
-* revert git sync script bump ([0f54ecd](https://github.com/windmill-labs/windmill/commit/0f54ecd34cf1ac86ded9305bc84a044eb6a86e72))
-
-## [1.704.0](https://github.com/windmill-labs/windmill/compare/v1.703.3...v1.704.0) (2026-05-18)
-
-
-### Features
-
-* add global ask user question tool ([#9217](https://github.com/windmill-labs/windmill/issues/9217)) ([f965512](https://github.com/windmill-labs/windmill/commit/f965512c7a9aca32c252ca0cda7ec00ab08a38e0))
-* add global chat selected context ([#9216](https://github.com/windmill-labs/windmill/issues/9216)) ([49ebf6f](https://github.com/windmill-labs/windmill/commit/49ebf6f8ba0ea55ea7987f40ecdd32738241a3f2))
-* show job status in favicon on the run page ([#9206](https://github.com/windmill-labs/windmill/issues/9206)) ([2e05bdd](https://github.com/windmill-labs/windmill/commit/2e05bdd73a664ddeec513653ab74e8c696ea1cfd))
-
-
-### Bug Fixes
-
-* don't fail flow on AlreadyCompleted after zombie restart ([#9214](https://github.com/windmill-labs/windmill/issues/9214)) ([8b7f7b3](https://github.com/windmill-labs/windmill/commit/8b7f7b37bdb91449cbd868bd3ee33a0ccbaf288f))
-* **git-sync:** bump default sync script to hub/28229 for extra_perms support ([#9223](https://github.com/windmill-labs/windmill/issues/9223)) ([0538412](https://github.com/windmill-labs/windmill/commit/0538412f1c370981be1915d8c724879d2c54fb83))
-* preserve ai reasoning content ([#9208](https://github.com/windmill-labs/windmill/issues/9208)) ([fec4008](https://github.com/windmill-labs/windmill/commit/fec40086961174fea25b4e1f796991152b84b211))
-* reject path traversal in MCP endpoint path parameters ([#9211](https://github.com/windmill-labs/windmill/issues/9211)) ([ad5ec29](https://github.com/windmill-labs/windmill/commit/ad5ec293b5a189135faea21e0d9c93637b77670f))
-* resolve absolute-path imports in monaco ts editor ([#9213](https://github.com/windmill-labs/windmill/issues/9213)) ([156eb0b](https://github.com/windmill-labs/windmill/commit/156eb0b045171e8d6990af9eeab752071bf7097b))
-
-## [1.703.3](https://github.com/windmill-labs/windmill/compare/v1.703.2...v1.703.3) (2026-05-18)
-
-
-### Bug Fixes
-
-* constrain unauthenticated get_public_resource to app_theme resources ([#9203](https://github.com/windmill-labs/windmill/issues/9203)) ([24eedef](https://github.com/windmill-labs/windmill/commit/24eedef918376d9d401335b6fada577916f8cc0e))
-* enforce folder ACL on flow run-by-version routes ([#9202](https://github.com/windmill-labs/windmill/issues/9202)) ([ab11c77](https://github.com/windmill-labs/windmill/commit/ab11c7747a9076e8121fcea6eafb8e88079ac987))
-* enforce jobs:run scope on job preview and inline endpoints ([#9198](https://github.com/windmill-labs/windmill/issues/9198)) ([664edcd](https://github.com/windmill-labs/windmill/commit/664edcdfb746f6c8513e2b487383b5d9ab9f5434))
-* **mcp:** validate oauth dynamic client registration redirect_uris ([#9197](https://github.com/windmill-labs/windmill/issues/9197)) ([8bc2295](https://github.com/windmill-labs/windmill/commit/8bc2295b94df159a7c8630cdbe02953b8b7c13a1))
-* validate entrypoint override to prevent worker code injection (GHSA-wxjq-w5pj-jqhx) ([#9204](https://github.com/windmill-labs/windmill/issues/9204)) ([bd05bca](https://github.com/windmill-labs/windmill/commit/bd05bcadde06b65fc4b732f576d89aae908b5a3f))
-
-## [1.703.2](https://github.com/windmill-labs/windmill/compare/v1.703.1...v1.703.2) (2026-05-17)
-
-
-### Bug Fixes
-
-* prevent cross-tenant DNS poisoning via writable /etc in nsjail ([#9194](https://github.com/windmill-labs/windmill/issues/9194)) ([f8467f3](https://github.com/windmill-labs/windmill/commit/f8467f38c8a053117ce62f96684cfb15ef792f08))
-
-## [1.703.1](https://github.com/windmill-labs/windmill/compare/v1.703.0...v1.703.1) (2026-05-16)
-
-
-### Bug Fixes
-
-* actionable error when a custom_path is taken by an app in another workspace ([#9190](https://github.com/windmill-labs/windmill/issues/9190)) ([dfeed9c](https://github.com/windmill-labs/windmill/commit/dfeed9c5c2e39bf3e10eea4f69ea140ee9e7832f))
-* atomic bundle cache writes to prevent parallel cold-load race ([#9186](https://github.com/windmill-labs/windmill/issues/9186)) ([81b5736](https://github.com/windmill-labs/windmill/commit/81b573610692b386e4861ef989fa7698b53fc861))
-* detect S3 assets passed as SDK object arg in ts parser ([#9181](https://github.com/windmill-labs/windmill/issues/9181)) ([6a334e9](https://github.com/windmill-labs/windmill/commit/6a334e9a07a7d0cffabde48be75263b0844d586c))
-* don't show ALLOW_PRIVATE_AI_BASE_URLS hint for malformed AI base URLs ([#9188](https://github.com/windmill-labs/windmill/issues/9188)) ([4e25954](https://github.com/windmill-labs/windmill/commit/4e259547225e13e5b51a166a84cdbbbfa35c3264))
-* reset parent_hash in auto_parent when all versions at path are archived ([#9172](https://github.com/windmill-labs/windmill/issues/9172)) ([52960ca](https://github.com/windmill-labs/windmill/commit/52960ca30ab9c019186a28b3ab054a1dfe72f451))
-
-## [1.703.0](https://github.com/windmill-labs/windmill/compare/v1.702.1...v1.703.0) (2026-05-15)
-
-
-### Features
-
-* **otel-tracing-proxy:** configurable tracing MITM NO_PROXY hosts ([#9169](https://github.com/windmill-labs/windmill/issues/9169)) ([d48d61c](https://github.com/windmill-labs/windmill/commit/d48d61cc79114f0b36736306d4015789be10c1f4))
-
-
-### Bug Fixes
-
-* aggregate wait time should target the true root job, not flow_innermost_root_job ([#9177](https://github.com/windmill-labs/windmill/issues/9177)) ([e181931](https://github.com/windmill-labs/windmill/commit/e1819313e15766007c959497a84fae5f5c78a46b))
-* apply pip_local_dependencies filtering to deployed scripts with populated lockfiles ([#9178](https://github.com/windmill-labs/windmill/issues/9178)) ([69b3141](https://github.com/windmill-labs/windmill/commit/69b3141e0370b95f2e13987503480d341608dbdf))
-* never mark failure/trigger/approval scripts as auto_kind=lib ([#9168](https://github.com/windmill-labs/windmill/issues/9168)) ([f414ffc](https://github.com/windmill-labs/windmill/commit/f414ffc4849cf4b92fcd5ca9611ecd246e59a7bd))
-
-## [1.702.1](https://github.com/windmill-labs/windmill/compare/v1.702.0...v1.702.1) (2026-05-14)
-
-
-### Bug Fixes
-
-* **nativets:** pass tracing-enabled OtelConfig to deno_telemetry::init ([#9163](https://github.com/windmill-labs/windmill/issues/9163)) ([bf99283](https://github.com/windmill-labs/windmill/commit/bf99283c3333bcdbc7679f4aea04ba29e41a48a5))
-
-## [1.702.0](https://github.com/windmill-labs/windmill/compare/v1.701.0...v1.702.0) (2026-05-14)
-
-
-### Features
-
-* **git-sync:** sync extra_perms for flows/scripts/apps ([#9162](https://github.com/windmill-labs/windmill/issues/9162)) ([5e909b2](https://github.com/windmill-labs/windmill/commit/5e909b2b4f2819f19deaf06d9e78e6458b324683))
-* include service accounts in instance settings users list ([#9157](https://github.com/windmill-labs/windmill/issues/9157)) ([e5286f4](https://github.com/windmill-labs/windmill/commit/e5286f46074cf2893e6ccd26175f929f16011c8f))
-
-
-### Bug Fixes
-
-* **mcp:** sanitize and enrich nested resource schemas ([#9158](https://github.com/windmill-labs/windmill/issues/9158)) ([d870edc](https://github.com/windmill-labs/windmill/commit/d870edc959481a06c894b4eda5e2be1a0269d7d0))
-
-## [1.701.0](https://github.com/windmill-labs/windmill/compare/v1.700.2...v1.701.0) (2026-05-13)
-
-
-### Features
-
-* **frontend:** unified EditorHeader with file picker for flow/script/app editors ([#9047](https://github.com/windmill-labs/windmill/issues/9047)) ([d0f23cc](https://github.com/windmill-labs/windmill/commit/d0f23cc5238b025208c61e983701894de28536d5))
-* read-only flag on API tokens ([#9144](https://github.com/windmill-labs/windmill/issues/9144)) ([d666e84](https://github.com/windmill-labs/windmill/commit/d666e8431cdbf14d9373d9ef625b5aafc50ac50a))
-
-
-### Bug Fixes
-
-* align script path existence check with deploy logic; hide Delete for non-admin ([#9152](https://github.com/windmill-labs/windmill/issues/9152)) ([c509206](https://github.com/windmill-labs/windmill/commit/c5092069cbeda2c4c18bea80dd629c7c087b30bf))
-* Allow devops role to use all_workspaces runs filter in admins workspace ([#9153](https://github.com/windmill-labs/windmill/issues/9153)) ([110bef0](https://github.com/windmill-labs/windmill/commit/110bef0a6e76615c7b371c5c0f5bc1f4e7a73a64))
-* **bun:** pass --preserve-symlinks on unbundled execution ([#9147](https://github.com/windmill-labs/windmill/issues/9147)) ([4d0f2c2](https://github.com/windmill-labs/windmill/commit/4d0f2c26a116a0f8a89a64231dc824eabda0a8c3))
-* **cli:** prevent !inline-corruption in flow push/pull ([#9142](https://github.com/windmill-labs/windmill/issues/9142)) ([79c5b7b](https://github.com/windmill-labs/windmill/commit/79c5b7b8b7676b0a06fa6480dd04b7105d39d250))
-* **operator:** refresh IAM RDS / Entra ID tokens in operator process ([#9141](https://github.com/windmill-labs/windmill/issues/9141)) ([7ebb081](https://github.com/windmill-labs/windmill/commit/7ebb08133cd4027bc00bacc4a0fc5865cd5709ec))
-* **python:** preserve strings containing Infinity/NaN in result JSON ([#9149](https://github.com/windmill-labs/windmill/issues/9149)) ([33bf01b](https://github.com/windmill-labs/windmill/commit/33bf01b627c8ea430c03dfc27a97a8f2d770582f))
-* scope promotion-mode debounce key per repo ([#9145](https://github.com/windmill-labs/windmill/issues/9145)) ([2ec1863](https://github.com/windmill-labs/windmill/commit/2ec1863340e759bba3408dbc4f41b16912b959ea))
-* send flow push-loop ping outside transaction so zombie monitor sees it ([#9136](https://github.com/windmill-labs/windmill/issues/9136)) ([818cb31](https://github.com/windmill-labs/windmill/commit/818cb31fbc731fa5c70ddf5942bb37bc4bc56e4d))
-
-
-### Performance Improvements
-
-* **dynselect:** only retrigger when helper args actually change ([#9148](https://github.com/windmill-labs/windmill/issues/9148)) ([dd19e52](https://github.com/windmill-labs/windmill/commit/dd19e52a84fb9a9f48e3ad061b084841c2ee7464))
-
-## [1.700.2](https://github.com/windmill-labs/windmill/compare/v1.700.1...v1.700.2) (2026-05-12)
-
-
-### Bug Fixes
-
-* preserve explicit nulls for typed fields in bulk instance config ([#9123](https://github.com/windmill-labs/windmill/issues/9123)) ([cab0000](https://github.com/windmill-labs/windmill/commit/cab0000f3a5e9a0b201a85da1a01b1f82df8a316))
-* preserve negative integers in Bedrock tool schema conversion ([#9116](https://github.com/windmill-labs/windmill/issues/9116)) ([01e21c7](https://github.com/windmill-labs/windmill/commit/01e21c7f913eaf7dffc3d6a31501418ff2104c8b))
-
-## [1.700.1](https://github.com/windmill-labs/windmill/compare/v1.700.0...v1.700.1) (2026-05-11)
-
-
-### Bug Fixes
-
-* CE build broken by enterprise-gated compute_instance_hash ([#9113](https://github.com/windmill-labs/windmill/issues/9113)) ([cd65de4](https://github.com/windmill-labs/windmill/commit/cd65de49285ff60abdd94c883180ded65609f382))
-
-## [1.700.0](https://github.com/windmill-labs/windmill/compare/v1.699.0...v1.700.0) (2026-05-11)
-
-
-### Features
-
-* **cli:** auto-infer args for `wmill app push` ([#9091](https://github.com/windmill-labs/windmill/issues/9091)) ([43b1800](https://github.com/windmill-labs/windmill/commit/43b18006f32fd5db54bbf8ae7ff0e0b314a517e5))
-* **forks:** prompt to delete forked children when deleting a fork ([#9097](https://github.com/windmill-labs/windmill/issues/9097)) ([e43a958](https://github.com/windmill-labs/windmill/commit/e43a958c5c6ae01a1fbecf3db63c6541a245be62))
-* **operators:** allow operators to access assets page ([#9095](https://github.com/windmill-labs/windmill/issues/9095)) ([20ecd90](https://github.com/windmill-labs/windmill/commit/20ecd904e7060c3cf90f2605740bb349b2a3e6ed))
-* **vault:** configurable JWT auth mount path and setup-doc fixes ([#9100](https://github.com/windmill-labs/windmill/issues/9100)) ([f8ba084](https://github.com/windmill-labs/windmill/commit/f8ba0840d74572c880cf458938365b3ec808c6fb))
-
-
-### Bug Fixes
-
-* add Input, Result, Trigger to reserved flow step IDs ([#9109](https://github.com/windmill-labs/windmill/issues/9109)) ([9f79a86](https://github.com/windmill-labs/windmill/commit/9f79a86a686708f66ccc512d4f132cb9a00397a7)), closes [#7139](https://github.com/windmill-labs/windmill/issues/7139)
-* **frontend:** mark Path dirty when folder picker changes selection ([#9096](https://github.com/windmill-labs/windmill/issues/9096)) ([23bb1b5](https://github.com/windmill-labs/windmill/commit/23bb1b541e78846d5978153fd8d9bb4f01cec72b))
-* mask oauth client secret in instance settings ([#9112](https://github.com/windmill-labs/windmill/issues/9112)) ([ac3c155](https://github.com/windmill-labs/windmill/commit/ac3c155541eb5ca20d65c38ad13dca6c10a572c9))
-* populate raw_code for flowscript and appscript runs ([#9104](https://github.com/windmill-labs/windmill/issues/9104)) ([05172ac](https://github.com/windmill-labs/windmill/commit/05172ac3bdfc3472da5e9d8a825cdd479ba9e375))
-
-
-### Performance Improvements
-
-* lazy-load script editor history and hit partial index ([#9107](https://github.com/windmill-labs/windmill/issues/9107)) ([03e8bc8](https://github.com/windmill-labs/windmill/commit/03e8bc8c14258355d7d695333c1588807fbf8cd6))
-
-## [1.699.0](https://github.com/windmill-labs/windmill/compare/v1.698.0...v1.699.0) (2026-05-08)
-
-
-### Features
-
-* parse windmill_failure field to tag run as failure ([#9073](https://github.com/windmill-labs/windmill/issues/9073)) ([dd53202](https://github.com/windmill-labs/windmill/commit/dd5320205f200dd058db2ff7d44d5c4bbcf25ec9))
-
-
-### Bug Fixes
-
-* **cli:** bump svelte version in `wmill app new` template ([#9084](https://github.com/windmill-labs/windmill/issues/9084)) ([4b4aa0e](https://github.com/windmill-labs/windmill/commit/4b4aa0e303f9c47c4f931511977107f42f93abc3))
-* **flows:** populate error handler input args from failure picker ([#9087](https://github.com/windmill-labs/windmill/issues/9087)) ([f37d360](https://github.com/windmill-labs/windmill/commit/f37d3606446d23f8b11a94ea1ce5f5d4836fae17))
-* hide _ENTRYPOINT_OVERRIDE jobs from script/flow history panel ([#9088](https://github.com/windmill-labs/windmill/issues/9088)) ([935c666](https://github.com/windmill-labs/windmill/commit/935c666d50ef30d89a3669c76094af7506fbb448))
-* **native-triggers:** serialize Google channel renewal across replicas ([#9060](https://github.com/windmill-labs/windmill/issues/9060)) ([ee3d82f](https://github.com/windmill-labs/windmill/commit/ee3d82f01f52d835218f544dad6de9b7c3184fbb))
-* **python:** verify wheel RECORD on cache pull/install, finalize piptar ([#9090](https://github.com/windmill-labs/windmill/issues/9090)) ([98ff146](https://github.com/windmill-labs/windmill/commit/98ff146cfabf45418c95c027ad6d07b08069cfcd))
-* reject root-rooted paths in ansible playbook validator on windows ([#9081](https://github.com/windmill-labs/windmill/issues/9081)) ([d37277d](https://github.com/windmill-labs/windmill/commit/d37277d2341c83faf72efa0035cbf70e2cfbd596))
-
-
-### Performance Improvements
-
-* **flows:** gate flow_env resolve on expr text and share cache with handle_flow ([#9085](https://github.com/windmill-labs/windmill/issues/9085)) ([23af6c2](https://github.com/windmill-labs/windmill/commit/23af6c2ea31265a1898d0632e72cd2fd826e4044))
-
-## [1.698.0](https://github.com/windmill-labs/windmill/compare/v1.697.0...v1.698.0) (2026-05-08)
-
-
-### Features
-
-* **cli:** add --parallel flag to generate-metadata ([#9074](https://github.com/windmill-labs/windmill/issues/9074)) ([bc527fd](https://github.com/windmill-labs/windmill/commit/bc527fd929577ac57d4e24196069ed236b702d71))
-
-
-### Bug Fixes
-
-* **cli-tests:** stabilize flow lock-gen race + Windows path ([#9080](https://github.com/windmill-labs/windmill/issues/9080)) ([1c56148](https://github.com/windmill-labs/windmill/commit/1c56148714861aafc4f489916c71aa4674e938c0))
-* **cli:** forward HEADERS env var on every backend fetch call ([#9075](https://github.com/windmill-labs/windmill/issues/9075)) ([d647686](https://github.com/windmill-labs/windmill/commit/d6476862b30692e450cceda09c58d47964f87d32))
-
-
-### Performance Improvements
-
-* **flows:** cache resolved flow_env per flow execution ([#9079](https://github.com/windmill-labs/windmill/issues/9079)) ([e1a7c75](https://github.com/windmill-labs/windmill/commit/e1a7c75e192b72b3b0d854c1901653e0b9386bf2))
-* **flows:** skip flow_env DB+transform work when no resolution is needed ([#9078](https://github.com/windmill-labs/windmill/issues/9078)) ([2067e07](https://github.com/windmill-labs/windmill/commit/2067e0719fd1fd1b899b015badec0f222c054e66))
-
-## [1.697.0](https://github.com/windmill-labs/windmill/compare/v1.696.2...v1.697.0) (2026-05-07)
-
-
-### Features
-
-* add workspace-specific flag for resources and variables ([#8836](https://github.com/windmill-labs/windmill/issues/8836)) ([4427a3d](https://github.com/windmill-labs/windmill/commit/4427a3d37f6e1edef93082322ee346077e0c1ff6))
-* **forks:** handle triggers and schedules in wmill workspace merge ([#9023](https://github.com/windmill-labs/windmill/issues/9023)) ([9de38f9](https://github.com/windmill-labs/windmill/commit/9de38f9a09d2a5759de98849c78f5470bffef94b))
-* **kafka-trigger:** OAUTHBEARER + SASL_SSL support ([#9054](https://github.com/windmill-labs/windmill/issues/9054)) ([80475f0](https://github.com/windmill-labs/windmill/commit/80475f011bf8f7ffbd5afa50fb5df37cdac0825d))
-* **secret-backend:** add Workload Identity Federation for Azure Key Vault ([#9061](https://github.com/windmill-labs/windmill/issues/9061)) ([0c203e8](https://github.com/windmill-labs/windmill/commit/0c203e8cf1cd23fa61675c5691c2649bf0664de0))
-
-
-### Bug Fixes
-
-* **cli:** stable auto-numbered inline-script names in app pull ([#9071](https://github.com/windmill-labs/windmill/issues/9071)) ([8c67e5f](https://github.com/windmill-labs/windmill/commit/8c67e5fdb78b4f17b8866e97bc04168519a65021))
-* **concurrency:** two-phase admit to skip FOR UPDATE on over-limit pulls ([#9064](https://github.com/windmill-labs/windmill/issues/9064)) ([153c4e6](https://github.com/windmill-labs/windmill/commit/153c4e6aff9344a9d6059737c8614c25c9b4443a))
-* handle singlestepflow zombies and stop filtering them from runs page ([#9055](https://github.com/windmill-labs/windmill/issues/9055)) ([e74f06c](https://github.com/windmill-labs/windmill/commit/e74f06cb561d2a2652477e2b6d2ea645e9c77964))
-* Log viewer fixed top bar ([#9070](https://github.com/windmill-labs/windmill/issues/9070)) ([23e081b](https://github.com/windmill-labs/windmill/commit/23e081b078df961d9c42ca12472e3daa7479e290))
-* scope dev server CSS reset to a layer so Tailwind utilities win ([#9069](https://github.com/windmill-labs/windmill/issues/9069)) ([6df79a4](https://github.com/windmill-labs/windmill/commit/6df79a457269a02747d8ec37ba5e97deec7a3e42))
-
-## [1.696.2](https://github.com/windmill-labs/windmill/compare/v1.696.1...v1.696.2) (2026-05-06)
-
-
-### Bug Fixes
-
-* bubble handle_flow chaining errors to parent flow ([#9058](https://github.com/windmill-labs/windmill/issues/9058)) ([5bca03e](https://github.com/windmill-labs/windmill/commit/5bca03eacdbf147268d8ad9079e9ec45b2819af0))
-* **bun:** make hub script cache resilient to malformed lockfiles ([#9063](https://github.com/windmill-labs/windmill/issues/9063)) ([c6f1c5e](https://github.com/windmill-labs/windmill/commit/c6f1c5e623716df703dd6b37be9981dbe5742550))
-* **cli:** detect upstream auth-gateway HTML responses and add poll heartbeat ([#9065](https://github.com/windmill-labs/windmill/issues/9065)) ([628ab56](https://github.com/windmill-labs/windmill/commit/628ab5692e1825001eac0aaa92638c0067a508ea))
-* **queue:** cap worker pull loop at 10 to avoid DB storm ([#9062](https://github.com/windmill-labs/windmill/issues/9062)) ([e3cc258](https://github.com/windmill-labs/windmill/commit/e3cc2584555d532d91aa888cc52af34a16277036))
-
-## [1.696.1](https://github.com/windmill-labs/windmill/compare/v1.696.0...v1.696.1) (2026-05-06)
-
-
-### Bug Fixes
-
-* **bun:** propagate non-zero exit from generate_bun_bundle on no-DB path ([#9051](https://github.com/windmill-labs/windmill/issues/9051)) ([eebaab9](https://github.com/windmill-labs/windmill/commit/eebaab9c87f975b70049e118a08665fc21653b13))
-* **workspaces:** validate fork id as a git branch name component ([#9049](https://github.com/windmill-labs/windmill/issues/9049)) ([f4553e8](https://github.com/windmill-labs/windmill/commit/f4553e8e7919b115a4239a62ee4587347cc82bb8))
-
-## [1.696.0](https://github.com/windmill-labs/windmill/compare/v1.695.0...v1.696.0) (2026-05-05)
-
-
-### Features
-
-* add ai chat resource action buttons ([#9016](https://github.com/windmill-labs/windmill/issues/9016)) ([502a029](https://github.com/windmill-labs/windmill/commit/502a02998685308e82fab95bae7d3c14efd77d6a))
-* add wac ai context for frontend chat ([#9021](https://github.com/windmill-labs/windmill/issues/9021)) ([0d0557f](https://github.com/windmill-labs/windmill/commit/0d0557fc9dc5addee887911ddfc0fd08a09bc92e))
-* **cli:** add --as-superadmin flag to workspace list-remote ([#9043](https://github.com/windmill-labs/windmill/issues/9043)) ([66c9063](https://github.com/windmill-labs/windmill/commit/66c90639191a77eb4f19da092167384565edb9b3))
-
-
-### Bug Fixes
-
-* **cli:** resolve cross-folder relative imports during lockgen on fresh DB ([#9048](https://github.com/windmill-labs/windmill/issues/9048)) ([40dbab5](https://github.com/windmill-labs/windmill/commit/40dbab531e5166b894f3f94b0d72b2ac456c0097))
-* **flows:** inherit flow_env in sub-flow predicates ([#9042](https://github.com/windmill-labs/windmill/issues/9042)) ([6e5a21a](https://github.com/windmill-labs/windmill/commit/6e5a21a9c7b5db77d325b5916a9ab8799a2eb6e7))
-* navigate home arrows ([#9024](https://github.com/windmill-labs/windmill/issues/9024)) ([c1e52ea](https://github.com/windmill-labs/windmill/commit/c1e52eab09794746bea7dc9adb94552641f87d5b))
-* open job detail header path links in a new tab ([#9039](https://github.com/windmill-labs/windmill/issues/9039)) ([fe68c06](https://github.com/windmill-labs/windmill/commit/fe68c066004d860088e09be32e7ff2e7438f78c4))
-* **rust-client:** re-export models module from wmill crate ([#9038](https://github.com/windmill-labs/windmill/issues/9038)) ([ca6efbf](https://github.com/windmill-labs/windmill/commit/ca6efbff74e7d7e85174b1d8394af79eda7d6535))
-* **windmill-utils-internal:** move config to subpath export ([#9045](https://github.com/windmill-labs/windmill/issues/9045)) ([b86f896](https://github.com/windmill-labs/windmill/commit/b86f8960fcd8a66bc6849638178ef45ac49e06f1))
-
-## [1.695.0](https://github.com/windmill-labs/windmill/compare/v1.694.0...v1.695.0) (2026-05-04)
-
-
-### Features
-
-* add separate filter searchbar for resource types tab ([#9019](https://github.com/windmill-labs/windmill/issues/9019)) ([f1fd245](https://github.com/windmill-labs/windmill/commit/f1fd245073d6bf97a6a6c64e64d545618cccc432))
-
-
-### Bug Fixes
-
-* **autoscaling:** consider dedicated workers in scale decisions ([#9020](https://github.com/windmill-labs/windmill/issues/9020)) ([42be1d4](https://github.com/windmill-labs/windmill/commit/42be1d46a632c23830f97995e9ab1b52a1ed5d3d))
-* bind MySQL table listing to configured database name ([#9007](https://github.com/windmill-labs/windmill/issues/9007)) ([44fad13](https://github.com/windmill-labs/windmill/commit/44fad139fe2076f5faa62de31aaeeb217466b25e))
-* **flows:** don't bubble error when continue_on_error is on the last step ([#9029](https://github.com/windmill-labs/windmill/issues/9029)) ([192866d](https://github.com/windmill-labs/windmill/commit/192866d5197c74ec930d6fe7bf9234fac76763f4))
-* **forks:** strip mode/enabled from merge-UI deploy payload ([#9008](https://github.com/windmill-labs/windmill/issues/9008)) ([da95588](https://github.com/windmill-labs/windmill/commit/da95588b253e8bb2a06b9792674b4d691384f55a))
-* stop sequential whileloop on iteration failure ([#9028](https://github.com/windmill-labs/windmill/issues/9028)) ([1be62ea](https://github.com/windmill-labs/windmill/commit/1be62ea926872882ddbd4c8ce81502d6e341b8c1))
-
-## [1.694.0](https://github.com/windmill-labs/windmill/compare/v1.693.4...v1.694.0) (2026-05-01)
-
-
-### Features
-
-* ansible delegate_to_git_repo install_requirements, dynamic fields, --limit ([#8997](https://github.com/windmill-labs/windmill/issues/8997)) ([96324ea](https://github.com/windmill-labs/windmill/commit/96324ea5aed4054d33895102ec9313f5dadc77a2))
-* **cli:** wmill-lock.yaml auto-fill + --rehash-only + path-prefix dedup ([#8978](https://github.com/windmill-labs/windmill/issues/8978)) ([0b959b8](https://github.com/windmill-labs/windmill/commit/0b959b8ec61b24d861c5a10a9242a7a0e6013707))
-* **forks:** handle triggers and schedules in workspace forks ([#8976](https://github.com/windmill-labs/windmill/issues/8976)) ([d60dd74](https://github.com/windmill-labs/windmill/commit/d60dd745e49853bb130b139f300fc0f2ab8ebe39))
-* support assigning a worker tag to app inline scripts ([#9002](https://github.com/windmill-labs/windmill/issues/9002)) ([0c22f52](https://github.com/windmill-labs/windmill/commit/0c22f52b46c56d3577309e37c1e81a1a1feb9b7c))
-
-
-### Bug Fixes
-
-* **cli:** only preserve case for raw-app runnableIds, not app/flow summaries ([#9000](https://github.com/windmill-labs/windmill/issues/9000)) ([5d5b853](https://github.com/windmill-labs/windmill/commit/5d5b853f70a73453f63d14edcd5d2fac8e3d804c))
-* distinguish AlreadyCompleted from execution failure on OTLP job span ([#9004](https://github.com/windmill-labs/windmill/issues/9004)) ([70a5880](https://github.com/windmill-labs/windmill/commit/70a5880d3619edece4ce67a00407ee0d2d523469))
-* nested-restart iteration count for step-id collisions across subflow boundaries ([#9003](https://github.com/windmill-labs/windmill/issues/9003)) ([ad9f1fa](https://github.com/windmill-labs/windmill/commit/ad9f1fa4541f2eefb0b013bac42cd626af424852))
-* omit empty assets array on scripts and raw app inline scripts ([#9006](https://github.com/windmill-labs/windmill/issues/9006)) ([419bc4b](https://github.com/windmill-labs/windmill/commit/419bc4b1757a7c20c2b7aa9b7d3b02e3515f934a))
-* pair PG arg type with actual Rust binding to keep query_typed_raw safe ([#8999](https://github.com/windmill-labs/windmill/issues/8999)) ([aedf369](https://github.com/windmill-labs/windmill/commit/aedf3691744748a307976ef01ea7f63b0961fad4))
-* route email trigger path through standard info channel ([#8996](https://github.com/windmill-labs/windmill/issues/8996)) ([2141128](https://github.com/windmill-labs/windmill/commit/21411282bb4a0442046bf71fdc7ea012fa2c3d3d))
-* surface scope errors as 403 and show real message in CLI ([#8953](https://github.com/windmill-labs/windmill/issues/8953)) ([66db873](https://github.com/windmill-labs/windmill/commit/66db873651a04b0701d8231fbf11ab973c3fc69b))
-* use otel.status_message for OTLP Status.message on failed jobs ([#8995](https://github.com/windmill-labs/windmill/issues/8995)) ([9cb777a](https://github.com/windmill-labs/windmill/commit/9cb777a6b6e969696cf9beade1cf07c86967dafc))
-
-## [1.693.4](https://github.com/windmill-labs/windmill/compare/v1.693.3...v1.693.4) (2026-04-30)
-
-
-### Bug Fixes
-
-* **cli:** pin wasm parser versions in published package.json ([#8993](https://github.com/windmill-labs/windmill/issues/8993)) ([7c227ec](https://github.com/windmill-labs/windmill/commit/7c227ece0d546ef0f521a963f927e4c913011e8e))
-
-## [1.693.3](https://github.com/windmill-labs/windmill/compare/v1.693.2...v1.693.3) (2026-04-30)
-
-
-### Bug Fixes
-
-* avoid named prepared statements in datatable/PG executor ([#8988](https://github.com/windmill-labs/windmill/issues/8988)) ([7bdfc0e](https://github.com/windmill-labs/windmill/commit/7bdfc0ea0636a93a893ab5f9f58a590e6a0b7be8))
-* improve raw app builder queue behavior for bigger apps ([e7deaf9](https://github.com/windmill-labs/windmill/commit/e7deaf988254eebbf99febc8a644622c9f809bd8))
-* sanitize underscores in agent worker suffix ([#8992](https://github.com/windmill-labs/windmill/issues/8992)) ([568d9cc](https://github.com/windmill-labs/windmill/commit/568d9cc8a086178314e15e3c27fb55ff4817f5cd))
-* **workspaces:** split get_settings into admin-only + public endpoint ([#8990](https://github.com/windmill-labs/windmill/issues/8990)) ([4483d0c](https://github.com/windmill-labs/windmill/commit/4483d0cab91e8a9df026b09c7908291f3fbbfd63))
-
-## [1.693.2](https://github.com/windmill-labs/windmill/compare/v1.693.1...v1.693.2) (2026-04-30)
-
-
-### Bug Fixes
-
-* avoid effect_update_depth_exceeded when clicking flow node on runs page ([#8986](https://github.com/windmill-labs/windmill/issues/8986)) ([3ebfc2b](https://github.com/windmill-labs/windmill/commit/3ebfc2b0af38f7eb17774de08a214d7813e07952))
-* OAuth popup login reliability + auto-login Safari edge cases ([#8971](https://github.com/windmill-labs/windmill/issues/8971)) ([3c3c034](https://github.com/windmill-labs/windmill/commit/3c3c03455d68fde982937787992e20c3f8eeeaaf))
-
-## [1.693.1](https://github.com/windmill-labs/windmill/compare/v1.693.0...v1.693.1) (2026-04-29)
-
-
-### Bug Fixes
-
-* include labels when loading flow with draft for editing ([#8981](https://github.com/windmill-labs/windmill/issues/8981)) ([485d1d1](https://github.com/windmill-labs/windmill/commit/485d1d1e3785b5ed7e5f1a5ee127c0fed15fba3e)), closes [#8963](https://github.com/windmill-labs/windmill/issues/8963)
-
 ## [1.693.0](https://github.com/windmill-labs/windmill/compare/v1.692.0...v1.693.0) (2026-04-29)
 
 

Dockerfile

@@ -66,7 +66,6 @@ RUN npm ci
 COPY frontend .
 RUN mkdir /backend
 COPY /backend/windmill-api/openapi.yaml /backend/windmill-api/openapi.yaml
-COPY /backend/oauth_connect.json /backend/oauth_connect.json
 COPY /openflow.openapi.yaml /openflow.openapi.yaml
 COPY /backend/windmill-api/build_openapi.sh /backend/windmill-api/build_openapi.sh
 COPY /system_prompts/auto-generated /system_prompts/auto-generated
@@ -233,14 +232,11 @@ ENV PATH="${PATH}:/usr/local/go/bin"
 ENV GO_PATH=/usr/local/go/bin/go
 
 # Install UV
-RUN curl --proto '=https' --tlsv1.2 -LsSf https://github.com/astral-sh/uv/releases/download/0.9.25/uv-installer.sh | sh && mv /root/.local/bin/uv /usr/local/bin/uv
+RUN curl --proto '=https' --tlsv1.2 -LsSf https://github.com/astral-sh/uv/releases/download/0.9.24/uv-installer.sh | sh && mv /root/.local/bin/uv /usr/local/bin/uv
 
 # Preinstall python runtimes to temp build location (will copy with world-writable perms later)
-# --compile-bytecode precompiles the stdlib to .pyc so jobs don't recompile it on every run
-# under the read-only nsjail runtime mount (uv >= 0.9.25). The copy below MUST preserve
-# timestamps or Python's mtime-based .pyc invalidation discards these compiled files.
-RUN UV_CACHE_DIR=/tmp/build_cache/uv UV_PYTHON_INSTALL_DIR=/tmp/build_cache/py_runtime uv python install 3.11 --compile-bytecode
-RUN UV_CACHE_DIR=/tmp/build_cache/uv UV_PYTHON_INSTALL_DIR=/tmp/build_cache/py_runtime uv python install $LATEST_STABLE_PY --compile-bytecode
+RUN UV_CACHE_DIR=/tmp/build_cache/uv UV_PYTHON_INSTALL_DIR=/tmp/build_cache/py_runtime uv python install 3.11
+RUN UV_CACHE_DIR=/tmp/build_cache/uv UV_PYTHON_INSTALL_DIR=/tmp/build_cache/py_runtime uv python install $LATEST_STABLE_PY
 
 
 RUN curl -sL https://deb.nodesource.com/setup_20.x | bash -
@@ -262,7 +258,7 @@ RUN export GOCACHE=/tmp/build_cache/go && \
 # chmod a+rw adds read+write WITHOUT removing execute bits (755->777, 644->666)
 # Note: uv python install only creates py_runtime, not uv cache - we create uv/go dirs for runtime
 RUN mkdir -p /tmp/windmill/cache && \
-    cp -r --preserve=timestamps /tmp/build_cache/* /tmp/windmill/cache/ && \
+    cp -r /tmp/build_cache/* /tmp/windmill/cache/ && \
     chmod -R a+rw /tmp/windmill/cache && \
     rm -rf /tmp/build_cache && \
     mkdir -p -m 777 /tmp/windmill/cache/uv /tmp/windmill/cache/go /tmp/windmill/cache/rustup /tmp/windmill/cache/cargo
@@ -303,20 +299,10 @@ ENV CARGO_HOME="/tmp/windmill/cache/cargo"
 ENV LD_LIBRARY_PATH="."
 
 # nsjail runtime deps and binary
-RUN apt-get update && apt-get install -y --no-install-recommends libprotobuf32 libnl-route-3-200 libnl-3-200 \
+RUN apt-get update && apt-get install -y libprotobuf-dev libnl-route-3-dev \
     && apt-get clean && rm -rf /var/lib/apt/lists/*
 COPY --from=nsjail /nsjail/nsjail /bin/nsjail
 
-# crane: pulls + flattens images for the sandboxed container runtime (`# sandbox <image>`).
-# Single static binary — no daemon/store/root needed. See docs/docker-v2-runtime.md.
-ARG CRANE_VERSION=v0.20.6
-RUN arch="$(dpkg --print-architecture)"; \
-    case "$arch" in amd64) crane_arch=x86_64 ;; arm64) crane_arch=arm64 ;; *) echo >&2 "error: unsupported arch '$arch' for crane"; exit 1 ;; esac; \
-    wget -O /tmp/crane.tgz "https://github.com/google/go-containerregistry/releases/download/${CRANE_VERSION}/go-containerregistry_Linux_${crane_arch}.tar.gz" \
-    && tar -xzf /tmp/crane.tgz -C /usr/local/bin crane \
-    && rm /tmp/crane.tgz \
-    && chmod +x /usr/local/bin/crane
-
 WORKDIR ${APP}
 
 RUN ln -s ${APP}/windmill /usr/local/bin/windmill

REVIEW.md

@@ -1,69 +0,0 @@
-# Pull request review — shared policy
-
-You are reviewing a GitHub pull request for this repository. Apply this policy alongside your tool's output requirements.
-
-## Read the project rules first
-
-- Read `AGENTS.md` (repo root) and any `AGENTS.md` in directories touched by the diff before reviewing — they are the canonical contributor guide.
-- Quote the exact rule from `AGENTS.md` when flagging a violation.
-
-## Verdict (first line of the review)
-
-Start every review with a single verdict line, before any other section (the only thing that may appear above the verdict is the optional `cc @<PR_AUTHOR>` ping described in "Pinging the author" below). Pick exactly one:
-
-- **Good to merge** — no blocking issues and no nits worth surfacing.
-- **Mergeable, but should ideally address nits: <short list>** — no blockers, but P2 findings that are worth a look. The list must name each nit briefly (e.g. "doc/code mismatch in `foo.rs`, half-finished `pub fn bar`").
-- **Should address issues before merging: <short list>** — at least one P0 or P1 finding. The list must name each blocking issue briefly (e.g. "missing auth check on new `/api/x` handler, SQL injection in `build_query`").
-
-The names in the list must match findings detailed later in the review. If you list a nit or issue here, it must appear with full context in the body. Do not invent items that aren't in the body, and do not bury blockers in the body without surfacing them in the verdict.
-
-## Pinging the author
-
-If the prompt context provides a `PR AUTHOR` (GitHub login) and the verdict is NOT "Good to merge" (i.e. it is "Mergeable, but should ideally address nits: ..." or "Should address issues before merging: ..."), prepend a single line `cc @<PR_AUTHOR>` to the top-level review comment, above the verdict line. This pings the author so they get a notification that there are items to address. Skip the ping entirely when the verdict is "Good to merge" — there is nothing for the author to act on. Do not add the ping to inline comments; the top-level summary comment is the only place it belongs.
-
-## Review policy
-
-- Only report issues you are confident are real and introduced by this pull request.
-- Focus on bugs, security problems, performance, and clear `AGENTS.md` violations.
-- Do not report style nits, speculative concerns, pre-existing issues, or anything a normal linter / typechecker would obviously catch.
-- Self-validate each finding before posting: "is this definitely a real issue?" If uncertain, discard it.
-- Read additional files only when the diff is not enough to validate a finding.
-- Do not modify any files.
-
-## Severity triage
-
-Tag each finding with a severity. Always report P0 and P1. Report P2 only when the diff invites it (a new `pub fn`, a new module, a new exported component, a meaningful refactor).
-
-- **P0** — RCE, auth bypass, data loss, secrets in code, SQL injection, path traversal, broken auth on a public surface.
-- **P1** — significant bug, missing auth/authorization check on a new public surface, blocking I/O on a likely async path, race condition, missing input validation on caller-controlled parameters, observable performance regression.
-- **P2** — wrong module placement, doc/code mismatch, half-finished public abstractions (`pub fn` + `#[allow(dead_code)]` + `TODO`), `AGENTS.md` style violations, naming that contradicts the function's behavior.
-
-## Checklist for new public surfaces
-
-For any new `pub fn` / `pub async fn` / exported Svelte component / exported prop introduced by this PR, verify:
-
-- (a) auth/authorization expectations are documented in the doc comment OR enforced in the function body. A new `pub fn` that touches workspace data, secrets, files, or processes without an auth check or documented "caller MUST verify" contract is a P1.
-- (b) the function is placed in a module whose stated purpose matches what it does. Check the module-level doc comment (`//!`) — a config-file reader inside `external_ip.rs` is a P2.
-- (c) it is not half-finished. `pub fn` + `#[allow(dead_code)]` + a `TODO` is a smell that says the function should land together with its caller, not separately. Cite the relevant `AGENTS.md` rule.
-- (d) input validation defends against injection / traversal / overflow / NUL bytes at every parameter that may be caller-controlled.
-
-## Test coverage assessment
-
-End your review with a short "Test coverage" section calibrated to the layers actually changed by the diff. Skip categories the PR does not touch.
-
-- **Backend** (Rust under `backend/`) — expect Rust unit tests for new logic. For new or modified API handlers, worker steps, queue/cron behavior, or DB access, also expect or note the absence of integration tests. Pure-refactor backend PRs don't need new tests if existing tests cover the surface.
-- **Frontend** (Svelte / TS under `frontend/`) — the codebase does not generally test Svelte components, so do not ask for component tests. Only flag missing tests for new pure-logic utilities (the kind of file that already has a sibling `*.test.ts`, e.g. `flowDiff`, `previousResults`, copilot logic).
-- **CI / workflows / docs / config-only** — no automated tests expected; say so explicitly so the reader knows you considered it.
-
-Then state what manual verification, if any, is still needed before merge:
-
-- Describe each manual scenario as a short paragraph (not a numbered list): what page / action / input, and what observable outcome confirms correctness.
-- If the diff has no in-app surface to exercise (purely backend internals, CI, docs, or refactor), say that plainly.
-
-## Additional reviewer instructions
-
-If the prompt or context includes an "Additional reviewer instructions" section, treat it as extra guidance from the human who triggered this review and follow it.
-
-## Prior PR discussion
-
-If the prompt or context includes a "Prior PR discussion" section, this PR has already received review activity. Look for your own previous comment, take it into account, focus on what changed in the latest commits, and do not repeat findings the human already pushed back on or addressed.

ai_evals/AGENTS.md

@@ -6,7 +6,6 @@ This folder contains black-box benchmark cases for:
 - `app`
 - `script`
 - `cli`
-- `global`
 
 The goal is to test the current production prompts and guidance with realistic user requests, not to test one exact implementation shape.
 
@@ -76,41 +75,6 @@ Still, avoid benchmark phrasing. The prompt should read like a repo task, not a
 
 When relevant, ask the assistant to tell the user which `wmill` commands to run next. That is part of the benchmarked behavior.
 
-## Global-specific rules
-
-Global prompts should exercise workspace-level drafting behavior:
-
-- inspecting existing scripts, flows, apps, schedules, triggers, resources, and variables when relevant
-- writing AI drafts rather than saving or deploying by default
-- producing coherent multi-artifact changes when the request crosses artifact boundaries
-
-Keep deterministic validation focused on the draft contract: required draft type/path, required content snippets, forbidden draft paths, and forbidden mutating tools such as deploy/delete unless the case explicitly asks for them.
-
-Datatable cases should set `skipJudge: true` and validate through tool-use
-(`requiredToolsUsed` / `forbiddenToolsUsed`) and SQL-argument assertions
-(`toolCallArgs` with `stringIncludesAnyOf`, e.g. `['select']`, `['create table']`,
-`['update', 'insert into']`). Two reasons the judge is unreliable here:
-
-- `list_datatables`, `get_datatable_table_schema`, and `exec_datatable_sql`
-  produce no drafts, and the global judge only sees the drafts artifact — it
-  scores a no-draft conversational answer as empty (same as the
-  `askUserQuestion` cases).
-- Even a case that *does* produce a draft (a script reading the data table via
-  `wmill.datatable()` at runtime) is mis-judged: the judge has no datatable SDK
-  reference and penalizes correct `wmill.datatable()` usage as wrong. Verify the
-  SDK call deterministically instead — `requiredDrafts.valueIncludes: ['wmill.datatable(']`
-  plus forbidding `exec_datatable_sql` (keeping chat-time SQL distinct from
-  runtime SDK use).
-
-`stringIncludesAnyOf` is existential over calls (at least one matching call), so a
-mutation case still passes when the model mixes its UPDATE/INSERT with
-verification SELECTs. The in-memory engine (`datatableSqlEngine.ts`) is stateful
-within a case — writes persist, so a model that re-queries to verify its
-CREATE/UPDATE sees the change and does not loop. But the engine is best-effort
-(SELECT returns all rows of the referenced/first table with no WHERE/projection),
-so still never assert specific returned row values. Seed data via
-`workspace.datatables` in the `initial` fixture (see README).
-
 ## Deterministic validation
 
 Use deterministic validation only for hard failures such as:

ai_evals/README.md

@@ -1,12 +1,11 @@
 # AI Evals
 
-Small benchmark runner for the Windmill AI generation modes:
+Small benchmark runner for the four Windmill AI generation modes:
 
 - `cli`
 - `flow`
 - `script`
 - `app`
-- `global`
 
 The benchmark always tests the current production prompts, tools, and guidance in this checkout.
 
@@ -56,9 +55,8 @@ bun run cli -- run flow flow-test4-order-processing-loop --model opus
 bun run cli -- run flow flow-test0-sum-two-numbers --models haiku,opus,4o
 bun run cli -- run flow flow-test0-sum-two-numbers --runs 3 --verbose
 bun run cli -- run flow --record
-GEMINI_API_KEY=... bun run cli -- run app app-test1-counter-create --model gemini-3-flash-preview
+GEMINI_API_KEY=... bun run cli -- run app app-test1-counter-create --model gemini-pro --transport proxy
 WMILL_AI_EVAL_BACKEND_URL=http://127.0.0.1:8000 bun run cli -- run flow --backend-validation preview
-bun run cli -- run global global-test1-script-create
 bun run cli -- run cli bun-hello-script
 ```
 
@@ -74,9 +72,8 @@ Public CLI surface:
 - `--output <path>`: custom result JSON path
 - `--model <alias>`: choose the model under test
 - `--models <a,b,c>`: run the same cases sequentially against several model aliases
+- `--transport <mode>`: frontend request transport (`direct` by default, `proxy` to exercise `/api/w/{workspace}/ai/proxy`)
 - `--verbose`: stream assistant output for frontend runs
-- `--skip-judge`: skip LLM judge scoring for the run
-- `--execution-only`: only require the model/proxy/frontend loop to complete; skip validators, tool expectations, backend artifact validation, and judge scoring
 - `--record`: append a compact tracked summary line to `ai_evals/history/<mode>.jsonl` for full-suite runs only
 - `--backend-validation <mode>`: optional backend smoke validation (`off` or `preview`) for `script` and `flow` evals
 
@@ -90,18 +87,17 @@ Today:
 - `sonnet`
 - `opus`
 - `4o`
-- `gpt-5.5`
+- `gemini-flash`
+- `gemini-pro`
 - `gemini-3-flash-preview`
 - `gemini-3.1-pro-preview`
-- `deepseek-v4-flash`
-- `deepseek-v4-pro`
 
 Notes:
 
-- the command also prints accepted alias spellings such as `gpt-4o`, `gpt-55`, `claude-opus-4.6`, and `claude-haiku-4.5`
-- frontend modes (`flow`, `script`, `app`, `global`) can use Anthropic, OpenAI, Gemini, and DeepSeek-backed aliases
+- the command also prints accepted alias spellings such as `gpt-4o`, `claude-opus-4.6`, and `claude-haiku-4.5`
+- frontend modes (`flow`, `script`, `app`) can use Anthropic, OpenAI, and Gemini-backed aliases
 - `cli` mode always uses the Anthropic agent SDK, so only Anthropic aliases are valid there
-- the judge model is separate and currently defaults to `claude-sonnet-4-6`; use `--skip-judge` for deterministic-only runs
+- the judge model is separate and currently defaults to `claude-sonnet-4-6`
 
 ## Case Format
 
@@ -138,39 +134,6 @@ For `app` mode, `validate` can express narrow hard requirements such as:
 - minimum datatable / datatable-table counts
 - specific required datatable tables
 
-For `global` mode, `validate` can express draft-level requirements such as:
-
-- required draft type/path/language
-- required or forbidden snippets in draft values
-- required or forbidden draft counts
-- forbidden draft paths
-
-Global initial fixtures can also seed `liveEditorDrafts` with `type`,
-`storagePath`, `effectivePath`, and `value` fields. These drafts emulate the
-currently open script, flow, or raw app editor so cases can test prompts that
-refer to "this" or the "current" item.
-
-Global (and flow) initial fixtures can seed `workspace.datatables` so the
-`list_datatables`, `get_datatable_table_schema`, and `exec_datatable_sql` tools
-return seeded data during evals. Each entry is
-`{ datatable_name, schemas: { <schema>: { <table>: { columns, rows? } } } }`.
-SQL runs through a small in-memory engine (`datatableSqlEngine.ts`), not a real
-database. Writes are **stateful within a case**: `CREATE`/`DROP`/`INSERT`/`UPDATE`/
-`DELETE` mutate the seeded datatable in place, so a later `list_datatables`,
-`get_datatable_table_schema`, `SELECT`, or `information_schema` query reflects them
-— this is what stops a model from looping when it re-queries to verify a write.
-The engine is best-effort: `SELECT` returns all rows of the referenced (or first)
-table with no WHERE filtering/projection/joins, `WHERE` on UPDATE/DELETE supports
-`col = value` predicates joined by `AND`, and anything unparseable is a no-op
-success. So validate datatable cases through tool-use and SQL-argument assertions
-(`requiredToolsUsed`, `stringIncludesAnyOf`) — not through exact returned row
-values. An empty/absent `datatables` seed makes `list_datatables` return `[]`,
-which is what the "no datatable configured" blocking cases rely on.
-
-Set `WMILL_AI_EVAL_DISABLE_ACTIVE_EDITOR_CONTEXT=1` to run those cases with
-the old behavior where the live editor is only discoverable through
-`list_workspace_items`.
-
 App fixtures can also include an optional `datatables.json` file at the fixture root.
 
 For `flow` mode, an `initial` fixture can also include a benchmark workspace catalog of
@@ -182,23 +145,26 @@ If `--backend-validation preview` is enabled:
 - `script` evals run a real backend script preview in an isolated temp workspace
 - `flow` evals run a real backend flow preview only for cases that define `runtime.backendPreview`
 - `flow` cases with `initial.workspace` fixtures seed those scripts and flows into the preview workspace before preview
-- when `WMILL_AI_EVAL_BACKEND_WORKSPACE` is set, `ai_evals` creates or reuses that workspace as a dedicated test workspace, clears managed eval assets under `f/evals/*` before each preview run, and then reseeds the current case fixtures
+- when `WMILL_AI_EVAL_BACKEND_WORKSPACE` is set, `ai_evals` treats that workspace as a dedicated test workspace, clears managed eval assets under `f/evals/*` before each preview run, and then reseeds the current case fixtures
 
-Supported backend env vars:
+Supported backend validation env vars:
 
 - `WMILL_AI_EVAL_BACKEND_VALIDATION=preview`
 - `WMILL_AI_EVAL_BACKEND_URL=http://127.0.0.1:8000`
 - `WMILL_AI_EVAL_BACKEND_EMAIL=admin@windmill.dev`
 - `WMILL_AI_EVAL_BACKEND_PASSWORD=changeme`
 - `WMILL_AI_EVAL_BACKEND_WORKSPACE=integration-tests` to reuse an existing workspace on CE installs with low workspace limits
+- `WMILL_AI_EVAL_KEEP_WORKSPACES=1`
+- `WMILL_AI_EVAL_WORKSPACE_PREFIX=ai-evals`
 
-Frontend modes require a reachable Windmill backend and send model requests through the workspace AI proxy at `/api/w/{workspace}/ai/proxy`. At startup, `ai_evals` checks the resolved backend URL and fails early with setup guidance if the backend cannot be reached or login fails.
+Frontend proxy transport uses the same backend auth/workspace env vars.
 
-For frontend modes:
+When `--transport proxy` is set:
 
-- `ai_evals` creates a temporary backend workspace, or creates/reuses `WMILL_AI_EVAL_BACKEND_WORKSPACE` when it is set
+- `ai_evals` creates or reuses a backend workspace
 - it upserts a provider resource under `f/evals/ai/<provider>`
 - frontend requests go through `/api/w/{workspace}/ai/proxy`
+- result JSON and history records include `transport` so direct vs proxy runs stay distinguishable
 
 ## Results And Artifacts
 
@@ -212,21 +178,16 @@ If `--record` is used, the CLI also appends one compact JSON line to:
 - `ai_evals/history/flow.jsonl`
 - `ai_evals/history/script.jsonl`
 - `ai_evals/history/app.jsonl`
-- `ai_evals/history/global.jsonl`
 - `ai_evals/history/cli.jsonl`
 
 Each recorded line contains:
 
-- run metadata (`createdAt`, `gitSha`, `mode`, `runModel`, `judgeModel`)
-- suite totals (`caseCount`, `attemptCount`, `passedAttempts`, `passRate`, `averageDurationMs`, `averagePassedDurationMs`, `averageJudgeScore`)
-- average token usage (`averageTokenUsagePerAttempt`, `averageTokenUsagePerPassedAttempt`)
-- per-case metrics under `cases[]` (`averageDurationMs`, `averagePassedDurationMs`, `averageJudgeScore`, `averageTokenUsagePerAttempt`, `averageTokenUsagePerPassedAttempt`, pass rate)
+- run metadata (`createdAt`, `gitSha`, `mode`, `runModel`, `transport`, `judgeModel`)
+- suite totals (`caseCount`, `attemptCount`, `passedAttempts`, `passRate`, `averageDurationMs`, `averageJudgeScore`)
+- average token usage (`averageTokenUsagePerAttempt`)
+- per-case metrics under `cases[]` (`averageDurationMs`, `averageJudgeScore`, `averageTokenUsagePerAttempt`, pass rate)
 - `failedCaseIds`
 
-The CLI headline duration and token averages use passed attempts only.
-All-attempt averages are still recorded to make failures auditable without
-letting failed attempts skew success cost comparisons.
-
 Example:
 
 - summary: `ai_evals/results/2026-04-09T09-40-33.051Z__flow.json`
@@ -237,7 +198,6 @@ Typical artifacts by mode:
 - `flow`: `flow.json`
 - `script`: `script.json` plus the generated script file
 - `app`: `app.json` plus frontend/backend files
-- `global`: `global-drafts.json`
 - `cli`: `assistant-output.txt`, `trace.json`, `wmill-invocations.jsonl`, plus generated workspace files
 - backend-validated attempts also include `backend-preview.json`
 
@@ -253,7 +213,6 @@ Typical artifacts by mode:
 ## Notes
 
 - Frontend modes reuse the production frontend chat code through the Vitest bridge.
-- Global mode evaluates the production global AI tools and validates the resulting AI draft store.
 - CLI mode creates an isolated workspace, writes the current checkout guidance into it, and benchmarks the real skills / `AGENTS.md` flow.
 - CLI mode now also records a structured trace of invoked skills, tool calls, proposed `wmill` commands, and any attempted `wmill` executions.
 - Frontend progress streams live while the benchmark is running.

ai_evals/adapters/cli/runtime.ts

@@ -16,9 +16,6 @@ export interface PromptRunResult {
   output: string;
   durationMs: number;
   tokenUsage: BenchmarkTokenUsage | null;
-  // Input tokens on the last assistant turn. The SDK `result` message reports
-  // usage cumulatively, so the final context size comes from per-turn usage.
-  finalContextTokens: number | null;
   trace: CliTrace;
 }
 
@@ -147,7 +144,6 @@ export async function runPromptAndCapture(
   let output = "";
   let assistantMessageCount = 0;
   let tokenUsage: BenchmarkTokenUsage | null = null;
-  let finalContextTokens: number | null = null;
   const startedAt = Date.now();
   const stubBinDir = join(cwd, WMILL_STUB_DIR_NAME);
   const wmillLogPath = join(cwd, WMILL_LOG_FILE_NAME);
@@ -170,12 +166,6 @@ export async function runPromptAndCapture(
   for await (const message of query({ prompt, options })) {
     if (message.type === "assistant") {
       assistantMessageCount += 1;
-      const turnContext = anthropicUsageToBenchmarkTokenUsage(
-        message.message?.usage
-      )?.prompt;
-      if (turnContext && turnContext > 0) {
-        finalContextTokens = turnContext;
-      }
       const content = message.message?.content;
       if (Array.isArray(content)) {
         for (const block of content) {
@@ -220,7 +210,6 @@ export async function runPromptAndCapture(
     output,
     durationMs: Date.now() - startedAt,
     tokenUsage,
-    finalContextTokens,
     trace: {
       toolsUsed,
       skillsInvoked,

ai_evals/adapters/frontend/backendPreview.test.ts

@@ -210,6 +210,8 @@ function buildSettings(
 		baseUrl: 'http://backend.test/default',
 		email: 'admin@windmill.dev',
 		password: 'changeme',
+		keepWorkspaces: true,
+		workspacePrefix: 'ai-evals',
 		pollIntervalMs: 1,
 		maxWaitMs: 50,
 		...overrides

ai_evals/adapters/frontend/backendPreview.ts

@@ -24,7 +24,6 @@ export interface CompletedPreviewJob {
 const tokenCache = new Map<string, Promise<string>>()
 const sharedWorkspaceQueue = new Map<string, Promise<void>>()
 const managedSharedWorkspacePrefixes = ['f/evals/']
-const DEFAULT_WORKSPACE_PREFIX = 'ai-evals'
 
 export class BackendPreviewClient {
 	constructor(private readonly settings: BackendValidationSettings) {}
@@ -36,7 +35,7 @@ export class BackendPreviewClient {
 	): Promise<T> {
 		const workspaceId =
 			this.settings.workspaceOverride ??
-			buildWorkspaceId(caseId, attempt)
+			buildWorkspaceId(this.settings.workspacePrefix, caseId, attempt)
 
 		const run = async () => {
 			await this.ensureWorkspace(workspaceId)
@@ -47,7 +46,7 @@ export class BackendPreviewClient {
 			try {
 				return await body(workspaceId)
 			} finally {
-				if (!this.settings.workspaceOverride) {
+				if (!this.settings.keepWorkspaces && !this.settings.workspaceOverride) {
 					await this.deleteWorkspace(workspaceId).catch(() => undefined)
 				}
 			}
@@ -441,14 +440,14 @@ async function withSharedWorkspaceLock<T>(workspaceId: string, body: () => Promi
 	}
 }
 
-function buildWorkspaceId(caseId: string, attempt: number): string {
+function buildWorkspaceId(prefix: string, caseId: string, attempt: number): string {
 	const caseSlug = caseId
 		.toLowerCase()
 		.replace(/[^a-z0-9-]+/g, '-')
 		.replace(/^-+|-+$/g, '')
 		.slice(0, 30)
 	const suffix = randomUUID().slice(0, 8)
-	return `${DEFAULT_WORKSPACE_PREFIX}-${caseSlug || 'case'}-a${attempt}-${suffix}`
+	return `${prefix}-${caseSlug || 'case'}-a${attempt}-${suffix}`
 }
 
 function extractFolderName(path: string): string | null {

ai_evals/adapters/frontend/benchmarkRunner.ts

@@ -1,5 +1,6 @@
 import { loadSelectedCases } from "../../core/cases";
 import { resolveBackendValidationSettings } from "../../core/backendValidation";
+import { resolveFrontendEvalTransportSettings } from "../../core/frontendTransport";
 import {
   formatRunModelLabel,
   getFrontendEvalModel,
@@ -8,11 +9,13 @@ import {
 import { buildRunResult } from "../../core/results";
 import { runSuite } from "../../core/runSuite";
 import type { BenchmarkRunResult, ModeRunner } from "../../core/types";
-import { resolveWindmillBackendSettings } from "../../core/windmillBackendSettings";
 import { emitFrontendBenchmarkProgress } from "./progress";
+import { createAppModeRunner } from "../../modes/app";
+import { createFlowModeRunner } from "../../modes/flow";
+import { createScriptModeRunner } from "../../modes/script";
 import { DEFAULT_JUDGE_MODEL } from "../../core/judge";
 
-export type FrontendBenchmarkMode = "flow" | "app" | "script" | "global";
+export type FrontendBenchmarkMode = "flow" | "app" | "script";
 
 export async function runFrontendBenchmarkFromEnv(): Promise<BenchmarkRunResult> {
   const mode = parseMode(process.env.WMILL_FRONTEND_AI_EVAL_MODE);
@@ -25,12 +28,6 @@ export async function runFrontendBenchmarkFromEnv(): Promise<BenchmarkRunResult>
   );
   const emitProgress = process.env.WMILL_FRONTEND_AI_EVAL_PROGRESS === "1";
   const verbose = process.env.WMILL_FRONTEND_AI_EVAL_VERBOSE === "1";
-  const executionOnly =
-    process.env.WMILL_FRONTEND_AI_EVAL_EXECUTION_ONLY === "1";
-  const judgeModel =
-    process.env.WMILL_FRONTEND_AI_EVAL_SKIP_JUDGE === "1" || executionOnly
-      ? null
-      : DEFAULT_JUDGE_MODEL;
   const model = resolveEvalModel(
     mode,
     process.env.WMILL_FRONTEND_AI_EVAL_MODEL,
@@ -39,14 +36,17 @@ export async function runFrontendBenchmarkFromEnv(): Promise<BenchmarkRunResult>
     evalMode: mode,
     requestedMode: process.env.WMILL_FRONTEND_AI_EVAL_BACKEND_VALIDATION,
   });
-  const backendSettings = resolveWindmillBackendSettings();
+  const transportSettings = resolveFrontendEvalTransportSettings({
+    evalMode: mode,
+    requestedTransport: process.env.WMILL_FRONTEND_AI_EVAL_TRANSPORT,
+  });
 
   const selectedCases = await loadSelectedCases(mode, caseIds);
-  const modeRunner = await getModeRunner(
+  const modeRunner = getModeRunner(
     mode,
     getFrontendEvalModel(model),
     backendValidation,
-    backendSettings,
+    transportSettings,
   );
   const runModel = formatRunModelLabel(mode, model);
   const caseResults = await runSuite({
@@ -54,8 +54,7 @@ export async function runFrontendBenchmarkFromEnv(): Promise<BenchmarkRunResult>
     cases: selectedCases,
     runs,
     runModel,
-    judgeModel,
-    executionOnly,
+    judgeModel: DEFAULT_JUDGE_MODEL,
     concurrency: verbose ? 1 : undefined,
     verbose,
     onProgress: emitProgress
@@ -67,48 +66,34 @@ export async function runFrontendBenchmarkFromEnv(): Promise<BenchmarkRunResult>
     mode,
     runs,
     runModel,
-    judgeModel,
+    transport: transportSettings.transport,
+    judgeModel: DEFAULT_JUDGE_MODEL,
     caseResults,
   });
 }
 
-async function getModeRunner(
+function getModeRunner(
   mode: FrontendBenchmarkMode,
   model: ReturnType<typeof getFrontendEvalModel>,
   backendValidation: ReturnType<typeof resolveBackendValidationSettings>,
-  backendSettings: ReturnType<typeof resolveWindmillBackendSettings>,
-): Promise<ModeRunner<any, any, any>> {
+  transportSettings: ReturnType<typeof resolveFrontendEvalTransportSettings>,
+): ModeRunner<any, any, any> {
   switch (mode) {
-    case "flow": {
-      const { createFlowModeRunner } = await import("../../modes/flow");
-      return createFlowModeRunner(model, backendValidation, backendSettings);
-    }
-    case "app": {
-      const { createAppModeRunner } = await import("../../modes/app");
-      return createAppModeRunner(model, backendSettings);
-    }
-    case "script": {
-      const { createScriptModeRunner } = await import("../../modes/script");
+    case "flow":
+      return createFlowModeRunner(model, backendValidation, transportSettings);
+    case "app":
+      return createAppModeRunner(model, transportSettings);
+    case "script":
       return createScriptModeRunner(
         model,
         backendValidation,
-        backendSettings,
+        transportSettings,
       );
-    }
-    case "global": {
-      const { createGlobalModeRunner } = await import("../../modes/global");
-      return createGlobalModeRunner(model, backendSettings);
-    }
   }
 }
 
 function parseMode(value: string | undefined): FrontendBenchmarkMode {
-  if (
-    value === "flow" ||
-    value === "app" ||
-    value === "script" ||
-    value === "global"
-  ) {
+  if (value === "flow" || value === "app" || value === "script") {
     return value;
   }
   throw new Error(`Unsupported frontend benchmark mode: ${String(value)}`);

ai_evals/adapters/frontend/core/app/appEvalRunner.ts

@@ -12,7 +12,7 @@ import {
   prepareAppUserMessage,
 } from "../../../../../frontend/src/lib/components/copilot/chat/app/core";
 import type { Tool as ProductionTool } from "../../../../../frontend/src/lib/components/copilot/chat/shared";
-import { createAppFileHelpers, type AppEvalChatHelpers } from "./fileHelpers";
+import { createAppFileHelpers } from "./fileHelpers";
 import { runEval } from "../shared";
 import type { AIProvider } from "$lib/gen/types.gen";
 import type {
@@ -22,6 +22,7 @@ import type {
 } from "../../../../core/types";
 import type { TokenUsage } from "../shared/types";
 import type { AppFilesState } from "../../../../core/validators";
+import type { FrontendEvalTransport } from "../../../../core/frontendTransport";
 import type { WindmillBackendSettings } from "../../../../core/windmillBackendSettings";
 import {
   createAppBackendRunnableContextElement,
@@ -38,7 +39,6 @@ export interface AppEvalResult {
   toolCallCount: number;
   toolsUsed: string[];
   tokenUsage: TokenUsage;
-  finalContextTokens: number | null;
 }
 
 export interface AppEvalOptions {
@@ -49,7 +49,8 @@ export interface AppEvalOptions {
   model?: string;
   maxIterations?: number;
   provider?: AIProvider;
-  backend: WindmillBackendSettings;
+  transport?: FrontendEvalTransport;
+  backend?: WindmillBackendSettings;
   workspaceRoot?: string;
   runContext?: ModeRunContext;
 }
@@ -57,7 +58,7 @@ export interface AppEvalOptions {
 export async function runAppEval(
   userPrompt: string,
   apiKey: string,
-  options: AppEvalOptions,
+  options?: AppEvalOptions,
 ): Promise<AppEvalResult> {
   const workspaceRoot =
     options?.workspaceRoot ??
@@ -100,9 +101,10 @@ export async function runAppEval(
         model,
         workspace: workspaceRoot,
         provider: options?.provider,
-        backend: options.backend,
-        caseId: options?.runContext?.caseId,
-        attempt: options?.runContext?.attempt,
+        transport: options?.transport,
+        backend: options?.backend,
+        proxyCaseId: options?.runContext?.caseId,
+        proxyAttempt: options?.runContext?.attempt,
       },
     });
 
@@ -114,7 +116,6 @@ export async function runAppEval(
       toolCallCount: rawResult.toolCallsCount,
       toolsUsed: rawResult.toolsCalled,
       tokenUsage: rawResult.tokenUsage,
-      finalContextTokens: rawResult.finalContextTokens,
     };
   } finally {
     await cleanup();
@@ -123,7 +124,7 @@ export async function runAppEval(
 
 async function buildAdditionalContext(
   appContext: EvalCaseRuntimeAppContextSpec | undefined,
-  helpers: AppEvalChatHelpers,
+  helpers: AppAIChatHelpers,
 ): Promise<ContextElement[]> {
   const entries = appContext?.additional ?? [];
   if (entries.length === 0) {

ai_evals/adapters/frontend/core/app/fileHelpers.ts

@@ -2,7 +2,6 @@ import { mkdir, rm, writeFile } from 'fs/promises'
 import { dirname, join } from 'path'
 import type {
 	AppAIChatHelpers,
-	AppDatatableMetadata,
 	AppFiles,
 	BackendRunnable,
 	DataTableSchema,
@@ -11,10 +10,6 @@ import type {
 } from '../../../../../frontend/src/lib/components/copilot/chat/app/core'
 import { buildAppWmillTypes, collectAppDiagnostics } from '../../../../core/appDiagnostics'
 
-export interface AppEvalChatHelpers extends AppAIChatHelpers {
-	getDatatables: () => Promise<DataTableSchema[]>
-}
-
 async function writeFrontendFile(
 	workspaceRoot: string | undefined,
 	path: string,
@@ -97,7 +92,7 @@ export async function createAppFileHelpers(
 	initialDatatables: DataTableSchema[] = [],
 	workspaceRoot?: string
 ): Promise<{
-	helpers: AppEvalChatHelpers
+	helpers: AppAIChatHelpers
 	getFiles: () => AppFiles
 	getEvalState: () => {
 		frontend: Record<string, string>
@@ -142,7 +137,7 @@ export async function createAppFileHelpers(
 	}
 	await persistDatatables(workspaceRoot, datatables)
 
-	const helpers: AppEvalChatHelpers = {
+	const helpers: AppAIChatHelpers = {
 		listFrontendFiles: () => [
 			...Object.keys(frontend).filter((path) => path !== '/wmill.d.ts'),
 			'/wmill.d.ts'
@@ -216,34 +211,6 @@ export async function createAppFileHelpers(
 		},
 		lint,
 		getDatatables: async () => structuredClone(datatables),
-		listDatatableTables: async () =>
-			datatables.map(
-				(datatable): AppDatatableMetadata => {
-					const schemas = Object.fromEntries(
-						Object.entries(datatable.schemas).map(([schemaName, tables]) => [
-							schemaName,
-							Object.keys(tables)
-						])
-					)
-					return {
-						datatable_name: datatable.datatable_name,
-						schemas,
-						tableCount: Object.values(schemas).reduce(
-							(sum, tableNames) => sum + tableNames.length,
-							0
-						),
-						error: datatable.error
-					}
-				}
-			),
-		getDatatableTableSchema: async (
-			datatableName: string,
-			schemaName: string,
-			tableName: string
-		) => {
-			const datatable = datatables.find((entry) => entry.datatable_name === datatableName)
-			return structuredClone(datatable?.schemas?.[schemaName]?.[tableName] ?? {})
-		},
 		getAvailableDatatableNames: () => datatables.map((datatable) => datatable.datatable_name),
 		execDatatableSql: async (
 			datatableName: string,

ai_evals/adapters/frontend/core/flow/flowEvalRunner.ts

@@ -18,6 +18,7 @@ import {
 import { runEval } from "../shared";
 import type { ModeRunContext } from "../../../../core/types";
 import type { TokenUsage, ToolCallDetail } from "../shared/types";
+import type { FrontendEvalTransport } from "../../../../core/frontendTransport";
 import type { WindmillBackendSettings } from "../../../../core/windmillBackendSettings";
 
 export interface FlowFixture {
@@ -39,7 +40,6 @@ export interface FlowEvalResult {
   toolsUsed: string[];
   toolCallDetails: ToolCallDetail[];
   tokenUsage: TokenUsage;
-  finalContextTokens: number | null;
 }
 
 export interface FlowEvalOptions {
@@ -48,7 +48,8 @@ export interface FlowEvalOptions {
   model?: string;
   maxIterations?: number;
   provider?: AIProvider;
-  backend: WindmillBackendSettings;
+  transport?: FrontendEvalTransport;
+  backend?: WindmillBackendSettings;
   workspaceRoot?: string;
   runContext?: ModeRunContext;
 }
@@ -56,7 +57,7 @@ export interface FlowEvalOptions {
 export async function runFlowEval(
   userPrompt: string,
   apiKey: string,
-  options: FlowEvalOptions,
+  options?: FlowEvalOptions,
 ): Promise<FlowEvalResult> {
   const workspaceRoot =
     options?.workspaceRoot ??
@@ -99,9 +100,10 @@ export async function runFlowEval(
         model,
         workspace: workspaceRoot,
         provider: options?.provider,
-        backend: options.backend,
-        caseId: options?.runContext?.caseId,
-        attempt: options?.runContext?.attempt,
+        transport: options?.transport,
+        backend: options?.backend,
+        proxyCaseId: options?.runContext?.caseId,
+        proxyAttempt: options?.runContext?.attempt,
       },
     });
 
@@ -114,7 +116,6 @@ export async function runFlowEval(
       toolsUsed: rawResult.toolsCalled,
       toolCallDetails: rawResult.toolCallDetails,
       tokenUsage: rawResult.tokenUsage,
-      finalContextTokens: rawResult.finalContextTokens,
     };
   } finally {
     await cleanup();

ai_evals/adapters/frontend/core/global/globalEvalRunner.ts

@@ -1,241 +0,0 @@
-import { mkdtemp, rm } from "fs/promises";
-import { tmpdir } from "os";
-import { join } from "path";
-import type { AIProvider } from "$lib/gen/types.gen";
-import {
-  globalTools,
-  prepareGlobalSystemMessage,
-  prepareGlobalUserMessage,
-} from "../../../../../frontend/src/lib/components/copilot/chat/global/core";
-import {
-  clearGlobalDrafts,
-  getGlobalDraft,
-  listGlobalDrafts,
-} from "../../../../../frontend/src/lib/components/copilot/chat/global/userDraftAdapter";
-import type { Tool as ProductionTool } from "../../../../../frontend/src/lib/components/copilot/chat/shared";
-import { UserDraft } from "../../../../../frontend/src/lib/userDraft.svelte";
-import type { ModeRunContext } from "../../../../core/types";
-import type { GlobalDraftState } from "../../../../core/validators";
-import type { WindmillBackendSettings } from "../../../../core/windmillBackendSettings";
-import {
-  registerBenchmarkWorkspaceRunnables,
-  seedBenchmarkDraft,
-  unregisterBenchmarkWorkspaceRunnables,
-  type BenchmarkWorkspaceRunnables,
-} from "../../mockBackend";
-import { runEval } from "../shared";
-import type { TokenUsage, ToolCallDetail } from "../shared/types";
-
-const MUTATING_GLOBAL_TOOLS = new Set([
-  "deploy_workspace_item",
-  "delete_workspace_item",
-]);
-const DISABLE_ACTIVE_EDITOR_CONTEXT_ENV =
-  "WMILL_AI_EVAL_DISABLE_ACTIVE_EDITOR_CONTEXT";
-// A/B gate for the search_app read tool: set to "1" to run the baseline arm
-// (toolset without search_app) so its token cost can be compared against the arm
-// that offers it.
-const DISABLE_SEARCH_APP_ENV = "WMILL_AI_EVAL_DISABLE_SEARCH_APP";
-
-const LIVE_EDITOR_ITEM_KINDS = {
-  script: "script",
-  flow: "flow",
-  app: "raw_app",
-} as const;
-
-export interface GlobalLiveEditorDraftFixture {
-  type: keyof typeof LIVE_EDITOR_ITEM_KINDS;
-  storagePath?: string;
-  effectivePath?: string;
-  value?: unknown;
-}
-
-// Identity the global system prompt builds paths from. Production reads
-// `userStore` (whoami) to fill `u/{username}/...`; the eval harness never logs
-// in, so without this the prompt sees an empty username (`u//...`) and no
-// path-selection case is meaningful. Seeded per-case via the initial fixture and
-// passed straight to `prepareGlobalSystemMessage` (no global-store mutation).
-export interface GlobalUserFixture {
-  username: string;
-  is_admin?: boolean;
-  /** Folders the user can write to (the writable set whoami returns). */
-  folders?: string[];
-  /** Folders the user can read; read-only folders = folders_read \ folders. */
-  folders_read?: string[];
-}
-
-export interface GlobalEvalResult {
-  success: boolean;
-  state: GlobalDraftState;
-  error?: string;
-  assistantMessageCount: number;
-  toolCallCount: number;
-  toolsUsed: string[];
-  toolCallDetails: ToolCallDetail[];
-  tokenUsage: TokenUsage;
-  finalContextTokens: number | null;
-}
-
-export interface GlobalEvalOptions {
-  workspaceFixtures?: BenchmarkWorkspaceRunnables;
-  liveEditorDrafts?: GlobalLiveEditorDraftFixture[];
-  user?: GlobalUserFixture;
-  model?: string;
-  maxIterations?: number;
-  provider?: AIProvider;
-  backend: WindmillBackendSettings;
-  workspaceRoot?: string;
-  runContext?: ModeRunContext;
-}
-
-export async function runGlobalEval(
-  userPrompt: string,
-  apiKey: string,
-  options: GlobalEvalOptions,
-): Promise<GlobalEvalResult> {
-  const workspaceRoot =
-    options.workspaceRoot ??
-    (await mkdtemp(join(tmpdir(), "wmill-frontend-global-benchmark-")));
-
-  clearGlobalDrafts(workspaceRoot);
-  registerBenchmarkWorkspaceRunnables(workspaceRoot, options.workspaceFixtures ?? {});
-  seedLiveEditorDrafts(workspaceRoot, options.liveEditorDrafts ?? []);
-
-  try {
-    const model = options.model ?? "claude-haiku-4-5-20251001";
-    const injectActiveEditorContext =
-      process.env[DISABLE_ACTIVE_EDITOR_CONTEXT_ENV] !== "1";
-    // Pass the seeded identity straight to the prompt builder rather than mutating
-    // the process-global `userStore`, so concurrent cases never race on it.
-    const rawResult = await runEval({
-      userPrompt,
-      systemMessage: prepareGlobalSystemMessage(undefined, { user: options.user }),
-      userMessage: prepareGlobalUserMessage(
-        userPrompt,
-        [],
-        injectActiveEditorContext ? { workspace: workspaceRoot } : {},
-      ),
-      tools: getGlobalEvalTools(),
-      helpers: {},
-      apiKey,
-      getOutput: () => collectGlobalDraftState(workspaceRoot),
-      onAssistantMessageStart: options.runContext?.onAssistantMessageStart,
-      onAssistantToken: options.runContext?.onAssistantChunk,
-      onAssistantMessageEnd: options.runContext?.onAssistantMessageEnd,
-      onToolCall: options.runContext?.onToolCall,
-      options: {
-        maxIterations: options.maxIterations,
-        model,
-        workspace: workspaceRoot,
-        provider: options.provider,
-        backend: options.backend,
-        caseId: options.runContext?.caseId,
-        attempt: options.runContext?.attempt,
-      },
-    });
-
-    return {
-      state: rawResult.output,
-      success: rawResult.success,
-      error: rawResult.error,
-      assistantMessageCount: rawResult.iterations,
-      toolCallCount: rawResult.toolCallsCount,
-      toolsUsed: rawResult.toolsCalled,
-      toolCallDetails: rawResult.toolCallDetails,
-      tokenUsage: rawResult.tokenUsage,
-      finalContextTokens: rawResult.finalContextTokens,
-    };
-  } finally {
-    clearGlobalDrafts(workspaceRoot);
-    clearLiveEditorDrafts(workspaceRoot, options.liveEditorDrafts ?? []);
-    unregisterBenchmarkWorkspaceRunnables(workspaceRoot);
-    if (!options.workspaceRoot) {
-      await rm(workspaceRoot, { recursive: true, force: true });
-    }
-  }
-}
-
-// Build the harness output from the DB-backed drafts. `listGlobalDrafts` returns
-// metadata-only rows for backend drafts (the model's `write_script` etc. persist
-// straight to the backend with no in-tab editor cell), so re-read each such row
-// with `getGlobalDraft` to attach the full value the validators assert on. A row
-// that already carries a value (the production in-tab cell overlay) is kept as-is.
-async function collectGlobalDraftState(
-  workspace: string,
-): Promise<GlobalDraftState> {
-  const items = await listGlobalDrafts(workspace);
-  const drafts = await Promise.all(
-    items.map(async (item) => {
-      if (item.value !== undefined) {
-        return item;
-      }
-      const full = await getGlobalDraft(
-        workspace,
-        item.type,
-        item.path,
-        item.triggerKind,
-      );
-      return full ?? item;
-    }),
-  );
-  return { drafts: drafts as GlobalDraftState["drafts"] };
-}
-
-function seedLiveEditorDrafts(
-  workspace: string,
-  fixtures: GlobalLiveEditorDraftFixture[],
-): void {
-  for (const fixture of fixtures) {
-    const itemKind = LIVE_EDITOR_ITEM_KINDS[fixture.type];
-    const storagePath = fixture.storagePath ?? fixture.effectivePath ?? "";
-    if (fixture.value !== undefined) {
-      // Seed as a backend draft row, not an in-tab cell: a cell would shadow the
-      // model's DB-backed edit when the output is read back via listGlobalDrafts.
-      seedBenchmarkDraft(workspace, itemKind, storagePath, fixture.value);
-    }
-    UserDraft.setLiveEditorDraft({
-      workspace,
-      itemKind,
-      storagePath,
-      effectivePath: fixture.effectivePath ?? fixture.storagePath,
-    });
-  }
-}
-
-function clearLiveEditorDrafts(
-  workspace: string,
-  fixtures: GlobalLiveEditorDraftFixture[],
-): void {
-  for (const fixture of fixtures) {
-    const itemKind = LIVE_EDITOR_ITEM_KINDS[fixture.type];
-    const storagePath = fixture.storagePath ?? fixture.effectivePath ?? "";
-    UserDraft.clearLiveEditorDraft(itemKind, { workspace, storagePath });
-  }
-}
-
-function getGlobalEvalTools(): ProductionTool<{}>[] {
-  const disableSearchApp = process.env[DISABLE_SEARCH_APP_ENV] === "1";
-  return (globalTools as ProductionTool<{}>[])
-    .filter((tool) => !(disableSearchApp && tool.def.function.name === "search_app"))
-    .map((tool) => {
-      if (!MUTATING_GLOBAL_TOOLS.has(tool.def.function.name)) {
-        return tool;
-      }
-
-      return {
-        ...tool,
-        requiresConfirmation: false,
-        validateBeforeConfirmation: undefined,
-        fn: async () =>
-          JSON.stringify(
-            {
-              success: false,
-              error:
-                "This mutating workspace tool is disabled during ai_evals global mode.",
-            },
-            null,
-            2,
-          ),
-      };
-    });
-}

ai_evals/adapters/frontend/core/script/scriptEvalRunner.ts

@@ -14,6 +14,7 @@ import { createScriptFileHelpers, type ScriptEvalState } from "./fileHelpers";
 import { runEval } from "../shared";
 import type { ModeRunContext } from "../../../../core/types";
 import type { TokenUsage, ToolCallDetail } from "../shared/types";
+import type { FrontendEvalTransport } from "../../../../core/frontendTransport";
 import type { WindmillBackendSettings } from "../../../../core/windmillBackendSettings";
 
 export interface ScriptEvalResult {
@@ -25,7 +26,6 @@ export interface ScriptEvalResult {
   toolsUsed: string[];
   toolCallDetails: ToolCallDetail[];
   tokenUsage: TokenUsage;
-  finalContextTokens: number | null;
 }
 
 export interface ScriptEvalOptions {
@@ -33,7 +33,8 @@ export interface ScriptEvalOptions {
   model?: string;
   maxIterations?: number;
   provider?: AIProvider;
-  backend: WindmillBackendSettings;
+  transport?: FrontendEvalTransport;
+  backend?: WindmillBackendSettings;
   workspaceRoot?: string;
   runContext?: ModeRunContext;
 }
@@ -97,9 +98,10 @@ export async function runScriptEval(
         model,
         workspace: workspaceRoot,
         provider: modelProvider.provider,
+        transport: options.transport,
         backend: options.backend,
-        caseId: options.runContext?.caseId,
-        attempt: options.runContext?.attempt,
+        proxyCaseId: options.runContext?.caseId,
+        proxyAttempt: options.runContext?.attempt,
       },
     });
 
@@ -112,7 +114,6 @@ export async function runScriptEval(
       toolsUsed: rawResult.toolsCalled,
       toolCallDetails: rawResult.toolCallDetails,
       tokenUsage: rawResult.tokenUsage,
-      finalContextTokens: rawResult.finalContextTokens,
     };
   } finally {
     await cleanup();

ai_evals/adapters/frontend/core/shared/baseEvalRunner.ts

@@ -38,11 +38,10 @@ export interface RunEvalParams<THelpers, TOutput> {
   helpers: THelpers;
   /** API key for the provider */
   apiKey: string;
-  /** Function to get the current output state. May be async — global mode reads
-   * DB-backed drafts back through the (mocked) backend to build its output. */
-  getOutput: () => TOutput | Promise<TOutput>;
-  /** Model and Windmill backend configuration */
-  options: EvalRunnerOptions;
+  /** Function to get the current output state */
+  getOutput: () => TOutput;
+  /** Optional configuration */
+  options?: EvalRunnerOptions;
   onAssistantMessageStart?: () => void;
   onAssistantToken?: (token: string) => void;
   onAssistantMessageEnd?: () => void;
@@ -71,10 +70,10 @@ export async function runEval<THelpers, TOutput>(
   } = params;
   let shouldEmitMessageStart = true;
 
-  const model = options.model ?? "gpt-4o";
-  const maxIterations = options.maxIterations ?? 20;
-  const workspace = options.workspace ?? "test-workspace";
-  const provider = toFrontendEvalProvider(options.provider);
+  const model = options?.model ?? "gpt-4o";
+  const maxIterations = options?.maxIterations ?? 20;
+  const workspace = options?.workspace ?? "test-workspace";
+  const provider = toFrontendEvalProvider(options?.provider);
 
   const modelProvider = resolveEvalModelProvider(model, provider);
 
@@ -155,10 +154,9 @@ export async function runEval<THelpers, TOutput>(
       if (result.hitMaxIterations) {
         return {
           success: false,
-          output: (await getOutput()) as TOutput,
+          output: getOutput(),
           error: `Reached max turns (${maxIterations})`,
           tokenUsage: result.tokenUsage,
-          finalContextTokens: result.lastIterationUsage?.prompt ?? null,
           toolCallsCount,
           toolsCalled,
           toolCallDetails,
@@ -172,9 +170,8 @@ export async function runEval<THelpers, TOutput>(
 
       return {
         success: true,
-        output: (await getOutput()) as TOutput,
+        output: getOutput(),
         tokenUsage: result.tokenUsage,
-        finalContextTokens: result.lastIterationUsage?.prompt ?? null,
         toolCallsCount,
         toolsCalled,
         toolCallDetails,
@@ -194,10 +191,9 @@ export async function runEval<THelpers, TOutput>(
 
       return {
         success: false,
-        output: (await getOutput()) as TOutput,
+        output: getOutput(),
         error: errorMessage,
         tokenUsage: { prompt: 0, completion: 0, total: 0 },
-        finalContextTokens: null,
         toolCallsCount,
         toolsCalled,
         toolCallDetails,
@@ -207,31 +203,45 @@ export async function runEval<THelpers, TOutput>(
     }
   };
 
-  const backendSettings = options.backend;
-  const backendClient = new WindmillBackendClient(backendSettings);
-  return await backendClient.withWorkspace(
-    options.caseId ?? "eval",
-    options.attempt ?? 1,
-    async (proxyWorkspaceId) => {
-      const resourcePath = buildProxyResourcePath(modelProvider.provider);
-      await backendClient.upsertResource({
-        workspaceId: proxyWorkspaceId,
-        path: resourcePath,
-        resourceType: modelProvider.provider,
-        value: { api_key: apiKey },
-      });
-      const token = await backendClient.getToken();
-      const clients = createEvalClients({
-        provider: modelProvider.provider,
-        proxy: {
-          baseURL: `${backendSettings.baseUrl}/api/w/${encodeURIComponent(proxyWorkspaceId)}/ai/proxy`,
-          bearerToken: token,
-          resourcePath,
-        },
-      }) as unknown as ChatClients;
-      return await executeChatLoop(clients);
-    },
-  );
+  if (options?.transport === "proxy") {
+    const backendSettings = options.backend;
+    if (!backendSettings) {
+      throw new Error("Missing backend settings for proxy transport");
+    }
+
+    const backendClient = new WindmillBackendClient(backendSettings);
+    return await backendClient.withWorkspace(
+      options.proxyCaseId ?? "eval",
+      options.proxyAttempt ?? 1,
+      async (proxyWorkspaceId) => {
+        const resourcePath = buildProxyResourcePath(modelProvider.provider);
+        await backendClient.upsertResource({
+          workspaceId: proxyWorkspaceId,
+          path: resourcePath,
+          resourceType: modelProvider.provider,
+          value: { api_key: apiKey },
+        });
+        const token = await backendClient.getToken();
+        const clients = createEvalClients({
+          provider: modelProvider.provider,
+          apiKey,
+          transport: "proxy",
+          proxy: {
+            baseURL: `${backendSettings.baseUrl}/api/w/${encodeURIComponent(proxyWorkspaceId)}/ai/proxy`,
+            bearerToken: token,
+            resourcePath,
+          },
+        }) as unknown as ChatClients;
+        return await executeChatLoop(clients);
+      },
+    );
+  }
+
+  const clients = createEvalClients({
+    provider: modelProvider.provider,
+    apiKey,
+  }) as unknown as ChatClients;
+  return await executeChatLoop(clients);
 }
 
 function toFrontendEvalProvider(
@@ -240,8 +250,7 @@ function toFrontendEvalProvider(
   if (
     provider === "anthropic" ||
     provider === "openai" ||
-    provider === "googleai" ||
-    provider === "deepseek"
+    provider === "googleai"
   ) {
     return provider;
   }

ai_evals/adapters/frontend/core/shared/providerConfig.test.ts

@@ -2,9 +2,35 @@ import { describe, expect, it } from "bun:test";
 import {
   buildProxyHeaders,
   buildProxyResourcePath,
+  buildOpenAICompatibleClientOptions,
   resolveEvalModelProvider,
 } from "./providerConfig";
 
+describe("buildOpenAICompatibleClientOptions", () => {
+  it("adds Gemini's OpenAI-compatible base URL and client header", () => {
+    const options = buildOpenAICompatibleClientOptions(
+      "googleai",
+      "gemini-test-key",
+    );
+
+    expect(options).toMatchObject({
+      apiKey: "gemini-test-key",
+      baseURL: "https://generativelanguage.googleapis.com/v1beta/openai/",
+      defaultHeaders: {
+        "x-goog-api-client": "windmill-ai-evals/1.0",
+      },
+    });
+  });
+
+  it("keeps the default OpenAI-compatible config for OpenAI", () => {
+    expect(
+      buildOpenAICompatibleClientOptions("openai", "openai-test-key"),
+    ).toEqual({
+      apiKey: "openai-test-key",
+    });
+  });
+});
+
 describe("proxy helpers", () => {
   it("builds provider-scoped proxy resource paths", () => {
     expect(buildProxyResourcePath("googleai")).toBe("f/evals/ai/googleai");
@@ -21,25 +47,16 @@ describe("proxy helpers", () => {
 
 describe("resolveEvalModelProvider", () => {
   it("infers googleai from Gemini model ids", () => {
-    expect(resolveEvalModelProvider("gemini-3-flash-preview")).toEqual({
+    expect(resolveEvalModelProvider("gemini-2.5-flash")).toEqual({
       provider: "googleai",
-      model: "gemini-3-flash-preview",
-    });
-  });
-
-  it("infers deepseek from DeepSeek model ids", () => {
-    expect(resolveEvalModelProvider("deepseek-v4-flash")).toEqual({
-      provider: "deepseek",
-      model: "deepseek-v4-flash",
+      model: "gemini-2.5-flash",
     });
   });
 
   it("preserves an explicit provider", () => {
-    expect(
-      resolveEvalModelProvider("gemini-3.1-pro-preview", "googleai"),
-    ).toEqual({
+    expect(resolveEvalModelProvider("gemini-2.5-pro", "googleai")).toEqual({
       provider: "googleai",
-      model: "gemini-3.1-pro-preview",
+      model: "gemini-2.5-pro",
     });
   });
 });

ai_evals/adapters/frontend/core/shared/providerConfig.ts

@@ -1,6 +1,7 @@
 import Anthropic from "@anthropic-ai/sdk";
 import OpenAI from "openai";
 import type { FrontendEvalModelConfig } from "../../../../core/models";
+import type { FrontendEvalTransport } from "../../../../core/frontendTransport";
 
 export type FrontendEvalProvider = FrontendEvalModelConfig["provider"];
 
@@ -14,12 +15,15 @@ export interface ResolvedEvalModelProvider {
   model: string;
 }
 
-export interface WindmillAiProxyClientConfig {
+export interface EvalProxyClientConfig {
   baseURL: string;
   bearerToken: string;
   resourcePath: string;
 }
 
+const GEMINI_OPENAI_BASE_URL =
+  "https://generativelanguage.googleapis.com/v1beta/openai/";
+const GEMINI_GOOG_API_CLIENT = "windmill-ai-evals/1.0";
 const EVAL_PROXY_RESOURCE_PREFIX = "f/evals/ai";
 
 export function buildProxyHeaders(
@@ -36,8 +40,25 @@ export function buildProxyResourcePath(provider: FrontendEvalProvider): string {
   return `${EVAL_PROXY_RESOURCE_PREFIX}/${provider}`;
 }
 
+export function buildOpenAICompatibleClientOptions(
+  provider: Exclude<FrontendEvalProvider, "anthropic">,
+  apiKey: string,
+): ConstructorParameters<typeof OpenAI>[0] {
+  if (provider === "googleai") {
+    return {
+      apiKey,
+      baseURL: GEMINI_OPENAI_BASE_URL,
+      defaultHeaders: {
+        "x-goog-api-client": GEMINI_GOOG_API_CLIENT,
+      },
+    };
+  }
+
+  return { apiKey };
+}
+
 function buildProxyOpenAIClientOptions(
-  proxy: WindmillAiProxyClientConfig,
+  proxy: EvalProxyClientConfig,
 ): ConstructorParameters<typeof OpenAI>[0] {
   return {
     apiKey: "unused",
@@ -48,24 +69,52 @@ function buildProxyOpenAIClientOptions(
 
 export function createEvalClients(input: {
   provider: FrontendEvalProvider;
-  proxy: WindmillAiProxyClientConfig;
+  apiKey: string;
+  transport?: FrontendEvalTransport;
+  proxy?: EvalProxyClientConfig;
 }): EvalClients {
+  const transport = input.transport ?? "direct";
+
   if (input.provider === "anthropic") {
+    if (transport === "proxy") {
+      if (!input.proxy) {
+        throw new Error(
+          "Missing proxy client configuration for proxy transport",
+        );
+      }
+      return {
+        openai: new OpenAI({ apiKey: "unused" }),
+        anthropic: new Anthropic({
+          apiKey: "unused",
+          baseURL: input.proxy.baseURL,
+          defaultHeaders: buildProxyHeaders(
+            input.proxy.bearerToken,
+            input.proxy.resourcePath,
+          ),
+        }),
+      };
+    }
+
     return {
       openai: new OpenAI({ apiKey: "unused" }),
-      anthropic: new Anthropic({
-        apiKey: "unused",
-        baseURL: input.proxy.baseURL,
-        defaultHeaders: buildProxyHeaders(
-          input.proxy.bearerToken,
-          input.proxy.resourcePath,
-        ),
-      }),
+      anthropic: new Anthropic({ apiKey: input.apiKey }),
+    };
+  }
+
+  if (transport === "proxy") {
+    if (!input.proxy) {
+      throw new Error("Missing proxy client configuration for proxy transport");
+    }
+    return {
+      openai: new OpenAI(buildProxyOpenAIClientOptions(input.proxy)),
+      anthropic: new Anthropic({ apiKey: "unused" }),
     };
   }
 
   return {
-    openai: new OpenAI(buildProxyOpenAIClientOptions(input.proxy)),
+    openai: new OpenAI(
+      buildOpenAICompatibleClientOptions(input.provider, input.apiKey),
+    ),
     anthropic: new Anthropic({ apiKey: "unused" }),
   };
 }
@@ -83,9 +132,6 @@ export function resolveEvalModelProvider(
   if (model.startsWith("gemini")) {
     return { provider: "googleai", model };
   }
-  if (model.startsWith("deepseek")) {
-    return { provider: "deepseek", model };
-  }
   if (model.startsWith("gpt") || model.startsWith("o")) {
     return { provider: "openai", model };
   }

ai_evals/adapters/frontend/core/shared/types.ts

@@ -1,5 +1,6 @@
 import type { ChatCompletionMessageParam } from "openai/resources/chat/completions.mjs";
 import type { AIProvider } from "$lib/gen/types.gen";
+import type { FrontendEvalTransport } from "../../../../core/frontendTransport";
 import type { WindmillBackendSettings } from "../../../../core/windmillBackendSettings";
 
 export interface TokenUsage {
@@ -14,13 +15,14 @@ export interface ToolCallDetail {
 }
 
 export interface EvalRunnerOptions {
-  backend: WindmillBackendSettings;
   maxIterations?: number;
   model?: string;
   workspace?: string;
   provider?: AIProvider;
-  caseId?: string;
-  attempt?: number;
+  transport?: FrontendEvalTransport;
+  backend?: WindmillBackendSettings;
+  proxyCaseId?: string;
+  proxyAttempt?: number;
 }
 
 export interface RawEvalResult<TOutput> {
@@ -28,8 +30,6 @@ export interface RawEvalResult<TOutput> {
   output: TOutput;
   error?: string;
   tokenUsage: TokenUsage;
-  /** Input tokens on the last model request of the loop (see BenchmarkAttemptResult.finalContextTokens). */
-  finalContextTokens: number | null;
   toolCallsCount: number;
   toolsCalled: string[];
   toolCallDetails: ToolCallDetail[];

ai_evals/adapters/frontend/datatableSqlEngine.test.ts

@@ -1,262 +0,0 @@
-import { describe, expect, it } from 'bun:test'
-import { applyDatatableSql, type BenchmarkDatatableSeed } from './datatableSqlEngine'
-
-function makeDatatable(): BenchmarkDatatableSeed {
-	return {
-		datatable_name: 'main',
-		schemas: {
-			public: {
-				orders: {
-					columns: { id: 'int4', customer_id: 'int4', total: 'numeric', status: 'text' },
-					rows: [
-						{ id: 1, customer_id: 1, total: 42.5, status: 'shipped' },
-						{ id: 2, customer_id: 2, total: 19.99, status: 'pending' },
-						{ id: 3, customer_id: 1, total: 88, status: 'shipped' }
-					]
-				},
-				customers: {
-					columns: { id: 'int4', name: 'text' },
-					rows: [{ id: 1, name: 'Alice' }]
-				}
-			}
-		}
-	}
-}
-
-describe('SELECT', () => {
-	it('returns the referenced table rows', () => {
-		const dt = makeDatatable()
-		expect(applyDatatableSql(dt, 'SELECT id, name FROM customers').rows).toEqual([
-			{ id: 1, name: 'Alice' }
-		])
-	})
-
-	it('falls back to the first table when no known table is referenced', () => {
-		const dt = makeDatatable()
-		expect(applyDatatableSql(dt, 'select 1').rows).toHaveLength(3)
-	})
-
-	it('resolves a schema-qualified table', () => {
-		const dt = makeDatatable()
-		expect(applyDatatableSql(dt, 'SELECT * FROM public.customers').rows).toEqual([
-			{ id: 1, name: 'Alice' }
-		])
-	})
-})
-
-describe('CREATE TABLE', () => {
-	it('adds a table with parsed columns, skipping table constraints and FK clauses', () => {
-		const dt = makeDatatable()
-		const result = applyDatatableSql(
-			dt,
-			'CREATE TABLE public.refunds (\n  order_id int4 NOT NULL REFERENCES public.orders(id),\n  amount numeric(10,2),\n  PRIMARY KEY (order_id)\n)'
-		)
-		expect(result.rows).toEqual([])
-		expect(dt.schemas.public.refunds).toEqual({
-			columns: { order_id: 'int4', amount: 'numeric(10,2)' },
-			rows: []
-		})
-	})
-
-	it('defaults an unqualified table to the public schema', () => {
-		const dt = makeDatatable()
-		applyDatatableSql(dt, 'CREATE TABLE notes (id int4, body text)')
-		expect(dt.schemas.public.notes.columns).toEqual({ id: 'int4', body: 'text' })
-	})
-
-	it('is a no-op for an existing table with IF NOT EXISTS', () => {
-		const dt = makeDatatable()
-		applyDatatableSql(dt, 'CREATE TABLE IF NOT EXISTS public.orders (x int4)')
-		expect(Object.keys(dt.schemas.public.orders.columns)).toContain('status')
-	})
-})
-
-describe('DROP TABLE', () => {
-	it('removes the table', () => {
-		const dt = makeDatatable()
-		applyDatatableSql(dt, 'DROP TABLE IF EXISTS public.customers')
-		expect(dt.schemas.public.customers).toBeUndefined()
-	})
-})
-
-describe('INSERT', () => {
-	it('appends a row using an explicit column list', () => {
-		const dt = makeDatatable()
-		applyDatatableSql(dt, "INSERT INTO customers (id, name) VALUES (2, 'Bob')")
-		expect(dt.schemas.public.customers.rows).toContainEqual({ id: 2, name: 'Bob' })
-	})
-
-	it('infers columns from the table when none are given, and appends multiple tuples', () => {
-		const dt = makeDatatable()
-		applyDatatableSql(dt, "INSERT INTO customers VALUES (2, 'Bob'), (3, 'Carol')")
-		expect(dt.schemas.public.customers.rows).toHaveLength(3)
-	})
-
-	it('returns the inserted rows when RETURNING is present', () => {
-		const dt = makeDatatable()
-		const result = applyDatatableSql(
-			dt,
-			"INSERT INTO customers (id, name) VALUES (2, 'Bob') RETURNING *"
-		)
-		expect(result.rows).toEqual([{ id: 2, name: 'Bob' }])
-	})
-})
-
-describe('UPDATE', () => {
-	it('updates only the rows matching an equality WHERE', () => {
-		const dt = makeDatatable()
-		const result = applyDatatableSql(
-			dt,
-			"UPDATE public.orders SET status = 'shipped' WHERE id = 2"
-		)
-		expect(result.rows).toEqual([])
-		expect(dt.schemas.public.orders.rows?.find((r) => r.id === 2)?.status).toBe('shipped')
-		expect(dt.schemas.public.orders.rows?.find((r) => r.id === 1)?.status).toBe('shipped')
-	})
-
-	it('strips a Postgres cast in the WHERE value', () => {
-		const dt = makeDatatable()
-		applyDatatableSql(dt, "UPDATE orders SET status = 'done' WHERE id = 2::int4")
-		expect(dt.schemas.public.orders.rows?.find((r) => r.id === 2)?.status).toBe('done')
-	})
-
-	it('matches multiple AND predicates including a numeric literal', () => {
-		const dt = makeDatatable()
-		applyDatatableSql(
-			dt,
-			"UPDATE orders SET status = 'done' WHERE customer_id = 2 AND total = 19.99"
-		)
-		expect(dt.schemas.public.orders.rows?.find((r) => r.id === 2)?.status).toBe('done')
-		expect(dt.schemas.public.orders.rows?.find((r) => r.id === 1)?.status).toBe('shipped')
-	})
-
-	it('updates every row when there is no WHERE', () => {
-		const dt = makeDatatable()
-		applyDatatableSql(dt, "UPDATE orders SET status = 'archived'")
-		expect(dt.schemas.public.orders.rows?.every((r) => r.status === 'archived')).toBe(true)
-	})
-
-	it('returns the affected rows when RETURNING is present', () => {
-		const dt = makeDatatable()
-		const result = applyDatatableSql(
-			dt,
-			"UPDATE orders SET status = 'shipped' WHERE id = 2 RETURNING *"
-		)
-		expect(result.rows).toHaveLength(1)
-		expect(result.rows[0]).toMatchObject({ id: 2, status: 'shipped' })
-	})
-
-	it('affects no rows when the WHERE clause cannot be parsed', () => {
-		const dt = makeDatatable()
-		applyDatatableSql(dt, "UPDATE orders SET status = 'x' WHERE total > 20")
-		expect(dt.schemas.public.orders.rows?.some((r) => r.status === 'x')).toBe(false)
-	})
-})
-
-describe('DELETE', () => {
-	it('removes only the matching rows', () => {
-		const dt = makeDatatable()
-		applyDatatableSql(dt, 'DELETE FROM orders WHERE id = 2')
-		expect(dt.schemas.public.orders.rows?.map((r) => r.id)).toEqual([1, 3])
-	})
-
-	it('returns the removed rows when RETURNING is present', () => {
-		const dt = makeDatatable()
-		const result = applyDatatableSql(dt, 'DELETE FROM orders WHERE id = 2 RETURNING *')
-		expect(result.rows).toEqual([{ id: 2, customer_id: 2, total: 19.99, status: 'pending' }])
-	})
-})
-
-describe('writes are reflected by later reads', () => {
-	it('UPDATE then SELECT sees the new value (the verify-loop fix)', () => {
-		const dt = makeDatatable()
-		applyDatatableSql(dt, "UPDATE orders SET status = 'shipped' WHERE id = 2")
-		const seen = applyDatatableSql(dt, 'SELECT * FROM orders').rows
-		expect(seen.find((r) => r.id === 2)?.status).toBe('shipped')
-	})
-
-	it('INSERT then SELECT sees the new row', () => {
-		const dt = makeDatatable()
-		applyDatatableSql(dt, "INSERT INTO customers (id, name) VALUES (9, 'Zed')")
-		const seen = applyDatatableSql(dt, 'SELECT * FROM customers').rows
-		expect(seen).toContainEqual({ id: 9, name: 'Zed' })
-	})
-
-	it('CREATE then SELECT on the new table returns its (empty) rows', () => {
-		const dt = makeDatatable()
-		applyDatatableSql(dt, 'CREATE TABLE public.refunds (order_id int4, amount numeric)')
-		expect(applyDatatableSql(dt, 'SELECT * FROM refunds').rows).toEqual([])
-	})
-})
-
-describe('system-catalog queries reflect the current tables/columns', () => {
-	it('lists current tables (including a freshly created one) via information_schema.tables', () => {
-		const dt = makeDatatable()
-		applyDatatableSql(dt, 'CREATE TABLE public.refunds (order_id int4)')
-		const rows = applyDatatableSql(
-			dt,
-			"SELECT table_name FROM information_schema.tables WHERE table_name = 'refunds'"
-		).rows
-		expect(rows.map((r) => r.table_name)).toContain('refunds')
-	})
-
-	it('does not list a dropped table', () => {
-		const dt = makeDatatable()
-		applyDatatableSql(dt, 'DROP TABLE public.customers')
-		const rows = applyDatatableSql(dt, 'SELECT table_name FROM information_schema.tables').rows
-		expect(rows.map((r) => r.table_name)).not.toContain('customers')
-	})
-
-	it('reports columns via information_schema.columns', () => {
-		const dt = makeDatatable()
-		const rows = applyDatatableSql(
-			dt,
-			"SELECT column_name FROM information_schema.columns WHERE table_name = 'orders'"
-		).rows
-		expect(rows.map((r) => r.column_name)).toContain('status')
-	})
-})
-
-describe('parser robustness (string/paren-aware splitting)', () => {
-	it('does not treat the word "returning" inside a string value as a RETURNING clause', () => {
-		const dt = makeDatatable()
-		const result = applyDatatableSql(
-			dt,
-			"INSERT INTO customers (id, name) VALUES (5, 'is returning soon')"
-		)
-		expect(result.rows).toEqual([])
-		expect(dt.schemas.public.customers.rows).toContainEqual({ id: 5, name: 'is returning soon' })
-	})
-
-	it('does not split on the word "where" inside a SET string value', () => {
-		const dt = makeDatatable()
-		applyDatatableSql(dt, "UPDATE orders SET status = 'ship where ordered' WHERE id = 2")
-		expect(dt.schemas.public.orders.rows?.find((r) => r.id === 2)?.status).toBe('ship where ordered')
-		expect(dt.schemas.public.orders.rows?.find((r) => r.id === 1)?.status).toBe('shipped')
-	})
-
-	it('keeps INSERT tuples intact when a value contains a function call', () => {
-		const dt = makeDatatable()
-		applyDatatableSql(dt, "INSERT INTO customers (id, name) VALUES (6, coalesce(NULL, 'x'))")
-		expect(dt.schemas.public.customers.rows).toHaveLength(2)
-		expect(dt.schemas.public.customers.rows?.[1]).toMatchObject({ id: 6 })
-	})
-
-	it('CREATE TABLE ignores a trailing semicolon-separated statement', () => {
-		const dt = makeDatatable()
-		applyDatatableSql(
-			dt,
-			'CREATE TABLE public.refunds (id int4, amount numeric); INSERT INTO refunds VALUES (1, 5)'
-		)
-		expect(dt.schemas.public.refunds.columns).toEqual({ id: 'int4', amount: 'numeric' })
-		expect(dt.schemas.public.refunds.rows).toEqual([])
-	})
-})
-
-describe('unparseable statements are a safe no-op', () => {
-	it('returns [] and does not throw', () => {
-		const dt = makeDatatable()
-		expect(applyDatatableSql(dt, 'VACUUM ANALYZE').rows).toEqual([])
-		expect(applyDatatableSql(dt, 'GRANT SELECT ON orders TO someone').rows).toEqual([])
-	})
-})

ai_evals/adapters/frontend/datatableSqlEngine.ts

@@ -1,541 +0,0 @@
-/**
- * A deliberately small, best-effort SQL engine for the benchmark datatable mock.
- *
- * This is NOT a real SQL implementation — it exists only so that writes a model
- * issues during an eval (`CREATE TABLE`, `INSERT`, `UPDATE`, `DELETE`, `DROP`)
- * become visible to its later reads (`list_datatables`, `get_datatable_table_schema`,
- * `SELECT`). Without that, a model that re-queries to verify a write sees stale
- * seed data, concludes the write failed, and loops until it exhausts its turns.
- *
- * It parses only the common statement shapes models produce. Anything it cannot
- * parse is a no-op success (it never throws) — behavioral evals assert that the
- * right statement was issued, not its exact data effects. Notable limits:
- *  - `SELECT` returns all rows of the referenced (or first) table — no WHERE
- *    filtering, projection, joins, or aggregation.
- *  - `WHERE` supports `col = value` predicates joined by `AND` only; an
- *    unparseable WHERE on UPDATE/DELETE affects zero rows (never the whole table).
- */
-
-/** One seeded datatable table: its columns (col -> compact_type) and optional rows. */
-export interface BenchmarkDatatableTableSeed {
-	columns: Record<string, string>
-	rows?: Record<string, unknown>[]
-}
-
-/** A seeded datatable: `datatable_name` plus a `schema -> table -> seed` map. */
-export interface BenchmarkDatatableSeed {
-	datatable_name: string
-	schemas: {
-		[schema: string]: {
-			[table: string]: BenchmarkDatatableTableSeed
-		}
-	}
-}
-
-export interface DatatableSqlResult {
-	rows: Record<string, unknown>[]
-}
-
-const DEFAULT_SCHEMA = 'public'
-
-type ParsedRef = { schema: string; table: string }
-type Predicate = { column: string; value: unknown }
-
-/**
- * Apply one SQL statement to `datatable` IN PLACE and return the result rows.
- * SELECT returns the referenced/first table's rows; a mutation returns its
- * affected rows when it has a RETURNING clause, otherwise `[]`.
- */
-export function applyDatatableSql(
-	datatable: BenchmarkDatatableSeed,
-	sql: string
-): DatatableSqlResult {
-	const statement = stripTrailingSemicolon(sql.trim())
-	if (/^\s*(with|select)\b/i.test(statement)) {
-		return { rows: selectRows(datatable, statement) }
-	}
-	if (/^\s*create\s+table\b/i.test(statement)) {
-		return { rows: applyCreateTable(datatable, statement) }
-	}
-	if (/^\s*drop\s+table\b/i.test(statement)) {
-		return { rows: applyDropTable(datatable, statement) }
-	}
-	if (/^\s*insert\s+into\b/i.test(statement)) {
-		return { rows: applyInsert(datatable, statement) }
-	}
-	if (/^\s*update\b/i.test(statement)) {
-		return { rows: applyUpdate(datatable, statement) }
-	}
-	if (/^\s*delete\s+from\b/i.test(statement)) {
-		return { rows: applyDelete(datatable, statement) }
-	}
-	return { rows: [] }
-}
-
-// ============= Reads =============
-
-function selectRows(
-	datatable: BenchmarkDatatableSeed,
-	sql: string
-): Record<string, unknown>[] {
-	const fromRef = sql.match(/\bfrom\s+([a-zA-Z_"][\w."]*)/i)?.[1]
-	if (fromRef) {
-		const catalog = catalogRows(datatable, fromRef)
-		if (catalog) {
-			return catalog
-		}
-	}
-	const table = fromRef ? resolveTable(datatable, fromRef) : undefined
-	const seed = table ?? firstTable(datatable)
-	return seed?.rows ?? []
-}
-
-/**
- * Synthesize rows for a system-catalog query so a model verifying a `CREATE`/`DROP`
- * via `information_schema.tables` / `.columns` (or `pg_tables`) sees the current
- * tables/columns instead of fallback data. WHERE is not applied, so the model gets
- * the full set and finds (or no longer finds) the table it just changed.
- * Returns `undefined` for non-catalog refs so normal table resolution proceeds.
- */
-function catalogRows(
-	datatable: BenchmarkDatatableSeed,
-	ref: string
-): Record<string, unknown>[] | undefined {
-	const normalized = ref.toLowerCase().replace(/"/g, '')
-	const name = normalized.split('.').pop()
-	const isCatalog = normalized.includes('information_schema.') || normalized.startsWith('pg_')
-	if (!isCatalog) {
-		return undefined
-	}
-	const tables = allTables(datatable)
-	if (name === 'tables' || name === 'pg_tables') {
-		return tables.map(({ schema, table }) => ({
-			table_schema: schema,
-			table_name: table,
-			schemaname: schema,
-			tablename: table
-		}))
-	}
-	if (name === 'columns') {
-		return tables.flatMap(({ schema, table, seed }) =>
-			Object.entries(seed.columns).map(([column, type]) => ({
-				table_schema: schema,
-				table_name: table,
-				column_name: column,
-				data_type: type
-			}))
-		)
-	}
-	return undefined
-}
-
-function allTables(
-	datatable: BenchmarkDatatableSeed
-): { schema: string; table: string; seed: BenchmarkDatatableTableSeed }[] {
-	return Object.entries(datatable.schemas).flatMap(([schema, tables]) =>
-		Object.entries(tables).map(([table, seed]) => ({ schema, table, seed }))
-	)
-}
-
-// ============= DDL =============
-
-function applyCreateTable(
-	datatable: BenchmarkDatatableSeed,
-	sql: string
-): Record<string, unknown>[] {
-	const head = sql.match(
-		/^\s*create\s+table\s+(?:if\s+not\s+exists\s+)?([a-zA-Z_"][\w."]*)/i
-	)
-	// The first top-level paren group is the column-definition list; using it (rather
-	// than a greedy `(...)` capture) ignores any trailing `;`-separated statement.
-	const columnText = extractParenGroups(sql)[0]
-	if (!head || columnText === undefined) {
-		return []
-	}
-	const { schema, table } = parseRef(head[1])
-	const existing = datatable.schemas[schema]?.[table]
-	if (existing) {
-		return []
-	}
-	const columns: Record<string, string> = {}
-	for (const rawDef of splitTopLevel(columnText)) {
-		const def = rawDef.trim()
-		if (!def || isTableConstraint(def)) {
-			continue
-		}
-		const tokens = def.split(/\s+/)
-		const column = unquoteIdentifier(tokens[0])
-		if (!column) {
-			continue
-		}
-		columns[column] = tokens[1] ?? 'text'
-	}
-	if (!datatable.schemas[schema]) {
-		datatable.schemas[schema] = {}
-	}
-	datatable.schemas[schema][table] = { columns, rows: [] }
-	return []
-}
-
-function applyDropTable(
-	datatable: BenchmarkDatatableSeed,
-	sql: string
-): Record<string, unknown>[] {
-	const match = sql.match(
-		/^\s*drop\s+table\s+(?:if\s+exists\s+)?([a-zA-Z_"][\w."]*)/i
-	)
-	if (!match) {
-		return []
-	}
-	const { schema, table } = parseRef(match[1])
-	if (datatable.schemas[schema]?.[table]) {
-		delete datatable.schemas[schema][table]
-	}
-	return []
-}
-
-// ============= DML =============
-
-function applyInsert(
-	datatable: BenchmarkDatatableSeed,
-	sql: string
-): Record<string, unknown>[] {
-	const { body, returning } = splitOffReturning(sql)
-	const match = body.match(
-		/^\s*insert\s+into\s+([a-zA-Z_"][\w."]*)\s*(?:\(([^)]*)\))?\s*values\s*([\s\S]+)$/i
-	)
-	if (!match) {
-		return []
-	}
-	const table = resolveTable(datatable, match[1])
-	if (!table) {
-		return []
-	}
-	const columns = match[2]
-		? splitTopLevel(match[2]).map((entry) => unquoteIdentifier(entry.trim()))
-		: Object.keys(table.columns)
-	const inserted: Record<string, unknown>[] = []
-	for (const tuple of extractParenGroups(match[3])) {
-		const values = splitTopLevel(tuple).map((entry) => parseValue(entry))
-		const row: Record<string, unknown> = {}
-		columns.forEach((column, index) => {
-			row[column] = values[index]
-		})
-		inserted.push(row)
-	}
-	table.rows ??= []
-	table.rows.push(...inserted)
-	return returning ? inserted : []
-}
-
-function applyUpdate(
-	datatable: BenchmarkDatatableSeed,
-	sql: string
-): Record<string, unknown>[] {
-	const { body, returning } = splitOffReturning(sql)
-	const match = body.match(/^\s*update\s+([a-zA-Z_"][\w."]*)\s+set\s+([\s\S]+)$/i)
-	if (!match) {
-		return []
-	}
-	const table = resolveTable(datatable, match[1])
-	if (!table) {
-		return []
-	}
-	let assignmentText = match[2]
-	let whereText: string | undefined
-	const whereMatch = maskForClauseScan(assignmentText).match(/\swhere\s/i)
-	if (whereMatch && whereMatch.index !== undefined) {
-		whereText = assignmentText.slice(whereMatch.index + whereMatch[0].length)
-		assignmentText = assignmentText.slice(0, whereMatch.index)
-	}
-	const predicates = parsePredicates(whereText)
-	if (predicates === null) {
-		return []
-	}
-	const assignments: Record<string, unknown> = {}
-	for (const entry of splitTopLevel(assignmentText)) {
-		const pair = entry.match(/^\s*([a-zA-Z_"][\w."]*)\s*=\s*([\s\S]+?)\s*$/)
-		if (pair) {
-			assignments[lastIdentifier(pair[1])] = parseValue(pair[2])
-		}
-	}
-	const affected = (table.rows ?? []).filter((row) => rowMatches(row, predicates))
-	for (const row of affected) {
-		Object.assign(row, assignments)
-	}
-	return returning ? affected : []
-}
-
-function applyDelete(
-	datatable: BenchmarkDatatableSeed,
-	sql: string
-): Record<string, unknown>[] {
-	const { body, returning } = splitOffReturning(sql)
-	const match = body.match(/^\s*delete\s+from\s+([a-zA-Z_"][\w."]*)\s*([\s\S]*)$/i)
-	if (!match) {
-		return []
-	}
-	const table = resolveTable(datatable, match[1])
-	if (!table) {
-		return []
-	}
-	const whereText = match[2].replace(/^\s*where\s+/i, '').trim() || undefined
-	const predicates = parsePredicates(whereText)
-	if (predicates === null) {
-		return []
-	}
-	const rows = table.rows ?? []
-	const removed = rows.filter((row) => rowMatches(row, predicates))
-	table.rows = rows.filter((row) => !rowMatches(row, predicates))
-	return returning ? removed : []
-}
-
-// ============= Parsing helpers =============
-
-function resolveTable(
-	datatable: BenchmarkDatatableSeed,
-	ref: string
-): BenchmarkDatatableTableSeed | undefined {
-	const { schema, table } = parseRef(ref)
-	const direct = datatable.schemas[schema]?.[table]
-	if (direct) {
-		return direct
-	}
-	// Bare table name: fall back to searching every schema for a matching table.
-	if (!ref.includes('.')) {
-		for (const tables of Object.values(datatable.schemas)) {
-			if (tables[table]) {
-				return tables[table]
-			}
-		}
-	}
-	return undefined
-}
-
-function firstTable(
-	datatable: BenchmarkDatatableSeed
-): BenchmarkDatatableTableSeed | undefined {
-	for (const tables of Object.values(datatable.schemas)) {
-		for (const seed of Object.values(tables)) {
-			return seed
-		}
-	}
-	return undefined
-}
-
-function parseRef(ref: string): ParsedRef {
-	const parts = ref.split('.').map(unquoteIdentifier)
-	if (parts.length >= 2) {
-		return { schema: parts[parts.length - 2], table: parts[parts.length - 1] }
-	}
-	return { schema: DEFAULT_SCHEMA, table: parts[0] }
-}
-
-/** A WHERE clause with no parseable form returns `null`; absent WHERE returns `[]` (match all). */
-function parsePredicates(whereText: string | undefined): Predicate[] | null {
-	if (whereText === undefined || whereText.trim() === '') {
-		return []
-	}
-	const predicates: Predicate[] = []
-	for (const part of whereText.split(/\s+and\s+/i)) {
-		const match = part.match(/^\s*([a-zA-Z_"][\w."]*)\s*=\s*([\s\S]+?)\s*$/)
-		if (!match) {
-			return null
-		}
-		predicates.push({ column: lastIdentifier(match[1]), value: parseValue(match[2]) })
-	}
-	return predicates
-}
-
-function rowMatches(row: Record<string, unknown>, predicates: Predicate[]): boolean {
-	return predicates.every((predicate) => looseEquals(row[predicate.column], predicate.value))
-}
-
-function looseEquals(left: unknown, right: unknown): boolean {
-	if (left === null || left === undefined) {
-		return right === null || right === undefined
-	}
-	if (typeof left === 'number' && typeof right === 'number') {
-		return left === right
-	}
-	return String(left) === String(right)
-}
-
-function parseValue(raw: string): unknown {
-	// Drop a trailing Postgres cast (e.g. `2::int4`) before interpreting the literal.
-	const token = raw.trim().replace(/::\s*[a-zA-Z_][\w]*(\([^)]*\))?\s*$/, '').trim()
-	const stringMatch = token.match(/^'([\s\S]*)'$/)
-	if (stringMatch) {
-		return stringMatch[1].replace(/''/g, "'")
-	}
-	if (/^-?\d+(\.\d+)?$/.test(token)) {
-		return Number(token)
-	}
-	if (/^true$/i.test(token)) {
-		return true
-	}
-	if (/^false$/i.test(token)) {
-		return false
-	}
-	if (/^null$/i.test(token)) {
-		return null
-	}
-	return token
-}
-
-function splitOffReturning(sql: string): { body: string; returning: boolean } {
-	const match = maskForClauseScan(sql).match(/\sreturning\s/i)
-	if (!match || match.index === undefined) {
-		return { body: sql, returning: false }
-	}
-	return { body: sql.slice(0, match.index), returning: true }
-}
-
-/**
- * A same-length copy of `sql` with the contents of single-quoted strings and
- * parenthesized groups blanked to spaces, so a top-level keyword scan
- * (WHERE / RETURNING) cannot match inside a string literal or a subquery. Index
- * positions in the result map 1:1 back onto the original.
- */
-function maskForClauseScan(sql: string): string {
-	let masked = ''
-	let depth = 0
-	let inString = false
-	for (let i = 0; i < sql.length; i++) {
-		const char = sql[i]
-		if (inString) {
-			if (char === "'") {
-				if (sql[i + 1] === "'") {
-					masked += '  '
-					i++
-					continue
-				}
-				inString = false
-			}
-			masked += ' '
-			continue
-		}
-		if (char === "'") {
-			inString = true
-			masked += ' '
-		} else if (char === '(') {
-			depth++
-			masked += ' '
-		} else if (char === ')') {
-			depth = Math.max(0, depth - 1)
-			masked += ' '
-		} else {
-			masked += depth > 0 ? ' ' : char
-		}
-	}
-	return masked
-}
-
-/**
- * Inner text of each top-level `( ... )` group in `input`, honoring nested parens
- * (e.g. `now()`, `numeric(10,2)`) and single-quoted strings. Used for the CREATE
- * column-definition group and INSERT value tuples.
- */
-function extractParenGroups(input: string): string[] {
-	const groups: string[] = []
-	let depth = 0
-	let inString = false
-	let current = ''
-	for (let i = 0; i < input.length; i++) {
-		const char = input[i]
-		if (inString) {
-			current += char
-			if (char === "'") {
-				if (input[i + 1] === "'") {
-					current += input[++i]
-				} else {
-					inString = false
-				}
-			}
-			continue
-		}
-		if (char === "'") {
-			inString = true
-			current += char
-		} else if (char === '(') {
-			depth++
-			if (depth === 1) {
-				current = ''
-			} else {
-				current += char
-			}
-		} else if (char === ')') {
-			depth = Math.max(0, depth - 1)
-			if (depth === 0) {
-				groups.push(current)
-				current = ''
-			} else {
-				current += char
-			}
-		} else if (depth > 0) {
-			current += char
-		}
-	}
-	return groups
-}
-
-/** Split on commas that are not inside parentheses or single-quoted strings. */
-function splitTopLevel(input: string): string[] {
-	const parts: string[] = []
-	let depth = 0
-	let inString = false
-	let current = ''
-	for (let i = 0; i < input.length; i++) {
-		const char = input[i]
-		if (inString) {
-			current += char
-			if (char === "'") {
-				if (input[i + 1] === "'") {
-					current += input[++i]
-				} else {
-					inString = false
-				}
-			}
-			continue
-		}
-		if (char === "'") {
-			inString = true
-			current += char
-		} else if (char === '(') {
-			depth++
-			current += char
-		} else if (char === ')') {
-			depth = Math.max(0, depth - 1)
-			current += char
-		} else if (char === ',' && depth === 0) {
-			parts.push(current)
-			current = ''
-		} else {
-			current += char
-		}
-	}
-	if (current.trim() !== '') {
-		parts.push(current)
-	}
-	return parts
-}
-
-function isTableConstraint(def: string): boolean {
-	return /^(primary\s+key|foreign\s+key|constraint|unique|check|exclude|like)\b/i.test(def)
-}
-
-function unquoteIdentifier(identifier: string): string {
-	const trimmed = identifier.trim()
-	const quoted = trimmed.match(/^"([\s\S]*)"$/)
-	return quoted ? quoted[1] : trimmed
-}
-
-/** For a qualified reference like `orders.id`, keep only the final identifier. */
-function lastIdentifier(reference: string): string {
-	const parts = reference.split('.')
-	return unquoteIdentifier(parts[parts.length - 1])
-}
-
-function stripTrailingSemicolon(sql: string): string {
-	return sql.replace(/;\s*$/, '')
-}

ai_evals/adapters/frontend/mockBackend.ts

@@ -1,25 +1,7 @@
 import { randomUUID } from 'node:crypto'
-import type {
-	AppWithLastVersion,
-	CompletedJob,
-	Flow,
-	Job,
-	ListableApp,
-	Script
-} from '../../../frontend/src/lib/gen'
-import type {
-	DataTableTables,
-	DataTableTableSchema,
-	GetDraftForUserResponse,
-	ListDraftsResponse,
-	ScriptLang,
-	UpdateDraftResponse,
-	UserDraftItemKind
-} from '../../../frontend/src/lib/gen/types.gen'
+import type { CompletedJob, Flow, Script } from '../../../frontend/src/lib/gen'
+import type { ScriptLang } from '../../../frontend/src/lib/gen/types.gen'
 import { buildScriptLintResult } from './core/script/preview'
-import { applyDatatableSql, type BenchmarkDatatableSeed } from './datatableSqlEngine'
-
-export type { BenchmarkDatatableSeed, BenchmarkDatatableTableSeed } from './datatableSqlEngine'
 
 const BENCHMARK_TIMESTAMP = '1970-01-01T00:00:00.000Z'
 
@@ -40,54 +22,21 @@ export interface BenchmarkWorkspaceFlow {
 	value: Flow['value']
 }
 
-export interface BenchmarkWorkspaceApp {
-	path: string
-	summary: string
-	value: {
-		files: Record<string, string>
-		runnables: Record<string, unknown>
-		data?: unknown
-		policy?: unknown
-		custom_path?: unknown
-	}
-}
-
-export interface BenchmarkWorkspaceJob {
-	/** Stable id so a case prompt can reference a specific run (e.g. for get_job_logs). */
-	id?: string
-	jobKind?: CompletedJob['job_kind']
-	scriptPath?: string
-	createdBy?: string
-	label?: string
-	success?: boolean
-	logs?: string
-}
-
 export interface BenchmarkWorkspaceRunnables {
 	scripts?: BenchmarkWorkspaceScript[]
 	flows?: BenchmarkWorkspaceFlow[]
-	apps?: BenchmarkWorkspaceApp[]
-	datatables?: BenchmarkDatatableSeed[]
-	jobs?: BenchmarkWorkspaceJob[]
 }
 
 type BenchmarkCompletedJob = CompletedJob & { type: 'CompletedJob' }
 
 const benchmarkWorkspaces = new Set<string>()
 const benchmarkWorkspaceRunnables = new Map<string, BenchmarkWorkspaceRunnables>()
-// Keyed by `${workspace}::${jobId}` so concurrent attempts (or distinct cases)
-// can seed the same fixed job id without clobbering each other's entry.
 const benchmarkJobs = new Map<string, { workspace: string; job: BenchmarkCompletedJob }>()
 
-function benchmarkJobKey(workspace: string, jobId: string): string {
-	return `${workspace}::${jobId}`
-}
-
 export function resetBenchmarkMockBackend(): void {
 	benchmarkWorkspaces.clear()
 	benchmarkWorkspaceRunnables.clear()
 	benchmarkJobs.clear()
-	benchmarkDrafts.clear()
 }
 
 export function registerBenchmarkWorkspace(workspace: string): void {
@@ -99,33 +48,12 @@ export function registerBenchmarkWorkspaceRunnables(
 	runnables: BenchmarkWorkspaceRunnables
 ): void {
 	benchmarkWorkspaces.add(workspace)
-	// Fresh case: drop any drafts left from a prior run on this workspace id.
-	clearBenchmarkDrafts(workspace)
-	// Datatables are mutated in place by exec_datatable_sql (a write must be visible
-	// to later reads), so store an isolated deep copy — never mutate the caller's seed.
-	benchmarkWorkspaceRunnables.set(workspace, {
-		...runnables,
-		datatables: runnables.datatables ? structuredClone(runnables.datatables) : undefined
-	})
-	// Seed any fixture jobs so list_runs / get_job_logs have data to return.
-	for (const seed of runnables.jobs ?? []) {
-		createBenchmarkCompletedJob({
-			workspace,
-			id: seed.id,
-			jobKind: seed.jobKind ?? 'script',
-			success: seed.success,
-			scriptPath: seed.scriptPath,
-			createdBy: seed.createdBy,
-			label: seed.label,
-			logs: seed.logs
-		})
-	}
+	benchmarkWorkspaceRunnables.set(workspace, runnables)
 }
 
 export function unregisterBenchmarkWorkspace(workspace: string): void {
 	benchmarkWorkspaces.delete(workspace)
 	benchmarkWorkspaceRunnables.delete(workspace)
-	clearBenchmarkDrafts(workspace)
 	for (const [jobId, entry] of benchmarkJobs.entries()) {
 		if (entry.workspace === workspace) {
 			benchmarkJobs.delete(jobId)
@@ -181,22 +109,6 @@ export function getBenchmarkFlowByPath(workspace: string, path: string): Flow |
 	return flow ? buildBenchmarkFlow(flow) : null
 }
 
-export function listBenchmarkApps(workspace: string): ListableApp[] | null {
-	const runnables = benchmarkWorkspaceRunnables.get(workspace)
-	if (!runnables) {
-		return null
-	}
-	return (runnables.apps ?? []).map(buildBenchmarkListableApp)
-}
-
-export function getBenchmarkAppByPath(workspace: string, path: string): AppWithLastVersion | null {
-	const app = benchmarkWorkspaceRunnables
-		.get(workspace)
-		?.apps?.find((entry) => entry.path === path)
-
-	return app ? buildBenchmarkApp(app) : null
-}
-
 export function createBenchmarkCompletedJob(input: {
 	workspace: string
 	jobKind: CompletedJob['job_kind']
@@ -206,17 +118,14 @@ export function createBenchmarkCompletedJob(input: {
 	scriptPath?: string
 	scriptHash?: string
 	args?: Record<string, unknown>
-	id?: string
-	createdBy?: string
-	label?: string
 }): string {
-	const jobId = input.id ?? `benchmark-job-${randomUUID()}`
+	const jobId = `benchmark-job-${randomUUID()}`
 	const now = new Date().toISOString()
 	const job: BenchmarkCompletedJob = {
 		type: 'CompletedJob',
 		id: jobId,
 		workspace_id: input.workspace,
-		created_by: input.createdBy ?? 'ai-evals',
+		created_by: 'ai-evals',
 		created_at: now,
 		started_at: now,
 		completed_at: now,
@@ -234,11 +143,10 @@ export function createBenchmarkCompletedJob(input: {
 		is_skipped: false,
 		email: 'ai-evals@local',
 		visible_to_owner: true,
-		tag: 'benchmark',
-		labels: input.label ? [input.label] : undefined
+		tag: 'benchmark'
 	}
 
-	benchmarkJobs.set(benchmarkJobKey(input.workspace, jobId), { workspace: input.workspace, job })
+	benchmarkJobs.set(jobId, { workspace: input.workspace, job })
 	return jobId
 }
 
@@ -246,239 +154,13 @@ export function getBenchmarkCompletedJob(
 	workspace: string,
 	jobId: string
 ): BenchmarkCompletedJob | null {
-	const entry = benchmarkJobs.get(benchmarkJobKey(workspace, jobId))
-	if (!entry) {
+	const entry = benchmarkJobs.get(jobId)
+	if (!entry || entry.workspace !== workspace) {
 		return null
 	}
 	return structuredClone(entry.job)
 }
 
-/**
- * List seeded/recorded jobs for a benchmark workspace, most recent first —
- * the shape `JobService.listJobs` returns. Returns `null` for a non-benchmark
- * workspace so the caller can fall through to the real backend. Server-side
- * filters (path/creator/status/limit) are intentionally not applied: global
- * eval cases assert on the recorded `list_runs` tool call, not on filtering.
- */
-export function listBenchmarkJobs(workspace: string): Job[] | null {
-	if (!hasBenchmarkWorkspace(workspace)) {
-		return null
-	}
-	return [...benchmarkJobs.values()]
-		.filter((entry) => entry.workspace === workspace)
-		.map((entry) => structuredClone(entry.job) as Job)
-		.sort((a, b) => (b.created_at ?? '').localeCompare(a.created_at ?? ''))
-}
-
-/**
- * Mirror `JobService.getJobLogs` (response is the raw log string). Throws a
- * "not found" error for an unknown id, matching the backend 404.
- */
-export function getBenchmarkJobLogs(workspace: string, jobId: string): string {
-	const job = getBenchmarkCompletedJob(workspace, jobId)
-	if (!job) {
-		throw new Error(`Job Logs not found for "${jobId}"`)
-	}
-	return job.logs ?? ''
-}
-
-// ============= Drafts (per-user, DB-backed in production) =============
-
-/**
- * In-memory stand-in for the per-user draft backend (`DraftService`). The global
- * AI chat now persists and reads drafts through the backend DB instead of an
- * in-tab `UserDraft` cell, so the eval mocks the three draft endpoints it
- * exercises (`updateDraft` / `getDraftForUser` / `listDrafts`) and keeps the
- * saved values here, keyed by workspace + draft kind + storage path. Mirrors the
- * semantics of the production unit test's mock in
- * `frontend/src/lib/components/copilot/chat/global/core.test.ts`.
- */
-const benchmarkDrafts = new Map<
-	string,
-	{ workspace: string; kind: UserDraftItemKind; path: string; value: unknown }
->()
-
-// Fixed timestamp so artifacts stay deterministic. No eval simulates a
-// concurrent writer, so every save is accepted and the conflict branch is
-// never taken — the syncer just records this as its `last_sync` baseline.
-const BENCHMARK_DRAFT_TIMESTAMP = '1970-01-01T00:00:00.000Z'
-
-function benchmarkDraftKey(workspace: string, kind: string, path: string): string {
-	return `${workspace}::${kind}::${path}`
-}
-
-export function clearBenchmarkDrafts(workspace: string): void {
-	for (const [key, entry] of benchmarkDrafts.entries()) {
-		if (entry.workspace === workspace) {
-			benchmarkDrafts.delete(key)
-		}
-	}
-}
-
-/**
- * Seed a draft straight into the store — used by the eval's live-editor draft
- * fixtures, which model "the user already has this draft open/saved". Writing it
- * here (instead of through `UserDraft.save`) keeps it a backend draft row with no
- * shadowing in-tab cell, so a model edit that persists to the backend is what the
- * output read-back captures — not the stale seed.
- */
-export function seedBenchmarkDraft(
-	workspace: string,
-	kind: UserDraftItemKind,
-	path: string,
-	value: unknown
-): void {
-	benchmarkDrafts.set(benchmarkDraftKey(workspace, kind, path), {
-		workspace,
-		kind,
-		path,
-		value
-	})
-}
-
-/** Mirror `DraftService.updateDraft`: a `null`/omitted value deletes the row. */
-export function updateBenchmarkDraft(input: {
-	workspace: string
-	kind: UserDraftItemKind
-	path: string
-	requestBody?: { value?: unknown }
-}): UpdateDraftResponse {
-	const key = benchmarkDraftKey(input.workspace, input.kind, input.path)
-	const value = input.requestBody?.value
-	if (value == null) {
-		benchmarkDrafts.delete(key)
-	} else {
-		benchmarkDrafts.set(key, {
-			workspace: input.workspace,
-			kind: input.kind,
-			path: input.path,
-			value
-		})
-	}
-	return { status: 'saved', current_timestamp: BENCHMARK_DRAFT_TIMESTAMP }
-}
-
-/** Mirror `DraftService.getDraftForUser`: 404-shaped throw when absent so the
- * adapter's narrowed catch treats it as "no draft" instead of re-throwing. */
-export function getBenchmarkDraftForUser(input: {
-	workspace: string
-	kind: UserDraftItemKind
-	path: string
-}): GetDraftForUserResponse {
-	const entry = benchmarkDrafts.get(benchmarkDraftKey(input.workspace, input.kind, input.path))
-	if (!entry) {
-		throw Object.assign(new Error(`no draft for "${input.path}"`), { status: 404 })
-	}
-	return { value: entry.value, created_at: BENCHMARK_DRAFT_TIMESTAMP }
-}
-
-/** Mirror `DraftService.listDrafts`: metadata rows (no value) for a workspace. */
-export function listBenchmarkDrafts(workspace: string): ListDraftsResponse {
-	return [...benchmarkDrafts.values()]
-		.filter((entry) => entry.workspace === workspace)
-		.map((entry) => ({
-			kind: entry.kind,
-			path: entry.path,
-			summary: (entry.value as { summary?: string } | null)?.summary,
-			draft_only: true,
-			legacy_draft: false,
-			created_at: BENCHMARK_DRAFT_TIMESTAMP
-		}))
-}
-
-// ============= Datatables (best-effort in-memory SQL) =============
-
-/**
- * Project the seeded datatables down to the `list_datatable_tables` response:
- * `datatable_name` + `schema -> table_names`, with no column detail.
- * Returns `null` for a non-benchmark workspace so callers can fall through to
- * the real backend; an empty seed yields `[]`.
- */
-export function listBenchmarkDatatables(workspace: string): DataTableTables[] | null {
-	const runnables = benchmarkWorkspaceRunnables.get(workspace)
-	if (!runnables) {
-		return null
-	}
-	return (runnables.datatables ?? []).map((datatable) => ({
-		datatable_name: datatable.datatable_name,
-		schemas: Object.fromEntries(
-			Object.entries(datatable.schemas).map(([schema, tables]) => [schema, Object.keys(tables)])
-		)
-	}))
-}
-
-export function getBenchmarkDatatableSchema(input: {
-	workspace: string
-	datatableName: string
-	schemaName: string
-	tableName: string
-}): DataTableTableSchema {
-	const runnables = benchmarkWorkspaceRunnables.get(input.workspace)
-	const datatable = (runnables?.datatables ?? []).find(
-		(entry) => entry.datatable_name === input.datatableName
-	)
-	if (!datatable) {
-		// Message MUST match the production `isDatatableNotConfiguredError` regex
-		// (/datatable\s+\S+\s+not found/i in datatableTools.ts) so the
-		// get_datatable_table_schema not-configured mapping is actually exercised.
-		throw new Error(`datatable "${input.datatableName}" not found`)
-	}
-	const table = datatable.schemas?.[input.schemaName]?.[input.tableName]
-	if (!table) {
-		throw new Error(
-			`table "${input.schemaName}.${input.tableName}" not found in datatable "${input.datatableName}"`
-		)
-	}
-	return {
-		datatable_name: input.datatableName,
-		schema_name: input.schemaName,
-		table_name: input.tableName,
-		columns: table.columns
-	}
-}
-
-/**
- * Execute SQL against a seeded datatable through the best-effort in-memory engine
- * (`applyDatatableSql`). Writes (CREATE/INSERT/UPDATE/DELETE/DROP) mutate the
- * stored datatable in place so a later list/schema/SELECT reflects them; SELECT
- * (and RETURNING) yield rows, other statements yield `[]`. Creates a benchmark
- * completed job and returns its id, like `runBenchmarkScriptPreview`.
- */
-export function runBenchmarkDatatableSql(input: {
-	workspace: string
-	datatableName: string
-	sql: string
-}): string {
-	const runnables = benchmarkWorkspaceRunnables.get(input.workspace)
-	const datatable = (runnables?.datatables ?? []).find(
-		(entry) => entry.datatable_name === input.datatableName
-	)
-	const rows = datatable ? applyDatatableSql(datatable, input.sql).rows : []
-	return createBenchmarkCompletedJob({
-		workspace: input.workspace,
-		jobKind: 'preview',
-		success: true,
-		args: { database: `datatable://${input.datatableName}` },
-		result: rows
-	})
-}
-
-/**
- * Mirror `JobService.getCompletedJobResultMaybe` for benchmark workspaces — the
- * shape `pollJobResult` consumes. The job is created synchronously before
- * polling, so it is always present and completed.
- */
-export function getBenchmarkCompletedJobResultMaybe(input: {
-	workspace: string
-	id: string
-}): { success: boolean; completed: boolean; result: unknown } {
-	const job = getBenchmarkCompletedJob(input.workspace, input.id)
-	if (!job) {
-		throw new Error(`Job "${input.id}" not found in benchmark workspace`)
-	}
-	return { success: job.success, completed: true, result: job.result }
-}
-
 export function runBenchmarkScriptPreview(input: {
 	workspace: string
 	requestBody: {
@@ -640,35 +322,3 @@ function buildBenchmarkFlow(flow: BenchmarkWorkspaceFlow): Flow {
 		extra_perms: {}
 	} as Flow
 }
-
-function buildBenchmarkListableApp(app: BenchmarkWorkspaceApp): ListableApp {
-	return {
-		id: 0,
-		workspace_id: 'benchmark',
-		path: app.path,
-		summary: app.summary,
-		version: 1,
-		extra_perms: {},
-		edited_at: BENCHMARK_TIMESTAMP,
-		execution_mode: 'viewer',
-		raw_app: true
-	}
-}
-
-function buildBenchmarkApp(app: BenchmarkWorkspaceApp): AppWithLastVersion {
-	return {
-		id: 0,
-		workspace_id: 'benchmark',
-		path: app.path,
-		summary: app.summary,
-		versions: [1],
-		created_by: 'benchmark',
-		created_at: BENCHMARK_TIMESTAMP,
-		value: app.value,
-		policy: (app.value.policy ?? {}) as AppWithLastVersion['policy'],
-		execution_mode: 'viewer',
-		extra_perms: {},
-		custom_path: app.value.custom_path as string | undefined,
-		raw_app: true
-	}
-}

ai_evals/adapters/frontend/mockBackendDatatables.test.ts

@@ -1,175 +0,0 @@
-import { afterEach, beforeEach, describe, expect, it } from 'bun:test'
-import {
-	getBenchmarkCompletedJobResultMaybe,
-	getBenchmarkDatatableSchema,
-	listBenchmarkDatatables,
-	registerBenchmarkWorkspaceRunnables,
-	resetBenchmarkMockBackend,
-	runBenchmarkDatatableSql,
-	type BenchmarkWorkspaceRunnables
-} from './mockBackend'
-
-const WORKSPACE = 'benchmark-datatable-ws'
-
-// Mirrors the production `isDatatableNotConfiguredError` regex in
-// datatableTools.ts. The schema mock's "not configured" message MUST match it,
-// otherwise the not-configured mapping in get_datatable_table_schema is silently
-// untested.
-const NOT_CONFIGURED_RE = /datatable\s+\S+\s+not found/i
-
-const SEED: BenchmarkWorkspaceRunnables = {
-	datatables: [
-		{
-			datatable_name: 'main',
-			schemas: {
-				public: {
-					orders: {
-						columns: { id: 'int', total: 'numeric' },
-						rows: [
-							{ id: 1, total: 10 },
-							{ id: 2, total: 20 }
-						]
-					},
-					customers: {
-						columns: { id: 'int', name: 'text' },
-						rows: [{ id: 1, name: 'alice' }]
-					}
-				}
-			}
-		}
-	]
-}
-
-beforeEach(() => resetBenchmarkMockBackend())
-afterEach(() => resetBenchmarkMockBackend())
-
-describe('listBenchmarkDatatables', () => {
-	it('returns null for a non-benchmark workspace (caller falls through to real backend)', () => {
-		expect(listBenchmarkDatatables('unregistered')).toBeNull()
-	})
-
-	it('returns [] for a registered workspace with no datatables seed', () => {
-		registerBenchmarkWorkspaceRunnables(WORKSPACE, {})
-		expect(listBenchmarkDatatables(WORKSPACE)).toEqual([])
-	})
-
-	it('projects seeded datatables to schema -> table names only (no columns)', () => {
-		registerBenchmarkWorkspaceRunnables(WORKSPACE, SEED)
-		expect(listBenchmarkDatatables(WORKSPACE)).toEqual([
-			{ datatable_name: 'main', schemas: { public: ['orders', 'customers'] } }
-		])
-	})
-})
-
-describe('getBenchmarkDatatableSchema', () => {
-	beforeEach(() => registerBenchmarkWorkspaceRunnables(WORKSPACE, SEED))
-
-	it('returns the columns for a seeded table', () => {
-		expect(
-			getBenchmarkDatatableSchema({
-				workspace: WORKSPACE,
-				datatableName: 'main',
-				schemaName: 'public',
-				tableName: 'orders'
-			})
-		).toEqual({
-			datatable_name: 'main',
-			schema_name: 'public',
-			table_name: 'orders',
-			columns: { id: 'int', total: 'numeric' }
-		})
-	})
-
-	it('throws a not-configured error matching the production regex for an unknown datatable', () => {
-		let error: Error | undefined
-		try {
-			getBenchmarkDatatableSchema({
-				workspace: WORKSPACE,
-				datatableName: 'ghost',
-				schemaName: 'public',
-				tableName: 'orders'
-			})
-		} catch (e) {
-			error = e as Error
-		}
-		expect(error).toBeDefined()
-		expect(error!.message).toMatch(NOT_CONFIGURED_RE)
-	})
-
-	it('throws a table-not-found error that does NOT match the datatable-not-configured regex', () => {
-		// The datatable IS configured; only the table is missing. Production maps
-		// this to a generic "error getting schema", not the blocking message.
-		let error: Error | undefined
-		try {
-			getBenchmarkDatatableSchema({
-				workspace: WORKSPACE,
-				datatableName: 'main',
-				schemaName: 'public',
-				tableName: 'ghost'
-			})
-		} catch (e) {
-			error = e as Error
-		}
-		expect(error).toBeDefined()
-		expect(error!.message).not.toMatch(NOT_CONFIGURED_RE)
-	})
-})
-
-describe('runBenchmarkDatatableSql + getBenchmarkCompletedJobResultMaybe', () => {
-	beforeEach(() => registerBenchmarkWorkspaceRunnables(WORKSPACE, SEED))
-
-	function exec(sql: string): { success: boolean; completed: boolean; result: unknown } {
-		const jobId = runBenchmarkDatatableSql({ workspace: WORKSPACE, datatableName: 'main', sql })
-		return getBenchmarkCompletedJobResultMaybe({ workspace: WORKSPACE, id: jobId })
-	}
-
-	it('returns the canned rows of the table named in a SELECT FROM clause', () => {
-		expect(exec('SELECT * FROM customers')).toEqual({
-			success: true,
-			completed: true,
-			result: [{ id: 1, name: 'alice' }]
-		})
-	})
-
-	it('falls back to the first seeded table when the SELECT references no known table', () => {
-		expect(exec('select 1').result).toEqual([
-			{ id: 1, total: 10 },
-			{ id: 2, total: 20 }
-		])
-	})
-
-	it('returns [] success for DDL and DML statements without RETURNING', () => {
-		expect(exec('CREATE TABLE foo (id int)').result).toEqual([])
-		expect(exec('INSERT INTO orders VALUES (3, 30)').result).toEqual([])
-		expect(exec('update orders set total = 0').result).toEqual([])
-	})
-
-	it('reflects a write in a later SELECT, isolated from the shared seed', () => {
-		exec('UPDATE orders SET total = 999 WHERE id = 1')
-		expect((exec('SELECT * FROM orders').result as Record<string, unknown>[])).toContainEqual({
-			id: 1,
-			total: 999
-		})
-		// Registration deep-clones the seed, so the shared SEED const stays pristine.
-		expect(SEED.datatables![0].schemas.public.orders.rows).toContainEqual({ id: 1, total: 10 })
-	})
-
-	it('reflects a CREATE in list_datatables and get_datatable_table_schema', () => {
-		exec('CREATE TABLE public.refunds (order_id int4, amount numeric)')
-		expect(listBenchmarkDatatables(WORKSPACE)?.[0].schemas.public).toContain('refunds')
-		expect(
-			getBenchmarkDatatableSchema({
-				workspace: WORKSPACE,
-				datatableName: 'main',
-				schemaName: 'public',
-				tableName: 'refunds'
-			}).columns
-		).toEqual({ order_id: 'int4', amount: 'numeric' })
-	})
-
-	it('throws for an unknown job id', () => {
-		expect(() =>
-			getBenchmarkCompletedJobResultMaybe({ workspace: WORKSPACE, id: 'does-not-exist' })
-		).toThrow()
-	})
-})

ai_evals/adapters/frontend/mockBackendDrafts.test.ts

@@ -1,94 +0,0 @@
-import { afterEach, beforeEach, describe, expect, it } from 'bun:test'
-import {
-	clearBenchmarkDrafts,
-	getBenchmarkDraftForUser,
-	listBenchmarkDrafts,
-	resetBenchmarkMockBackend,
-	seedBenchmarkDraft,
-	updateBenchmarkDraft
-} from './mockBackend'
-
-const WORKSPACE = 'benchmark-drafts-ws'
-
-// Drives the in-memory stand-in for the per-user draft backend (`DraftService`)
-// that the global AI-chat eval round-trips its drafts through. Mirrors the
-// production-unit-test mock in
-// `frontend/src/lib/components/copilot/chat/global/core.test.ts`.
-describe('mockBackend drafts', () => {
-	beforeEach(() => resetBenchmarkMockBackend())
-	afterEach(() => resetBenchmarkMockBackend())
-
-	it('round-trips a saved draft through update / get / list', () => {
-		const value = { summary: 'Greet a user', content: 'export async function main() {}' }
-		const res = updateBenchmarkDraft({
-			workspace: WORKSPACE,
-			kind: 'script',
-			path: 'f/evals/greet',
-			requestBody: { value }
-		})
-		expect(res.status).toBe('saved')
-
-		expect(getBenchmarkDraftForUser({ workspace: WORKSPACE, kind: 'script', path: 'f/evals/greet' }).value).toEqual(
-			value
-		)
-
-		const rows = listBenchmarkDrafts(WORKSPACE)
-		expect(rows).toHaveLength(1)
-		expect(rows[0]).toMatchObject({ kind: 'script', path: 'f/evals/greet', summary: 'Greet a user', draft_only: true })
-	})
-
-	it('treats a null value as a delete', () => {
-		updateBenchmarkDraft({
-			workspace: WORKSPACE,
-			kind: 'variable',
-			path: 'f/evals/token',
-			requestBody: { value: { summary: 'token' } }
-		})
-		updateBenchmarkDraft({
-			workspace: WORKSPACE,
-			kind: 'variable',
-			path: 'f/evals/token',
-			requestBody: { value: null }
-		})
-
-		expect(listBenchmarkDrafts(WORKSPACE)).toHaveLength(0)
-		expect(() => getBenchmarkDraftForUser({ workspace: WORKSPACE, kind: 'variable', path: 'f/evals/token' })).toThrow()
-	})
-
-	it('throws a 404-shaped error when no draft exists', () => {
-		try {
-			getBenchmarkDraftForUser({ workspace: WORKSPACE, kind: 'script', path: 'f/evals/missing' })
-			throw new Error('expected a throw')
-		} catch (e) {
-			expect((e as { status?: number }).status).toBe(404)
-		}
-	})
-
-	it('seeds a draft as a backend row that a later edit overwrites', () => {
-		seedBenchmarkDraft(WORKSPACE, 'script', 'f/evals/current', { content: 'seed' })
-		expect(getBenchmarkDraftForUser({ workspace: WORKSPACE, kind: 'script', path: 'f/evals/current' }).value).toEqual({
-			content: 'seed'
-		})
-
-		// A model edit persists the same path and must win over the seed.
-		updateBenchmarkDraft({
-			workspace: WORKSPACE,
-			kind: 'script',
-			path: 'f/evals/current',
-			requestBody: { value: { content: 'edited' } }
-		})
-		expect(getBenchmarkDraftForUser({ workspace: WORKSPACE, kind: 'script', path: 'f/evals/current' }).value).toEqual({
-			content: 'edited'
-		})
-	})
-
-	it('clears only the targeted workspace', () => {
-		seedBenchmarkDraft(WORKSPACE, 'script', 'f/a', { content: 'a' })
-		seedBenchmarkDraft('other-ws', 'script', 'f/b', { content: 'b' })
-
-		clearBenchmarkDrafts(WORKSPACE)
-
-		expect(listBenchmarkDrafts(WORKSPACE)).toHaveLength(0)
-		expect(listBenchmarkDrafts('other-ws')).toHaveLength(1)
-	})
-})

ai_evals/adapters/frontend/progress.ts

@@ -1,4 +1,4 @@
-export type FrontendBenchmarkProgressSurface = 'flow' | 'app' | 'script' | 'global'
+export type FrontendBenchmarkProgressSurface = 'flow' | 'app' | 'script'
 
 export type FrontendBenchmarkProgressEvent =
 	| {

ai_evals/adapters/frontend/runtime.ts

@@ -16,16 +16,15 @@ const FRONTEND_BENCHMARK_TEST =
 const FRONTEND_BENCHMARK_CONFIG =
   "../ai_evals/adapters/frontend/vitest.config.ts";
 
-export type FrontendMode = "flow" | "app" | "script" | "global";
+export type FrontendMode = "flow" | "app" | "script";
 
 export async function runFrontendBenchmarkAdapter(input: {
   mode: FrontendMode;
   caseIds: string[];
   runs: number;
   model?: string;
+  transport?: string;
   verbose?: boolean;
-  skipJudge?: boolean;
-  executionOnly?: boolean;
   backendValidation?: string;
 }): Promise<BenchmarkRunResult> {
   const tempDir = await mkdtemp(
@@ -42,12 +41,13 @@ export async function runFrontendBenchmarkAdapter(input: {
     WMILL_FRONTEND_AI_EVAL_MODEL: input.model ?? "",
     WMILL_FRONTEND_AI_EVAL_PROGRESS: "1",
     WMILL_FRONTEND_AI_EVAL_VERBOSE: input.verbose ? "1" : "0",
-    WMILL_FRONTEND_AI_EVAL_SKIP_JUDGE:
-      input.skipJudge || input.executionOnly ? "1" : "0",
-    WMILL_FRONTEND_AI_EVAL_EXECUTION_ONLY: input.executionOnly ? "1" : "0",
     WMILL_FRONTEND_AI_EVAL_BACKEND_VALIDATION: input.backendValidation ?? "",
   };
 
+  if (input.transport) {
+    env.WMILL_FRONTEND_AI_EVAL_TRANSPORT = input.transport;
+  }
+
   try {
     await runVitestBenchmark(
       path.join(FRONTEND_DIR, "node_modules", ".bin", "vitest"),

ai_evals/adapters/frontend/vitestAdapter.test.ts

@@ -33,29 +33,18 @@ vi.mock('$lib/components/vscode', () => ({}))
 vi.mock('$lib/gen', async () => {
 	const actual = await vi.importActual<any>('$lib/gen')
 	const {
-		getBenchmarkAppByPath,
 		getBenchmarkCompletedJob,
-		getBenchmarkCompletedJobResultMaybe,
-		getBenchmarkDatatableSchema,
-		getBenchmarkDraftForUser,
 		getBenchmarkFlowByPath,
-		getBenchmarkJobLogs,
 		getBenchmarkScriptByHash,
 		getBenchmarkScriptByPath,
 		hasBenchmarkWorkspace,
-		listBenchmarkApps,
-		listBenchmarkDatatables,
-		listBenchmarkDrafts,
 		listBenchmarkFlows,
-		listBenchmarkJobs,
 		listBenchmarkScripts,
 		createBenchmarkHttpTrigger,
 		createBenchmarkSchedule,
 		previewBenchmarkSchedule,
-		runBenchmarkDatatableSql,
 		runBenchmarkFlowByPath,
-		runBenchmarkScriptPreview,
-		updateBenchmarkDraft
+		runBenchmarkScriptPreview
 	} = await import('./mockBackend')
 
 	function wrapService<T extends object>(target: T, overrides: Record<string, unknown>): T {
@@ -71,34 +60,11 @@ vi.mock('$lib/gen', async () => {
 
 	return {
 		...actual,
-		DraftService: wrapService(actual.DraftService, {
-			updateDraft: async (data: {
-				workspace: string
-				kind: any
-				path: string
-				requestBody?: { value?: unknown }
-			}) =>
-				hasBenchmarkWorkspace(data.workspace)
-					? updateBenchmarkDraft(data)
-					: actual.DraftService.updateDraft(data),
-			getDraftForUser: async (data: { workspace: string; kind: any; path: string }) =>
-				hasBenchmarkWorkspace(data.workspace)
-					? getBenchmarkDraftForUser(data)
-					: actual.DraftService.getDraftForUser(data),
-			listDrafts: async (data: { workspace: string }) =>
-				hasBenchmarkWorkspace(data.workspace)
-					? listBenchmarkDrafts(data.workspace)
-					: actual.DraftService.listDrafts(data)
-		}),
 		ScriptService: wrapService(actual.ScriptService, {
 			listScripts: async (data: { workspace: string }) =>
 				hasBenchmarkWorkspace(data.workspace)
 					? (listBenchmarkScripts(data.workspace) ?? [])
 					: actual.ScriptService.listScripts(data),
-			existsScriptByPath: async (data: { workspace: string; path: string }) =>
-				hasBenchmarkWorkspace(data.workspace)
-					? Boolean(getBenchmarkScriptByPath(data.workspace, data.path))
-					: actual.ScriptService.existsScriptByPath(data),
 			getScriptByPath: async (data: { workspace: string; path: string }) => {
 				if (hasBenchmarkWorkspace(data.workspace)) {
 					const script = getBenchmarkScriptByPath(data.workspace, data.path)
@@ -109,16 +75,6 @@ vi.mock('$lib/gen', async () => {
 				}
 				return actual.ScriptService.getScriptByPath(data)
 			},
-			getScriptByPathWithDraft: async (data: { workspace: string; path: string }) => {
-				if (hasBenchmarkWorkspace(data.workspace)) {
-					const script = getBenchmarkScriptByPath(data.workspace, data.path)
-					if (!script) {
-						throw new Error(`Script "${data.path}" not found in benchmark workspace`)
-					}
-					return script
-				}
-				return actual.ScriptService.getScriptByPathWithDraft(data)
-			},
 			getScriptByHash: async (data: { workspace: string; hash: string }) => {
 				if (hasBenchmarkWorkspace(data.workspace)) {
 					const script = getBenchmarkScriptByHash(data.workspace, data.hash)
@@ -135,10 +91,6 @@ vi.mock('$lib/gen', async () => {
 				hasBenchmarkWorkspace(data.workspace)
 					? (listBenchmarkFlows(data.workspace) ?? [])
 					: actual.FlowService.listFlows(data),
-			existsFlowByPath: async (data: { workspace: string; path: string }) =>
-				hasBenchmarkWorkspace(data.workspace)
-					? Boolean(getBenchmarkFlowByPath(data.workspace, data.path))
-					: actual.FlowService.existsFlowByPath(data),
 			getFlowByPath: async (data: { workspace: string; path: string }) => {
 				if (hasBenchmarkWorkspace(data.workspace)) {
 					const flow = getBenchmarkFlowByPath(data.workspace, data.path)
@@ -148,26 +100,6 @@ vi.mock('$lib/gen', async () => {
 					return flow
 				}
 				return actual.FlowService.getFlowByPath(data)
-			},
-			getFlowByPathWithDraft: async (data: { workspace: string; path: string }) => {
-				if (hasBenchmarkWorkspace(data.workspace)) {
-					const flow = getBenchmarkFlowByPath(data.workspace, data.path)
-					if (!flow) {
-						throw new Error(`Flow "${data.path}" not found in benchmark workspace`)
-					}
-					return flow
-				}
-				return actual.FlowService.getFlowByPathWithDraft(data)
-			},
-			getFlowLatestVersion: async (data: { workspace: string; path: string }) => {
-				if (hasBenchmarkWorkspace(data.workspace)) {
-					const flow = getBenchmarkFlowByPath(data.workspace, data.path)
-					if (!flow) {
-						throw new Error(`Flow "${data.path}" not found in benchmark workspace`)
-					}
-					return { id: 1 }
-				}
-				return actual.FlowService.getFlowLatestVersion(data)
 			}
 		}),
 		JobService: wrapService(actual.JobService, {
@@ -179,27 +111,13 @@ vi.mock('$lib/gen', async () => {
 					args?: Record<string, unknown>
 					path?: string
 				}
-			}) => {
-				if (!hasBenchmarkWorkspace(data.workspace)) {
-					return actual.JobService.runScriptPreview(data)
-				}
-				const requestBody = data.requestBody ?? {}
-				const database = requestBody.args?.database
-				// Datatable SQL runs as a `postgresql` preview against `datatable://<name>`.
-				// Execute it through the canned-SQL mock instead of linting it as a script.
-				if (
-					requestBody.language === 'postgresql' &&
-					typeof database === 'string' &&
-					database.startsWith('datatable://')
-				) {
-					return runBenchmarkDatatableSql({
-						workspace: data.workspace,
-						datatableName: database.slice('datatable://'.length),
-						sql: requestBody.content ?? ''
-					})
-				}
-				return runBenchmarkScriptPreview({ workspace: data.workspace, requestBody })
-			},
+			}) =>
+				hasBenchmarkWorkspace(data.workspace)
+					? runBenchmarkScriptPreview({
+							workspace: data.workspace,
+							requestBody: data.requestBody ?? {}
+						})
+					: actual.JobService.runScriptPreview(data),
 			runFlowByPath: async (data: {
 				workspace: string
 				path: string
@@ -221,51 +139,9 @@ vi.mock('$lib/gen', async () => {
 					return job
 				}
 				return actual.JobService.getJob(data)
-			},
-			getCompletedJobResultMaybe: async (data: { workspace: string; id: string }) =>
-				hasBenchmarkWorkspace(data.workspace)
-					? getBenchmarkCompletedJobResultMaybe({ workspace: data.workspace, id: data.id })
-					: actual.JobService.getCompletedJobResultMaybe(data),
-			listJobs: async (data: { workspace: string }) =>
-				hasBenchmarkWorkspace(data.workspace)
-					? (listBenchmarkJobs(data.workspace) ?? [])
-					: actual.JobService.listJobs(data),
-			getJobLogs: async (data: { workspace: string; id: string }) =>
-				hasBenchmarkWorkspace(data.workspace)
-					? getBenchmarkJobLogs(data.workspace, data.id)
-					: actual.JobService.getJobLogs(data)
-		}),
-		WorkspaceService: wrapService(actual.WorkspaceService, {
-			listDataTableTables: async (data: { workspace: string }) =>
-				hasBenchmarkWorkspace(data.workspace)
-					? (listBenchmarkDatatables(data.workspace) ?? [])
-					: actual.WorkspaceService.listDataTableTables(data),
-			getDataTableTableSchema: async (data: {
-				workspace: string
-				datatableName: string
-				schemaName: string
-				tableName: string
-			}) =>
-				hasBenchmarkWorkspace(data.workspace)
-					? getBenchmarkDatatableSchema({
-							workspace: data.workspace,
-							datatableName: data.datatableName,
-							schemaName: data.schemaName,
-							tableName: data.tableName
-						})
-					: actual.WorkspaceService.getDataTableTableSchema(data)
+			}
 		}),
 		ScheduleService: wrapService(actual.ScheduleService, {
-			existsSchedule: async (data: { workspace: string; path: string }) =>
-				hasBenchmarkWorkspace(data.workspace) ? false : actual.ScheduleService.existsSchedule(data),
-			listSchedules: async (data: { workspace: string }) =>
-				hasBenchmarkWorkspace(data.workspace) ? [] : actual.ScheduleService.listSchedules(data),
-			getSchedule: async (data: { workspace: string; path: string }) => {
-				if (hasBenchmarkWorkspace(data.workspace)) {
-					throw new Error(`Schedule "${data.path}" not found in benchmark workspace`)
-				}
-				return actual.ScheduleService.getSchedule(data)
-			},
 			previewSchedule: async (data: { requestBody?: Record<string, unknown> }) =>
 				previewBenchmarkSchedule(data),
 			createSchedule: async (data: { workspace: string; requestBody: Record<string, unknown> }) =>
@@ -273,175 +149,11 @@ vi.mock('$lib/gen', async () => {
 					? createBenchmarkSchedule(data)
 					: actual.ScheduleService.createSchedule(data)
 		}),
-		ResourceService: wrapService(actual.ResourceService, {
-			existsResource: async (data: { workspace: string; path: string }) =>
-				hasBenchmarkWorkspace(data.workspace) ? false : actual.ResourceService.existsResource(data),
-			listResource: async (data: { workspace: string }) =>
-				hasBenchmarkWorkspace(data.workspace) ? [] : actual.ResourceService.listResource(data),
-			getResource: async (data: { workspace: string; path: string }) => {
-				if (hasBenchmarkWorkspace(data.workspace)) {
-					throw new Error(`Resource "${data.path}" not found in benchmark workspace`)
-				}
-				return actual.ResourceService.getResource(data)
-			},
-			queryResourceTypes: async (data: { workspace: string }) =>
-				hasBenchmarkWorkspace(data.workspace) ? [] : actual.ResourceService.queryResourceTypes(data)
-		}),
-		VariableService: wrapService(actual.VariableService, {
-			existsVariable: async (data: { workspace: string; path: string }) =>
-				hasBenchmarkWorkspace(data.workspace) ? false : actual.VariableService.existsVariable(data),
-			listVariable: async (data: { workspace: string }) =>
-				hasBenchmarkWorkspace(data.workspace) ? [] : actual.VariableService.listVariable(data),
-			getVariable: async (data: { workspace: string; path: string }) => {
-				if (hasBenchmarkWorkspace(data.workspace)) {
-					throw new Error(`Variable "${data.path}" not found in benchmark workspace`)
-				}
-				return actual.VariableService.getVariable(data)
-			}
-		}),
-		AppService: wrapService(actual.AppService, {
-			existsApp: async (data: { workspace: string; path: string }) =>
-				hasBenchmarkWorkspace(data.workspace)
-					? Boolean(getBenchmarkAppByPath(data.workspace, data.path))
-					: actual.AppService.existsApp(data),
-			listApps: async (data: { workspace: string }) =>
-				hasBenchmarkWorkspace(data.workspace)
-					? (listBenchmarkApps(data.workspace) ?? [])
-					: actual.AppService.listApps(data),
-			getAppByPath: async (data: { workspace: string; path: string }) => {
-				if (hasBenchmarkWorkspace(data.workspace)) {
-					const app = getBenchmarkAppByPath(data.workspace, data.path)
-					if (!app) {
-						throw new Error(`App "${data.path}" not found in benchmark workspace`)
-					}
-					return app
-				}
-				return actual.AppService.getAppByPath(data)
-			}
-		}),
 		HttpTriggerService: wrapService(actual.HttpTriggerService, {
-			existsHttpTrigger: async (data: { workspace: string; path: string }) =>
-				hasBenchmarkWorkspace(data.workspace) ? false : actual.HttpTriggerService.existsHttpTrigger(data),
-			listHttpTriggers: async (data: { workspace: string }) =>
-				hasBenchmarkWorkspace(data.workspace) ? [] : actual.HttpTriggerService.listHttpTriggers(data),
-			getHttpTrigger: async (data: { workspace: string; path: string }) => {
-				if (hasBenchmarkWorkspace(data.workspace)) {
-					throw new Error(`HTTP trigger "${data.path}" not found in benchmark workspace`)
-				}
-				return actual.HttpTriggerService.getHttpTrigger(data)
-			},
 			createHttpTrigger: async (data: { workspace: string; requestBody: Record<string, unknown> }) =>
 				hasBenchmarkWorkspace(data.workspace)
 					? createBenchmarkHttpTrigger(data)
 					: actual.HttpTriggerService.createHttpTrigger(data)
-		}),
-		WebsocketTriggerService: wrapService(actual.WebsocketTriggerService, {
-			existsWebsocketTrigger: async (data: { workspace: string; path: string }) =>
-				hasBenchmarkWorkspace(data.workspace)
-					? false
-					: actual.WebsocketTriggerService.existsWebsocketTrigger(data),
-			listWebsocketTriggers: async (data: { workspace: string }) =>
-				hasBenchmarkWorkspace(data.workspace)
-					? []
-					: actual.WebsocketTriggerService.listWebsocketTriggers(data),
-			getWebsocketTrigger: async (data: { workspace: string; path: string }) => {
-				if (hasBenchmarkWorkspace(data.workspace)) {
-					throw new Error(`Websocket trigger "${data.path}" not found in benchmark workspace`)
-				}
-				return actual.WebsocketTriggerService.getWebsocketTrigger(data)
-			}
-		}),
-		KafkaTriggerService: wrapService(actual.KafkaTriggerService, {
-			existsKafkaTrigger: async (data: { workspace: string; path: string }) =>
-				hasBenchmarkWorkspace(data.workspace)
-					? false
-					: actual.KafkaTriggerService.existsKafkaTrigger(data),
-			listKafkaTriggers: async (data: { workspace: string }) =>
-				hasBenchmarkWorkspace(data.workspace) ? [] : actual.KafkaTriggerService.listKafkaTriggers(data),
-			getKafkaTrigger: async (data: { workspace: string; path: string }) => {
-				if (hasBenchmarkWorkspace(data.workspace)) {
-					throw new Error(`Kafka trigger "${data.path}" not found in benchmark workspace`)
-				}
-				return actual.KafkaTriggerService.getKafkaTrigger(data)
-			}
-		}),
-		NatsTriggerService: wrapService(actual.NatsTriggerService, {
-			existsNatsTrigger: async (data: { workspace: string; path: string }) =>
-				hasBenchmarkWorkspace(data.workspace) ? false : actual.NatsTriggerService.existsNatsTrigger(data),
-			listNatsTriggers: async (data: { workspace: string }) =>
-				hasBenchmarkWorkspace(data.workspace) ? [] : actual.NatsTriggerService.listNatsTriggers(data),
-			getNatsTrigger: async (data: { workspace: string; path: string }) => {
-				if (hasBenchmarkWorkspace(data.workspace)) {
-					throw new Error(`NATS trigger "${data.path}" not found in benchmark workspace`)
-				}
-				return actual.NatsTriggerService.getNatsTrigger(data)
-			}
-		}),
-		PostgresTriggerService: wrapService(actual.PostgresTriggerService, {
-			existsPostgresTrigger: async (data: { workspace: string; path: string }) =>
-				hasBenchmarkWorkspace(data.workspace)
-					? false
-					: actual.PostgresTriggerService.existsPostgresTrigger(data),
-			listPostgresTriggers: async (data: { workspace: string }) =>
-				hasBenchmarkWorkspace(data.workspace)
-					? []
-					: actual.PostgresTriggerService.listPostgresTriggers(data),
-			getPostgresTrigger: async (data: { workspace: string; path: string }) => {
-				if (hasBenchmarkWorkspace(data.workspace)) {
-					throw new Error(`Postgres trigger "${data.path}" not found in benchmark workspace`)
-				}
-				return actual.PostgresTriggerService.getPostgresTrigger(data)
-			}
-		}),
-		MqttTriggerService: wrapService(actual.MqttTriggerService, {
-			existsMqttTrigger: async (data: { workspace: string; path: string }) =>
-				hasBenchmarkWorkspace(data.workspace) ? false : actual.MqttTriggerService.existsMqttTrigger(data),
-			listMqttTriggers: async (data: { workspace: string }) =>
-				hasBenchmarkWorkspace(data.workspace) ? [] : actual.MqttTriggerService.listMqttTriggers(data),
-			getMqttTrigger: async (data: { workspace: string; path: string }) => {
-				if (hasBenchmarkWorkspace(data.workspace)) {
-					throw new Error(`MQTT trigger "${data.path}" not found in benchmark workspace`)
-				}
-				return actual.MqttTriggerService.getMqttTrigger(data)
-			}
-		}),
-		SqsTriggerService: wrapService(actual.SqsTriggerService, {
-			existsSqsTrigger: async (data: { workspace: string; path: string }) =>
-				hasBenchmarkWorkspace(data.workspace) ? false : actual.SqsTriggerService.existsSqsTrigger(data),
-			listSqsTriggers: async (data: { workspace: string }) =>
-				hasBenchmarkWorkspace(data.workspace) ? [] : actual.SqsTriggerService.listSqsTriggers(data),
-			getSqsTrigger: async (data: { workspace: string; path: string }) => {
-				if (hasBenchmarkWorkspace(data.workspace)) {
-					throw new Error(`SQS trigger "${data.path}" not found in benchmark workspace`)
-				}
-				return actual.SqsTriggerService.getSqsTrigger(data)
-			}
-		}),
-		GcpTriggerService: wrapService(actual.GcpTriggerService, {
-			existsGcpTrigger: async (data: { workspace: string; path: string }) =>
-				hasBenchmarkWorkspace(data.workspace) ? false : actual.GcpTriggerService.existsGcpTrigger(data),
-			listGcpTriggers: async (data: { workspace: string }) =>
-				hasBenchmarkWorkspace(data.workspace) ? [] : actual.GcpTriggerService.listGcpTriggers(data),
-			getGcpTrigger: async (data: { workspace: string; path: string }) => {
-				if (hasBenchmarkWorkspace(data.workspace)) {
-					throw new Error(`GCP trigger "${data.path}" not found in benchmark workspace`)
-				}
-				return actual.GcpTriggerService.getGcpTrigger(data)
-			}
-		}),
-		AzureTriggerService: wrapService(actual.AzureTriggerService, {
-			existsAzureTrigger: async (data: { workspace: string; path: string }) =>
-				hasBenchmarkWorkspace(data.workspace)
-					? false
-					: actual.AzureTriggerService.existsAzureTrigger(data),
-			listAzureTriggers: async (data: { workspace: string }) =>
-				hasBenchmarkWorkspace(data.workspace) ? [] : actual.AzureTriggerService.listAzureTriggers(data),
-			getAzureTrigger: async (data: { workspace: string; path: string }) => {
-				if (hasBenchmarkWorkspace(data.workspace)) {
-					throw new Error(`Azure trigger "${data.path}" not found in benchmark workspace`)
-				}
-				return actual.AzureTriggerService.getAzureTrigger(data)
-			}
 		})
 	}
 })
@@ -466,6 +178,5 @@ benchmarkIt(
 			resetBenchmarkMockBackend()
 		}
 	},
-	// Full-suite runs (30+ cases at concurrency 2-3) routinely exceed 10 minutes.
-	7_200_000
+	600_000
 )

ai_evals/adapters/frontend/windmillBackend.test.ts

@@ -1,104 +0,0 @@
-import { afterEach, describe, expect, it } from "bun:test";
-import type { WindmillBackendSettings } from "../../core/windmillBackendSettings";
-import {
-  WindmillBackendClient,
-  assertWindmillBackendReachable,
-} from "./windmillBackend";
-
-const ORIGINAL_FETCH = globalThis.fetch;
-
-afterEach(() => {
-  globalThis.fetch = ORIGINAL_FETCH;
-});
-
-describe("assertWindmillBackendReachable", () => {
-  it("logs in to verify backend reachability", async () => {
-    const requests: Array<{ url: string; init?: RequestInit }> = [];
-    globalThis.fetch = mockFetch(requests, textResponse(200, "token"));
-
-    await expect(
-      assertWindmillBackendReachable(
-        buildSettings({ baseUrl: "http://backend.test/reachable" }),
-      ),
-    ).resolves.toBeUndefined();
-
-    expect(requests.map((entry) => entry.url)).toEqual([
-      "http://backend.test/reachable/api/auth/login",
-    ]);
-  });
-
-  it("adds setup guidance when the backend cannot be initialized", async () => {
-    globalThis.fetch = mockFetch(
-      [],
-      textResponse(401, "invalid password"),
-    );
-
-    await expect(
-      assertWindmillBackendReachable(
-        buildSettings({ baseUrl: "http://backend.test/auth-failure" }),
-      ),
-    ).rejects.toThrow(
-      "Start a Windmill backend at that URL, or set WMILL_AI_EVAL_BACKEND_URL=<url>.",
-    );
-  });
-});
-
-describe("WindmillBackendClient", () => {
-  it("creates or reuses the specified backend workspace without deleting it", async () => {
-    const requests: Array<{ url: string; init?: RequestInit }> = [];
-    globalThis.fetch = mockFetch(
-      requests,
-      textResponse(200, "token"),
-      textResponse(200, "false"),
-      textResponse(200, ""),
-    );
-
-    const client = new WindmillBackendClient(
-      buildSettings({
-        baseUrl: "http://backend.test/shared-workspace",
-        workspaceOverride: "shared-evals",
-      }),
-    );
-
-    await expect(
-      client.withWorkspace("case-a", 1, async (workspaceId) => workspaceId),
-    ).resolves.toBe("shared-evals");
-
-    expect(requests.map((entry) => entry.url)).toEqual([
-      "http://backend.test/shared-workspace/api/auth/login",
-      "http://backend.test/shared-workspace/api/workspaces/exists",
-      "http://backend.test/shared-workspace/api/workspaces/create",
-    ]);
-  });
-});
-
-function buildSettings(
-  overrides: Partial<WindmillBackendSettings> = {},
-): WindmillBackendSettings {
-  return {
-    baseUrl: "http://backend.test/default",
-    email: "admin@windmill.dev",
-    password: "changeme",
-    ...overrides,
-  };
-}
-
-function mockFetch(
-  requests: Array<{ url: string; init?: RequestInit }>,
-  ...responses: Response[]
-): typeof fetch {
-  const queue = [...responses];
-  return async (input, init) => {
-    const url = String(input);
-    requests.push({ url, init });
-    const next = queue.shift();
-    if (!next) {
-      throw new Error(`Unexpected fetch: ${url}`);
-    }
-    return next;
-  };
-}
-
-function textResponse(status: number, body: string): Response {
-  return new Response(body, { status });
-}

ai_evals/adapters/frontend/windmillBackend.ts

@@ -3,7 +3,6 @@ import type { WindmillBackendSettings } from "../../core/windmillBackendSettings
 
 const tokenCache = new Map<string, Promise<string>>();
 const sharedWorkspaceQueue = new Map<string, Promise<void>>();
-const DEFAULT_WORKSPACE_PREFIX = "ai-evals";
 
 export class WindmillBackendClient {
   constructor(private readonly settings: WindmillBackendSettings) {}
@@ -15,7 +14,7 @@ export class WindmillBackendClient {
   ): Promise<T> {
     const workspaceId =
       this.settings.workspaceOverride ??
-      buildWorkspaceId(caseId, attempt);
+      buildWorkspaceId(this.settings.workspacePrefix, caseId, attempt);
 
     const run = async () => {
       await this.ensureWorkspace(workspaceId);
@@ -23,7 +22,7 @@ export class WindmillBackendClient {
       try {
         return await body(workspaceId);
       } finally {
-        if (!this.settings.workspaceOverride) {
+        if (!this.settings.keepWorkspaces && !this.settings.workspaceOverride) {
           await this.deleteWorkspace(workspaceId).catch(() => undefined);
         }
       }
@@ -137,24 +136,6 @@ export class WindmillBackendClient {
   }
 }
 
-export async function assertWindmillBackendReachable(
-  settings: WindmillBackendSettings,
-): Promise<void> {
-  try {
-    await new WindmillBackendClient(settings).getToken();
-  } catch (error) {
-    const details = error instanceof Error ? error.message : String(error);
-    throw new Error(
-      [
-        `Could not initialize the Windmill backend for AI eval proxy at ${settings.baseUrl}.`,
-        "Start a Windmill backend at that URL, or set WMILL_AI_EVAL_BACKEND_URL=<url>.",
-        `Using login ${settings.email}; if authentication failed, set WMILL_AI_EVAL_BACKEND_EMAIL and WMILL_AI_EVAL_BACKEND_PASSWORD.`,
-        `Details: ${details}`,
-      ].join("\n"),
-    );
-  }
-}
-
 async function withSharedWorkspaceLock<T>(
   workspaceId: string,
   body: () => Promise<T>,
@@ -179,14 +160,18 @@ async function withSharedWorkspaceLock<T>(
   }
 }
 
-function buildWorkspaceId(caseId: string, attempt: number): string {
+function buildWorkspaceId(
+  prefix: string,
+  caseId: string,
+  attempt: number,
+): string {
   const caseSlug = caseId
     .toLowerCase()
     .replace(/[^a-z0-9-]+/g, "-")
     .replace(/^-+|-+$/g, "")
     .slice(0, 30);
   const suffix = randomUUID().slice(0, 8);
-  return `${DEFAULT_WORKSPACE_PREFIX}-${caseSlug || "case"}-a${attempt}-${suffix}`;
+  return `${prefix}-${caseSlug || "case"}-a${attempt}-${suffix}`;
 }
 
 async function expectOk(response: Response, context: string): Promise<void> {

ai_evals/cases/cli.yaml

@@ -129,70 +129,6 @@
     - does not duplicate the greeting logic in a new inline script
     - wires the name input into the reused script
 
-- id: wac-typescript-order-workflow
-  prompt: |-
-    Create a Windmill Workflow-as-Code TypeScript script at `f/evals/order_workflow.ts`.
-    It should take an `orderId` string, load the order in a durable task, checkpoint a processing timestamp with `step`, and return `{ orderId, processedAt, status }`.
-  cliExpect:
-    requiredSkills:
-      - write-workflow-as-code
-    requiredSkillsBeforeFirstMutation:
-      - write-workflow-as-code
-    forbiddenSkills:
-      - write-flow
-      - write-script-bun
-      - write-script-python3
-  judgeChecklist:
-    - creates the requested TypeScript WAC script at f/evals/order_workflow.ts
-    - uses the Workflow-as-Code SDK from windmill-client
-    - wraps the entrypoint with workflow
-    - uses a durable task for loading the order
-    - uses step to checkpoint the processing timestamp
-    - does not create an OpenFlow flow.yaml or flow folder
-
-- id: wac-python-approval-workflow
-  prompt: |-
-    Create a Windmill Workflow-as-Code Python script at `f/evals/approval_workflow.py`.
-    It should take a `request_id` string, prepare an approval summary in a task, create resume URLs inside a durable step, wait for approval, and return the approval result.
-  cliExpect:
-    requiredSkills:
-      - write-workflow-as-code
-    requiredSkillsBeforeFirstMutation:
-      - write-workflow-as-code
-    forbiddenSkills:
-      - write-flow
-      - write-script-bun
-      - write-script-python3
-  judgeChecklist:
-    - creates the requested Python WAC script at f/evals/approval_workflow.py
-    - imports Workflow-as-Code helpers from wmill
-    - decorates an async entrypoint with @workflow
-    - uses @task for the approval summary work
-    - gets resume URLs inside step before waiting for approval
-    - uses wait_for_approval
-    - does not create an OpenFlow flow.yaml or flow folder
-
-- id: wac-not-openflow-disambiguation
-  prompt: |-
-    Create this as Workflow-as-Code, not an OpenFlow YAML flow: a TypeScript script at `f/evals/fanout_workflow.ts`.
-    It should take an array of customer IDs, process each customer with a WAC task, run the independent customer tasks in parallel, and return the collected results.
-  cliExpect:
-    requiredSkills:
-      - write-workflow-as-code
-    requiredSkillsBeforeFirstMutation:
-      - write-workflow-as-code
-    forbiddenSkills:
-      - write-flow
-      - write-script-bun
-      - write-script-python3
-  judgeChecklist:
-    - creates the requested TypeScript script at f/evals/fanout_workflow.ts
-    - treats the request as Workflow-as-Code rather than an OpenFlow flow
-    - uses workflow for the script entrypoint
-    - uses task for each customer processing unit
-    - runs independent customer tasks in parallel
-    - does not create a flow folder or flow.yaml
-
 - id: cli-job-debug-guidance
   prompt: |-
     A Windmill job failed.

ai_evals/cases/flow.yaml

@@ -8,9 +8,6 @@
       args:
         a: 4
         b: 5
-  toolExpect:
-    requiredToolsUsed:
-      - test_run_flow
   judgeChecklist:
     - "the flow takes `a` and `b` as inputs"
     - "the main step is named `sum_numbers`"
@@ -28,9 +25,6 @@
       args:
         a: 2
         b: 3
-  toolExpect:
-    requiredToolsUsed:
-      - test_run_flow
   judgeChecklist:
     - "the flow takes `a` and `b` as inputs"
     - "the main step is named `sum_numbers`"
@@ -48,9 +42,6 @@
       args:
         a: 7
         b: 8
-  toolExpect:
-    requiredToolsUsed:
-      - test_run_flow
   judgeChecklist:
     - "the parent flow takes `a` and `b` as inputs"
     - "the main step is named `call_add_numbers`"
@@ -435,7 +426,6 @@
       - return_schedule_status
   toolExpect:
     requiredToolsUsed:
-      - test_run_flow
       - create_schedule
     toolCallArgs:
       - tool: create_schedule
@@ -463,7 +453,6 @@
       - webhook_response
   toolExpect:
     requiredToolsUsed:
-      - test_run_flow
       - create_trigger
     toolCallArgs:
       - tool: create_trigger

ai_evals/cases/script.yaml

@@ -5,9 +5,6 @@
     Keep it simple and do not add external dependencies.
   initial: ai_evals/fixtures/frontend/script/initial/test1_empty_bun.json
   expected: ai_evals/fixtures/frontend/script/expected/test1_greet_user.json
-  toolExpect:
-    requiredToolsUsed:
-      - test_run_script
   judgeChecklist:
     - uses the existing `name` input
     - returns a plain greeting string
@@ -23,7 +20,6 @@
   expected: ai_evals/fixtures/frontend/script/expected/test1_greet_user.json
   toolExpect:
     requiredToolsUsed:
-      - test_run_script
       - create_schedule
     toolCallArgs:
       - tool: create_schedule
@@ -48,7 +44,6 @@
   expected: ai_evals/fixtures/frontend/script/expected/test1_greet_user.json
   toolExpect:
     requiredToolsUsed:
-      - test_run_script
       - create_trigger
     toolCallArgs:
       - tool: create_trigger

ai_evals/cli/index.ts

@@ -25,12 +25,13 @@ import {
 import { runSuite } from "../core/runSuite";
 import { EVAL_MODES, type EvalMode } from "../core/types";
 import { DEFAULT_JUDGE_MODEL } from "../core/judge";
-// createCliModeRunner is imported lazily in runCliBenchmark so the non-cli modes
-// (global/flow/script/app) don't pull in the wmill CLI toolchain and its JSR deps
-// (e.g. @cliffy/*) just to load this entrypoint.
+import { createCliModeRunner } from "../modes/cli";
 import { runFrontendBenchmarkAdapter } from "../adapters/frontend/runtime";
-import { resolveWindmillBackendSettings } from "../core/windmillBackendSettings";
-import { assertWindmillBackendReachable } from "../adapters/frontend/windmillBackend";
+import {
+  FRONTEND_EVAL_TRANSPORTS,
+  type FrontendEvalTransport,
+  parseFrontendEvalTransport,
+} from "../core/frontendTransport";
 
 async function main() {
   const program = new Command()
@@ -55,7 +56,6 @@ async function main() {
         "  bun run cli -- run flow --record",
         "  bun run cli -- run flow --backend-validation preview",
         "  bun run cli -- run flow flow-test5-simple-modification --runs 3",
-        "  bun run cli -- run global global-test1-script-create",
         "  bun run cli -- run cli bun-hello-script",
         "",
         "Models:",
@@ -73,7 +73,7 @@ async function main() {
   program
     .command("cases")
     .description("List available cases")
-    .argument("[mode]", "cli, flow, script, app, or global", parseOptionalMode)
+    .argument("[mode]", "cli, flow, script, or app", parseOptionalMode)
     .action(async (mode?: EvalMode) => {
       await handleCases(mode);
     });
@@ -81,7 +81,7 @@ async function main() {
   program
     .command("run")
     .description("Run one benchmark mode")
-    .argument("<mode>", "cli, flow, script, app, or global", parseMode)
+    .argument("<mode>", "cli, flow, script, or app", parseMode)
     .argument("[caseIds...]", "specific case ids to run")
     .option(
       "--runs <n>",
@@ -98,12 +98,11 @@ async function main() {
       "--models <names>",
       "comma-separated model aliases to run sequentially",
     )
-    .option("--verbose", "stream assistant output during frontend runs")
-    .option("--skip-judge", "skip LLM judge scoring for this run")
     .option(
-      "--execution-only",
-      "only require the model/proxy/frontend loop to complete",
+      "--transport <mode>",
+      `frontend transport (${FRONTEND_EVAL_TRANSPORTS.join(", ")})`,
     )
+    .option("--verbose", "stream assistant output during frontend runs")
     .option(
       "--record",
       "append a compact summary line to ai_evals/history/<mode>.jsonl",
@@ -121,9 +120,8 @@ async function main() {
           output?: string;
           model?: string;
           models?: string;
+          transport?: string;
           verbose?: boolean;
-          skipJudge?: boolean;
-          executionOnly?: boolean;
           record?: boolean;
           backendValidation?: string;
         },
@@ -135,9 +133,10 @@ async function main() {
           outputPath: options.output,
           model: options.model,
           models: options.models,
+          transport: options.transport
+            ? parseFrontendEvalTransport(options.transport)
+            : undefined,
           verbose: options.verbose ?? false,
-          skipJudge: options.skipJudge ?? false,
-          executionOnly: options.executionOnly ?? false,
           record: options.record ?? false,
           backendValidation: options.backendValidation,
         });
@@ -164,7 +163,7 @@ function handleModels() {
   process.stdout.write("Available models\n");
   for (const model of EVAL_MODELS) {
     const supports = [
-      ...(model.frontend ? ["flow", "script", "app", "global"] : []),
+      ...(model.frontend ? ["flow", "script", "app"] : []),
       ...(model.cli ? ["cli"] : []),
     ];
     const aliases = [
@@ -185,9 +184,8 @@ async function handleRun(input: {
   outputPath?: string;
   model?: string;
   models?: string;
+  transport?: FrontendEvalTransport;
   verbose: boolean;
-  skipJudge: boolean;
-  executionOnly: boolean;
   record: boolean;
   backendValidation?: string;
 }) {
@@ -199,6 +197,11 @@ async function handleRun(input: {
   if (input.model && input.models) {
     throw new Error("Use either --model or --models, not both");
   }
+  if (input.mode === "cli" && input.transport === "proxy") {
+    throw new Error(
+      "--transport proxy is only supported for flow, script, and app modes",
+    );
+  }
 
   const selectedCases = await loadSelectedCases(input.mode, input.caseIds);
   const models = resolveRequestedModels(input.mode, input.model, input.models);
@@ -217,14 +220,11 @@ async function handleRun(input: {
       "--backend-validation currently supports only flow and script modes",
     );
   }
-  if (input.mode !== "cli") {
-    await assertWindmillBackendReachable(resolveWindmillBackendSettings());
-  }
 
   const summaries: Array<{
     label: string;
     passRate: number;
-    averagePassedDurationMs: number | null;
+    averageDurationMs: number;
   }> = [];
 
   for (const [index, model] of models.entries()) {
@@ -243,17 +243,14 @@ async function handleRun(input: {
             input.runs,
             getCliEvalModel(model),
             runModel,
-            input.skipJudge,
-            input.executionOnly,
           )
         : await runFrontendBenchmarkAdapter({
             mode: input.mode,
             caseIds: input.caseIds,
             runs: input.runs,
             model: model.id,
+            transport: input.transport,
             verbose: input.verbose,
-            skipJudge: input.skipJudge,
-            executionOnly: input.executionOnly,
             backendValidation,
           });
 
@@ -276,7 +273,7 @@ async function handleRun(input: {
     summaries.push({
       label: `${model.id} (${runModel})`,
       passRate: result.passRate,
-      averagePassedDurationMs: result.averagePassedDurationMs ?? null,
+      averageDurationMs: result.averageDurationMs,
     });
   }
 
@@ -284,7 +281,7 @@ async function handleRun(input: {
     process.stdout.write("\nModel summary\n");
     for (const summary of summaries) {
       process.stdout.write(
-        `- ${summary.label}: ${formatPercent(summary.passRate)} | passed avg ${formatNullableDuration(summary.averagePassedDurationMs)}\n`,
+        `- ${summary.label}: ${formatPercent(summary.passRate)} | ${Math.round(summary.averageDurationMs)}ms\n`,
       );
     }
   }
@@ -295,25 +292,20 @@ async function runCliBenchmark(
   runs: number,
   model: ReturnType<typeof getCliEvalModel>,
   runModel: string,
-  skipJudge: boolean,
-  executionOnly: boolean,
 ) {
-  const { createCliModeRunner } = await import("../modes/cli");
-  const judgeModel = skipJudge || executionOnly ? null : DEFAULT_JUDGE_MODEL;
   const caseResults = await runSuite({
     modeRunner: createCliModeRunner(model),
     cases,
     runs,
     runModel,
-    judgeModel,
-    executionOnly,
+    judgeModel: DEFAULT_JUDGE_MODEL,
   });
 
   return buildRunResult({
     mode: "cli",
     runs,
     runModel,
-    judgeModel,
+    judgeModel: DEFAULT_JUDGE_MODEL,
     caseResults,
   });
 }
@@ -373,10 +365,6 @@ function formatPercent(value: number): string {
   return `${(value * 100).toFixed(1)}%`;
 }
 
-function formatNullableDuration(value: number | null): string {
-  return value === null ? "n/a" : `${Math.round(value)}ms`;
-}
-
 void main().catch((error) => {
   const message = error instanceof Error ? error.message : String(error);
   process.stderr.write(`${message}\n`);

ai_evals/core/backendValidation.ts

@@ -13,7 +13,9 @@ export interface BackendValidationSettings {
   baseUrl: string;
   email: string;
   password: string;
+  keepWorkspaces: boolean;
   workspaceOverride?: string;
+  workspacePrefix: string;
   pollIntervalMs: number;
   maxWaitMs: number;
 }

ai_evals/core/cases.test.ts

@@ -14,21 +14,6 @@ describe("loadCases", () => {
         },
       },
     });
-    expect(caseEntry?.toolExpect).toEqual({
-      requiredToolsUsed: ["test_run_flow"],
-    });
-  });
-
-  it("loads script and flow test tool expectations", async () => {
-    const scriptCases = await loadCases("script");
-    const flowCases = await loadCases("flow");
-
-    expect(scriptCases.find((entry) => entry.id === "script-test1-greet-user")?.toolExpect).toEqual({
-      requiredToolsUsed: ["test_run_script"],
-    });
-    expect(flowCases.find((entry) => entry.id === "flow-test0-sum-two-numbers")?.toolExpect).toEqual({
-      requiredToolsUsed: ["test_run_flow"],
-    });
   });
 
   it("loads the workspace-flow preference benchmark case", async () => {
@@ -198,69 +183,6 @@ describe("loadCases", () => {
     });
   });
 
-  it("loads global draft validation and forbidden tool expectations", async () => {
-    const globalCases = await loadCases("global");
-    const caseEntry = globalCases.find((entry) => entry.id === "global-test1-script-create");
-
-    expect(caseEntry?.validate).toMatchObject({
-      draftCountExactly: 1,
-      requiredDrafts: [
-        {
-          type: "script",
-          path: "f/evals/global/greet_user",
-          language: "bun",
-        },
-      ],
-    });
-    expect(caseEntry?.toolExpect).toMatchObject({
-      requiredToolsUsed: ["write_script"],
-      forbiddenToolsUsed: ["deploy_workspace_item", "delete_workspace_item"],
-    });
-  });
-
-  it("loads global active-editor eval cases", async () => {
-    const globalCases = await loadCases("global");
-    const scriptCase = globalCases.find(
-      (entry) => entry.id === "global-test12-current-live-script-edit"
-    );
-    const flowCase = globalCases.find(
-      (entry) => entry.id === "global-test13-current-live-flow-edit"
-    );
-
-    expect(scriptCase?.initialPath).toContain(
-      "ai_evals/fixtures/frontend/global/initial/current_greeting_live_script.json"
-    );
-    expect(scriptCase?.toolExpect).toMatchObject({
-      requiredToolsUsed: ["read_workspace_item"],
-    });
-    expect(flowCase?.initialPath).toContain(
-      "ai_evals/fixtures/frontend/global/initial/current_invoice_live_flow.json"
-    );
-    expect(flowCase?.validate).toMatchObject({
-      requiredDrafts: [
-        {
-          type: "flow",
-          path: "f/evals/global/current_invoice_flow",
-        },
-      ],
-    });
-  });
-
-  it("loads global docs-search cases as tool-use checks", async () => {
-    const globalCases = await loadCases("global");
-    const docsCases = globalCases.filter((entry) =>
-      entry.id.startsWith("global-docs-"),
-    );
-    expect(docsCases.length).toBeGreaterThanOrEqual(3);
-
-    // Each docs case verifies the assistant reaches for search_docs and does not
-    // draft anything; with no draft, the global judge is skipped.
-    for (const entry of docsCases) {
-      expect(entry.skipJudge).toBe(true);
-      expect(entry.toolExpect?.requiredToolsUsed).toContain("search_docs");
-    }
-  });
-
   it("loads tool expectations for workspace mutation cases", async () => {
     const scriptCases = await loadCases("script");
     const caseEntry = scriptCases.find(
@@ -268,7 +190,7 @@ describe("loadCases", () => {
     );
 
     expect(caseEntry?.toolExpect).toEqual({
-      requiredToolsUsed: ["test_run_script", "create_schedule"],
+      requiredToolsUsed: ["create_schedule"],
       toolCallArgs: [
         {
           tool: "create_schedule",

ai_evals/core/frontendTransport.test.ts

@@ -0,0 +1,64 @@
+import { afterEach, describe, expect, it } from "bun:test";
+import {
+  parseFrontendEvalTransport,
+  resolveFrontendEvalTransportSettings,
+} from "./frontendTransport";
+
+const ORIGINAL_ENV = {
+  WMILL_AI_EVAL_BACKEND_URL: process.env.WMILL_AI_EVAL_BACKEND_URL,
+};
+
+afterEach(() => {
+  if (ORIGINAL_ENV.WMILL_AI_EVAL_BACKEND_URL === undefined) {
+    delete process.env.WMILL_AI_EVAL_BACKEND_URL;
+  } else {
+    process.env.WMILL_AI_EVAL_BACKEND_URL =
+      ORIGINAL_ENV.WMILL_AI_EVAL_BACKEND_URL;
+  }
+});
+
+describe("parseFrontendEvalTransport", () => {
+  it("defaults to direct when unset", () => {
+    expect(parseFrontendEvalTransport(undefined)).toBe("direct");
+  });
+
+  it("accepts proxy explicitly", () => {
+    expect(parseFrontendEvalTransport("proxy")).toBe("proxy");
+  });
+
+  it("rejects unsupported values", () => {
+    expect(() => parseFrontendEvalTransport("worker")).toThrow(
+      "Unsupported frontend eval transport: worker",
+    );
+  });
+});
+
+describe("resolveFrontendEvalTransportSettings", () => {
+  it("includes backend settings for proxy transport", () => {
+    process.env.WMILL_AI_EVAL_BACKEND_URL = "http://127.0.0.1:8000/";
+
+    expect(
+      resolveFrontendEvalTransportSettings({
+        evalMode: "app",
+        requestedTransport: "proxy",
+      }),
+    ).toMatchObject({
+      transport: "proxy",
+      backend: {
+        baseUrl: "http://127.0.0.1:8000",
+      },
+    });
+  });
+
+  it("keeps direct transport for cli runs", () => {
+    expect(
+      resolveFrontendEvalTransportSettings({
+        evalMode: "cli",
+        requestedTransport: "direct",
+      }),
+    ).toEqual({
+      transport: "direct",
+      backend: undefined,
+    });
+  });
+});

ai_evals/core/frontendTransport.ts

@@ -0,0 +1,49 @@
+import type { EvalMode } from "./types";
+import type { WindmillBackendSettings } from "./windmillBackendSettings";
+import { resolveWindmillBackendSettings } from "./windmillBackendSettings";
+
+export const FRONTEND_EVAL_TRANSPORTS = ["direct", "proxy"] as const;
+
+export type FrontendEvalTransport = (typeof FRONTEND_EVAL_TRANSPORTS)[number];
+
+export interface FrontendEvalTransportSettings {
+  transport: FrontendEvalTransport;
+  backend?: WindmillBackendSettings;
+}
+
+export function parseFrontendEvalTransport(
+  value?: string | null,
+): FrontendEvalTransport {
+  const normalized = value?.trim().toLowerCase();
+
+  if (!normalized || normalized === "direct") {
+    return "direct";
+  }
+
+  if (normalized === "proxy") {
+    return "proxy";
+  }
+
+  throw new Error(
+    `Unsupported frontend eval transport: ${value}. Use one of: ${FRONTEND_EVAL_TRANSPORTS.join(", ")}`,
+  );
+}
+
+export function resolveFrontendEvalTransportSettings(input: {
+  evalMode: EvalMode;
+  requestedTransport?: string | null;
+}): FrontendEvalTransportSettings {
+  const transport = parseFrontendEvalTransport(input.requestedTransport);
+
+  if (transport === "proxy" && input.evalMode === "cli") {
+    throw new Error(
+      'Frontend eval transport "proxy" is only supported for flow, script, and app evals',
+    );
+  }
+
+  return {
+    transport,
+    backend:
+      transport === "proxy" ? resolveWindmillBackendSettings() : undefined,
+  };
+}

ai_evals/core/models.test.ts

@@ -2,50 +2,28 @@ import { describe, expect, it } from "bun:test";
 import { resolveEvalModel } from "./models";
 
 describe("resolveEvalModel", () => {
-  it("supports GPT-5.5 aliases for frontend evals", () => {
-    expect(resolveEvalModel("flow", "gpt-5.5").frontend).toEqual({
-      provider: "openai",
-      model: "gpt-5.5",
-    });
-    expect(resolveEvalModel("app", "gpt-55").frontend).toEqual({
-      provider: "openai",
-      model: "gpt-5.5",
-    });
-    expect(resolveEvalModel("script", "5.5").frontend).toEqual({
-      provider: "openai",
-      model: "gpt-5.5",
-    });
-  });
-
   it("supports Gemini aliases for frontend evals", () => {
-    expect(
-      resolveEvalModel("script", "gemini-3-flash-preview").frontend,
-    ).toEqual({
+    expect(resolveEvalModel("flow", "gemini").frontend).toEqual({
+      provider: "googleai",
+      model: "gemini-2.5-flash",
+    });
+    expect(resolveEvalModel("app", "gemini-pro").frontend).toEqual({
+      provider: "googleai",
+      model: "gemini-2.5-pro",
+    });
+    expect(resolveEvalModel("script", "gemini-3-flash-preview").frontend).toEqual({
       provider: "googleai",
       model: "gemini-3-flash-preview",
     });
-    expect(resolveEvalModel("flow", "gemini-3.1-pro-preview").frontend).toEqual(
-      {
-        provider: "googleai",
-        model: "gemini-3.1-pro-preview",
-      },
-    );
-  });
-
-  it("supports DeepSeek aliases for frontend evals", () => {
-    expect(resolveEvalModel("flow", "deepseek").frontend).toEqual({
-      provider: "deepseek",
-      model: "deepseek-v4-flash",
-    });
-    expect(resolveEvalModel("script", "deepseek-v4-pro").frontend).toEqual({
-      provider: "deepseek",
-      model: "deepseek-v4-pro",
+    expect(resolveEvalModel("flow", "gemini-3.1-pro-preview").frontend).toEqual({
+      provider: "googleai",
+      model: "gemini-3.1-pro-preview",
     });
   });
 
   it("rejects Gemini aliases for cli evals", () => {
-    expect(() => resolveEvalModel("cli", "gemini-3-flash-preview")).toThrow(
-      "Model gemini-3-flash-preview is not supported for cli mode",
+    expect(() => resolveEvalModel("cli", "gemini")).toThrow(
+      "Model gemini-flash is not supported for cli mode"
     );
   });
 });

ai_evals/core/models.ts

@@ -1,7 +1,7 @@
 import type { EvalMode } from "./types";
 
 export interface FrontendEvalModelConfig {
-  provider: "anthropic" | "openai" | "googleai" | "deepseek";
+  provider: "anthropic" | "openai" | "googleai";
   model: string;
 }
 
@@ -88,12 +88,21 @@ export const EVAL_MODELS: EvalModelSpec[] = [
     },
   },
   {
-    id: "gpt-5.5",
-    label: "GPT-5.5",
-    aliases: ["gpt-5.5", "gpt-55", "5.5"],
+    id: "gemini-flash",
+    label: "Gemini 2.5 Flash",
+    aliases: ["gemini", "gemini-flash", "gemini-2.5-flash"],
     frontend: {
-      provider: "openai",
-      model: "gpt-5.5",
+      provider: "googleai",
+      model: "gemini-2.5-flash",
+    },
+  },
+  {
+    id: "gemini-pro",
+    label: "Gemini 2.5 Pro",
+    aliases: ["gemini-pro", "gemini-2.5-pro"],
+    frontend: {
+      provider: "googleai",
+      model: "gemini-2.5-pro",
     },
   },
   {
@@ -108,40 +117,15 @@ export const EVAL_MODELS: EvalModelSpec[] = [
   {
     id: "gemini-3.1-pro-preview",
     label: "Gemini 3.1 Pro Preview",
-    aliases: [
-      "gemini-3.1-pro-preview",
-      "gemini-3.1-pro",
-      "gemini-3-pro-preview",
-    ],
+    aliases: ["gemini-3.1-pro-preview", "gemini-3.1-pro", "gemini-3-pro-preview"],
     frontend: {
       provider: "googleai",
       model: "gemini-3.1-pro-preview",
     },
   },
-  {
-    id: "deepseek-v4-flash",
-    label: "DeepSeek V4 Flash",
-    aliases: ["deepseek", "deepseek-v4", "deepseek-v4-flash"],
-    frontend: {
-      provider: "deepseek",
-      model: "deepseek-v4-flash",
-    },
-  },
-  {
-    id: "deepseek-v4-pro",
-    label: "DeepSeek V4 Pro",
-    aliases: ["deepseek-pro", "deepseek-v4-pro"],
-    frontend: {
-      provider: "deepseek",
-      model: "deepseek-v4-pro",
-    },
-  },
 ];
 
-export function resolveEvalModel(
-  mode: EvalMode,
-  alias?: string,
-): EvalModelSpec {
+export function resolveEvalModel(mode: EvalMode, alias?: string): EvalModelSpec {
   const spec = alias ? findEvalModel(alias) : getDefaultEvalModel(mode);
   if (!spec) {
     throw new Error(`Unknown model: ${alias}`);
@@ -161,26 +145,21 @@ export function resolveEvalModel(
 export function getEvalModelHelpText(): string {
   return EVAL_MODELS.map((model) => {
     const modes = [
-      ...(model.frontend ? ["flow", "script", "app", "global"] : []),
+      ...(model.frontend ? ["flow", "script", "app"] : []),
       ...(model.cli ? ["cli"] : []),
     ];
     return `  ${model.id.padEnd(8)} ${model.label} (${modes.join(", ")})`;
   }).join("\n");
 }
 
-export function formatRunModelLabel(
-  mode: EvalMode,
-  model: EvalModelSpec,
-): string {
+export function formatRunModelLabel(mode: EvalMode, model: EvalModelSpec): string {
   if (mode === "cli") {
     return `${model.cli!.provider}:${model.cli!.model}`;
   }
   return `${model.frontend!.provider}:${model.frontend!.model}`;
 }
 
-export function getFrontendEvalModel(
-  model: EvalModelSpec,
-): FrontendEvalModelConfig {
+export function getFrontendEvalModel(model: EvalModelSpec): FrontendEvalModelConfig {
   if (!model.frontend) {
     throw new Error(`Model ${model.id} does not support frontend evals`);
   }
@@ -201,8 +180,6 @@ function getDefaultEvalModel(mode: EvalMode): EvalModelSpec {
 function findEvalModel(alias: string): EvalModelSpec | undefined {
   const normalized = alias.trim().toLowerCase();
   return EVAL_MODELS.find((model) =>
-    [model.id, ...model.aliases].some(
-      (candidate) => candidate.toLowerCase() === normalized,
-    ),
+    [model.id, ...model.aliases].some((candidate) => candidate.toLowerCase() === normalized)
   );
 }

ai_evals/core/results.test.ts

@@ -1,307 +0,0 @@
-import { mkdtemp, readFile, rm } from "node:fs/promises";
-import { join } from "node:path";
-import { tmpdir } from "node:os";
-import { describe, expect, it } from "bun:test";
-import {
-  appendHistoryRecord,
-  buildRunResult,
-  formatRunSummary,
-} from "./results";
-import type { BenchmarkCaseResult } from "./types";
-
-function caseResult(
-  attempts: BenchmarkCaseResult["attempts"],
-): BenchmarkCaseResult {
-  return {
-    id: "case-1",
-    prompt: "Do the thing",
-    attempts,
-  };
-}
-
-describe("benchmark results", () => {
-  it("keeps success cost metrics separate from failed attempts", () => {
-    const result = buildRunResult({
-      mode: "global",
-      runs: 1,
-      runModel: "model-under-test",
-      judgeModel: "judge-model",
-      caseResults: [
-        caseResult([
-          {
-            attempt: 1,
-            passed: true,
-            durationMs: 1000,
-            assistantMessageCount: 1,
-            toolCallCount: 1,
-            toolsUsed: ["edit_script"],
-            skillsInvoked: [],
-            checks: [{ name: "edited", passed: true }],
-            judgeScore: 100,
-            judgeSummary: "ok",
-            error: null,
-            tokenUsage: { prompt: 100, completion: 20, total: 120 },
-          },
-          {
-            attempt: 2,
-            passed: false,
-            durationMs: 100,
-            assistantMessageCount: 1,
-            toolCallCount: 0,
-            toolsUsed: [],
-            skillsInvoked: [],
-            checks: [{ name: "edited", passed: false }],
-            judgeScore: 10,
-            judgeSummary: "missed",
-            error: "failed",
-            tokenUsage: { prompt: 10, completion: 5, total: 15 },
-          },
-        ]),
-      ],
-    });
-
-    expect(result.attemptCount).toBe(2);
-    expect(result.passedAttempts).toBe(1);
-    expect(result.passRate).toBe(0.5);
-    expect(result.averageDurationMs).toBe(550);
-    expect(result.averagePassedDurationMs).toBe(1000);
-    expect(result.totalTokenUsage).toEqual({
-      prompt: 110,
-      completion: 25,
-      total: 135,
-    });
-    expect(result.totalPassedTokenUsage).toEqual({
-      prompt: 100,
-      completion: 20,
-      total: 120,
-    });
-    expect(result.averageTokenUsagePerAttempt).toEqual({
-      prompt: 55,
-      completion: 12.5,
-      total: 67.5,
-    });
-    expect(result.averageTokenUsagePerPassedAttempt).toEqual({
-      prompt: 100,
-      completion: 20,
-      total: 120,
-    });
-
-    const summary = formatRunSummary(result);
-    expect(summary).toContain("Average duration (passed): 1000ms");
-    expect(summary).toContain("Average tokens (passed): 120 total");
-    expect(summary).toContain("Average duration (all attempts): 550ms");
-  });
-
-  it("aggregates final context size over passed attempts only", () => {
-    const result = buildRunResult({
-      mode: "global",
-      runs: 1,
-      runModel: "model-under-test",
-      judgeModel: "judge-model",
-      caseResults: [
-        caseResult([
-          {
-            attempt: 1,
-            passed: true,
-            durationMs: 1000,
-            assistantMessageCount: 1,
-            toolCallCount: 1,
-            toolsUsed: ["edit_script"],
-            skillsInvoked: [],
-            checks: [{ name: "edited", passed: true }],
-            judgeScore: 100,
-            judgeSummary: "ok",
-            error: null,
-            tokenUsage: { prompt: 12000, completion: 200, total: 12200 },
-            finalContextTokens: 5000,
-          },
-          {
-            attempt: 2,
-            passed: true,
-            durationMs: 1100,
-            assistantMessageCount: 1,
-            toolCallCount: 1,
-            toolsUsed: ["edit_script"],
-            skillsInvoked: [],
-            checks: [{ name: "edited", passed: true }],
-            judgeScore: 100,
-            judgeSummary: "ok",
-            error: null,
-            tokenUsage: { prompt: 18000, completion: 300, total: 18300 },
-            finalContextTokens: 7000,
-          },
-          {
-            attempt: 3,
-            passed: false,
-            durationMs: 100,
-            assistantMessageCount: 1,
-            toolCallCount: 0,
-            toolsUsed: [],
-            skillsInvoked: [],
-            checks: [{ name: "edited", passed: false }],
-            judgeScore: 10,
-            judgeSummary: "missed",
-            error: "failed",
-            tokenUsage: { prompt: 20000, completion: 100, total: 20100 },
-            finalContextTokens: 9000,
-          },
-        ]),
-      ],
-    });
-
-    // Final context size stays below cumulative prompt and ignores the failed attempt.
-    expect(result.averageFinalContextTokensPassed).toBe(6000);
-    expect(result.maxFinalContextTokensPassed).toBe(7000);
-    expect(formatRunSummary(result)).toContain(
-      "Final context size (passed): 6000 tokens (max 7000)",
-    );
-  });
-
-  it("reports passed averages as unavailable when no attempt passes", () => {
-    const result = buildRunResult({
-      mode: "global",
-      runs: 1,
-      runModel: "model-under-test",
-      judgeModel: "judge-model",
-      caseResults: [
-        caseResult([
-          {
-            attempt: 1,
-            passed: false,
-            durationMs: 100,
-            assistantMessageCount: 1,
-            toolCallCount: 0,
-            toolsUsed: [],
-            skillsInvoked: [],
-            checks: [{ name: "edited", passed: false }],
-            judgeScore: 10,
-            judgeSummary: "missed",
-            error: "failed",
-            tokenUsage: { prompt: 10, completion: 5, total: 15 },
-          },
-        ]),
-      ],
-    });
-
-    expect(result.averagePassedDurationMs).toBeNull();
-    expect(result.totalPassedTokenUsage).toBeNull();
-    expect(result.averageTokenUsagePerPassedAttempt).toBeNull();
-    expect(formatRunSummary(result)).toContain(
-      "Average duration (passed): n/a",
-    );
-  });
-
-  it("normalizes passed token averages by passed attempts", () => {
-    const result = buildRunResult({
-      mode: "global",
-      runs: 1,
-      runModel: "model-under-test",
-      judgeModel: "judge-model",
-      caseResults: [
-        caseResult([
-          {
-            attempt: 1,
-            passed: true,
-            durationMs: 1000,
-            assistantMessageCount: 1,
-            toolCallCount: 1,
-            toolsUsed: ["edit_script"],
-            skillsInvoked: [],
-            checks: [{ name: "edited", passed: true }],
-            judgeScore: 100,
-            judgeSummary: "ok",
-            error: null,
-            tokenUsage: { prompt: 100, completion: 20, total: 120 },
-          },
-          {
-            attempt: 2,
-            passed: true,
-            durationMs: 1200,
-            assistantMessageCount: 1,
-            toolCallCount: 1,
-            toolsUsed: ["edit_script"],
-            skillsInvoked: [],
-            checks: [{ name: "edited", passed: true }],
-            judgeScore: 100,
-            judgeSummary: "ok",
-            error: null,
-            tokenUsage: null,
-          },
-        ]),
-      ],
-    });
-
-    expect(result.passedAttempts).toBe(2);
-    expect(result.totalPassedTokenUsage).toEqual({
-      prompt: 100,
-      completion: 20,
-      total: 120,
-    });
-    expect(result.averageTokenUsagePerPassedAttempt).toEqual({
-      prompt: 50,
-      completion: 10,
-      total: 60,
-    });
-  });
-
-  it("records passed-attempt metrics in history", async () => {
-    const tempDir = await mkdtemp(join(tmpdir(), "windmill-ai-evals-"));
-    try {
-      const historyPath = join(tempDir, "history.jsonl");
-      const result = buildRunResult({
-        mode: "global",
-        runs: 1,
-        runModel: "model-under-test",
-        judgeModel: "judge-model",
-        caseResults: [
-          caseResult([
-            {
-              attempt: 1,
-              passed: true,
-              durationMs: 1000,
-              assistantMessageCount: 1,
-              toolCallCount: 1,
-              toolsUsed: ["edit_script"],
-              skillsInvoked: [],
-              checks: [{ name: "edited", passed: true }],
-              judgeScore: 100,
-              judgeSummary: "ok",
-              error: null,
-              tokenUsage: { prompt: 100, completion: 20, total: 120 },
-            },
-            {
-              attempt: 2,
-              passed: false,
-              durationMs: 100,
-              assistantMessageCount: 1,
-              toolCallCount: 0,
-              toolsUsed: [],
-              skillsInvoked: [],
-              checks: [{ name: "edited", passed: false }],
-              judgeScore: 10,
-              judgeSummary: "missed",
-              error: "failed",
-              tokenUsage: { prompt: 10, completion: 5, total: 15 },
-            },
-          ]),
-        ],
-      });
-
-      await appendHistoryRecord(result, historyPath);
-      const record = JSON.parse(await readFile(historyPath, "utf8"));
-
-      expect(record.averageDurationMs).toBe(550);
-      expect(record.averagePassedDurationMs).toBe(1000);
-      expect(record.averageTokenUsagePerAttempt.total).toBe(67.5);
-      expect(record.averageTokenUsagePerPassedAttempt.total).toBe(120);
-      expect(record.cases[0].averageDurationMs).toBe(550);
-      expect(record.cases[0].averagePassedDurationMs).toBe(1000);
-      expect(record.cases[0].averageTokenUsagePerAttempt.total).toBe(67.5);
-      expect(record.cases[0].averageTokenUsagePerPassedAttempt.total).toBe(
-        120,
-      );
-    } finally {
-      await rm(tempDir, { recursive: true, force: true });
-    }
-  });
-});

ai_evals/core/results.ts

@@ -4,23 +4,12 @@ import { execFileSync } from "node:child_process";
 import { getAiEvalsRoot, getRepoRoot } from "./cases";
 import type {
   BenchmarkArtifactFile,
-  BenchmarkAttemptResult,
   BenchmarkCaseResult,
   BenchmarkRunResult,
   BenchmarkTokenUsage,
   EvalMode,
 } from "./types";
 
-type AttemptAggregate = {
-  attemptCount: number;
-  durationTotal: number;
-  tokenUsageAttemptCount: number;
-  tokenUsageTotal: BenchmarkTokenUsage | null;
-  finalContextAttemptCount: number;
-  finalContextTotal: number;
-  finalContextMax: number | null;
-};
-
 export async function writeRunResult(
   result: BenchmarkRunResult,
   outputPath?: string,
@@ -85,15 +74,40 @@ export function buildRunResult(input: {
   mode: EvalMode;
   runs: number;
   runModel: string | null;
+  transport?: BenchmarkRunResult["transport"];
   judgeModel: string | null;
   caseResults: BenchmarkCaseResult[];
 }): BenchmarkRunResult {
-  const attempts = input.caseResults.flatMap((entry) => entry.attempts);
-  const passedAttemptResults = attempts.filter((attempt) => attempt.passed);
-  const attemptAggregate = aggregateAttempts(attempts);
-  const passedAttemptAggregate = aggregateAttempts(passedAttemptResults);
-  const attemptCount = attemptAggregate.attemptCount;
-  const passedAttempts = passedAttemptAggregate.attemptCount;
+  const attemptCount = input.caseResults.reduce(
+    (sum, entry) => sum + entry.attempts.length,
+    0,
+  );
+  const passedAttempts = input.caseResults.reduce(
+    (sum, entry) =>
+      sum + entry.attempts.filter((attempt) => attempt.passed).length,
+    0,
+  );
+  const durationTotal = input.caseResults.reduce(
+    (sum, entry) =>
+      sum +
+      entry.attempts.reduce((inner, attempt) => inner + attempt.durationMs, 0),
+    0,
+  );
+  const tokenUsageTotal = input.caseResults.reduce<BenchmarkTokenUsage | null>(
+    (sum, entry) => {
+      for (const attempt of entry.attempts) {
+        if (!attempt.tokenUsage) {
+          continue;
+        }
+        sum ??= { prompt: 0, completion: 0, total: 0 };
+        sum.prompt += attempt.tokenUsage.prompt;
+        sum.completion += attempt.tokenUsage.completion;
+        sum.total += attempt.tokenUsage.total;
+      }
+      return sum;
+    },
+    null,
+  );
 
   return {
     version: 1,
@@ -102,26 +116,22 @@ export function buildRunResult(input: {
     gitSha: getGitSha(),
     runs: input.runs,
     runModel: input.runModel,
+    transport: input.transport ?? null,
     judgeModel: input.judgeModel,
     caseCount: input.caseResults.length,
     attemptCount,
     passedAttempts,
     passRate: attemptCount === 0 ? 0 : passedAttempts / attemptCount,
-    averageDurationMs:
-      attemptCount === 0 ? 0 : attemptAggregate.durationTotal / attemptCount,
-    averagePassedDurationMs: averageDuration(passedAttemptAggregate),
-    totalTokenUsage: attemptAggregate.tokenUsageTotal,
-    totalPassedTokenUsage: passedAttemptAggregate.tokenUsageTotal,
+    averageDurationMs: attemptCount === 0 ? 0 : durationTotal / attemptCount,
+    totalTokenUsage: tokenUsageTotal,
     averageTokenUsagePerAttempt:
-      attemptCount === 0
+      attemptCount === 0 || !tokenUsageTotal
         ? null
-        : averageTokenUsage(attemptAggregate, attemptCount),
-    averageTokenUsagePerPassedAttempt: averageTokenUsage(
-      passedAttemptAggregate,
-      passedAttempts,
-    ),
-    averageFinalContextTokensPassed: averageFinalContext(passedAttemptAggregate),
-    maxFinalContextTokensPassed: passedAttemptAggregate.finalContextMax,
+        : {
+            prompt: tokenUsageTotal.prompt / attemptCount,
+            completion: tokenUsageTotal.completion / attemptCount,
+            total: tokenUsageTotal.total / attemptCount,
+          },
     cases: input.caseResults,
   };
 }
@@ -130,28 +140,10 @@ export function formatRunSummary(result: BenchmarkRunResult): string {
   const lines = [
     `${result.mode} benchmark complete`,
     `Pass rate: ${formatPercent(result.passRate)} (${result.passedAttempts}/${result.attemptCount})`,
-    `Average duration (passed): ${formatNullableDuration(result.averagePassedDurationMs ?? null)}`,
+    `Average duration: ${Math.round(result.averageDurationMs)}ms`,
   ];
-
-  if (result.averageTokenUsagePerPassedAttempt) {
-    lines.push(
-      `Average tokens (passed): ${formatTokenUsage(result.averageTokenUsagePerPassedAttempt)}`,
-    );
-  }
-  if (result.averageFinalContextTokensPassed != null) {
-    lines.push(
-      `Final context size (passed): ${Math.round(result.averageFinalContextTokensPassed)} tokens (max ${Math.round(result.maxFinalContextTokensPassed ?? 0)})`,
-    );
-  }
-  if (result.passedAttempts < result.attemptCount) {
-    lines.push(
-      `Average duration (all attempts): ${Math.round(result.averageDurationMs)}ms`,
-    );
-    if (result.averageTokenUsagePerAttempt) {
-      lines.push(
-        `Average tokens (all attempts): ${formatTokenUsage(result.averageTokenUsagePerAttempt)}`,
-      );
-    }
+  if (result.transport) {
+    lines.splice(1, 0, `Transport: ${result.transport}`);
   }
 
   const failures = collectFailures(result);
@@ -185,77 +177,6 @@ function collectFailures(result: BenchmarkRunResult): string[] {
   return failures;
 }
 
-function aggregateAttempts(attempts: BenchmarkAttemptResult[]): AttemptAggregate {
-  const aggregate: AttemptAggregate = {
-    attemptCount: attempts.length,
-    durationTotal: 0,
-    tokenUsageAttemptCount: 0,
-    tokenUsageTotal: null,
-    finalContextAttemptCount: 0,
-    finalContextTotal: 0,
-    finalContextMax: null,
-  };
-
-  for (const attempt of attempts) {
-    aggregate.durationTotal += attempt.durationMs;
-    if (typeof attempt.finalContextTokens === "number") {
-      aggregate.finalContextAttemptCount += 1;
-      aggregate.finalContextTotal += attempt.finalContextTokens;
-      aggregate.finalContextMax = Math.max(
-        aggregate.finalContextMax ?? 0,
-        attempt.finalContextTokens,
-      );
-    }
-    if (!attempt.tokenUsage) {
-      continue;
-    }
-    aggregate.tokenUsageAttemptCount += 1;
-    aggregate.tokenUsageTotal ??= { prompt: 0, completion: 0, total: 0 };
-    aggregate.tokenUsageTotal.prompt += attempt.tokenUsage.prompt;
-    aggregate.tokenUsageTotal.completion += attempt.tokenUsage.completion;
-    aggregate.tokenUsageTotal.total += attempt.tokenUsage.total;
-  }
-
-  return aggregate;
-}
-
-function averageDuration(aggregate: AttemptAggregate): number | null {
-  return aggregate.attemptCount === 0
-    ? null
-    : aggregate.durationTotal / aggregate.attemptCount;
-}
-
-function averageFinalContext(aggregate: AttemptAggregate): number | null {
-  return aggregate.finalContextAttemptCount === 0
-    ? null
-    : aggregate.finalContextTotal / aggregate.finalContextAttemptCount;
-}
-
-function averageTokenUsage(
-  aggregate: AttemptAggregate,
-  denominator: number,
-): BenchmarkTokenUsage | null {
-  if (denominator === 0 || !aggregate.tokenUsageTotal) {
-    return null;
-  }
-  return {
-    prompt: aggregate.tokenUsageTotal.prompt / denominator,
-    completion: aggregate.tokenUsageTotal.completion / denominator,
-    total: aggregate.tokenUsageTotal.total / denominator,
-  };
-}
-
-function formatNullableDuration(value: number | null): string {
-  return value === null ? "n/a" : `${Math.round(value)}ms`;
-}
-
-function formatTokenUsage(value: BenchmarkTokenUsage): string {
-  const total = Math.round(value.total);
-  const prompt = Math.round(value.prompt);
-  const completion = Math.round(value.completion);
-  return `${total} total (${prompt} prompt, ${completion} completion)`;
-}
-
 function defaultFileName(mode: EvalMode): string {
   return `${new Date().toISOString().replaceAll(":", "-")}__${mode}.json`;
 }
@@ -330,24 +251,19 @@ function toHistoryRecord(result: BenchmarkRunResult) {
     mode: result.mode,
     runs: result.runs,
     runModel: result.runModel,
+    transport: result.transport,
     judgeModel: result.judgeModel,
     caseCount: result.caseCount,
     attemptCount: result.attemptCount,
     passedAttempts: result.passedAttempts,
     passRate: result.passRate,
     averageDurationMs: result.averageDurationMs,
-    averagePassedDurationMs: result.averagePassedDurationMs ?? null,
     averageJudgeScore:
       judgeScores.length === 0
         ? null
         : judgeScores.reduce((sum, score) => sum + score, 0) /
           judgeScores.length,
     averageTokenUsagePerAttempt: result.averageTokenUsagePerAttempt ?? null,
-    averageTokenUsagePerPassedAttempt:
-      result.averageTokenUsagePerPassedAttempt ?? null,
-    averageFinalContextTokensPassed:
-      result.averageFinalContextTokensPassed ?? null,
-    maxFinalContextTokensPassed: result.maxFinalContextTokensPassed ?? null,
     failedCaseIds: Array.from(
       new Set(
         result.cases
@@ -358,15 +274,31 @@ function toHistoryRecord(result: BenchmarkRunResult) {
       ),
     ),
     cases: result.cases.map((caseResult) => {
-      const attemptAggregate = aggregateAttempts(caseResult.attempts);
-      const passedAttemptAggregate = aggregateAttempts(
-        caseResult.attempts.filter((attempt) => attempt.passed),
+      const attemptCount = caseResult.attempts.length;
+      const passedAttempts = caseResult.attempts.filter(
+        (attempt) => attempt.passed,
+      ).length;
+      const totalDurationMs = caseResult.attempts.reduce(
+        (sum, attempt) => sum + attempt.durationMs,
+        0,
       );
-      const attemptCount = attemptAggregate.attemptCount;
-      const passedAttempts = passedAttemptAggregate.attemptCount;
       const judgeScores = caseResult.attempts.flatMap((attempt) =>
         typeof attempt.judgeScore === "number" ? [attempt.judgeScore] : [],
       );
+      const totalTokenUsage =
+        caseResult.attempts.reduce<BenchmarkTokenUsage | null>(
+          (sum, attempt) => {
+            if (!attempt.tokenUsage) {
+              return sum;
+            }
+            sum ??= { prompt: 0, completion: 0, total: 0 };
+            sum.prompt += attempt.tokenUsage.prompt;
+            sum.completion += attempt.tokenUsage.completion;
+            sum.total += attempt.tokenUsage.total;
+            return sum;
+          },
+          null,
+        );
 
       return {
         id: caseResult.id,
@@ -374,27 +306,20 @@ function toHistoryRecord(result: BenchmarkRunResult) {
         passedAttempts,
         passRate: attemptCount === 0 ? 0 : passedAttempts / attemptCount,
         averageDurationMs:
-          attemptCount === 0
-            ? 0
-            : attemptAggregate.durationTotal / attemptCount,
-        averagePassedDurationMs: averageDuration(passedAttemptAggregate),
+          attemptCount === 0 ? 0 : totalDurationMs / attemptCount,
         averageJudgeScore:
           judgeScores.length === 0
             ? null
             : judgeScores.reduce((sum, score) => sum + score, 0) /
               judgeScores.length,
         averageTokenUsagePerAttempt:
-          attemptCount === 0
+          attemptCount === 0 || !totalTokenUsage
             ? null
-            : averageTokenUsage(attemptAggregate, attemptCount),
-        averageTokenUsagePerPassedAttempt: averageTokenUsage(
-          passedAttemptAggregate,
-          passedAttempts,
-        ),
-        averageFinalContextTokensPassed: averageFinalContext(
-          passedAttemptAggregate,
-        ),
-        maxFinalContextTokensPassed: passedAttemptAggregate.finalContextMax,
+            : {
+                prompt: totalTokenUsage.prompt / attemptCount,
+                completion: totalTokenUsage.completion / attemptCount,
+                total: totalTokenUsage.total / attemptCount,
+              },
       };
     }),
   };

ai_evals/core/runSuite.test.ts

@@ -1,102 +0,0 @@
-import { describe, expect, it } from "bun:test";
-import { runSuite } from "./runSuite";
-import type { ModeRunner } from "./types";
-
-const modeRunner: ModeRunner<undefined, undefined, { ok: boolean }> = {
-  mode: "global",
-  concurrency: 1,
-  loadInitial: async () => undefined,
-  loadExpected: async () => undefined,
-  run: async () => ({
-    success: true,
-    actual: { ok: true },
-    assistantMessageCount: 1,
-    toolCallCount: 0,
-    toolsUsed: [],
-    skillsInvoked: [],
-    tokenUsage: null,
-  }),
-  validate: () => [],
-};
-
-describe("runSuite", () => {
-  it("skips judge checks when the run disables judge scoring", async () => {
-    const [caseResult] = await runSuite({
-      modeRunner,
-      cases: [
-        {
-          id: "case-1",
-          prompt: "Create a draft script",
-          judgeChecklist: ["the output satisfies the prompt"],
-        },
-      ],
-      runs: 1,
-      runModel: "model-under-test",
-      judgeModel: null,
-    });
-
-    const [attempt] = caseResult.attempts;
-    expect(attempt.passed).toBe(true);
-    expect(attempt.judgeScore).toBeNull();
-    expect(attempt.judgeSummary).toBeNull();
-    expect(attempt.checks.map((check) => check.name)).toEqual([
-      "run succeeded",
-    ]);
-  });
-
-  it("only requires run success when execution-only is enabled", async () => {
-    let loadExpectedCalls = 0;
-    let validateCalls = 0;
-    let backendValidateCalls = 0;
-
-    const executionOnlyRunner: ModeRunner<
-      undefined,
-      undefined,
-      { ok: boolean }
-    > = {
-      ...modeRunner,
-      loadExpected: async () => {
-        loadExpectedCalls++;
-        return undefined;
-      },
-      validate: () => {
-        validateCalls++;
-        return [{ name: "validator failed", passed: false }];
-      },
-      backendValidate: async () => {
-        backendValidateCalls++;
-        return {
-          checks: [{ name: "backend validation failed", passed: false }],
-        };
-      },
-    };
-
-    const [caseResult] = await runSuite({
-      modeRunner: executionOnlyRunner,
-      cases: [
-        {
-          id: "case-1",
-          prompt: "Create a draft script",
-          expectedPath: "fixtures/expected.json",
-          toolExpect: { requiredToolsUsed: ["write_script"] },
-          judgeChecklist: ["the output satisfies the prompt"],
-        },
-      ],
-      runs: 1,
-      runModel: "model-under-test",
-      judgeModel: "judge-model",
-      executionOnly: true,
-    });
-
-    const [attempt] = caseResult.attempts;
-    expect(attempt.passed).toBe(true);
-    expect(attempt.judgeScore).toBeNull();
-    expect(attempt.judgeSummary).toBeNull();
-    expect(attempt.checks.map((check) => check.name)).toEqual([
-      "run succeeded",
-    ]);
-    expect(loadExpectedCalls).toBe(0);
-    expect(validateCalls).toBe(0);
-    expect(backendValidateCalls).toBe(0);
-  });
-});

ai_evals/core/runSuite.ts

@@ -15,13 +15,11 @@ export async function runSuite<TInitial, TExpected, TActual>(input: {
   runs: number;
   runModel: string | null;
   judgeModel?: string | null;
-  executionOnly?: boolean;
   concurrency?: number;
   verbose?: boolean;
   onProgress?: (event: FrontendBenchmarkProgressEvent) => void;
 }): Promise<BenchmarkCaseResult[]> {
-  const judgeModel =
-    input.judgeModel === undefined ? DEFAULT_JUDGE_MODEL : input.judgeModel;
+  const judgeModel = input.judgeModel ?? DEFAULT_JUDGE_MODEL;
   const concurrency = Math.max(1, input.concurrency ?? input.modeRunner.concurrency);
   const results = new Array<BenchmarkCaseResult>(input.cases.length);
   let cursor = 0;
@@ -54,7 +52,6 @@ export async function runSuite<TInitial, TExpected, TActual>(input: {
           runs: input.runs,
           judgeModel,
           judgeThreshold: input.modeRunner.judgeThreshold ?? 80,
-          executionOnly: input.executionOnly ?? false,
           modeRunner: input.modeRunner,
           totalCases: input.cases.length,
           verbose: input.verbose ?? false,
@@ -75,9 +72,8 @@ async function runCaseAttempts<TInitial, TExpected, TActual>(input: {
   caseIndex: number;
   evalCase: EvalCase;
   runs: number;
-  judgeModel: string | null;
+  judgeModel: string;
   judgeThreshold: number;
-  executionOnly: boolean;
   modeRunner: ModeRunner<TInitial, TExpected, TActual>;
   totalCases: number;
   verbose: boolean;
@@ -103,9 +99,7 @@ async function runCaseAttempts<TInitial, TExpected, TActual>(input: {
 
     try {
       const initial = await input.modeRunner.loadInitial(input.evalCase.initialPath);
-      const expected = input.executionOnly
-        ? undefined
-        : await input.modeRunner.loadExpected(input.evalCase.expectedPath);
+      const expected = await input.modeRunner.loadExpected(input.evalCase.expectedPath);
       const run = await input.modeRunner.run(input.evalCase.prompt, initial, {
         evalCase: input.evalCase,
         caseId: input.evalCase.id,
@@ -168,30 +162,22 @@ async function runCaseAttempts<TInitial, TExpected, TActual>(input: {
       });
       const checks: BenchmarkCheck[] = [
         buildCheck("run succeeded", run.success, run.error),
+        ...input.modeRunner.validate({
+          evalCase: input.evalCase,
+          prompt: input.evalCase.prompt,
+          initial,
+          expected,
+          actual: run.actual,
+          run,
+        }),
+        ...validateToolExpectations({
+          run,
+          toolExpect: input.evalCase.toolExpect,
+        }),
       ];
-      if (!input.executionOnly) {
-        checks.push(
-          ...input.modeRunner.validate({
-            evalCase: input.evalCase,
-            prompt: input.evalCase.prompt,
-            initial,
-            expected,
-            actual: run.actual,
-            run,
-          }),
-          ...validateToolExpectations({
-            run,
-            toolExpect: input.evalCase.toolExpect,
-          })
-        );
-      }
       const artifactFiles = input.modeRunner.buildArtifacts?.(run.actual) ?? [];
 
-      if (
-        run.success &&
-        !input.executionOnly &&
-        input.modeRunner.backendValidate
-      ) {
+      if (run.success && input.modeRunner.backendValidate) {
         try {
           const backendValidation = await input.modeRunner.backendValidate({
             evalCase: input.evalCase,
@@ -232,21 +218,14 @@ async function runCaseAttempts<TInitial, TExpected, TActual>(input: {
       let judgeScore: number | null = null;
       let judgeSummary: string | null = null;
 
-      if (
-        run.success &&
-        !input.executionOnly &&
-        input.judgeModel !== null &&
-        !input.evalCase.skipJudge
-      ) {
+      if (run.success && !input.evalCase.skipJudge) {
         const judge = await judgeOutput({
           mode: input.modeRunner.mode,
           prompt: input.evalCase.prompt,
           checklist: input.evalCase.judgeChecklist,
           initial,
           expected: input.modeRunner.mode === "cli" ? undefined : expected,
-          actual: input.modeRunner.prepareJudgeActual
-            ? input.modeRunner.prepareJudgeActual(run.actual)
-            : run.actual,
+          actual: run.actual,
           model: input.judgeModel,
         });
 
@@ -276,7 +255,6 @@ async function runCaseAttempts<TInitial, TExpected, TActual>(input: {
         judgeSummary,
         error: run.error ?? null,
         tokenUsage: run.tokenUsage ?? null,
-        finalContextTokens: run.finalContextTokens ?? null,
         artifactsPath: null,
         artifactFiles,
       };
@@ -313,7 +291,6 @@ async function runCaseAttempts<TInitial, TExpected, TActual>(input: {
         judgeSummary: null,
         error: message,
         tokenUsage: null,
-        finalContextTokens: null,
       };
       if (surface) {
         input.onProgress?.({

ai_evals/core/types.ts

@@ -1,6 +1,7 @@
-export const EVAL_MODES = ["cli", "flow", "script", "app", "global"] as const;
+export const EVAL_MODES = ["cli", "flow", "script", "app"] as const;
 
 export type EvalMode = (typeof EVAL_MODES)[number];
+export type FrontendEvalTransport = "direct" | "proxy";
 
 export interface EvalCaseRuntimeBackendPreview {
   args?: Record<string, unknown>;
@@ -108,29 +109,6 @@ export interface AppValidationSpec {
   forbiddenAppContent?: string[];
 }
 
-export interface GlobalDraftRequirement {
-  type: string;
-  path?: string;
-  pathIncludes?: string[];
-  pathStartsWith?: string;
-  triggerKind?: string;
-  language?: string;
-  summaryIncludes?: string[];
-  valueIncludes?: string[];
-  valueExcludes?: string[];
-}
-
-export interface GlobalValidationSpec {
-  draftCountAtLeast?: number;
-  draftCountExactly?: number;
-  requiredDrafts?: GlobalDraftRequirement[];
-  forbiddenDrafts?: Array<{
-    type: string;
-    path: string;
-    triggerKind?: string;
-  }>;
-}
-
 export interface CliValidationSpec {
   requiredSkills?: string[];
   forbiddenSkills?: string[];
@@ -155,34 +133,14 @@ export interface ToolCallArgumentRule {
   field: string;
   stringStartsWithAnyOf?: string[];
   stringMustNotStartWithAnyOf?: string[];
-  /**
-   * Case-insensitive "contains", existential over calls: at least one recorded
-   * call to `tool` must have `field` containing one of these substrings. Other
-   * calls to the same tool may do anything. Use instead of `stringStartsWithAnyOf`
-   * (which is universal over calls) when the meaningful token can appear anywhere
-   * in the value and the model may make additional, unrelated calls to the same
-   * tool — e.g. SQL where a mutation is mixed with verification SELECTs.
-   */
-  stringIncludesAnyOf?: string[];
 }
 
 export interface ToolValidationSpec {
   requiredToolsUsed?: string[];
-  /**
-   * Each inner array is an alternatives group: the check passes when at least
-   * one tool in the group was used. Use when several tools satisfy the same
-   * intent so a model that picks any valid path passes — e.g. inspecting an
-   * app's files via either `read_app_file` or `search_app`.
-   */
-  requiredToolsAnyOf?: string[][];
-  forbiddenToolsUsed?: string[];
   toolCallArgs?: ToolCallArgumentRule[];
 }
 
-export type EvalValidationSpec =
-  | FlowValidationSpec
-  | AppValidationSpec
-  | GlobalValidationSpec;
+export type EvalValidationSpec = FlowValidationSpec | AppValidationSpec;
 
 export interface EvalCase {
   id: string;
@@ -259,12 +217,6 @@ export interface ModeRunOutput<TActual> {
   toolCallDetails?: ToolCallDetail[];
   skillsInvoked: string[];
   tokenUsage?: BenchmarkTokenUsage | null;
-  /**
-   * Total input tokens occupying the context window on the LAST model request
-   * of the agentic loop (input + cache-creation + cache-read). Complements the
-   * cumulative `tokenUsage.prompt`, which sums every iteration's input.
-   */
-  finalContextTokens?: number | null;
 }
 
 export interface ModeRunContext {
@@ -310,12 +262,6 @@ export interface ModeRunner<TInitial, TExpected, TActual> {
     context: ModeRunContext;
   }): Promise<BackendValidationResult | null>;
   buildArtifacts?(actual: TActual): BenchmarkArtifactFile[];
-  /**
-   * Optional transform applied to `actual` before it is handed to the LLM judge.
-   * Use it to strip fields the judge must stay blind to (e.g. which docs-tool
-   * arm produced an answer). When omitted, the judge receives `actual` as-is.
-   */
-  prepareJudgeActual?(actual: TActual): unknown;
 }
 
 export interface BenchmarkAttemptResult {
@@ -332,7 +278,6 @@ export interface BenchmarkAttemptResult {
   judgeSummary: string | null;
   error: string | null;
   tokenUsage?: BenchmarkTokenUsage | null;
-  finalContextTokens?: number | null;
   artifactsPath?: string | null;
   artifactFiles?: BenchmarkArtifactFile[];
 }
@@ -352,19 +297,15 @@ export interface BenchmarkRunResult {
   gitSha: string | null;
   runs: number;
   runModel: string | null;
+  transport: FrontendEvalTransport | null;
   judgeModel: string | null;
   caseCount: number;
   attemptCount: number;
   passedAttempts: number;
   passRate: number;
   averageDurationMs: number;
-  averagePassedDurationMs?: number | null;
   totalTokenUsage?: BenchmarkTokenUsage | null;
-  totalPassedTokenUsage?: BenchmarkTokenUsage | null;
   averageTokenUsagePerAttempt?: BenchmarkTokenUsage | null;
-  averageTokenUsagePerPassedAttempt?: BenchmarkTokenUsage | null;
-  averageFinalContextTokensPassed?: number | null;
-  maxFinalContextTokensPassed?: number | null;
   artifactsPath?: string | null;
   cases: BenchmarkCaseResult[];
 }

ai_evals/core/validators.test.ts

@@ -2,7 +2,6 @@ import { describe, expect, it } from "bun:test";
 import {
   validateAppState,
   validateCliWorkspace,
-  validateGlobalState,
   validateScriptState,
   validateToolExpectations,
 } from "./validators";
@@ -118,441 +117,6 @@ describe("validateToolExpectations", () => {
       details: 'rejected prefixes: schedules/; values: "schedules/greet_user_daily"',
     });
   });
-
-  it("rejects forbidden tool usage", () => {
-    const checks = validateToolExpectations({
-      run: {
-        success: true,
-        actual: {},
-        assistantMessageCount: 1,
-        toolCallCount: 1,
-        toolsUsed: ["write_script", "deploy_workspace_item"],
-        skillsInvoked: [],
-      },
-      toolExpect: {
-        forbiddenToolsUsed: ["deploy_workspace_item"],
-      },
-    });
-
-    expect(checks).toContainEqual({
-      name: "does not use deploy_workspace_item",
-      passed: false,
-      details: "tools used: write_script, deploy_workspace_item",
-    });
-  });
-
-  it("accepts a stringIncludesAnyOf substring regardless of case or position", () => {
-    const checks = validateToolExpectations({
-      run: {
-        success: true,
-        actual: {},
-        assistantMessageCount: 1,
-        toolCallCount: 1,
-        toolsUsed: ["exec_datatable_sql"],
-        toolCallDetails: [
-          {
-            name: "exec_datatable_sql",
-            arguments: {
-              sql: "WITH recent AS (SELECT * FROM orders) SELECT count(*) FROM recent",
-            },
-          },
-        ],
-        skillsInvoked: [],
-      },
-      toolExpect: {
-        requiredToolsUsed: ["exec_datatable_sql"],
-        toolCallArgs: [
-          {
-            tool: "exec_datatable_sql",
-            field: "sql",
-            stringIncludesAnyOf: ["select"],
-          },
-        ],
-      },
-    });
-
-    expect(checks.every((check) => check.passed)).toBe(true);
-  });
-
-  it("accepts stringIncludesAnyOf when only one of several calls matches", () => {
-    // Existential: a mutation mixed with verification SELECTs still passes.
-    const checks = validateToolExpectations({
-      run: {
-        success: true,
-        actual: {},
-        assistantMessageCount: 1,
-        toolCallCount: 2,
-        toolsUsed: ["exec_datatable_sql"],
-        toolCallDetails: [
-          {
-            name: "exec_datatable_sql",
-            arguments: { sql: "UPDATE orders SET status = 'shipped' WHERE id = 2" },
-          },
-          {
-            name: "exec_datatable_sql",
-            arguments: { sql: "SELECT * FROM orders WHERE id = 2" },
-          },
-        ],
-        skillsInvoked: [],
-      },
-      toolExpect: {
-        toolCallArgs: [
-          {
-            tool: "exec_datatable_sql",
-            field: "sql",
-            stringIncludesAnyOf: ["insert into", "update"],
-          },
-        ],
-      },
-    });
-
-    expect(checks.every((check) => check.passed)).toBe(true);
-  });
-
-  it("rejects stringIncludesAnyOf when no call matches any substring", () => {
-    const checks = validateToolExpectations({
-      run: {
-        success: true,
-        actual: {},
-        assistantMessageCount: 1,
-        toolCallCount: 1,
-        toolsUsed: ["exec_datatable_sql"],
-        toolCallDetails: [
-          {
-            name: "exec_datatable_sql",
-            arguments: {
-              sql: "DROP TABLE orders",
-            },
-          },
-        ],
-        skillsInvoked: [],
-      },
-      toolExpect: {
-        toolCallArgs: [
-          {
-            tool: "exec_datatable_sql",
-            field: "sql",
-            stringIncludesAnyOf: ["insert into", "update"],
-          },
-        ],
-      },
-    });
-
-    expect(checks).toContainEqual({
-      name: "exec_datatable_sql.sql includes a required substring",
-      passed: false,
-      details:
-        'accepted substrings: insert into, update; values: "DROP TABLE orders"',
-    });
-  });
-
-  it("passes requiredToolsAnyOf when any alternative in the group is used", () => {
-    const checks = validateToolExpectations({
-      run: {
-        success: true,
-        actual: {},
-        assistantMessageCount: 1,
-        toolCallCount: 1,
-        toolsUsed: ["search_app", "patch_app_file"],
-        skillsInvoked: [],
-      },
-      toolExpect: {
-        requiredToolsAnyOf: [["read_app_file", "search_app"]],
-      },
-    });
-
-    expect(checks).toContainEqual({
-      name: "uses one of read_app_file, search_app",
-      passed: true,
-    });
-  });
-
-  it("fails requiredToolsAnyOf when no alternative in the group is used", () => {
-    const checks = validateToolExpectations({
-      run: {
-        success: true,
-        actual: {},
-        assistantMessageCount: 1,
-        toolCallCount: 1,
-        toolsUsed: ["patch_app_file"],
-        skillsInvoked: [],
-      },
-      toolExpect: {
-        requiredToolsAnyOf: [["read_app_file", "search_app"]],
-      },
-    });
-
-    expect(checks).toContainEqual({
-      name: "uses one of read_app_file, search_app",
-      passed: false,
-      details: "tools used: patch_app_file",
-    });
-  });
-});
-
-describe("validateGlobalState", () => {
-  it("accepts a required script draft", () => {
-    const checks = validateGlobalState({
-      actual: {
-        drafts: [
-          {
-            type: "script",
-            path: "f/evals/global/greet_user",
-            language: "bun",
-            value:
-              "export async function main(name: string) {\n  return `Hello, ${name}!`\n}\n",
-            isDraft: true,
-          },
-        ],
-      },
-      validate: {
-        draftCountExactly: 1,
-        requiredDrafts: [
-          {
-            type: "script",
-            path: "f/evals/global/greet_user",
-            language: "bun",
-            valueIncludes: ["Hello"],
-          },
-        ],
-      },
-    });
-
-    expect(checks.every((check) => check.passed)).toBe(true);
-  });
-
-  it("fails when a required draft is missing", () => {
-    const checks = validateGlobalState({
-      actual: {
-        drafts: [],
-      },
-      validate: {
-        requiredDrafts: [
-          {
-            type: "script",
-            path: "f/evals/global/greet_user",
-          },
-        ],
-      },
-    });
-
-    expect(checks).toContainEqual({
-      name: "global includes script draft f/evals/global/greet_user",
-      passed: false,
-      details: "drafts: none",
-    });
-  });
-
-  it("accepts a required script draft without an exact path", () => {
-    const checks = validateGlobalState({
-      actual: {
-        drafts: [
-          {
-            type: "script",
-            path: "f/team_tools/friendly_greeting",
-            language: "bun",
-            summary: "Friendly greeting helper",
-            value:
-              "export async function main(name: string) {\n  return `Hello, ${name}!`\n}\n",
-            isDraft: true,
-          },
-        ],
-      },
-      validate: {
-        draftCountExactly: 1,
-        requiredDrafts: [
-          {
-            type: "script",
-            pathIncludes: ["greeting"],
-            language: "bun",
-            summaryIncludes: ["Friendly"],
-            valueIncludes: ["Hello"],
-          },
-        ],
-      },
-    });
-
-    expect(checks.every((check) => check.passed)).toBe(true);
-  });
-
-  it("reports flexible global draft path filters when no draft matches", () => {
-    const checks = validateGlobalState({
-      actual: {
-        drafts: [
-          {
-            type: "script",
-            path: "f/team_tools/friendly_greeting",
-            language: "bun",
-            value:
-              "export async function main(name: string) {\n  return `Hello, ${name}!`\n}\n",
-            isDraft: true,
-          },
-        ],
-      },
-      validate: {
-        requiredDrafts: [
-          {
-            type: "script",
-            pathIncludes: ["invoice"],
-          },
-        ],
-      },
-    });
-
-    expect(checks).toContainEqual({
-      name: "global includes script draft (path includes invoice)",
-      passed: false,
-      details: "drafts: script:f/team_tools/friendly_greeting",
-    });
-  });
-
-  it("does not require a TypeScript entrypoint for non-TypeScript script drafts", () => {
-    const checks = validateGlobalState({
-      actual: {
-        drafts: [
-          {
-            type: "script",
-            path: "f/evals/global/greet_python",
-            language: "python3",
-            value: "def main(name: str):\n    return f'Hello, {name}!'\n",
-            isDraft: true,
-          },
-        ],
-      },
-    });
-
-    expect(checks.some((check) => check.name.includes("exports entrypoint"))).toBe(
-      false
-    );
-    expect(checks.every((check) => check.passed)).toBe(true);
-  });
-
-  it("allows read-only global cases without draft expectations", () => {
-    const checks = validateGlobalState({
-      actual: {
-        drafts: [],
-      },
-    });
-
-    expect(
-      checks.some(
-        (check) => check.name === "global produced at least one draft"
-      )
-    ).toBe(false);
-    expect(checks.every((check) => check.passed)).toBe(true);
-  });
-
-  it("matches expected global draft fixtures", () => {
-    const checks = validateGlobalState({
-      actual: {
-        drafts: [
-          {
-            type: "script",
-            path: "f/evals/global/greet_user",
-            language: "bun",
-            value:
-              "export async function main(name: string) {\r\n  return `Hello, ${name}!`\r\n}\r\n",
-            isDraft: true,
-          },
-        ],
-      },
-      expected: {
-        drafts: [
-          {
-            type: "script",
-            path: "f/evals/global/greet_user",
-            language: "bun",
-            value:
-              "export async function main(name: string) {\n  return `Hello, ${name}!`\n}\n",
-            isDraft: true,
-          },
-        ],
-      },
-    });
-
-    expect(checks).toContainEqual({
-      name: "global drafts match expected",
-      passed: true,
-    });
-  });
-
-  it("fails when expected global draft fixtures differ", () => {
-    const checks = validateGlobalState({
-      actual: {
-        drafts: [
-          {
-            type: "script",
-            path: "f/evals/global/greet_user",
-            language: "bun",
-            value:
-              "export async function main(name: string) {\n  return `Hello, ${name}!`\n}\n",
-            isDraft: true,
-          },
-        ],
-      },
-      expected: {
-        drafts: [
-          {
-            type: "script",
-            path: "f/evals/global/greet_user",
-            language: "bun",
-            value:
-              "export async function main(name: string) {\n  return `Bonjour, ${name}!`\n}\n",
-            isDraft: true,
-          },
-        ],
-      },
-    });
-
-    const expectedMatchCheck = checks.find(
-      (check) => check.name === "global drafts match expected"
-    );
-    expect(expectedMatchCheck?.passed).toBe(false);
-    expect(expectedMatchCheck?.details).toContain(
-      "script:f/evals/global/greet_user value differs"
-    );
-    expect(expectedMatchCheck?.details).toContain("Hello");
-    expect(expectedMatchCheck?.details).toContain("Bonjour");
-  });
-
-  it("explains expected global draft metadata mismatches", () => {
-    const checks = validateGlobalState({
-      actual: {
-        drafts: [
-          {
-            type: "script",
-            path: "f/evals/global/greet_user",
-            language: "bun",
-            value:
-              "export async function main(name: string) {\n  return `Hello, ${name}!`\n}\n",
-            isDraft: true,
-          },
-        ],
-      },
-      expected: {
-        drafts: [
-          {
-            type: "script",
-            path: "f/evals/global/greet_user",
-            language: "python3",
-            value:
-              "export async function main(name: string) {\n  return `Hello, ${name}!`\n}\n",
-            isDraft: true,
-          },
-        ],
-      },
-    });
-
-    const expectedMatchCheck = checks.find(
-      (check) => check.name === "global drafts match expected"
-    );
-    expect(expectedMatchCheck?.passed).toBe(false);
-    expect(expectedMatchCheck?.details).toContain(
-      "script:f/evals/global/greet_user language differs"
-    );
-    expect(expectedMatchCheck?.details).toContain('actual="bun"');
-    expect(expectedMatchCheck?.details).toContain('expected="python3"');
-  });
 });
 
 describe("validateAppState", () => {

ai_evals/core/validators.ts

@@ -6,7 +6,6 @@ import type {
   CliTrace,
   CliValidationSpec,
   FlowValidationSpec,
-  GlobalValidationSpec,
   ModeRunOutput,
   ToolValidationSpec,
 } from "./types";
@@ -52,20 +51,6 @@ export interface AppDatatableState {
   error?: string;
 }
 
-export interface GlobalDraftState {
-  drafts: GlobalDraft[];
-}
-
-export interface GlobalDraft {
-  type: string;
-  path: string;
-  triggerKind?: string;
-  summary?: string;
-  language?: string;
-  value?: unknown;
-  isDraft?: boolean;
-}
-
 const TS_LIKE_LANGUAGES = new Set(["bun", "deno", "nativets", "bunnative", "ts", "typescript"]);
 const CONTROL_FLOW_MODULE_TYPES = new Set(["branchone", "branchall", "forloopflow", "whileloopflow"]);
 
@@ -169,26 +154,6 @@ export function validateToolExpectations(input: {
     );
   }
 
-  for (const group of expect.requiredToolsAnyOf ?? []) {
-    checks.push(
-      check(
-        `uses one of ${group.join(", ")}`,
-        group.some((toolName) => input.run.toolsUsed.includes(toolName)),
-        `tools used: ${input.run.toolsUsed.join(", ") || "none"}`
-      )
-    );
-  }
-
-  for (const toolName of expect.forbiddenToolsUsed ?? []) {
-    checks.push(
-      check(
-        `does not use ${toolName}`,
-        !input.run.toolsUsed.includes(toolName),
-        `tools used: ${input.run.toolsUsed.join(", ") || "none"}`
-      )
-    );
-  }
-
   for (const rule of expect.toolCallArgs ?? []) {
     const calls = toolCallDetails.filter((call) => call.name === rule.tool);
     checks.push(
@@ -232,181 +197,6 @@ export function validateToolExpectations(input: {
         )
       );
     }
-
-    if (rule.stringIncludesAnyOf && rule.stringIncludesAnyOf.length > 0) {
-      // Existential: at least one call must contain one of the substrings.
-      // Other calls to the same tool may do anything — this suits SQL, where a
-      // model mixes the requested statement (e.g. an UPDATE) with verification
-      // SELECTs that would otherwise fail an "all calls" check.
-      const needles = rule.stringIncludesAnyOf.map((needle) => needle.toLowerCase());
-      const hasMatch = values.some(
-        (value) =>
-          typeof value === "string" && needles.some((needle) => value.toLowerCase().includes(needle))
-      );
-      checks.push(
-        check(
-          `${rule.tool}.${rule.field} includes a required substring`,
-          hasMatch,
-          `accepted substrings: ${rule.stringIncludesAnyOf.join(", ")}; values: ${summarizeToolValues(values)}`
-        )
-      );
-    }
-  }
-
-  return checks;
-}
-
-export function validateGlobalState(input: {
-  actual: GlobalDraftState;
-  expected?: GlobalDraftState;
-  validate?: GlobalValidationSpec;
-}): BenchmarkCheck[] {
-  const drafts = input.actual.drafts ?? [];
-  const checks: BenchmarkCheck[] = [];
-
-  // Read-only global cases are valid; only enforce draft production when the
-  // case explicitly asks for draft output.
-  if (globalValidationExpectsDrafts(input)) {
-    checks.push(
-      check(
-        "global produced at least one draft",
-        drafts.length > 0,
-        `drafts=${drafts.length}`
-      )
-    );
-  }
-
-  checks.push(
-    check(
-      "all global outputs are drafts",
-      drafts.every((draft) => draft.isDraft === true),
-      summarizeGlobalDrafts(drafts)
-    )
-  );
-
-  for (const draft of drafts) {
-    if (draft.type !== "script" || typeof draft.value !== "string") {
-      continue;
-    }
-
-    const language = (draft.language ?? "bun").toLowerCase();
-    const syntaxErrors = getScriptSyntaxErrors(draft.value, language);
-    if (TS_LIKE_LANGUAGES.has(language)) {
-      checks.push(
-        check(
-          `script draft ${draft.path} exports entrypoint`,
-          hasSupportedEntrypoint(draft.value)
-        )
-      );
-    }
-    checks.push(
-      check(
-        `script draft ${draft.path} has no syntax errors`,
-        syntaxErrors.length === 0,
-        summarizeProblems(syntaxErrors)
-      )
-    );
-  }
-
-  if (input.expected) {
-    checks.push(
-      check(
-        "global drafts match expected",
-        globalDraftStatesEqual(input.actual, input.expected),
-        describeGlobalDraftStateMismatch(input.actual, input.expected)
-      )
-    );
-  }
-
-  const validate = input.validate;
-  if (!validate) {
-    return checks;
-  }
-
-  if (validate.draftCountAtLeast !== undefined) {
-    checks.push(
-      check(
-        `global includes at least ${validate.draftCountAtLeast} draft(s)`,
-        drafts.length >= validate.draftCountAtLeast,
-        `drafts=${drafts.length}`
-      )
-    );
-  }
-
-  if (validate.draftCountExactly !== undefined) {
-    checks.push(
-      check(
-        `global includes exactly ${validate.draftCountExactly} draft(s)`,
-        drafts.length === validate.draftCountExactly,
-        `drafts=${drafts.length}`
-      )
-    );
-  }
-
-  for (const required of validate.requiredDrafts ?? []) {
-    const requirementLabel = formatGlobalDraftRequirement(required);
-    const draft = findGlobalDraft(drafts, required);
-    checks.push(
-      check(
-        `global includes ${requirementLabel}`,
-        Boolean(draft),
-        summarizeGlobalDrafts(drafts)
-      )
-    );
-    if (!draft) {
-      continue;
-    }
-
-    if (required.language !== undefined) {
-      checks.push(
-        check(
-          `${requirementLabel} uses ${required.language}`,
-          draft.language === required.language,
-          `language=${draft.language ?? "(none)"}`
-        )
-      );
-    }
-
-    for (const snippet of required.summaryIncludes ?? []) {
-      checks.push(
-        check(
-          `${requirementLabel} summary includes '${snippet}'`,
-          normalizeText(draft.summary ?? "").includes(normalizeText(snippet)),
-          `summary=${draft.summary ?? ""}`
-        )
-      );
-    }
-
-    const valueText = stringifyGlobalDraftValue(draft.value);
-    for (const snippet of required.valueIncludes ?? []) {
-      checks.push(
-        check(
-          `${requirementLabel} value includes '${snippet}'`,
-          normalizeText(valueText).includes(normalizeText(snippet)),
-          truncateForDetails(valueText)
-        )
-      );
-    }
-
-    for (const snippet of required.valueExcludes ?? []) {
-      checks.push(
-        check(
-          `${requirementLabel} value excludes '${snippet}'`,
-          !normalizeText(valueText).includes(normalizeText(snippet)),
-          truncateForDetails(valueText)
-        )
-      );
-    }
-  }
-
-  for (const forbidden of validate.forbiddenDrafts ?? []) {
-    checks.push(
-      check(
-        `global does not include ${forbidden.type} draft ${forbidden.path}`,
-        !findGlobalDraft(drafts, forbidden),
-        summarizeGlobalDrafts(drafts)
-      )
-    );
   }
 
   return checks;
@@ -643,286 +433,6 @@ function summarizeProblems(problems: string[], limit = 5): string | undefined {
   return `${problems.slice(0, limit).join("; ")}; ...and ${problems.length - limit} more`;
 }
 
-function findGlobalDraft(
-  drafts: GlobalDraft[],
-  requirement: {
-    type: string;
-    path?: string;
-    pathIncludes?: string[];
-    pathStartsWith?: string;
-    triggerKind?: string;
-    summaryIncludes?: string[];
-    valueIncludes?: string[];
-    valueExcludes?: string[];
-  }
-): GlobalDraft | undefined {
-  const candidates = drafts.filter((draft) =>
-    globalDraftMatchesLocator(draft, requirement)
-  );
-  return (
-    candidates.find((draft) => globalDraftMatchesContent(draft, requirement)) ??
-    candidates[0]
-  );
-}
-
-function globalDraftMatchesLocator(
-  draft: GlobalDraft,
-  requirement: {
-    type: string;
-    path?: string;
-    pathIncludes?: string[];
-    pathStartsWith?: string;
-    triggerKind?: string;
-  }
-): boolean {
-  return (
-    draft.type === requirement.type &&
-    (requirement.path === undefined || draft.path === requirement.path) &&
-    (requirement.pathStartsWith === undefined ||
-      draft.path.startsWith(requirement.pathStartsWith)) &&
-    (requirement.pathIncludes ?? []).every((snippet) =>
-      normalizeText(draft.path).includes(normalizeText(snippet))
-    ) &&
-    (requirement.triggerKind === undefined ||
-      draft.triggerKind === requirement.triggerKind)
-  );
-}
-
-function globalDraftMatchesContent(
-  draft: GlobalDraft,
-  requirement: {
-    summaryIncludes?: string[];
-    valueIncludes?: string[];
-    valueExcludes?: string[];
-  }
-): boolean {
-  const summary = normalizeText(draft.summary ?? "");
-  const value = normalizeText(stringifyGlobalDraftValue(draft.value));
-  return (
-    (requirement.summaryIncludes ?? []).every((snippet) =>
-      summary.includes(normalizeText(snippet))
-    ) &&
-    (requirement.valueIncludes ?? []).every((snippet) =>
-      value.includes(normalizeText(snippet))
-    ) &&
-    (requirement.valueExcludes ?? []).every(
-      (snippet) => !value.includes(normalizeText(snippet))
-    )
-  );
-}
-
-function formatGlobalDraftRequirement(
-  requirement: {
-    type: string;
-    path?: string;
-    pathIncludes?: string[];
-    pathStartsWith?: string;
-    triggerKind?: string;
-  }
-): string {
-  const typeLabel =
-    requirement.triggerKind === undefined
-      ? requirement.type
-      : `${requirement.triggerKind} ${requirement.type}`;
-  if (requirement.path !== undefined) {
-    return `${typeLabel} draft ${requirement.path}`;
-  }
-
-  const filters = [
-    ...(requirement.pathStartsWith === undefined
-      ? []
-      : [`path starts with ${requirement.pathStartsWith}`]),
-    ...(requirement.pathIncludes ?? []).map(
-      (snippet) => `path includes ${snippet}`
-    ),
-  ];
-  return filters.length === 0
-    ? `${typeLabel} draft`
-    : `${typeLabel} draft (${filters.join(", ")})`;
-}
-
-function summarizeGlobalDrafts(drafts: GlobalDraft[]): string {
-  const summary = drafts
-    .map((draft) => formatGlobalDraftKey(draft))
-    .join(", ");
-  return `drafts: ${summary || "none"}`;
-}
-
-function formatGlobalDraftKey(draft: GlobalDraft): string {
-  return `${draft.type}${draft.triggerKind ? `:${draft.triggerKind}` : ""}:${draft.path}`;
-}
-
-function globalValidationExpectsDrafts(input: {
-  expected?: GlobalDraftState;
-  validate?: GlobalValidationSpec;
-}): boolean {
-  const validate = input.validate;
-  return (
-    (input.expected?.drafts?.length ?? 0) > 0 ||
-    (validate?.requiredDrafts?.length ?? 0) > 0 ||
-    (validate?.draftCountAtLeast ?? 0) > 0 ||
-    (validate?.draftCountExactly ?? 0) > 0
-  );
-}
-
-function globalDraftStatesEqual(left: GlobalDraftState, right: GlobalDraftState): boolean {
-  return (
-    JSON.stringify(canonicalizeGlobalDrafts(left.drafts ?? [])) ===
-    JSON.stringify(canonicalizeGlobalDrafts(right.drafts ?? []))
-  );
-}
-
-function describeGlobalDraftStateMismatch(
-  actual: GlobalDraftState,
-  expected: GlobalDraftState
-): string {
-  const actualDrafts = actual.drafts ?? [];
-  const expectedDrafts = expected.drafts ?? [];
-  const actualByKey = new Map(
-    actualDrafts.map((draft) => [globalDraftSortKey(draft), draft] as const)
-  );
-  const expectedByKey = new Map(
-    expectedDrafts.map((draft) => [globalDraftSortKey(draft), draft] as const)
-  );
-
-  for (const key of Array.from(expectedByKey.keys()).sort()) {
-    const expectedDraft = expectedByKey.get(key);
-    if (expectedDraft && !actualByKey.has(key)) {
-      return `missing expected draft ${formatGlobalDraftKey(expectedDraft)}; actual=${summarizeGlobalDrafts(actualDrafts)}`;
-    }
-  }
-
-  for (const key of Array.from(actualByKey.keys()).sort()) {
-    const actualDraft = actualByKey.get(key);
-    if (actualDraft && !expectedByKey.has(key)) {
-      return `unexpected draft ${formatGlobalDraftKey(actualDraft)}; expected=${summarizeGlobalDrafts(expectedDrafts)}`;
-    }
-  }
-
-  for (const key of Array.from(expectedByKey.keys()).sort()) {
-    const actualDraft = actualByKey.get(key);
-    const expectedDraft = expectedByKey.get(key);
-    if (!actualDraft || !expectedDraft) {
-      continue;
-    }
-
-    const fieldMismatch = describeGlobalDraftFieldMismatch(
-      formatGlobalDraftKey(expectedDraft),
-      actualDraft,
-      expectedDraft
-    );
-    if (fieldMismatch) {
-      return fieldMismatch;
-    }
-  }
-
-  return `actual=${summarizeGlobalDrafts(actualDrafts)}; expected=${summarizeGlobalDrafts(expectedDrafts)}`;
-}
-
-function describeGlobalDraftFieldMismatch(
-  key: string,
-  actual: GlobalDraft,
-  expected: GlobalDraft
-): string | undefined {
-  const fields: Array<"language" | "summary" | "value" | "isDraft"> = [
-    "language",
-    "summary",
-    "value",
-    "isDraft",
-  ];
-
-  for (const field of fields) {
-    const actualValue = comparableGlobalDraftFieldValue(actual, field);
-    const expectedValue = comparableGlobalDraftFieldValue(expected, field);
-    if (JSON.stringify(actualValue) === JSON.stringify(expectedValue)) {
-      continue;
-    }
-
-    return `${key} ${field} differs: actual=${formatGlobalDraftFieldValue(
-      actualValue
-    )}; expected=${formatGlobalDraftFieldValue(expectedValue)}`;
-  }
-
-  return undefined;
-}
-
-function comparableGlobalDraftFieldValue(
-  draft: GlobalDraft,
-  field: "language" | "summary" | "value" | "isDraft"
-): unknown {
-  if (field === "summary" && typeof draft.summary === "string") {
-    return normalizeText(draft.summary);
-  }
-  if (field === "value" && typeof draft.value === "string") {
-    return normalizeText(draft.value);
-  }
-  if (field === "value") {
-    return canonicalizeJsonValue(draft.value);
-  }
-  return draft[field];
-}
-
-function formatGlobalDraftFieldValue(value: unknown): string {
-  if (value === undefined) {
-    return "(missing)";
-  }
-  return truncateForDetails(JSON.stringify(value), 300);
-}
-
-function canonicalizeGlobalDrafts(drafts: GlobalDraft[]): unknown[] {
-  return drafts
-    .slice()
-    .sort((left, right) => globalDraftSortKey(left).localeCompare(globalDraftSortKey(right)))
-    .map((draft) =>
-      canonicalizeJsonValue({
-        type: draft.type,
-        path: draft.path,
-        triggerKind: draft.triggerKind,
-        language: draft.language,
-        summary:
-          typeof draft.summary === "string" ? normalizeText(draft.summary) : draft.summary,
-        value:
-          typeof draft.value === "string"
-            ? normalizeText(draft.value)
-            : canonicalizeJsonValue(draft.value),
-        isDraft: draft.isDraft,
-      })
-    );
-}
-
-function globalDraftSortKey(draft: GlobalDraft): string {
-  return `${draft.type}:${draft.triggerKind ?? ""}:${draft.path}`;
-}
-
-function canonicalizeJsonValue(value: unknown): unknown {
-  if (Array.isArray(value)) {
-    return value.map(canonicalizeJsonValue);
-  }
-  if (value && typeof value === "object") {
-    return Object.fromEntries(
-      Object.entries(value as Record<string, unknown>)
-        .sort(([left], [right]) => left.localeCompare(right))
-        .map(([key, nested]) => [key, canonicalizeJsonValue(nested)])
-    );
-  }
-  return value;
-}
-
-function stringifyGlobalDraftValue(value: unknown): string {
-  if (typeof value === "string") {
-    return value;
-  }
-  return JSON.stringify(value ?? null, null, 2);
-}
-
-function truncateForDetails(value: string, maxLength = 500): string {
-  const normalized = value.replace(/\s+/g, " ").trim();
-  if (normalized.length <= maxLength) {
-    return normalized;
-  }
-  return `${normalized.slice(0, Math.max(0, maxLength - 3))}...`;
-}
-
 function validateCliExpectations(
   assistantOutput: string,
   trace: CliTrace | undefined,

ai_evals/core/windmillBackendSettings.test.ts

@@ -1,63 +0,0 @@
-import { afterEach, describe, expect, it } from "bun:test";
-import { resolveWindmillBackendSettings } from "./windmillBackendSettings";
-
-const ENV_KEYS = [
-  "WMILL_AI_EVAL_BACKEND_URL",
-  "WINDMILL_URL",
-  "WINDMILL_BASE_URL",
-  "REMOTE",
-  "WMILL_AI_EVAL_BACKEND_EMAIL",
-  "WMILL_AI_EVAL_BACKEND_PASSWORD",
-  "WMILL_AI_EVAL_BACKEND_WORKSPACE",
-] as const;
-
-const ORIGINAL_ENV = Object.fromEntries(
-  ENV_KEYS.map((key) => [key, process.env[key]]),
-) as Record<(typeof ENV_KEYS)[number], string | undefined>;
-
-afterEach(() => {
-  for (const key of ENV_KEYS) {
-    const value = ORIGINAL_ENV[key];
-    if (value === undefined) {
-      delete process.env[key];
-    } else {
-      process.env[key] = value;
-    }
-  }
-});
-
-describe("resolveWindmillBackendSettings", () => {
-  it("uses backend URL/auth defaults and the optional explicit workspace", () => {
-    delete process.env.WMILL_AI_EVAL_BACKEND_URL;
-    delete process.env.WINDMILL_URL;
-    delete process.env.WINDMILL_BASE_URL;
-    delete process.env.REMOTE;
-    process.env.WMILL_AI_EVAL_BACKEND_WORKSPACE = "shared-evals";
-
-    expect(resolveWindmillBackendSettings()).toEqual({
-      baseUrl: "http://127.0.0.1:8000",
-      email: "admin@windmill.dev",
-      password: "changeme",
-      workspaceOverride: "shared-evals",
-    });
-  });
-
-  it("does not expose workspace retention knobs", () => {
-    process.env.WMILL_AI_EVAL_BACKEND_URL = "http://backend.test/";
-
-    const settings = resolveWindmillBackendSettings();
-
-    expect(settings).toEqual({
-      baseUrl: "http://backend.test",
-      email: "admin@windmill.dev",
-      password: "changeme",
-      workspaceOverride: undefined,
-    });
-    expect(Object.keys(settings).sort()).toEqual([
-      "baseUrl",
-      "email",
-      "password",
-      "workspaceOverride",
-    ]);
-  });
-});

ai_evals/core/windmillBackendSettings.ts

@@ -2,7 +2,9 @@ export interface WindmillBackendSettings {
   baseUrl: string;
   email: string;
   password: string;
+  keepWorkspaces: boolean;
   workspaceOverride?: string;
+  workspacePrefix: string;
 }
 
 export function resolveWindmillBackendSettings(): WindmillBackendSettings {
@@ -16,9 +18,13 @@ export function resolveWindmillBackendSettings(): WindmillBackendSettings {
     ),
     email: process.env.WMILL_AI_EVAL_BACKEND_EMAIL ?? "admin@windmill.dev",
     password: process.env.WMILL_AI_EVAL_BACKEND_PASSWORD ?? "changeme",
+    keepWorkspaces: isTruthy(process.env.WMILL_AI_EVAL_KEEP_WORKSPACES),
     workspaceOverride: sanitizeOptionalWorkspaceId(
       process.env.WMILL_AI_EVAL_BACKEND_WORKSPACE,
     ),
+    workspacePrefix: sanitizeWorkspacePrefix(
+      process.env.WMILL_AI_EVAL_WORKSPACE_PREFIX ?? "ai-evals",
+    ),
   };
 }
 
@@ -37,9 +43,25 @@ function normalizeBaseUrl(value: string): string {
   return value.replace(/\/+$/, "");
 }
 
+function sanitizeWorkspacePrefix(value: string): string {
+  const sanitized = value
+    .trim()
+    .toLowerCase()
+    .replace(/[^a-z0-9-]+/g, "-")
+    .replace(/^-+|-+$/g, "");
+  return sanitized.length > 0 ? sanitized : "ai-evals";
+}
+
 function sanitizeOptionalWorkspaceId(
   value: string | undefined,
 ): string | undefined {
   const trimmed = value?.trim();
   return trimmed ? trimmed : undefined;
 }
+
+function isTruthy(value: string | undefined): boolean {
+  if (!value) {
+    return false;
+  }
+  return ["1", "true", "yes", "on"].includes(value.trim().toLowerCase());
+}