fix

* fix: prevent variable push from corrupting is_secret variables

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* test(cli): unit-test looksLikeWorkspaceCiphertext shape detection

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* fix(cli): scope is_secret downgrade to single-file push, not sync push

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* fix(cli): warn when variable push stores a secret value as already-encrypted

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* fix(cli): route workspace-resolution and auth diagnostics to stderr

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* docs(cli): rephrase comments to describe current behavior, not history

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

.agents/skills/adding-a-trigger/SKILL.md

@@ -0,0 +1,267 @@
+---
+name: adding-a-trigger
+description: Checklist for adding a new TriggerCrud-based trigger type to Windmill (Azure, GCP, Kafka, etc.). Use when wiring a new trigger kind across backend, frontend, CLI, and capture infrastructure.
+---
+
+# Skill: Adding a New Trigger Type
+
+Use this skill when adding a trigger kind that implements `TriggerCrud` (Kafka, GCP, Azure, MQTT, SQS, NATS, Postgres, Email…). For native triggers (Nextcloud, Google Drive — things wired through `windmill-native-triggers`), use the `native-trigger` skill instead.
+
+The goal of this doc is to enumerate every file that needs to change. Missing any one of them leads to silent regressions: sync drops the trigger, capture button does nothing, workspace forks lose it, sidebar counters undercount. Follow the checklist top-to-bottom — each section is independent enough to be validated on its own.
+
+Throughout this doc, substitute `{kind}` for the new trigger kind (`azure`, `kafka`, …), `{Kind}` for PascalCase (`Azure`, `Kafka`), `{KIND}` for SCREAMING (`AZURE`, `KAFKA`).
+
+## Reference implementations
+
+- **GCP** — closest analogue to Azure. Has push + pull, OIDC auth, ARM-like resource paths, capture handler. Grep for `gcp_trigger` / `GcpTrigger`.
+- **Kafka** — simpler (pull-only, streaming). Good for trivial integrations.
+- **Azure** — most recently added (2026). Shared-secret push auth, Event Grid namespaces + basic topics, ARM resource discovery, Namespace-pull data-plane. Grep for `azure_trigger` / `AzureTrigger`.
+
+## 1. Database migration
+
+Create a migration: `cargo sqlx migrate add -r add_{kind}_trigger` from `backend/`. Never write timestamps manually.
+
+The `up.sql` usually defines:
+- An optional enum type (e.g. `AZURE_MODE`) if the trigger has sub-kinds
+- The `{kind}_trigger` table with at minimum these columns (mirrored from kafka/gcp):
+  - primary: `(workspace_id, path)`
+  - `script_path`, `is_flow`, `enabled`, `mode`, `permissioned_as`, `edited_by`, `email`
+  - `edited_at`, `error`, `server_id`, `last_server_ping`
+  - `error_handler_path`, `error_handler_args jsonb`, `retry jsonb`
+  - trigger-specific fields
+- Indexes on foreign keys + any frequently-filtered columns
+- Foreign key to `workspace`
+
+Down migration drops the table and any enum types.
+
+## 2. Backend crate (`windmill-trigger-{kind}`)
+
+Create a new crate under `backend/windmill-trigger-{kind}/` with:
+
+- `Cargo.toml`: features `enterprise`, `private` if EE, standard deps
+- `src/lib.rs`: `pub use mod_ee::*;` behind `#[cfg(all(feature = "enterprise", feature = "private"))]`
+- `src/mod_ee.rs`: core types + helpers
+- `src/handler_ee.rs`: `TriggerCrud` impl + route handlers
+- `src/listener_ee.rs`: (only if streaming/pull-based) `Listener` trait impl
+
+Required in `mod_ee.rs`:
+- `{Kind}Config` struct (persisted shape, `FromRow`)
+- `{Kind}ConfigRequest` struct (what API receives — usually similar to Config but with validation fields)
+- `{Kind}Trigger` unit struct (implements the traits)
+- `impl TriggerJobArgs for {Kind}Trigger` — sets `TRIGGER_KIND`, `Payload`, `v1_payload_fn`
+
+Required in `handler_ee.rs`:
+- `#[async_trait] impl TriggerCrud for {Kind}Trigger` with:
+  - `type Trigger = Trigger<{Kind}Config>`
+  - `type TriggerConfigRequest = {Kind}ConfigRequest`
+  - `const ROUTE_PREFIX: &'static str = "/{kind}_triggers";`
+  - `const TABLE_NAME`, `ADDITIONAL_SELECT_FIELDS`
+  - `get_deployed_object`, `validate_config`, `create_trigger`, `update_trigger`, `delete_trigger`, `test_connection`
+  - `additional_routes` (optional — mount extra endpoints for things like ARM resource listing, topic discovery)
+
+Register the crate in `backend/Cargo.toml` as a workspace member and as a dep of `windmill-api` behind the feature flag.
+
+## 3. Wire into `windmill-api` (feature-gated everywhere)
+
+**`backend/windmill-api/src/triggers/handler.rs`** — mount the trigger crate:
+```rust
+#[cfg(all(feature = "enterprise", feature = "{kind}_trigger", feature = "private"))]
+{
+    use crate::triggers::{kind}::{Kind}Trigger;
+    router = router.nest({Kind}Trigger::ROUTE_PREFIX, complete_trigger_routes({Kind}Trigger));
+}
+```
+
+**`backend/windmill-api/src/triggers/{kind}/mod.rs`** — re-export the crate:
+```rust
+pub use windmill_trigger_{kind}::*;
+```
+
+**`backend/windmill-api/src/lib.rs`** — if the trigger receives inbound pushes, add a webhook route:
+```rust
+.nest("/{kind}/w/{workspace_id}", {
+    #[cfg(all(feature = "enterprise", feature = "{kind}_trigger", feature = "private"))]
+    { triggers::{kind}::handler_oss::{kind}_push_route_handler() }
+    #[cfg(not(...))]
+    { Router::new() }
+})
+```
+
+## 4. `TriggerKind` enum (`backend/windmill-types/src/triggers.rs`)
+
+Already has slots for most triggers but verify your variant exists:
+- Add `{Kind}` to the `TriggerKind` enum
+- Add match arm in `to_key()`
+- Add match arm in `from_str`
+- Add match arm in `JobTriggerKind` (if jobs need kind tagging)
+
+## 5. OpenAPI (`backend/windmill-api/openapi.yaml`)
+
+This file is huge and the single most-forgotten place. Add:
+
+- `/w/{workspace}/{kind}_triggers/create` + `/update/{path}` + `/delete/{path}` + `/get/{path}` + `/list` + `/exists/{path}` + `/setmode/{path}` + `/test` paths (mirror gcp section)
+- Any `additional_routes` your handler exposes (resource discovery, etc.)
+- Schemas: `{Kind}Trigger`, `{Kind}TriggerData`, `{Kind}Mode` (if enum), `{Kind}DeliveryConfig`, helper request/response types
+- Add `{kind}` to `CaptureTriggerKind` enum
+- Add `{kind}_used: boolean` to the `UsedTriggers` response schema
+
+Regenerate frontend client: `npm run generate-backend-client` from `frontend/`.
+
+## 6. `UsedTriggers` + workspace export
+
+**`backend/windmill-api-workspaces/src/workspaces.rs`** — add `{kind}_used: bool` to the `UsedTriggers` struct and add an `EXISTS(SELECT 1 FROM {kind}_trigger …)` to the `get_used_triggers` query.
+
+**`backend/windmill-api/src/workspaces_export.rs`** — add export block mirroring gcp's (export lists all triggers, serializes them to YAML/JSON). The block re-uses the `trigger_ignore_keys` variable so the new kind automatically participates in fork-export stripping (`mode` field is omitted when the source workspace is a fork — keeps fork→parent merges from flipping the parent's enabled state).
+
+**Fork cloning (`clone_triggers_and_schedules` in workspaces.rs)** — add an `INSERT INTO {kind}_trigger ... SELECT ...` block that copies all rows from the parent workspace, forcing `mode = 'disabled'::TRIGGER_MODE`. Always runs at fork creation; forgetting this means users can't carry `{kind}` triggers into their forks.
+
+## 6.5 Hardcoded trigger-kind arrays (silent-failure hotspots)
+
+Several files keep **hardcoded arrays** of trigger kind strings. Miss one and ACL checks / user offboarding / trash drop your kind:
+
+- **`backend/windmill-api-groups/src/granular_acls.rs`** — `KINDS: [&str; N]`. **Increment N** (the compile error is cryptic otherwise). Controls which kinds accept granular ACL operations.
+- **`backend/windmill-api-users/src/users.rs`** (`extra_perms_tables`) — which tables get `extra_perms` entries cleaned when a user is deleted.
+- **`backend/windmill-api/src/offboarding.rs`** — three separate arrays (enumeration, fork-copy, and delete paths). **All three** need the new kind.
+- **`backend/windmill-api/src/trash.rs`** — `valid_tables` for the trash / restore API.
+- **`backend/windmill-git-sync/src/lib.rs`** — add a test assertion for `DeployedObject::{Kind}Trigger.get_kind() == "{kind}_trigger"` (the `get_kind` match arm itself lives in the enum impl — already required by the Rust compiler).
+- **`backend/windmill-api-auth/src/scopes.rs`** — add the `{Kind}Triggers` variant to `ScopeDomain` enum + `as_str` match + `from_str` match. Required for the OAuth/token system to recognise `{kind}_triggers:read|write` scopes.
+- **`backend/windmill-api/src/token.rs`** (`build_trigger_scope_domains` → `TRIGGER_DOMAINS`) — add `("{kind}_triggers", "{Kind display name}")` so the CreateToken UI's scope selector surfaces the `read` / `write` checkboxes.
+
+**OpenAPI enums** to extend (do NOT forget — generated client will allow it but server rejects as 400):
+- `CaptureTriggerKind` enum
+- Three `kind` enums under `/w/{workspace}/acls/{get,add,remove}/{kind}/{path}` (yes, same list repeated three times)
+
+After editing any of these, run a full `cargo check` with your feature flag + `gcp_trigger` + other core flags — the `KINDS: [&str; N]` length mismatch only surfaces when the crate compiles.
+
+## 7. Capture infrastructure (`backend/windmill-api/src/capture.rs`)
+
+If the trigger supports push delivery, it also needs a capture endpoint so users can test it:
+
+- `{Kind}TriggerConfig` struct (gated by feature flags)
+- `TriggerConfig::{Kind}` variant
+- `set_{kind}_trigger_config` function (creates the subscription/equivalent pointing at the capture URL — use your `manage_{kind}_subscription` helper with `trigger_mode=false`)
+- Both real + no-op versions behind feature gates
+- `TriggerKind::{Kind} => set_{kind}_trigger_config(...)` arm in `set_config`
+- `{kind}_payload` async handler — validates auth (if any), processes payload, calls `insert_capture_payload`
+- Route: `.route("/{kind}/{runnable_kind}/{*path}", post({kind}_payload))` inside `workspaced_unauthed_service` — and expand the surrounding `#[cfg(any(...))]` to include your feature flag
+
+## 8. CLI (`cli/`) — easy to miss, breaks sync silently
+
+Check all of these:
+
+**`cli/src/types.ts`:**
+- Add `"{kind}"` to `TRIGGER_TYPES` array
+- Add `"{kind}_trigger"` to `getTypeStrFromPath` return union
+- Add match case in `getTypeStrFromPath`'s `typeEnding ===` chain
+- Add `pushTrigger("{kind}", ...)` branch in `pushObj`
+
+**`cli/src/commands/trigger/trigger.ts`:**
+- Import `{Kind}Trigger` type
+- Add `{kind}: {Kind}Trigger` to the `Trigger` type map
+- Add `{kind}: wmill.get{Kind}Trigger`, `update{Kind}Trigger`, `create{Kind}Trigger` to each function map
+- Add `{kind}: { ... }` template to `triggerTemplates`
+- Add `list{Kind}Triggers` call + spread in the `list` aggregation
+- Update `--kind` option descriptions to mention the new kind
+
+**`cli/src/commands/sync/sync.ts`:**
+- Add `path.endsWith(".{kind}_trigger" + ext)` in the file-type filter
+- Add `typ == "{kind}_trigger"` in `getTypeOrder`
+- Add `"{kind}_trigger"` to the delete-suffix regex (~line 3092)
+- Add a `case "{kind}_trigger"` in the delete switch
+
+**`cli/src/guidance/skills.ts`** — **DO NOT EDIT DIRECTLY**. It's auto-generated by `system_prompts/generate.py`. Instead:
+- Edit `system_prompts/utils.py` → append `('{Kind}Trigger', '{kind}_trigger')` to the `SCHEMA_MAPPINGS['triggers']` list (this is the master list — the one in `generate.py` is duplicated and `utils.py` wins)
+- Then run `python3 system_prompts/generate.py` — it regenerates `cli/src/guidance/skills.ts` with the schema extracted from `backend/windmill-api/openapi.yaml`
+- Commit the regenerated file
+
+## 9. Frontend — editor + drawer
+
+Under `frontend/src/lib/components/triggers/{kind}/`:
+
+- `{Kind}TriggerPanel.svelte` — the tile shown in the triggers listing
+- `{Kind}TriggerEditor.svelte` — outer drawer wrapper
+- `{Kind}TriggerEditorInner.svelte` — state + business logic; must expose:
+  - `openEdit(path, isFlow, defaultValues?)` method
+  - `isEditor` prop, `onConfigChange` + `onCaptureConfigChange` callbacks
+  - `get{Kind}Config()` + `get{Kind}CaptureConfig()` helpers
+  - `captureConfig = $derived.by(untrack(() => isEditor) ? get{Kind}CaptureConfig : () => ({}))`
+  - `$effect(() => { const args = [captureConfig, isValid] as const; untrack(() => onCaptureConfigChange?.(...args)) })`
+- `{Kind}TriggerEditorConfigSection.svelte` — form fields; use design-system components (`TextInput`, `Select`, `Toggle`, `ToggleButtonGroup`), never raw `<input>`
+- `{Kind}Capture.svelte` — capture panel; wraps `CaptureSection` with `captureType="{kind}"`
+- `utils.ts` — `requestBody` builders and any trigger-type-specific helpers
+
+## 10. Frontend — global integration
+
+Easy to miss:
+
+- **`frontend/src/lib/components/triggers.ts`** — add `'{kind}'` to the `TriggerKind` union
+- **`frontend/src/lib/components/triggers/CaptureWrapper.svelte`**:
+  - Import `{Kind}Capture`
+  - Add to `isStreamingCapture()` array (streaming = pull-style; push-style is typically `false`)
+  - Add `{:else if captureType === '{kind}'}` branch with the `<{Kind}Capture>` render
+- **`frontend/src/lib/components/sidebar/SidebarContent.svelte`** — import the icon, add the nav entry
+- **`frontend/src/lib/components/sidebar/OperatorMenu.svelte`** — add the operator-mode entry
+- **`frontend/src/routes/(root)/(logged)/+layout.svelte`** — destructure `{kind}_used` from `/get_used_triggers` response, push `'{kind}'` into `usedKinds`
+- **`frontend/src/lib/components/search/GlobalSearchModal.svelte`** — import icon, add "Go to {Kind} ..." entry
+- **`frontend/src/lib/components/offboarding-utils.ts`** — add mappings `{kind}_trigger: '{kind}_triggers'` and `{kind}_trigger: '{kind} trigger'`
+- **`frontend/src/lib/components/icons/{Kind}Icon.svelte`** — single-path SVG, `fill={color ?? 'currentColor'}`, `size` prop default 16 (match existing icons — don't hardcode colors, don't use `width`/`height` props)
+- **`frontend/src/routes/(root)/(logged)/{kind}_triggers/+page.svelte`** — listing page (mirror `gcp_triggers/+page.svelte` for push+pull, `kafka_triggers` for pure streaming)
+- **`frontend/src/lib/components/CompareWorkspaces.svelte`** — workspace fork / compare tool. Needs: service import, editor import, `{kind}Editor` `$state`, `case '{kind}'` in `openTriggerDetails()`, entry in `triggerServices` object (list/delete/normalize), and `<{Kind}TriggerEditor bind:this={{kind}Editor} />` in the template
+
+## 10.5 AI system prompts (`system_prompts/`)
+
+- **`system_prompts/utils.py`** — append `('{Kind}Trigger', '{kind}_trigger')` to `SCHEMA_MAPPINGS['triggers']` (master list used by code generation + CLI skills)
+- **`system_prompts/generate.py`** — also has a duplicated `schema_types` list (~line 903) for the AI `triggers` skill content. Add `('{Kind}Trigger', '{kind}_trigger')` there too
+- **`system_prompts/generate.py`** `schema_names` (~line 1192) — add `'{Kind}Trigger'` (add `'New{Kind}Trigger'` only if the OpenAPI declares one; GCP and Azure don't)
+- Run `python3 system_prompts/generate.py` — this rewrites `cli/src/guidance/skills.ts` and all `auto-generated/` docs. Commit the regenerated files
+
+## 11. Validation
+
+Run all of these before declaring done:
+
+```bash
+# Backend
+cd backend
+cargo check --features enterprise,{kind}_trigger,private   # minimal
+cargo check --features enterprise,azure_trigger,private,gcp_trigger,http_trigger,mqtt_trigger,postgres_trigger,sqs_trigger,kafka,nats,smtp,websocket  # full
+
+# SQLx offline data (never run `cargo sqlx prepare` directly — use the wrapper)
+./update_sqlx.sh
+
+# Frontend
+cd frontend
+npm run generate-backend-client
+npm run check:fast
+```
+
+Smoke test in the UI: create a trigger, save, check it appears in sidebar + search, delete, re-create via CLI `wmill sync`.
+
+## 12. Common pitfalls
+
+- **Forgetting feature gates in `workspaced_unauthed_service()`** — the surrounding `#[cfg(any(...))]` expression must include your feature flag, not just the inner `#[cfg]` on the route
+- **`.route(path, ...).route(path, ...)` with same path and different methods** — older axum replaced; use `.route(path, post(h1).options(h2))` to chain methods on the same `MethodRouter`
+- **`on:event` directives** — legacy Svelte 4, no-op in runes mode. Use callback props (`onSelected`, `onConfigChange`)
+- **`$bindable(default_value)` on optional props** — banned by project CLAUDE.md. Use `$bindable()` + `$derived(prop ?? default)` instead
+- **CORS layer intercepting OPTIONS** — tower-http CorsLayer short-circuits OPTIONS before reaching your handler. For server-to-server webhook endpoints, drop the CORS layer entirely (CORS is browser-only)
+- **DeliveryAttributeMappings / custom headers for auth** — prefer HMAC or sha256-hashed shared secrets over opaque JWTs when the provider doesn't support signed tokens natively. Store only the hash; regenerate secret on every save
+- **ARM / API resource-listing cascades** — if the trigger's resource type is deep (Azure: subscription → RG → namespace → topic), offer dropdowns in the UI populated from the provider's APIs using the user's credential resource
+- **Clearing stale selections on dependency change** — when a dropdown's underlying data reloads (e.g., user changes SP or edition), clear selections that no longer match the new list
+- **Workspace-scoped tag compatibility** — if the trigger has tags, verify forked workspaces handle them (see commit `0773b5bc85` for a historical fix)
+
+## 13. EE file split
+
+If the trigger is enterprise-only, the code lives in `windmill-ee-private__worktrees/.../windmill-trigger-{kind}/src/*_ee.rs` and is symlinked into the OSS tree. The `windmill-ee-private__worktrees/` directory holds the real files; changes propagate via symlinks. See `docs/enterprise.md` for the workflow.
+
+## 14. Final checklist before PR
+
+- [ ] Migration up/down tested (revert + re-apply)
+- [ ] `./update_sqlx.sh` committed the updated `.sqlx/` offline data
+- [ ] `cargo check` passes with your feature flag + with all trigger features
+- [ ] `npm run check:fast` passes
+- [ ] Trigger visible in sidebar with correct icon weight (not oversized/colored — use `currentColor`)
+- [ ] Create, edit, delete flow all work in the UI
+- [ ] Capture button works (if push-capable)
+- [ ] Trigger appears in `/get_used_triggers` → sidebar pulse
+- [ ] `wmill sync pull` + `wmill sync push` both round-trip the trigger
+- [ ] `wmill trigger list` includes it
+- [ ] OpenAPI schemas are complete (no `null` in generated types)

.agents/skills/commit/SKILL.md

@@ -0,0 +1,58 @@
+---
+name: commit
+user_invocable: true
+description: Create a git commit with conventional commit format. MUST use anytime you want to commit changes.
+---
+
+# Git Commit Skill
+
+Create a focused, single-line commit following conventional commit conventions.
+
+## Instructions
+
+1. **Analyze changes**: Run `git status` and `git diff` to understand what was modified
+2. **Stage only modified files**: Add files individually by name. NEVER use `git add -A` or `git add .`
+3. **Write commit message**: Follow the conventional commit format as a single line
+
+## Conventional Commit Format
+
+```
+<type>: <description>
+```
+
+### Types
+- `feat`: New feature or capability
+- `fix`: Bug fix
+- `refactor`: Code change that neither fixes a bug nor adds a feature
+- `docs`: Documentation only changes
+- `style`: Formatting, missing semicolons, etc (no code change)
+- `test`: Adding or correcting tests
+- `chore`: Maintenance tasks, dependency updates, etc
+- `perf`: Performance improvement
+
+### Rules
+- Message MUST be a single line (no multi-line messages)
+- Description should be lowercase, imperative mood ("add" not "added")
+- No period at the end
+- Keep under 72 characters total
+
+### Examples
+```
+feat: add token usage tracking for AI providers
+fix: resolve null pointer in job executor
+refactor: extract common validation logic
+docs: update API endpoint documentation
+chore: upgrade sqlx to 0.7
+```
+
+## Execution Steps
+
+1. Run `git status` to see all changes
+2. Run `git diff` to understand the changes in detail
+3. Run `git log --oneline -5` to see recent commit style
+4. Stage ONLY the modified/relevant files: `git add <file1> <file2> ...`
+5. Create the commit with conventional format:
+   ```bash
+   git commit -m "<type>: <description>"
+   ```
+6. Run `git status` to verify the commit succeeded

.agents/skills/local-review/SKILL.md

@@ -0,0 +1,98 @@
+---
+name: local-review
+description: Code review the current PR (or branch diff against main) for bugs, security, and AGENTS.md compliance. MUST use when asked to review code.
+---
+
+# Local Code Review
+
+Run the same review locally that the GitHub auto-review actions run on PRs (Claude / Codex / Pi). The review policy lives in `REVIEW.md`.
+
+**Why a subagent**: the review MUST run in a fresh context — not inline in the current session. If the user has been iterating on the diff, the main session has absorbed their reasoning and rationalizations, so it anchors and misses things CI catches. A subagent starts cold, like CI does.
+
+## Steps
+
+1. **Determine the PR scope** (cheap, do this in the main session):
+   - If an argument is provided, treat it as a PR number or branch.
+   - Otherwise, detect from the current branch vs `main`.
+   - Confirm the PR/branch exists (`gh pr view <n>` or `git rev-parse <branch>`).
+
+2. **Delegate the review to a fresh-context subagent** with a self-contained prompt. The prompt MUST include:
+   - The PR number or branch name to review.
+   - The instruction to read `REVIEW.md` first for the policy, then `AGENTS.md` files in directories touched by the diff.
+   - The exact output format (see below).
+   - Whether `--comment` was requested (so the subagent emits inline-comment payloads if needed).
+   - Any "Additional reviewer instructions" the user provided.
+
+   - **Claude Code**: use the `Agent` tool with `subagent_type: branch-diff-reviewer` (read-only tools, purpose-built for this). If unavailable, fall back to `general-purpose`.
+   - **Codex / Pi**: if the CLI exposes a fresh-session subagent mechanism, use it. Otherwise tell the user to run the skill in a fresh CLI session and stop — running inline in the current session defeats the purpose.
+
+3. **Receive the findings** from the subagent and relay them to the user verbatim. Do not re-summarize, re-judge, or filter — the whole point of fresh context is to surface what the main session would dismiss.
+
+4. **Post comments if `--comment` was requested**: use the `gh` commands below with the subagent's output as the body. The main session does the posting because the subagent is read-only.
+
+## Subagent prompt template
+
+```
+Review <PR #N | branch X> against main per the policy in REVIEW.md.
+
+Steps:
+1. Read REVIEW.md (repo root) for the full policy: severity triage, public-surface
+   checklist, AGENTS.md compliance, test coverage assessment.
+2. Read AGENTS.md (repo root) and any AGENTS.md in directories touched by the diff.
+3. Get the diff: `gh pr diff <N>` (if PR) or `git diff main...<branch>`.
+4. Get context: `gh pr view <N>` (if PR) or `git log main..<branch> --oneline`.
+5. Read changed files only when the diff alone is insufficient to validate a finding.
+6. Self-validate each finding: "is this definitely a real issue a senior engineer
+   would flag?" Discard if uncertain.
+7. Output findings in the exact format below. Do not modify any files.
+
+<paste output format from below>
+
+<if --comment requested:>
+Additionally emit a JSON array of inline comments suitable for the GitHub reviews
+API, one per finding that maps to a specific line:
+[{"path": "...", "line": N, "side": "RIGHT", "body": "[P1] ..."}, ...]
+```
+
+## Output format
+
+```
+## Code review
+
+<verdict line per REVIEW.md>
+
+Found N issues:
+
+1. [P0|P1|P2] <description>
+   <file_path:line_number>
+
+2. [P0|P1|P2] <description>
+   <file_path:line_number>
+```
+
+End with a `Test coverage` section per the shared policy.
+
+If no issues are found:
+
+```
+## Code review
+
+Good to merge.
+
+No issues found. Checked for bugs, security, and AGENTS.md compliance.
+```
+
+## Posting comments (`--comment`)
+
+For a top-level PR comment:
+
+```bash
+gh pr review --comment --body "<summary from subagent>"
+```
+
+For inline comments on specific lines (using the JSON the subagent emitted):
+
+```bash
+gh api repos/{owner}/{repo}/pulls/{pr}/reviews \
+  -f body="<summary>" -f event="COMMENT" -f comments="<json from subagent>"
+```

.agents/skills/native-trigger/SKILL.md

@@ -0,0 +1,793 @@
+---
+name: native-trigger
+description: Guidance for adding native trigger services to Windmill. Use when implementing or modifying native trigger integrations across the backend and frontend.
+---
+
+# Skill: Adding Native Trigger Services
+
+This skill provides comprehensive guidance for adding new native trigger services to Windmill. Native triggers allow external services (like Nextcloud, Google Drive, etc.) to trigger Windmill scripts/flows via webhooks or push notifications.
+
+## Architecture Overview
+
+The native trigger system consists of:
+
+1. **Database Layer** - PostgreSQL tables and enum types
+2. **Backend Rust Implementation** - Core trait, handlers, and service modules in the `windmill-native-triggers` crate
+3. **Frontend Svelte Components** - Configuration forms and UI components
+
+### Key Files
+
+| Component | Path |
+|-----------|------|
+| Core module with `External` trait | `backend/windmill-native-triggers/src/lib.rs` |
+| Generic CRUD handlers | `backend/windmill-native-triggers/src/handler.rs` |
+| Background sync logic | `backend/windmill-native-triggers/src/sync.rs` |
+| OAuth/workspace integration | `backend/windmill-native-triggers/src/workspace_integrations.rs` |
+| Re-export shim (windmill-api) | `backend/windmill-api/src/native_triggers/mod.rs` |
+| TriggerKind enum | `backend/windmill-common/src/triggers.rs` |
+| JobTriggerKind enum | `backend/windmill-common/src/jobs.rs` |
+| Frontend service registry | `frontend/src/lib/components/triggers/native/utils.ts` |
+| Frontend trigger utilities | `frontend/src/lib/components/triggers/utils.ts` |
+| Trigger badges (icons + counts) | `frontend/src/lib/components/graph/renderers/triggers/TriggersBadge.svelte` |
+| Workspace integrations UI | `frontend/src/lib/components/workspaceSettings/WorkspaceIntegrations.svelte` |
+| OAuth config form component | `frontend/src/lib/components/workspaceSettings/OAuthClientConfig.svelte` |
+| OpenAPI spec | `backend/windmill-api/openapi.yaml` |
+| Reference: Nextcloud module | `backend/windmill-native-triggers/src/nextcloud/` |
+| Reference: Google module | `backend/windmill-native-triggers/src/google/` |
+
+### Crate Structure
+
+The native trigger code lives in the `windmill-native-triggers` crate (`backend/windmill-native-triggers/`). The `windmill-api` crate re-exports everything via a shim:
+
+```rust
+// backend/windmill-api/src/native_triggers/mod.rs
+pub use windmill_native_triggers::*;
+```
+
+All new service modules go in `backend/windmill-native-triggers/src/`.
+
+---
+
+## Core Concepts
+
+### The `External` Trait
+
+Every native trigger service implements the `External` trait defined in `lib.rs`:
+
+```rust
+#[async_trait]
+pub trait External: Send + Sync + 'static {
+    // Associated types:
+    type ServiceConfig: Debug + DeserializeOwned + Serialize + Send + Sync;
+    type TriggerData: Debug + Serialize + Send + Sync;
+    type OAuthData: DeserializeOwned + Serialize + Clone + Send + Sync;
+    type CreateResponse: DeserializeOwned + Send + Sync;
+
+    // Constants:
+    const SUPPORT_WEBHOOK: bool;
+    const SERVICE_NAME: ServiceName;
+    const DISPLAY_NAME: &'static str;
+    const TOKEN_ENDPOINT: &'static str;
+    const REFRESH_ENDPOINT: &'static str;
+    const AUTH_ENDPOINT: &'static str;
+
+    // Required methods:
+    async fn create(&self, w_id, oauth_data, webhook_token, data, db, tx) -> Result<Self::CreateResponse>;
+    async fn update(&self, w_id, oauth_data, external_id, webhook_token, data, db, tx) -> Result<serde_json::Value>;
+    async fn get(&self, w_id, oauth_data, external_id, db, tx) -> Result<Self::TriggerData>;
+    async fn delete(&self, w_id, oauth_data, external_id, db, tx) -> Result<()>;
+    async fn exists(&self, w_id, oauth_data, external_id, db, tx) -> Result<bool>;
+    async fn maintain_triggers(&self, db, workspace_id, triggers, oauth_data, synced, errors);
+    fn external_id_and_metadata_from_response(&self, resp) -> (String, Option<serde_json::Value>);
+
+    // Methods with defaults:
+    async fn prepare_webhook(&self, db, w_id, headers, body, script_path, is_flow) -> Result<PushArgsOwned>;
+    fn service_config_from_create_response(&self, data, resp) -> Option<serde_json::Value>;
+    fn additional_routes(&self) -> axum::Router;
+    async fn http_client_request<T, B>(&self, url, method, workspace_id, tx, db, headers, body) -> Result<T>;
+}
+```
+
+Key design points:
+- **`update()` returns `serde_json::Value`** - the resolved service_config to store. Each service is responsible for building the final config.
+- **`maintain_triggers()`** - periodic background maintenance. Each service implements its own strategy (Nextcloud: reconcile with external state; Google: renew expiring channels).
+- **No `list_all()` in the trait** - services that need it (Nextcloud) implement it privately; services that don't (Google) use different maintenance strategies.
+- **No `get_external_id_from_trigger_data()` or `extract_service_config_from_trigger_data()`** - removed in favor of the `maintain_triggers` pattern.
+
+### Create Lifecycle: Two Paths
+
+The `create_native_trigger` handler in `handler.rs` supports two creation flows, controlled by `service_config_from_create_response()`:
+
+**Path A: Short (Google pattern)** - `service_config_from_create_response()` returns `Some(config)`:
+1. `create()` registers on external service
+2. `external_id_and_metadata_from_response()` extracts the ID
+3. `service_config_from_create_response()` builds the config directly from input data + response metadata
+4. Stores trigger in DB -- done, no extra round-trip
+
+Use this when the external_id is known before the create call (e.g., Google generates the channel_id as a UUID upfront and includes it in the webhook URL).
+
+**Path B: Long (Nextcloud pattern)** - `service_config_from_create_response()` returns `None` (default):
+1. `create()` registers on external service (webhook URL has no external_id yet)
+2. `external_id_and_metadata_from_response()` extracts the ID
+3. `update()` is called to fix the webhook URL with the now-known external_id
+4. `update()` returns the resolved service_config
+5. Stores trigger in DB
+
+Use this when the external_id is assigned by the remote service and the webhook URL needs to be corrected after creation.
+
+### OAuth Token Storage (Three-Table Pattern)
+
+OAuth tokens are stored across three tables, NOT in `workspace_integrations.oauth_data` directly:
+
+| Table | What's Stored |
+|-------|---------------|
+| `workspace_integrations` | `oauth_data` JSON with `base_url`, `client_id`, `client_secret`, `instance_shared` flag; `resource_path` pointing to the variable |
+| `variable` | Encrypted `access_token` (at the path stored in `resource_path`), linked to `account` via `account` column |
+| `account` | `refresh_token`, keyed by `workspace_id` + `client` (service name) + `is_workspace_integration = true` |
+
+The `decrypt_oauth_data()` function in `lib.rs` assembles these into a unified struct:
+```rust
+pub struct OAuthConfig {
+    pub base_url: String,
+    pub access_token: String,      // decrypted from variable
+    pub refresh_token: Option<String>, // from account table
+    pub client_id: String,         // from oauth_data or instance settings
+    pub client_secret: String,     // from oauth_data or instance settings
+}
+```
+
+Instance-level sharing: when `oauth_data.instance_shared == true`, `client_id` and `client_secret` are read from global settings instead of workspace_integrations.
+
+### URL Resolution
+
+The `resolve_endpoint()` helper handles both absolute and relative OAuth URLs:
+
+```rust
+pub fn resolve_endpoint(base_url: &str, endpoint: &str) -> String {
+    if endpoint.starts_with("http://") || endpoint.starts_with("https://") {
+        endpoint.to_string()  // Google: absolute URLs
+    } else {
+        format!("{}{}", base_url, endpoint)  // Nextcloud: relative paths
+    }
+}
+```
+
+### ServiceName Methods
+
+`ServiceName` is the central registry enum. Each variant must implement these match arms:
+
+| Method | Purpose |
+|--------|---------|
+| `as_str()` | Lowercase identifier (e.g., `"google"`) |
+| `as_trigger_kind()` | Maps to `TriggerKind` enum |
+| `as_job_trigger_kind()` | Maps to `JobTriggerKind` enum |
+| `token_endpoint()` | OAuth token endpoint (relative or absolute) |
+| `auth_endpoint()` | OAuth authorization endpoint |
+| `oauth_scopes()` | Space-separated OAuth scopes |
+| `resource_type()` | Resource type for token storage (e.g., `"gworkspace"`) |
+| `extra_auth_params()` | Extra OAuth params (e.g., Google needs `access_type=offline`, `prompt=consent`) |
+| `integration_service()` | Maps to the workspace integration service (usually `*self`) |
+| `TryFrom<String>` | Parse from string |
+| `Display` | Delegates to `as_str()` |
+
+---
+
+## Step-by-Step Implementation Guide
+
+### Step 1: Database Migration
+
+Create a new migration file: `backend/migrations/YYYYMMDDHHMMSS_newservice_trigger.up.sql`
+
+```sql
+-- Add the service to the native_trigger_service enum
+ALTER TYPE native_trigger_service ADD VALUE IF NOT EXISTS 'newservice';
+
+-- Add to TRIGGER_KIND enum (used for trigger tracking)
+ALTER TYPE TRIGGER_KIND ADD VALUE IF NOT EXISTS 'newservice';
+
+-- Add to job_trigger_kind enum (used for job tracking)
+ALTER TYPE job_trigger_kind ADD VALUE IF NOT EXISTS 'newservice';
+```
+
+Also create the corresponding down migration.
+
+### Step 2: Update windmill-common Enums
+
+#### `backend/windmill-common/src/triggers.rs`
+
+Add variant to `TriggerKind` enum, and update `to_key()` and `fmt()` implementations.
+
+#### `backend/windmill-common/src/jobs.rs`
+
+Add variant to `JobTriggerKind` enum and update the `Display` implementation.
+
+### Step 3: Backend Service Module
+
+Create a new directory: `backend/windmill-native-triggers/src/newservice/`
+
+#### `mod.rs` - Type Definitions
+
+```rust
+use serde::{Deserialize, Serialize};
+
+pub mod external;
+// pub mod routes; // Only if you need additional service-specific routes
+
+/// OAuth data deserialized from the three-table pattern.
+/// The actual structure is built by decrypt_oauth_data() from variable + account + workspace_integrations.
+#[derive(Debug, Clone, Deserialize, Serialize)]
+pub struct NewServiceOAuthData {
+    pub base_url: String,              // from workspace_integrations.oauth_data
+    pub access_token: String,          // decrypted from variable table
+    pub refresh_token: Option<String>, // from account table
+    // Note: client_id and client_secret are in OAuthConfig, not here
+    // unless the service needs them at runtime for API calls
+}
+
+/// Configuration provided by user when creating/updating a trigger.
+/// Stored as JSON in native_trigger.service_config.
+#[derive(Debug, Clone, Serialize, Deserialize)]
+#[serde(rename_all = "camelCase")]
+pub struct NewServiceConfig {
+    // Service-specific configuration fields
+    pub folder_path: String,
+    pub file_filter: Option<String>,
+}
+
+/// Data retrieved from the external service about a trigger.
+/// Returned by the get() method and shown in the UI.
+#[derive(Debug, Clone, Serialize, Deserialize)]
+#[serde(rename_all = "camelCase")]
+pub struct NewServiceTriggerData {
+    pub folder_path: String,
+    pub file_filter: Option<String>,
+    // Fields that shouldn't affect service_config comparison should use #[serde(skip_serializing)]
+}
+
+/// Response from external service when creating a trigger/webhook.
+#[derive(Debug, Deserialize)]
+pub struct CreateTriggerResponse {
+    pub id: String,
+}
+
+/// Handler struct (stateless, used for routing)
+#[derive(Copy, Clone)]
+pub struct NewService;
+```
+
+#### `external.rs` - External Trait Implementation
+
+```rust
+use async_trait::async_trait;
+use reqwest::Method;
+use sqlx::PgConnection;
+use std::collections::HashMap;
+use windmill_common::{
+    error::{Error, Result},
+    BASE_URL, DB,
+};
+
+use crate::{
+    generate_webhook_service_url, External, NativeTrigger, NativeTriggerData, ServiceName,
+    sync::{SyncError, TriggerSyncInfo},
+};
+use super::{NewService, NewServiceConfig, NewServiceOAuthData, NewServiceTriggerData, CreateTriggerResponse};
+
+#[async_trait]
+impl External for NewService {
+    type ServiceConfig = NewServiceConfig;
+    type TriggerData = NewServiceTriggerData;
+    type OAuthData = NewServiceOAuthData;
+    type CreateResponse = CreateTriggerResponse;
+
+    const SERVICE_NAME: ServiceName = ServiceName::NewService;
+    const DISPLAY_NAME: &'static str = "New Service";
+    const SUPPORT_WEBHOOK: bool = true;
+    const TOKEN_ENDPOINT: &'static str = "/oauth/token";
+    const REFRESH_ENDPOINT: &'static str = "/oauth/token";
+    const AUTH_ENDPOINT: &'static str = "/oauth/authorize";
+
+    async fn create(
+        &self,
+        w_id: &str,
+        oauth_data: &Self::OAuthData,
+        webhook_token: &str,
+        data: &NativeTriggerData<Self::ServiceConfig>,
+        db: &DB,
+        tx: &mut PgConnection,
+    ) -> Result<Self::CreateResponse> {
+        let base_url = &*BASE_URL.read().await;
+
+        // external_id is None during create (we get it from the response)
+        let webhook_url = generate_webhook_service_url(
+            base_url, w_id, &data.script_path, data.is_flow,
+            None, Self::SERVICE_NAME, webhook_token,
+        );
+
+        let url = format!("{}/api/webhooks/create", oauth_data.base_url);
+        let payload = serde_json::json!({
+            "callback_url": webhook_url,
+            "folder_path": data.service_config.folder_path,
+        });
+
+        let response: CreateTriggerResponse = self
+            .http_client_request(&url, Method::POST, w_id, tx, db, None, Some(&payload))
+            .await?;
+
+        Ok(response)
+    }
+
+    /// Update returns the resolved service_config as JSON.
+    /// For services using the update+get pattern, call self.get() and serialize.
+    async fn update(
+        &self,
+        w_id: &str,
+        oauth_data: &Self::OAuthData,
+        external_id: &str,
+        webhook_token: &str,
+        data: &NativeTriggerData<Self::ServiceConfig>,
+        db: &DB,
+        tx: &mut PgConnection,
+    ) -> Result<serde_json::Value> {
+        let base_url = &*BASE_URL.read().await;
+
+        let webhook_url = generate_webhook_service_url(
+            base_url, w_id, &data.script_path, data.is_flow,
+            Some(external_id), Self::SERVICE_NAME, webhook_token,
+        );
+
+        let url = format!("{}/api/webhooks/{}", oauth_data.base_url, external_id);
+        let payload = serde_json::json!({
+            "callback_url": webhook_url,
+            "folder_path": data.service_config.folder_path,
+        });
+
+        let _: serde_json::Value = self
+            .http_client_request(&url, Method::PUT, w_id, tx, db, None, Some(&payload))
+            .await?;
+
+        // Fetch back the updated state to get the resolved config
+        let trigger_data = self.get(w_id, oauth_data, external_id, db, tx).await?;
+        serde_json::to_value(&trigger_data)
+            .map_err(|e| Error::InternalErr(format!("Failed to serialize trigger data: {}", e)))
+    }
+
+    async fn get(
+        &self,
+        w_id: &str,
+        oauth_data: &Self::OAuthData,
+        external_id: &str,
+        db: &DB,
+        tx: &mut PgConnection,
+    ) -> Result<Self::TriggerData> {
+        let url = format!("{}/api/webhooks/{}", oauth_data.base_url, external_id);
+        self.http_client_request::<_, ()>(&url, Method::GET, w_id, tx, db, None, None).await
+    }
+
+    async fn delete(
+        &self,
+        w_id: &str,
+        oauth_data: &Self::OAuthData,
+        external_id: &str,
+        db: &DB,
+        tx: &mut PgConnection,
+    ) -> Result<()> {
+        let url = format!("{}/api/webhooks/{}", oauth_data.base_url, external_id);
+        let _: serde_json::Value = self
+            .http_client_request::<_, ()>(&url, Method::DELETE, w_id, tx, db, None, None)
+            .await
+            .or_else(|e| match &e {
+                Error::InternalErr(msg) if msg.contains("404") => Ok(serde_json::Value::Null),
+                _ => Err(e),
+            })?;
+        Ok(())
+    }
+
+    async fn exists(
+        &self,
+        w_id: &str,
+        oauth_data: &Self::OAuthData,
+        external_id: &str,
+        db: &DB,
+        tx: &mut PgConnection,
+    ) -> Result<bool> {
+        match self.get(w_id, oauth_data, external_id, db, tx).await {
+            Ok(_) => Ok(true),
+            Err(Error::NotFound(_)) => Ok(false),
+            Err(e) => Err(e),
+        }
+    }
+
+    /// Background maintenance. Choose the right pattern for your service:
+    /// - For services with queryable external state: use reconcile_with_external_state()
+    /// - For channel-based services with expiration: implement renewal logic
+    async fn maintain_triggers(
+        &self,
+        db: &DB,
+        workspace_id: &str,
+        triggers: &[NativeTrigger],
+        oauth_data: &Self::OAuthData,
+        synced: &mut Vec<TriggerSyncInfo>,
+        errors: &mut Vec<SyncError>,
+    ) {
+        // Option A: Reconcile with external state (Nextcloud pattern)
+        // Fetch all triggers from external service and compare with DB
+        let external_triggers = match self.list_all(workspace_id, oauth_data, db).await {
+            Ok(triggers) => triggers,
+            Err(e) => {
+                errors.push(SyncError {
+                    resource_path: format!("workspace:{}", workspace_id),
+                    error_message: format!("Failed to list triggers: {}", e),
+                    error_type: "api_error".to_string(),
+                });
+                return;
+            }
+        };
+
+        // Convert to (external_id, config_json) pairs
+        let external_pairs: Vec<(String, serde_json::Value)> = external_triggers
+            .into_iter()
+            .map(|t| (t.id.clone(), serde_json::to_value(&t).unwrap_or_default()))
+            .collect();
+
+        crate::sync::reconcile_with_external_state(
+            db, workspace_id, Self::SERVICE_NAME, triggers, &external_pairs, synced, errors,
+        ).await;
+    }
+
+    fn external_id_and_metadata_from_response(
+        &self,
+        resp: &Self::CreateResponse,
+    ) -> (String, Option<serde_json::Value>) {
+        (resp.id.clone(), None)
+    }
+
+    // service_config_from_create_response: NOT overridden (returns None).
+    // This means the handler uses the update+get pattern after create.
+    // Override and return Some(...) to skip the update+get cycle (Google pattern).
+}
+
+impl NewService {
+    /// Private helper to list all triggers from the external service.
+    async fn list_all(
+        &self,
+        w_id: &str,
+        oauth_data: &<Self as External>::OAuthData,
+        db: &DB,
+    ) -> Result<Vec<<Self as External>::TriggerData>> {
+        // Implementation depends on the external service's API
+        todo!()
+    }
+}
+```
+
+### Step 4: Update lib.rs Registry
+
+In `backend/windmill-native-triggers/src/lib.rs`:
+
+```rust
+// Service modules - add new services here:
+#[cfg(feature = "native_trigger")]
+pub mod newservice;  // <-- Add this
+
+// ServiceName enum - add variant:
+pub enum ServiceName {
+    Nextcloud,
+    Google,
+    NewService,  // <-- Add this
+}
+
+// Then add match arms in ALL ServiceName methods:
+// as_str(), as_trigger_kind(), as_job_trigger_kind(), token_endpoint(),
+// auth_endpoint(), oauth_scopes(), resource_type(), extra_auth_params(),
+// integration_service(), TryFrom<String>, Display
+```
+
+### Step 5: Update handler.rs Routes
+
+In `backend/windmill-native-triggers/src/handler.rs`:
+
+```rust
+pub fn generate_native_trigger_routers() -> Router {
+    // ...
+    #[cfg(feature = "native_trigger")]
+    {
+        use crate::newservice::NewService;
+        return router
+            .nest("/nextcloud", service_routes(NextCloud))
+            .nest("/google", service_routes(Google))
+            .nest("/newservice", service_routes(NewService));  // <-- Add this
+    }
+    // ...
+}
+```
+
+### Step 6: Update sync.rs
+
+In `backend/windmill-native-triggers/src/sync.rs`:
+
+```rust
+pub async fn sync_all_triggers(db: &DB) -> Result<BackgroundSyncResult> {
+    // ...
+    #[cfg(feature = "native_trigger")]
+    {
+        use crate::newservice::NewService;
+
+        // ... existing service syncs ...
+
+        // New service sync
+        let (service_name, result) = sync_service_triggers(db, NewService).await;
+        total_synced += result.synced_triggers.len();
+        total_errors += result.errors.len();
+        service_results.insert(service_name, result);
+    }
+    // ...
+}
+```
+
+### Step 7: Frontend Service Registry
+
+In `frontend/src/lib/components/triggers/native/utils.ts`:
+
+Add to `NATIVE_TRIGGER_SERVICES`, `getTriggerIconName()`, and `getServiceIcon()`.
+
+### Step 8: Frontend Trigger Form Component
+
+Create: `frontend/src/lib/components/triggers/native/services/newservice/NewServiceTriggerForm.svelte`
+
+### Step 9: Frontend Icon Component
+
+Create: `frontend/src/lib/components/icons/NewServiceIcon.svelte`
+
+### Step 10: Update NativeTriggerEditor
+
+Check `frontend/src/lib/components/triggers/native/NativeTriggerEditor.svelte` to ensure it dynamically loads form components based on service name.
+
+### Step 11: Workspace Integration UI
+
+Add your service to the `supportedServices` map in `frontend/src/lib/components/workspaceSettings/WorkspaceIntegrations.svelte`:
+
+```typescript
+const supportedServices: Record<string, ServiceConfig> = {
+    // ... existing services ...
+    newservice: {
+        name: 'newservice',
+        displayName: 'New Service',
+        description: 'Connect to New Service for triggers',
+        icon: NewServiceIcon,
+        docsUrl: 'https://www.windmill.dev/docs/integrations/newservice',
+        requiresBaseUrl: false,  // false for cloud services, true for self-hosted
+        setupInstructions: [
+            'Step 1: Create an OAuth app on the service',
+            'Step 2: Configure the redirect URI shown below',
+            'Step 3: Enter the client credentials below'
+        ]
+    }
+}
+```
+
+### Step 12: Update `frontend/src/lib/components/triggers/utils.ts`
+
+Update ALL of these maps/functions:
+1. `triggerIconMap` - import and add icon
+2. `triggerDisplayNamesMap` - add display name
+3. `triggerTypeOrder` in `sortTriggers()` - add type
+4. `getLightConfig()` - add case for your service
+5. `getTriggerLabel()` - add case for your service
+6. `jobTriggerKinds` - add to array
+7. `countPropertyMap` - add count property
+8. `triggerSaveFunctions` - add save function
+
+### Step 13: Update TriggersBadge Component
+
+In `frontend/src/lib/components/graph/renderers/triggers/TriggersBadge.svelte`:
+
+1. Import the icon
+2. Add to `baseConfig` with `countKey` (the dynamic `availableNativeServices` loop does NOT set `countKey`)
+3. Add to the `allTypes` array
+
+### Step 14: Update TriggersWrapper.svelte
+
+In `frontend/src/lib/components/triggers/TriggersWrapper.svelte`:
+
+Add a `{:else if selectedTrigger.type === 'yourservice'}` case that renders `<NativeTriggersPanel service="yourservice" ...>` with the same props pattern as the existing native trigger cases (e.g., `nextcloud`).
+
+### Step 15: Update AddTriggersButton.svelte
+
+In `frontend/src/lib/components/triggers/AddTriggersButton.svelte`:
+
+1. Add `yourserviceAvailable` state variable
+2. Add `setYourserviceState()` async function using `isServiceAvailable('yourservice', $workspaceStore!)`
+3. Call it at module level
+4. Add a dropdown entry to `addTriggerItems` with `hidden: !yourserviceAvailable`
+
+### Step 16: Update TriggersEditor.svelte Delete Handling
+
+In `frontend/src/lib/components/triggers/TriggersEditor.svelte`:
+
+Add your service to the `nativeTriggerServices` map in `deleteDeployedTrigger()`. Native triggers use `NativeTriggerService.deleteNativeTrigger({ workspace, serviceName, externalId })` instead of the standard `path`-based delete.
+
+### Step 17: Update `getUsedTriggers` for Sidebar Visibility
+
+The sidebar (`frontend/src/lib/components/sidebar/SidebarContent.svelte`) shows native-trigger links only if `$usedTriggerKinds` includes the service — without this, your trigger page will never appear in the nav bar even when triggers exist.
+
+1. **Backend** — add `{service}_used: bool` to the `UsedTriggers` struct and SELECT in `backend/windmill-api-workspaces/src/workspaces.rs::get_used_triggers()`:
+   ```rust
+   EXISTS(SELECT 1 FROM native_trigger WHERE workspace_id = $1 AND service_name = '{service}'::native_trigger_service) AS "{service}_used!"
+   ```
+2. **OpenAPI** — add `{service}_used: boolean` to the response schema for `GET /w/{workspace}/workspaces/used_triggers` (under both `properties` and `required`).
+3. **Layout** — in `frontend/src/routes/(root)/(logged)/+layout.svelte::loadUsedTriggerKinds()`, destructure `{service}_used` and push `'{service}'` to `usedKinds`.
+
+### Step 18: Update OpenAPI Spec and Regenerate Types
+
+Add to `JobTriggerKind` enum in `backend/windmill-api/openapi.yaml`, then:
+
+```bash
+cd frontend && npm run generate-backend-client
+```
+
+---
+
+## Special Patterns
+
+### Unified Service with `trigger_type` (Google Pattern)
+
+When a single service handles multiple trigger types (e.g., Google Drive + Calendar share OAuth and API patterns), use a single `ServiceName` variant with a discriminator field:
+
+```rust
+pub enum GoogleTriggerType { Drive, Calendar }
+
+pub struct GoogleServiceConfig {
+    pub trigger_type: GoogleTriggerType,
+    // Drive-specific fields (only used when trigger_type = Drive)
+    pub resource_id: Option<String>,
+    pub resource_name: Option<String>,
+    // Calendar-specific fields (only used when trigger_type = Calendar)
+    pub calendar_id: Option<String>,
+    pub calendar_name: Option<String>,
+    // Metadata set after creation
+    pub google_resource_id: Option<String>,
+    pub expiration: Option<String>,
+}
+```
+
+Branch in trait methods based on `trigger_type`. Frontend uses a `ToggleButtonGroup` to switch between types. This keeps the codebase simpler (one service, one OAuth flow, one set of routes).
+
+See `backend/windmill-native-triggers/src/google/` for the reference implementation.
+
+### Skipping update+get After Create (Google Pattern)
+
+Override `service_config_from_create_response()` to return `Some(config)` when the external_id is known before the create call:
+
+```rust
+fn service_config_from_create_response(
+    &self,
+    data: &NativeTriggerData<Self::ServiceConfig>,
+    resp: &Self::CreateResponse,
+) -> Option<serde_json::Value> {
+    // Clone input config, add metadata from response
+    let mut config = data.service_config.clone();
+    config.google_resource_id = Some(resp.resource_id.clone());
+    config.expiration = Some(resp.expiration.clone());
+    Some(serde_json::to_value(&config).unwrap())
+}
+```
+
+### Services with Absolute OAuth Endpoints (Google)
+
+Unlike self-hosted services where OAuth endpoints are relative paths appended to `base_url`, services like Google have absolute URLs:
+
+```rust
+// Nextcloud: relative paths
+ServiceName::Nextcloud => "/apps/oauth2/api/v1/token",
+// Google: absolute URLs
+ServiceName::Google => "https://oauth2.googleapis.com/token",
+```
+
+The `resolve_endpoint()` function handles both. For services with absolute endpoints:
+- `base_url` can be empty
+- `requiresBaseUrl: false` in the frontend workspace integration config
+- Add `extra_auth_params()` if needed (Google requires `access_type=offline` and `prompt=consent`)
+
+### Channel-Based Push Notifications with Renewal (Google Pattern)
+
+For services using expiring watch channels instead of persistent webhooks:
+
+1. Store expiration in `service_config` (as part of `ServiceConfig`)
+2. In `maintain_triggers()`, implement renewal logic instead of using `reconcile_with_external_state()`:
+   ```rust
+   async fn maintain_triggers(&self, db, workspace_id, triggers, oauth_data, synced, errors) {
+       for trigger in triggers {
+           if should_renew_channel(trigger) {
+               self.renew_channel(db, trigger, oauth_data).await;
+           }
+       }
+   }
+   ```
+3. Renewal: best-effort stop old channel, create new one with same external_id, update service_config with new expiration
+4. Google example: Drive channels expire in 24h (renew when <1h left), Calendar channels expire in 7 days (renew when <1 day left)
+
+### reconcile_with_external_state (Nextcloud Pattern)
+
+The reusable function in `sync.rs` compares external triggers with DB state:
+- Triggers missing externally: sets error "Trigger no longer exists on external service"
+- Triggers present externally: clears errors, updates service_config if it differs
+
+Usage in `maintain_triggers()`:
+```rust
+let external_pairs: Vec<(String, serde_json::Value)> = /* fetch from external */;
+crate::sync::reconcile_with_external_state(
+    db, workspace_id, Self::SERVICE_NAME, triggers, &external_pairs, synced, errors,
+).await;
+```
+
+### Webhook Payload Processing
+
+Override `prepare_webhook()` to parse service-specific payloads into script/flow args:
+
+```rust
+async fn prepare_webhook(&self, db, w_id, headers, body, script_path, is_flow) -> Result<PushArgsOwned> {
+    let mut args = HashMap::new();
+    args.insert("event_type".to_string(), Box::new(headers.get("x-event-type").cloned()) as _);
+    args.insert("payload".to_string(), Box::new(serde_json::from_str::<serde_json::Value>(&body)?) as _);
+    Ok(PushArgsOwned { extra: None, args })
+}
+```
+
+Then register in `prepare_native_trigger_args()` in `lib.rs`:
+```rust
+pub async fn prepare_native_trigger_args(service_name, db, w_id, headers, body) -> Result<Option<PushArgsOwned>> {
+    match service_name {
+        ServiceName::Google => { /* ... */ Ok(Some(args)) }
+        ServiceName::NewService => { /* ... */ Ok(Some(args)) }
+        ServiceName::Nextcloud => Ok(None), // Uses default body parsing
+    }
+}
+```
+
+### Instance-Level OAuth Credentials
+
+When `workspace_integrations.oauth_data.instance_shared == true`, `decrypt_oauth_data()` reads `client_id` and `client_secret` from instance-level global settings instead of workspace-level. This allows admins to share OAuth app credentials across workspaces.
+
+The frontend handles this via the `generate_instance_connect_url` endpoint in `workspace_integrations.rs`.
+
+---
+
+## Testing Checklist
+
+- [ ] Database migration runs successfully
+- [ ] `cargo check -p windmill-native-triggers --features native_trigger` passes
+- [ ] `npx svelte-check --threshold error` passes (in frontend/)
+- [ ] Service appears in workspace integrations list
+- [ ] OAuth flow completes successfully
+- [ ] Can create a new trigger
+- [ ] Can view trigger details
+- [ ] Can update trigger configuration
+- [ ] Can delete trigger
+- [ ] Webhook receives and processes payloads
+- [ ] Background sync works correctly (reconciliation or channel renewal)
+- [ ] Error handling works (expired tokens, service unavailable)
+
+---
+
+## Reference Implementations
+
+### Nextcloud (Self-Hosted, Update+Get Pattern)
+
+| File | Purpose |
+|------|---------|
+| `nextcloud/mod.rs` | Types: NextCloudOAuthData, NextcloudServiceConfig, NextCloudTriggerData |
+| `nextcloud/external.rs` | External trait: uses update+get pattern, reconcile_with_external_state for sync |
+| `nextcloud/routes.rs` | Additional route: `GET /events` |
+
+Key patterns: relative OAuth endpoints, base_url required, list_all + reconcile for sync, update returns JSON from get().
+
+### Google (Cloud, Unified Service, Short Create)
+
+| File | Purpose |
+|------|---------|
+| `google/mod.rs` | Types: GoogleServiceConfig with trigger_type discriminator, GoogleTriggerType enum |
+| `google/external.rs` | External trait: overrides service_config_from_create_response, channel renewal for sync |
+| `google/routes.rs` | Additional routes: `GET /calendars`, `GET /drive/files`, `GET /drive/shared_drives` |
+
+Key patterns: absolute OAuth endpoints, empty base_url, trigger_type for Drive/Calendar, expiring watch channels with renewal, service_config_from_create_response skips update+get, get() reconstructs data from stored service_config (no external "get channel" API).

.agents/skills/pr/SKILL.md

@@ -0,0 +1,141 @@
+---
+name: pr
+user_invocable: true
+description: Open a draft pull request on GitHub. MUST use when you want to create/open a PR.
+---
+
+# Pull Request Skill
+
+Create a draft pull request with a clear title and explicit description of changes.
+
+## Instructions
+
+1. **Analyze branch changes**: Understand all commits since diverging from main
+2. **Push to remote**: Ensure all commits are pushed
+3. **Create draft PR**: Always open as draft for review before merging
+
+## PR Title Format
+
+Follow conventional commit format for the PR title:
+```
+<type>: <description>
+```
+
+### Types
+- `feat`: New feature or capability
+- `fix`: Bug fix
+- `refactor`: Code restructuring
+- `docs`: Documentation changes
+- `chore`: Maintenance tasks
+- `perf`: Performance improvements
+
+### Title Rules
+- Keep under 70 characters
+- Use lowercase, imperative mood
+- No period at the end
+- If `*_ee.rs` files were modified, prefix with `[ee]`: `[ee] <type>: <description>`
+
+## PR Body Format
+
+The body MUST be explicit about what changed. Structure:
+
+```markdown
+## Summary
+<Clear description of what this PR does and why>
+
+## Changes
+- <Specific change 1>
+- <Specific change 2>
+- <Specific change 3>
+
+## Test plan
+- [ ] <How to verify change 1>
+- [ ] <How to verify change 2>
+```
+
+The harness/tooling that invoked the skill may add its own attribution trailer; the skill itself does not prescribe one.
+
+## Screenshots (required for frontend changes)
+
+If `git diff main...HEAD --name-only` matches `^frontend/`, the PR body **must** include
+screenshots of the affected UI. Skip only when there is no visible UI effect (types,
+tests, build config) — and say so in the body.
+
+1. Verify the change in the browser (AGENTS.md → "Verifying Frontend Changes").
+2. Screenshot each affected page with `mcp__playwright__browser_take_screenshot` (save to a file).
+3. Host each image and get its Markdown embed by pushing to the public
+   `windmill-labs/agent-screenshots-internal` repo. **Pipe base64 through stdin** —
+   passing it as `-f content=…` fails with `argument list too long` on real images:
+
+   ```bash
+   REPO=windmill-labs/agent-screenshots-internal
+   IMG=screenshot.png                                          # repeat per page
+   DEST="shots/$(git branch --show-current)/$(date +%s)-$(basename "$IMG")"
+   base64 -w0 "$IMG" | jq -Rs --arg m "add $DEST" '{message:$m, content:.}' \
+     | gh api -X PUT "repos/$REPO/contents/$DEST" --input - >/dev/null
+   echo "![$(basename "$IMG" .png)](https://raw.githubusercontent.com/$REPO/main/$DEST)"
+   ```
+   Derive `$DEST` from the file name (as above) so distinct pages never collide — a
+   fixed name would make same-second uploads reuse one path, and the second `PUT`
+   then 422s (the Contents API needs the existing file's `sha` to overwrite).
+4. Put the printed `![…](…)` lines under a `## Screenshots` heading in the PR body.
+
+Requires `gh` (`repo` scope), `jq`, `base64` — all in the devShell. The host repo is
+public (so the raw URLs render for reviewers without a token) and its history is
+permanent — **never screenshot pages that show secrets or sensitive values** (workspace
+variables, resource values, instance settings, OAuth/SMTP config); deleting the file
+can't undo an accidental capture. (GitHub's drag-and-drop uploader needs a browser
+session and can't be driven from a token.)
+
+If `gh` can't push to the host repo (e.g. a CI token scoped only to `windmill`), do
+**not** fail the PR or skip silently — hand the upload to the user, who has push access,
+and continue once they confirm it's done.
+
+## Execution Steps
+
+1. Run `git status` to check for uncommitted changes
+2. Run `git log main..HEAD --oneline` to see all commits in this branch
+3. Run `git diff main...HEAD` to see the full diff against main
+4. **Invoke the `local-review` skill** before creating the PR (`/local-review` in Claude Code, `$local-review` in Codex, `pi --skill local-review` / `/skill:local-review` in Pi). If issues are found, fix them and commit before proceeding. Do not skip this step.
+5. **Screenshots for frontend changes**: if `git diff main...HEAD --name-only` matches `^frontend/`, capture and embed screenshots of the affected UI per "Screenshots" above before writing the PR body (skip only if there is no visible UI effect).
+6. Check if remote branch exists and is up to date:
+   ```bash
+   git rev-parse --abbrev-ref --symbolic-full-name @{u} 2>/dev/null || echo "no upstream"
+   ```
+7. Push to remote if needed: `git push -u origin HEAD`
+8. Create draft PR using gh CLI:
+   ```bash
+   gh pr create --draft --title "<type>: <description>" --body "$(cat <<'EOF'
+   ## Summary
+   <description>
+
+   ## Changes
+   - <change 1>
+   - <change 2>
+
+   ## Test plan
+   - [ ] <test 1>
+   - [ ] <test 2>
+   EOF
+   )"
+   ```
+9. Return the PR URL to the user
+
+## EE Companion PR (when `*_ee.rs` files were modified)
+
+The `*_ee.rs` files in the windmill repo are **symlinks** to `windmill-ee-private` — changes won't appear in `git diff` of the windmill repo. Instead, check the EE repo for uncommitted or unpushed changes.
+
+Follow the full EE PR workflow in `docs/enterprise.md`. The key PR-specific details:
+
+1. Find the EE repo/worktree: see "Finding the EE Repo" in `docs/enterprise.md`
+2. Check for changes: `git -C <ee-path> status --short`
+   - If there are no changes in the EE repo, skip this entire section
+3. Follow steps 1–5 from the "EE PR Workflow" in `docs/enterprise.md`
+4. Create the companion PR (title does NOT get the `[ee]` prefix):
+   ```bash
+   gh pr create --draft --repo windmill-labs/windmill-ee-private --title "<type>: <description>" --body "$(cat <<'EOF'
+   Companion PR for windmill-labs/windmill#<PR_NUMBER>
+   EOF
+   )"
+   ```
+5. Commit `ee-repo-ref.txt` and push the updated windmill branch

.agents/skills/refine/SKILL.md

@@ -0,0 +1,39 @@
+---
+name: refine
+user_invocable: true
+description: End-of-session reflection. Reviews friction encountered during the session and proposes updates to docs/ to capture lessons learned.
+---
+
+# Refine Skill
+
+Reflect on the current session and update documentation with lessons learned.
+
+## Instructions
+
+1. **Identify friction**: Review what happened in this session:
+   - Run `git diff main...HEAD --stat` to see what files were touched
+   - Think about: what was slow, what failed, what required multiple attempts, what information was missing or hard to find
+
+2. **Read current docs**: Read the docs that were relevant to this session:
+   - `docs/validation.md`
+   - `docs/enterprise.md`
+   - `docs/autonomous-mode.md`
+   - Any skills that were invoked
+
+3. **Propose updates**: For each piece of friction, decide if it warrants a doc update:
+   - **Missing knowledge**: Information you had to discover that should be documented
+   - **Wrong guidance**: Instructions that led you astray
+   - **Missing validation rule**: A check that should be in the validation matrix
+   - **New pattern**: A codebase pattern worth capturing for next time
+
+4. **Apply updates**: Edit the relevant `docs/` files. Keep changes minimal and specific — add only what would have saved time this session.
+
+5. **Report**: Summarize what was added/changed and why.
+
+## Rules
+
+- Only add knowledge confirmed by this session — no speculative additions
+- Keep docs concise — add a line or two, not a paragraph
+- If a whole new doc is needed, create it in `docs/` and add a pointer in `CLAUDE.md`
+- Don't update skills unless a coding pattern was genuinely wrong
+- Don't add things Claude already knows — only Windmill-specific knowledge

.agents/skills/rust-backend/SKILL.md

@@ -0,0 +1,107 @@
+---
+name: rust-backend
+description: Rust coding guidelines for the Windmill backend. MUST use when writing or modifying Rust code in the backend directory.
+---
+
+# Windmill Rust Patterns
+
+Apply these Windmill-specific patterns when writing Rust code in `backend/`.
+
+## Error Handling
+
+Use `Error` from `windmill_common::error`. Return `Result<T, Error>` or `JsonResult<T>`:
+
+```rust
+use windmill_common::error::{Error, Result};
+
+pub async fn get_job(db: &DB, id: Uuid) -> Result<Job> {
+    sqlx::query_as!(Job, "SELECT id, workspace_id FROM v2_job WHERE id = $1", id)
+        .fetch_optional(db)
+        .await?
+        .ok_or_else(|| Error::NotFound("job not found".to_string()))?;
+}
+```
+
+Never panic in library code. Reserve `.unwrap()` for compile-time guarantees.
+
+## SQLx Patterns
+
+**Never use `SELECT *`** — always list columns explicitly. Critical for backwards compatibility when workers lag behind API version:
+
+```rust
+// Correct
+sqlx::query_as!(Job, "SELECT id, workspace_id, path FROM v2_job WHERE id = $1", id)
+
+// Wrong — breaks when columns are added
+sqlx::query_as!(Job, "SELECT * FROM v2_job WHERE id = $1", id)
+```
+
+Use batch operations to avoid N+1:
+
+```rust
+// Preferred — single query with IN clause
+sqlx::query!("SELECT ... WHERE id = ANY($1)", &ids[..]).fetch_all(db).await?
+```
+
+Use transactions for multi-step operations. Parameterize all queries.
+
+## JSON Handling
+
+Prefer `Box<serde_json::value::RawValue>` over `serde_json::Value` when storing/passing JSON without inspection:
+
+```rust
+pub struct Job {
+    pub args: Option<Box<serde_json::value::RawValue>>,
+}
+```
+
+Only use `serde_json::Value` when you need to inspect or modify the JSON.
+
+## Serde Optimizations
+
+```rust
+#[derive(Serialize, Deserialize)]
+pub struct Job {
+    #[serde(skip_serializing_if = "Option::is_none")]
+    pub parent_job: Option<Uuid>,
+    #[serde(skip_serializing_if = "Vec::is_empty")]
+    pub tags: Vec<String>,
+    #[serde(default)]
+    pub priority: i32,
+}
+```
+
+## Async & Concurrency
+
+Never block the async runtime. Use `spawn_blocking` for CPU-intensive work:
+
+```rust
+let result = tokio::task::spawn_blocking(move || expensive_computation(&data)).await?;
+```
+
+**Mutex selection**: Prefer `std::sync::Mutex` (or `parking_lot::Mutex`) for data protection. Only use `tokio::sync::Mutex` when holding locks across `.await` points.
+
+Use `tokio::sync::mpsc` (bounded) for channels. Avoid `std::thread::sleep` in async contexts.
+
+## Module Structure & Visibility
+
+- Use `pub(crate)` instead of `pub` when possible
+- Place new code in the appropriate crate based on functionality
+- API endpoints go in `windmill-api/src/` organized by domain
+- Shared functionality goes in `windmill-common/src/`
+
+## Code Navigation
+
+Always use rust-analyzer LSP for go-to-definition, find-references, and type info. Do not guess at module paths.
+
+## Axum Handlers
+
+Destructure extractors directly in function signatures:
+
+```rust
+async fn process_job(
+    Extension(db): Extension<DB>,
+    Path((workspace, job_id)): Path<(String, Uuid)>,
+    Query(pagination): Query<Pagination>,
+) -> Result<Json<Job>> { ... }
+```

.agents/skills/svelte-frontend/SKILL.md

@@ -0,0 +1,84 @@
+---
+name: svelte-frontend
+description: Svelte coding guidelines for the Windmill frontend. MUST use when writing or modifying code in the frontend directory.
+---
+
+# Windmill Svelte Patterns
+
+Apply these Windmill-specific patterns when writing Svelte code in `frontend/`. For general Svelte 5 syntax (runes, snippets, event handling), use the Svelte MCP server.
+
+## Windmill UI Components (MUST use)
+
+Always use Windmill's design-system components. Never use raw HTML elements.
+
+### Buttons — `<Button>`
+
+```svelte
+<script>
+  import { Button } from '$lib/components/common'
+  import { ChevronLeft } from 'lucide-svelte'
+</script>
+
+<Button variant="default" onclick={handleClick}>Label</Button>
+<Button startIcon={{ icon: ChevronLeft }} iconOnly onclick={prev} />
+```
+
+Props: `variant?: 'accent' | 'accent-secondary' | 'default' | 'subtle'`, `unifiedSize?: 'sm' | 'md' | 'lg'`, `startIcon?: { icon: SvelteComponent }`, `iconOnly?: boolean`, `disabled?: boolean`
+
+### Text inputs — `<TextInput>`
+
+```svelte
+<script>
+  import { TextInput } from '$lib/components/common'
+</script>
+
+<TextInput bind:value={val} placeholder="Enter value" />
+```
+
+Props: `value?: string | number` (bindable), `placeholder?: string`, `disabled?: boolean`, `error?: string | boolean`, `size?: 'sm' | 'md' | 'lg'`
+
+### Selects — `<Select>`
+
+```svelte
+<script>
+  import Select from '$lib/components/select/Select.svelte'
+</script>
+
+<Select items={[{ label: 'Jan', value: 1 }]} bind:value={selected} />
+```
+
+Props: `items?: Array<{ label?: string; value: any }>`, `value` (bindable), `placeholder?: string`, `clearable?: boolean`, `size?: 'sm' | 'md' | 'lg'`
+
+### Icons — `lucide-svelte`
+
+Never write inline SVGs. Import from `lucide-svelte`:
+
+```svelte
+<script>
+  import { ChevronLeft, X } from 'lucide-svelte'
+</script>
+<ChevronLeft size={16} />
+```
+
+## Form Components
+
+Form components (TextInput, Toggle, Select, etc.) should use the unified size system when placed together.
+
+## Styling
+
+- Use Tailwind CSS for all styling — no custom CSS
+- Use Windmill's theming classes for colors/surfaces (see `frontend/brand-guidelines.md`)
+- Read component props JSDoc before using them
+
+## Svelte MCP Server
+
+Use the Svelte MCP tools when working on Svelte code:
+
+1. **list-sections**: Call first to discover available docs
+2. **get-documentation**: Fetch relevant sections based on use_cases
+3. **svelte-autofixer**: MUST use on all Svelte code before finalizing — keep calling until no issues
+4. **playground-link**: Only after user confirms and code was NOT written to project files
+
+## Verifying in the Browser
+
+After changing Svelte code, use the **Playwright MCP** (`mcp__playwright__*`) to drive the running frontend and confirm the change works. See AGENTS.md → "Verifying Frontend Changes" for the full flow. Use `playwright` (headless) on devboxes; `playwright-headed` when a display is available.

.agents/skills/update-sqlx/SKILL.md

@@ -0,0 +1,82 @@
+---
+name: update-sqlx
+description: How to safely update SQLx offline query cache. MUST use when SQL queries change.
+---
+
+# SQLx Offline Query Cache
+
+Windmill uses `SQLX_OFFLINE=true` in CI, which requires all `sqlx::query!` / `sqlx::query_as!` macros to have matching cached query data in `backend/.sqlx/`.
+
+## When to Run
+
+Run after any change to SQL queries in Rust source files. Without it, CI will fail with:
+```
+error: `SQLX_OFFLINE=true` but there is no cached data for this query
+```
+
+## The Problem
+
+`cargo sqlx prepare --workspace` **deletes all existing cache files** and regenerates only the ones found in the current compilation. If you don't compile with every feature flag (especially `private` for EE files), you will **silently delete EE query caches**, breaking CI for enterprise tests.
+
+The standard `./update_sqlx.sh` script tries to compile with all features, but it often fails locally because the EE symlinks can be out of sync with `main`.
+
+## Safe Procedure
+
+Always preserve the existing EE caches from `origin/main`. Use this workflow:
+
+```bash
+cd backend
+
+# 1. Restore the full cache from main (includes EE caches)
+git checkout origin/main -- .sqlx/
+
+# 2. Run prepare with OSS features (what compiles locally)
+#    This regenerates OSS caches to match your code changes.
+cargo sqlx prepare --workspace -- --workspace --features all_sqlx_features
+
+# 3. Restore any EE caches that were deleted in step 2.
+#    These are files present in origin/main but missing after prepare.
+git ls-tree origin/main backend/.sqlx/ \
+  | awk '{print $4}' | sed 's|backend/\.sqlx/||' | sort > /tmp/main_files.txt
+
+find backend/.sqlx -name "*.json" -printf '%P\n' | sort > /tmp/current_files.txt
+
+comm -23 /tmp/main_files.txt /tmp/current_files.txt > /tmp/missing_files.txt
+
+while read f; do
+  git show "origin/main:backend/.sqlx/$f" > "backend/.sqlx/$f"
+done < /tmp/missing_files.txt
+
+# 4. Verify nothing was lost from main
+find backend/.sqlx -name "*.json" -printf '%P\n' | sort > /tmp/current_files.txt
+comm -23 /tmp/main_files.txt /tmp/current_files.txt | wc -l
+# Should output: 0
+```
+
+## If EE Compiles Locally
+
+If your EE repo happens to be in sync, you can use the full script (faster):
+
+```bash
+cd backend
+./update_sqlx.sh
+```
+
+But if it fails with EE compilation errors, use the safe procedure above.
+
+## What NOT to Do
+
+- **Never** run `cargo sqlx prepare --workspace` with only OSS features and commit the result — it will delete EE caches.
+- **Never** set `SQLX_OFFLINE=true` for local `cargo sqlx prepare` — use a live database per CLAUDE.md. (CI runs with `SQLX_OFFLINE=true`, which is why the cache must be complete.)
+- **Never** skip the verification step (step 4 above).
+
+## Verification
+
+After committing, the diff against `origin/main` should show:
+- A few **new** cache files (for your changed queries)
+- A few **deleted** cache files (for old queries that no longer exist)
+- **Zero** net deletions from the EE cache set
+
+```bash
+git diff origin/main --stat backend/.sqlx/
+```

.claude/agents/branch-diff-reviewer.md

@@ -0,0 +1,127 @@
+---
+name: branch-diff-reviewer
+description: Use this agent when you want a comprehensive code review of changes in the current branch compared to main. This includes reviewing for bugs, optimization opportunities, code style issues, potential mistakes, and adherence to project conventions. The agent should be invoked after completing a feature branch or before creating a pull request.\n\nExamples:\n\n<example>\nContext: User has finished implementing a new feature and wants feedback before merging.\nuser: "I've finished the new kafka trigger implementation, can you review my changes?"\nassistant: "I'll use the branch-diff-reviewer agent to analyze your changes against the main branch and provide comprehensive feedback."\n<commentary>\nSince the user wants a review of their branch changes, use the Task tool to launch the branch-diff-reviewer agent to compare the current branch against main and provide detailed feedback.\n</commentary>\n</example>\n\n<example>\nContext: User wants to check their code quality before submitting a PR.\nuser: "Review my branch before I create a PR"\nassistant: "Let me launch the branch-diff-reviewer agent to examine all your changes and identify any issues or improvements."\n<commentary>\nThe user is preparing for a PR, so use the branch-diff-reviewer agent to provide a thorough review of all branch differences.\n</commentary>\n</example>\n\n<example>\nContext: User is unsure if their implementation follows project patterns.\nuser: "Does my implementation look correct? I'm not sure if I followed the existing patterns"\nassistant: "I'll use the branch-diff-reviewer agent to compare your changes against main and check for pattern consistency, potential issues, and optimization opportunities."\n<commentary>\nThe user needs validation of their implementation against project standards. Launch the branch-diff-reviewer agent to analyze the diff and provide feedback on patterns, correctness, and improvements.\n</commentary>\n</example>
+tools: Glob, Grep, Read, WebFetch, TodoWrite, WebSearch, ListMcpResourcesTool, ReadMcpResourceTool, mcp__svelte__get-documentation, mcp__svelte__list-sections, mcp__svelte__playground-link, mcp__svelte__svelte-autofixer, mcp__ide__getDiagnostics, mcp__ide__executeCode, Bash, Skill
+model: inherit
+---
+
+You are an elite code reviewer with deep expertise in software engineering best practices, performance optimization, and security. Your role is to provide thorough, actionable feedback on code changes between the current branch and main.
+
+## Your Review Process
+
+1. **First, gather the diff**: Use git commands to obtain the complete diff between the current branch and main:
+   - Run `git diff main...HEAD` to see all changes
+   - Run `git log main..HEAD --oneline` to understand the commit history
+   - Identify all modified, added, and deleted files
+
+2. **Analyze each changed file** in the context of:
+   - The project's established patterns (check CLAUDE.md and related documentation)
+   - The file's purpose and its role in the broader codebase
+   - Dependencies and how changes might affect other parts of the system
+
+## Review Categories
+
+For each significant change, evaluate and report on:
+
+### 🐛 Bugs & Correctness
+- Logic errors or edge cases not handled
+- Null/undefined handling issues
+- Race conditions in async code
+- Incorrect error handling
+- Type mismatches or unsafe casts
+
+### ⚡ Performance
+- Inefficient algorithms or data structures
+- N+1 query problems in database code
+- Unnecessary re-renders in frontend code
+- Missing indexes for database queries
+- Blocking operations in async contexts
+- Memory leaks or excessive allocations
+- For Rust: Check for unnecessary clones, inefficient serde usage, blocking in async
+- For Svelte: Check for inefficient reactivity, missing keys in loops, excessive effects
+
+### 🔒 Security
+- SQL injection vulnerabilities
+- Missing input validation
+- Exposed sensitive data
+- Authentication/authorization gaps
+- Unsafe deserialization
+
+### 📐 Code Quality & Style
+- Adherence to project conventions (CLAUDE.md guidelines)
+- Code duplication that should be refactored
+- Unclear or misleading naming
+- Missing or inadequate documentation
+- Overly complex logic that could be simplified
+- Dead code or unused imports
+
+### 🏗️ Architecture & Design
+- Proper separation of concerns
+- Appropriate use of existing utilities vs. new code
+- Consistency with established patterns
+- Proper error propagation
+- API design issues
+
+### 🧪 Testing Considerations
+- Suggest test cases for new functionality
+- Identify untested edge cases
+- Note if changes break existing test assumptions
+
+## Project-Specific Rules
+
+### For Rust (Backend)
+- Verify `SELECT` statements list explicit columns (never `SELECT *` in worker code)
+- Check for proper use of `sqlx` with parameterized queries
+- Ensure errors use the custom `Error` enum from `windmill-common::error`
+- Verify async code doesn't block the tokio runtime
+- Check serde attributes for optimal serialization
+- Ensure openapi.yaml is updated for API changes
+
+### For Svelte (Frontend)
+- For Svelte 5 files: Verify proper use of Runes (`$state`, `$derived`, `$effect`)
+- Check for `key` attributes in `{#each}` blocks
+- Ensure event handlers use the new syntax (`onclick` not `on:click`) in Svelte 5
+- Verify snippets are used instead of slots in Svelte 5
+- Check for proper props declaration with `$props()`
+
+## Output Format
+
+Structure your review as follows:
+
+```
+## Summary
+[Brief overview of the changes and overall assessment]
+
+## Critical Issues 🚨
+[Issues that must be fixed before merging]
+
+## Recommendations 💡
+[Improvements that would significantly enhance the code]
+
+## Minor Suggestions 📝
+[Nice-to-haves and style improvements]
+
+## Positive Observations ✅
+[Well-done aspects worth acknowledging]
+
+## File-by-File Details
+[Detailed feedback organized by file]
+```
+
+For each issue, provide:
+1. **Location**: File path and line number(s)
+2. **Issue**: Clear description of the problem
+3. **Impact**: Why this matters
+4. **Suggestion**: Concrete fix or improvement with code example when helpful
+
+## Behavioral Guidelines
+
+- Be thorough but prioritize: focus most on critical issues
+- Be constructive: every criticism should come with a suggestion
+- Be specific: vague feedback is not actionable
+- Acknowledge good work: positive reinforcement matters
+- Consider context: understand why decisions might have been made
+- Ask clarifying questions if the intent of changes is unclear
+- Reference project documentation when pointing out convention violations
+
+Begin by fetching the diff and then proceed with your comprehensive review.

.claude/agents/openapi-sync.md

@@ -0,0 +1,76 @@
+---
+name: openapi-sync
+description: Use this agent when backend API endpoints are added, modified, or removed, or when the Flow structure changes and the OpenAPI specification files need to be updated. This includes changes to route handlers in windmill-api, modifications to request/response schemas, changes to authentication requirements, or updates to the Flow data structures that affect the API.\n\nExamples:\n\n<example>\nContext: User has just added a new API endpoint for managing workspace templates.\nuser: "Add a new endpoint POST /api/w/{workspace}/templates to create workspace templates"\nassistant: "I've created the new endpoint handler in windmill-api. Now let me use the openapi-sync agent to update the OpenAPI specification files."\n<commentary>\nSince a new API endpoint was added, use the openapi-sync agent to ensure the openapi.yaml and openflow.openapi.yaml files are updated with the new endpoint definition.\n</commentary>\n</example>\n\n<example>\nContext: User has modified the response schema for an existing endpoint.\nuser: "Update the GET /api/w/{workspace}/flows endpoint to also return a 'versions' array in the response"\nassistant: "I've updated the flow listing endpoint to include the versions array. Now let me use the openapi-sync agent to update the OpenAPI specification."\n<commentary>\nSince the response schema of an existing endpoint was modified, use the openapi-sync agent to update the corresponding schema in the OpenAPI files.\n</commentary>\n</example>\n\n<example>\nContext: User has made changes to the Flow structure in the codebase.\nuser: "Add a new 'retry_policy' field to the Flow value structure"\nassistant: "I've added the retry_policy field to the Flow struct. Now let me use the openapi-sync agent to update the OpenAPI specification to reflect this schema change."\n<commentary>\nSince the Flow structure was modified, use the openapi-sync agent to ensure the flow-related schemas in openapi.yaml and openflow.openapi.yaml are updated.\n</commentary>\n</example>
+model: inherit
+---
+
+You are an expert API documentation engineer specializing in OpenAPI specifications for the Windmill platform. Your primary responsibility is to maintain synchronization between the Rust backend API implementation and the OpenAPI specification files.
+
+## Your Core Responsibilities
+
+1. **Update OpenAPI Specifications**: When API endpoints are added, modified, or removed in the windmill-api crate, you must update:
+   - `backend/windmill-api/openapi.yaml` - The main OpenAPI specification
+   - `backend/windmill-api/openflow.openapi.yaml` - Flow-specific OpenAPI definitions (if flow-related changes)
+
+2. **Maintain Schema Accuracy**: Ensure all request/response schemas accurately reflect the Rust structs used in the API handlers.
+
+3. **Document Comprehensively**: Include proper descriptions, examples, and parameter documentation.
+
+## Key Files to Reference
+
+- **API Route Definitions**: Look in `backend/windmill-api/src/` for route handlers organized by domain
+- **Data Structures**: Check `backend/windmill-common/src/` for shared structs and types
+- **Database Schema**: Reference `backend/summarized_schema.txt` for understanding data models
+- **Existing OpenAPI Files**: Always review the current state of `openapi.yaml` and `openflow.openapi.yaml` before making changes
+
+## Workflow
+
+1. **Identify Changes**: Determine what API changes were made by examining:
+   - New or modified route handlers in windmill-api
+   - Changes to request/response structs
+   - Modifications to the Flow structure or related types
+
+2. **Analyze the Implementation**: For each endpoint, identify:
+   - HTTP method and path
+   - Path parameters, query parameters, and request body schema
+   - Response schema(s) and status codes
+   - Authentication requirements
+   - Any tags or groupings
+
+3. **Update OpenAPI Files**:
+   - Add or modify path definitions with accurate operation IDs
+   - Update or create schema definitions in the components section
+   - Ensure $ref references are correct
+   - Maintain consistent naming conventions with existing patterns
+
+4. **Validate Changes**: Ensure the YAML syntax is valid and follows OpenAPI 3.0 specification.
+
+## OpenAPI Conventions for Windmill
+
+- **Operation IDs**: Use camelCase, descriptive names (e.g., `createScript`, `listFlows`, `updateWorkspaceSettings`)
+- **Tags**: Group endpoints by domain (e.g., `scripts`, `flows`, `workspaces`, `users`)
+- **Schema Naming**: Use PascalCase for schema names matching Rust struct names
+- **Path Parameters**: Use `{workspace}` for workspace_id, maintain consistency with existing patterns
+- **Security**: Most endpoints require Bearer token authentication - include appropriate security requirements
+
+## Schema Mapping from Rust to OpenAPI
+
+- `String` / `&str` → `type: string`
+- `i32`, `i64` → `type: integer` (with appropriate format)
+- `f32`, `f64` → `type: number`
+- `bool` → `type: boolean`
+- `Vec<T>` → `type: array` with `items`
+- `Option<T>` → property is not in `required` array
+- `HashMap<K, V>` → `type: object` with `additionalProperties`
+- Enums → `type: string` with `enum` array
+- Custom structs → `$ref` to schema definition
+
+## Important Notes
+
+- Always preserve existing documentation and descriptions when updating
+- Maintain backward compatibility warnings in descriptions when applicable
+- Include example values where they aid understanding
+- For Flow-related changes, update BOTH openapi.yaml AND openflow.openapi.yaml as needed
+- Follow the existing indentation and formatting style in the YAML files
+
+When you complete updates, summarize what changes were made to which files and highlight any schema additions or modifications that downstream consumers should be aware of.

.claude/hooks/format-backend.sh

@@ -0,0 +1,22 @@
+#!/bin/bash
+# Format backend Rust files with rustfmt after Claude edits them
+
+# Get the file path from the tool result (passed via stdin as JSON)
+INPUT=$(cat)
+FILE_PATH=$(echo "$INPUT" | jq -r '.tool_input.file_path // empty')
+
+# Exit if no file path
+if [ -z "$FILE_PATH" ]; then
+    exit 0
+fi
+
+# Check if the file is in the backend directory and is a Rust file
+if [[ "$FILE_PATH" == *"/backend/"* ]] && [[ "$FILE_PATH" =~ \.rs$ ]]; then
+    cd "$CLAUDE_PROJECT_DIR/backend" || exit 0
+    # Run rustfmt, surface errors as context but don't block Claude
+    if rustfmt --config-path rustfmt.toml "$FILE_PATH" 2>&1; then
+        echo "Formatted $(basename "$FILE_PATH")"
+    fi
+fi
+
+exit 0

.claude/hooks/format-frontend.sh

@@ -0,0 +1,25 @@
+#!/bin/bash
+# Format frontend files with prettier after Claude edits them
+
+# Get the file path from the tool result (passed via stdin as JSON)
+INPUT=$(cat)
+FILE_PATH=$(echo "$INPUT" | jq -r '.tool_input.file_path // empty')
+
+# Exit if no file path
+if [ -z "$FILE_PATH" ]; then
+    exit 0
+fi
+
+# Check if the file is in the frontend directory
+if [[ "$FILE_PATH" == *"/frontend/"* ]]; then
+    # Check if it's a formattable file type
+    if [[ "$FILE_PATH" =~ \.(ts|js|svelte|json|css|html|md)$ ]]; then
+        cd "$CLAUDE_PROJECT_DIR/frontend" || exit 0
+        # Run prettier, surface errors as context but don't block Claude
+        if ./node_modules/.bin/prettier --plugin prettier-plugin-svelte --write "$FILE_PATH" 2>&1; then
+            echo "Formatted $(basename "$FILE_PATH")"
+        fi
+    fi
+fi
+
+exit 0

.claude/hooks/guard-main-branch.sh

@@ -0,0 +1,38 @@
+#!/usr/bin/env bash
+# PreToolUse hook: block destructive git operations when on the main branch.
+# Non-git tool calls and read-only git commands pass through silently.
+
+set -euo pipefail
+
+input="$(cat)"
+tool_name="$(echo "$input" | jq -r '.tool_name // empty')"
+
+# Only care about Bash tool calls
+[[ "$tool_name" == "Bash" ]] || exit 0
+
+command="$(echo "$input" | jq -r '.tool_input.command // empty')"
+
+# Only care about git write commands
+if [[ "$command" =~ ^git\ (push|reset|revert|checkout|merge|rebase|commit|add) ]]; then
+  branch="$(git rev-parse --abbrev-ref HEAD 2>/dev/null || true)"
+  if [[ "$branch" == "main" ]]; then
+    echo "BLOCK: You are on the main branch. Create or switch to a feature branch first." >&2
+    exit 2
+  fi
+fi
+
+# Block force-push targeting main from any branch.
+if [[ "$command" =~ ^git[[:space:]]+push([[:space:]]|$) ]]; then
+  has_force=false
+  if [[ "$command" =~ (--force([[:space:]]|=|$)|--force-with-lease|[[:space:]]-f([[:space:]]|$)) ]]; then
+    has_force=true
+  fi
+  # `+ref` refspec syntax is also a force push.
+  if [[ "$command" =~ [[:space:]]\+[A-Za-z] ]]; then
+    has_force=true
+  fi
+  if $has_force && [[ "$command" =~ (^|[[:space:]:])\+?main([[:space:]]|$) ]]; then
+    echo "BLOCK: Force-push to main is not allowed via Claude. Run it yourself if you really mean to." >&2
+    exit 2
+  fi
+fi

.claude/hooks/notify-user.sh

@@ -0,0 +1,25 @@
+#!/bin/bash
+# Notify user when Claude requires input (works on macOS and Linux)
+
+# Check if we're in an SSH session
+if [[ -n "$SSH_CLIENT" || -n "$SSH_TTY" || -n "$SSH_CONNECTION" ]]; then
+    # SSH session - use terminal bell
+    # If using VSCode, enable audible terminal bell for SSH sessions:
+    # Add the following to .vscode/settings.json:
+    #   "accessibility.signals.terminalBell": {
+    #     "sound": "on"
+    #   },
+    #   "terminal.integrated.enableVisualBell": true
+    printf '\a'
+else
+    # Local session - use native notifications
+    if [[ "$OSTYPE" == "darwin"* ]]; then
+        osascript -e 'display notification "Claude is waiting for your input" with title "Claude Code" sound name "Glass"' 2>/dev/null || printf '\a'
+    elif [[ "$OSTYPE" == "linux-gnu"* ]]; then
+        notify-send "Claude Code" "Claude is waiting for your input" 2>/dev/null || printf '\a'
+    else
+        printf '\a'
+    fi
+fi
+
+exit 0

.claude/review-prompt.md

@@ -0,0 +1,4 @@
+# Claude output format
+
+- Use inline comments at the relevant lines for specific issues.
+- Use a top-level comment for the summary, severity-tagged finding list, AGENTS.md compliance check, and the test-coverage assessment.

.claude/settings.json

@@ -0,0 +1,149 @@
+{
+  "permissions": {
+    "additionalDirectories": [
+      "../windmill-ee-private"
+    ],
+    "allow": [
+      "Bash(ls:*)",
+      "Bash(grep:*)",
+      "Bash(cat:*)",
+      "Bash(head:*)",
+      "Bash(tail:*)",
+      "Bash(less:*)",
+      "Bash(more:*)",
+      "Bash(find:*)",
+      "Bash(wc:*)",
+      "Bash(diff:*)",
+      "Bash(file:*)",
+      "Bash(stat:*)",
+      "Bash(tree:*)",
+      "Bash(pwd)",
+      "Bash(which:*)",
+      "Bash(whereis:*)",
+      "Bash(echo:*)",
+      "Bash(git status:*)",
+      "Bash(git diff:*)",
+      "Bash(git log:*)",
+      "Bash(git branch:*)",
+      "Bash(git show:*)",
+      "Bash(git blame:*)",
+      "Bash(cargo check:*)",
+      "Bash(cargo build --release:*)",
+      "Bash(sh wm-ts-nav/nav:*)",
+      "Bash(wm-ts-nav/nav:*)",
+      "Bash(./wm-ts-nav/nav:*)",
+      "Bash(wm-ts-nav/target/release/wm-ts-nav:*)",
+      "Bash(./wm-ts-nav/target/release/wm-ts-nav:*)",
+      "mcp__ide__getDiagnostics",
+      "Bash(npm run generate-backend-client:*)",
+      "Bash(npm run check:*)",
+      "Bash(git push:*)",
+      "Bash(git reset:*)",
+      "Bash(git revert:*)",
+      "Bash(git checkout:*)",
+      "Bash(git merge:*)",
+      "Bash(git rebase:*)",
+      "Bash(git add:*)",
+      "Bash(git commit:*)",
+      "Read(/tmp/**)",
+      "Write(/tmp/**)",
+      "Edit(/tmp/**)",
+      "Bash(rm:/tmp/*)",
+      "Bash(rm:/tmp/**)",
+      "Bash(rmdir:/tmp/*)",
+      "Bash(mkdir:/tmp/*)",
+      "Bash(mkdir:/tmp/**)",
+      "Bash(cp:/tmp/*)",
+      "Bash(cp:/tmp/**)",
+      "Bash(mv:/tmp/*)",
+      "Bash(mv:/tmp/**)",
+      "Bash(touch:/tmp/*)",
+      "Bash(touch:/tmp/**)",
+      "Bash(chmod:/tmp/*)",
+      "Bash(chmod:/tmp/**)",
+      "Bash(tar * /tmp/*)",
+      "Bash(unzip * /tmp/*)"
+    ],
+    "deny": [
+      "Read(.env)",
+      "Read(.env.*)",
+      "Read(**/.env)",
+      "Read(**/.env.*)",
+      "Read(**/secrets/**)",
+      "Read(**/*.pem)",
+      "Read(**/*.key)",
+      "Read(**/credentials.json)",
+      "Read(**/.secret*)",
+      "Read(**/.secrets*)",
+      "Read(**/*.secret)",
+      "Read(**/*.secrets)",
+      "Edit(.env)",
+      "Edit(.env.*)",
+      "Edit(**/.env)",
+      "Edit(**/.env.*)"
+    ],
+    "ask": [
+      "Bash(rm:*)",
+      "Bash(rmdir:*)",
+      "Bash(mv:*)",
+      "Bash(chmod:*)",
+      "Bash(chown:*)",
+      "Bash(truncate:*)",
+      "Bash(shred:*)",
+      "Bash(unlink:*)",
+      "mcp__claude_ai_Stripe",
+      "mcp__claude_ai_Gmail",
+      "mcp__claude_ai_Google_Calendar",
+      "mcp__claude_ai_Google_Drive",
+      "mcp__claude_ai_Slack",
+      "mcp__claude_ai_Linear"
+    ]
+  },
+  "enableAllProjectMcpServers": true,
+  "hooks": {
+    "PreToolUse": [
+      {
+        "matcher": "Bash",
+        "hooks": [
+          {
+            "type": "command",
+            "command": "\"$CLAUDE_PROJECT_DIR\"/.claude/hooks/guard-main-branch.sh",
+            "timeout": 5
+          }
+        ]
+      }
+    ],
+    "PostToolUse": [
+      {
+        "matcher": "Edit|Write",
+        "hooks": [
+          {
+            "type": "command",
+            "command": "\"$CLAUDE_PROJECT_DIR\"/.claude/hooks/format-frontend.sh",
+            "timeout": 30
+          },
+          {
+            "type": "command",
+            "command": "\"$CLAUDE_PROJECT_DIR\"/.claude/hooks/format-backend.sh",
+            "timeout": 30
+          }
+        ]
+      }
+    ],
+    "Notification": [
+      {
+        "hooks": [
+          {
+            "type": "command",
+            "command": "\"$CLAUDE_PROJECT_DIR\"/.claude/hooks/notify-user.sh",
+            "timeout": 10
+          }
+        ]
+      }
+    ]
+  },
+  "enabledPlugins": {
+    "typescript-lsp@claude-plugins-official": true,
+    "code-review@claude-plugins-official": true
+  }
+}

.claude/skills/adding-a-trigger/SKILL.md

@@ -0,0 +1 @@
+../../../.agents/skills/adding-a-trigger/SKILL.md

.claude/skills/commit/SKILL.md

@@ -0,0 +1 @@
+../../../.agents/skills/commit/SKILL.md

.claude/skills/local-review/SKILL.md

@@ -0,0 +1 @@
+../../../.agents/skills/local-review/SKILL.md

.claude/skills/native-trigger/SKILL.md

@@ -0,0 +1 @@
+../../../.agents/skills/native-trigger/SKILL.md

.claude/skills/pr/SKILL.md

@@ -0,0 +1 @@
+../../../.agents/skills/pr/SKILL.md

.claude/skills/refine/SKILL.md

@@ -0,0 +1 @@
+../../../.agents/skills/refine/SKILL.md

.claude/skills/rust-backend/SKILL.md

@@ -0,0 +1 @@
+../../../.agents/skills/rust-backend/SKILL.md

.claude/skills/svelte-frontend/SKILL.md

@@ -0,0 +1 @@
+../../../.agents/skills/svelte-frontend/SKILL.md

.claude/skills/update-sqlx/SKILL.md

@@ -0,0 +1 @@
+../../../.agents/skills/update-sqlx/SKILL.md

.codex/config.toml

@@ -0,0 +1,4 @@
+#:schema https://developers.openai.com/codex/config-schema.json
+
+[mcp_servers.svelte]
+url = "https://mcp.svelte.dev/mcp"

.dockerignore

@@ -3,3 +3,4 @@ frontend/build/
 frontend/.svelte-kit/
 
 backend/target/
+backend/windmill-duckdb-ffi-internal/target/

.envrc

@@ -1 +1,7 @@
 use flake
+
+# Per-worktree overrides (ports, DATABASE_URL, etc.) written by webmux/workmux
+# post-create hooks. Must come after `use flake` so they take precedence over
+# the flake's defaults.
+# shellcheck source=/dev/null
+[ -f .env.local ] && source .env.local

.github/DockerfileBackendTests

@@ -28,7 +28,7 @@ ENV PATH="${PATH}:/usr/local/go/bin"
 ENV GO_PATH=/usr/local/go/bin/go
 
 # UV
-RUN curl --proto '=https' --tlsv1.2 -LsSf https://github.com/astral-sh/uv/releases/download/0.4.18/uv-installer.sh | sh && mv /usr/local/cargo/bin/uv /usr/local/bin/uv
+RUN curl --proto '=https' --tlsv1.2 -LsSf https://github.com/astral-sh/uv/releases/download/0.9.25/uv-installer.sh | sh && mv /usr/local/cargo/bin/uv /usr/local/bin/uv
 
 ENV TZ=Etc/UTC
 
@@ -42,7 +42,11 @@ RUN wget https://www.python.org/ftp/python/${PYTHON_VERSION}/Python-${PYTHON_VER
 RUN /usr/local/bin/python3 -m pip install pip-tools
 
 # Bun
-COPY --from=oven/bun:1.2.18 /usr/local/bin/bun /usr/bin/bun
+COPY --from=oven/bun:1.3.10 /usr/local/bin/bun /usr/bin/bun
+
+# Install windmill CLI
+RUN bun install -g windmill-cli \
+    && ln -s $(bun pm bin -g)/wmill /usr/bin/wmill
 
 ARG TARGETPLATFORM
 
@@ -57,8 +61,12 @@ RUN apt-get update \
 RUN rustup component add rustfmt
 
 # C#
-COPY --from=bitnami/dotnet-sdk:9.0.101-debian-12-r0 /opt/bitnami/dotnet-sdk /opt/dotnet-sdk
-RUN ln -s /opt/dotnet-sdk/bin/dotnet /usr/bin/dotnet
+RUN wget https://dot.net/v1/dotnet-install.sh -O dotnet-install.sh \
+    && chmod +x dotnet-install.sh \
+    && ./dotnet-install.sh --channel 9.0 --install-dir /usr/share/dotnet \
+    && ln -s /usr/share/dotnet/dotnet /usr/bin/dotnet \
+    && rm dotnet-install.sh
+
 
 # Nushell
 COPY --from=ghcr.io/nushell/nushell:0.101.0-bookworm /usr/bin/nu /usr/bin/nu

.github/change-versions-mac.sh

@@ -7,7 +7,7 @@ VERSION=$1
 echo "Updating versions to: $VERSION"
 
 sed -i '' -e "/^version =/s/= .*/= \"$VERSION\"/" ${root_dirpath}/backend/Cargo.toml
-sed -i '' -e "/^export const VERSION =/s/= .*/= \"v$VERSION\";/"  ${root_dirpath}/cli/src/main.ts
+sed -i '' -e "/^export const VERSION =/s/= .*/= \"v$VERSION\";/"  ${root_dirpath}/cli/src/core/constants.ts
 sed -i '' -e "/^export const VERSION =/s/= .*/= \"v$VERSION\";/" ${root_dirpath}/benchmarks/lib.ts
 sed -i '' -e "/version: /s/: .*/: $VERSION/" ${root_dirpath}/backend/windmill-api/openapi.yaml
 sed -i '' -e "/version: /s/: .*/: $VERSION/" ${root_dirpath}/openflow.openapi.yaml
@@ -15,12 +15,15 @@ sed -i '' -e "/\"version\": /s/: .*,/: \"$VERSION\",/" ${root_dirpath}/typescrip
 sed -i '' -e "/\"version\": /s/: .*,/: \"$VERSION\",/" ${root_dirpath}/frontend/package.json
 sed -i '' -e "/^version =/s/= .*/= \"$VERSION\"/" ${root_dirpath}/python-client/wmill/pyproject.toml
 sed -i '' -e "/^windmill-api =/s/= .*/= \"\\^$VERSION\"/" ${root_dirpath}/python-client/wmill/pyproject.toml
-sed -i '' -e "/^version =/s/= .*/= \"$VERSION\"/" ${root_dirpath}/python-client/wmill_pg/pyproject.toml
 sed -i '' -e "/^[[:space:]]*ModuleVersion[[:space:]]*=/s/= .*/= '$VERSION'/" ${root_dirpath}/powershell-client/WindmillClient/WindmillClient.psd1
-# sed -i '' -e "/^wmill =/s/= .*/= \"\\^$VERSION\"/" python-client/wmill_pg/pyproject.toml
 sed -i '' -e "/^wmill =/s/= .*/= \">=$VERSION\"/" ${root_dirpath}/lsp/Pipfile
-sed -i '' -e "/^wmill_pg =/s/= .*/= \">=$VERSION\"/" ${root_dirpath}/lsp/Pipfile
 
 sed -i '' -E "s/name = \"windmill\"\nversion = \"[^\"]*\"\\n(.*)/name = \"windmill\"\nversion = \"$VERSION\"\\n\\1/" ${root_dirpath}/backend/Cargo.lock
 
+# windmill-parser-wasm is its own workspace (excluded from the backend workspace
+# because of nightly-only cargo-features), so its version lives in
+# [workspace.package] and its Cargo.lock is not regenerated by the backend step.
+sed -i '' -e "/^version =/s/= .*/= \"$VERSION\"/" ${root_dirpath}/backend/parsers/windmill-parser-wasm/Cargo.toml
+sed -i '' -E "s/(name = \"windmill[^\"]*\"\nversion = )\"[^\"]*\"/\\1\"$VERSION\"/g" ${root_dirpath}/backend/parsers/windmill-parser-wasm/Cargo.lock
+
 cd ${root_dirpath}/frontend && npm i --package-lock-only

.github/change-versions.sh

@@ -7,7 +7,7 @@ VERSION=$1
 echo "Updating versions to: $VERSION"
 
 sed -i -e "/^version =/s/= .*/= \"$VERSION\"/" ${root_dirpath}/backend/Cargo.toml
-sed -i -e "/^export const VERSION =/s/= .*/= \"$VERSION\";/" ${root_dirpath}/cli/src/main.ts
+sed -i -e "/^export const VERSION =/s/= .*/= \"$VERSION\";/" ${root_dirpath}/cli/src/core/constants.ts
 sed -i -e "/^export const VERSION =/s/= .*/= \"v$VERSION\";/" ${root_dirpath}/benchmarks/lib.ts
 sed -i -e "/version: /s/: .*/: $VERSION/" ${root_dirpath}/backend/windmill-api/openapi.yaml
 sed -i -e "/version: /s/: .*/: $VERSION/" ${root_dirpath}/openflow.openapi.yaml
@@ -16,12 +16,15 @@ sed -i -e "/\"version\": /s/: .*,/: \"$VERSION\",/" ${root_dirpath}/typescript-c
 sed -i -e "/\"version\": /s/: .*,/: \"$VERSION\",/" ${root_dirpath}/frontend/package.json
 sed -i -e "/^version =/s/= .*/= \"$VERSION\"/" ${root_dirpath}/python-client/wmill/pyproject.toml
 sed -i -e "/^windmill-api =/s/= .*/= \"\\^$VERSION\"/" ${root_dirpath}/python-client/wmill/pyproject.toml
-sed -i -e "/^version =/s/= .*/= \"$VERSION\"/" ${root_dirpath}/python-client/wmill_pg/pyproject.toml
 sed -i -e "/^[[:space:]]*ModuleVersion[[:space:]]*=/s/= .*/= '$VERSION'/" ${root_dirpath}/powershell-client/WindmillClient/WindmillClient.psd1
-# sed -i -e "/^wmill =/s/= .*/= \"\\^$VERSION\"/" ${root_dirpath}/python-client/wmill_pg/pyproject.toml
 sed -i -e "/^wmill =/s/= .*/= \">=$VERSION\"/" ${root_dirpath}/lsp/Pipfile
-sed -i -e "/^wmill_pg =/s/= .*/= \">=$VERSION\"/" ${root_dirpath}/lsp/Pipfile
 
 sed -i -zE "s/name = \"windmill\"\nversion = \"[^\"]*\"\\n(.*)/name = \"windmill\"\nversion = \"$VERSION\"\\n\\1/" ${root_dirpath}/backend/Cargo.lock
 
+# windmill-parser-wasm is its own workspace (excluded from the backend workspace
+# because of nightly-only cargo-features), so its version lives in
+# [workspace.package] and its Cargo.lock is not regenerated by the backend step.
+sed -i -e "/^version =/s/= .*/= \"$VERSION\"/" ${root_dirpath}/backend/parsers/windmill-parser-wasm/Cargo.toml
+sed -i -zE "s/(name = \"windmill[^\"]*\"\nversion = )\"[^\"]*\"/\\1\"$VERSION\"/g" ${root_dirpath}/backend/parsers/windmill-parser-wasm/Cargo.lock
+
 cd ${root_dirpath}/frontend && npm i --package-lock-only --ignore-scripts

.github/codex/pr-review.prompt.md

@@ -0,0 +1,5 @@
+# Codex output format
+
+- Read `./.github/codex/pr-review-context.md` for PR metadata and the diff commands.
+- Return a markdown PR comment starting with `## Codex Review`.
+- Tag each finding with a severity (P0 / P1 / P2), file path, and line number when known confidently.

.github/dependabot.yml

@@ -31,9 +31,3 @@ updates:
     directory: "/python-client/wmill"
     schedule:
       interval: "weekly"
-      
-  # Maintain dependencies for wmill_pg python client
-  - package-ecosystem: "pip"
-    directory: "/python-client/wmill_pg"
-    schedule:
-      interval: "weekly"

.github/pi/pr-review.prompt.md

@@ -0,0 +1,6 @@
+# Pi output format
+
+- Read `./.github/pi/pr-review-context.md` for PR metadata and the diff commands.
+- Return a markdown PR comment starting with `## Pi Review`.
+- Tag each finding with a severity (P0 / P1 / P2), file path, and line number when known confidently.
+- Output ONLY the final review markdown — no preamble, no thinking, no tool transcripts.

.github/workflows/ai-agent-tests.yml

@@ -0,0 +1,132 @@
+name: AI Agent Integration Tests
+
+# Exercises the AI agent flow path (preview_flow with `aiagent` modules) against
+# real LLM providers. Runs only when AI-agent backend code or the tests change,
+# because each run makes real (paid) LLM calls. To avoid spending on every commit,
+# the PR side triggers only when a PR is marked ready for review (out of draft) —
+# not on `synchronize` — plus push to main and manual dispatch.
+on:
+  workflow_dispatch:
+  push:
+    branches: [main]
+    paths:
+      - "integration_tests/ai_agent_tests/**"
+      - "backend/windmill-ai/**"
+      - "backend/windmill-api/src/ai.rs"
+      - "backend/windmill-worker/src/ai_executor.rs"
+      - "backend/windmill-worker/src/ai/**"
+      - "backend/windmill-worker/src/memory_common.rs"
+      - "backend/windmill-common/src/flow_conversations.rs"
+      - ".github/workflows/ai-agent-tests.yml"
+  pull_request:
+    types: [opened, reopened, ready_for_review]
+    paths:
+      - "integration_tests/ai_agent_tests/**"
+      - "backend/windmill-ai/**"
+      - "backend/windmill-api/src/ai.rs"
+      - "backend/windmill-worker/src/ai_executor.rs"
+      - "backend/windmill-worker/src/ai/**"
+      - "backend/windmill-worker/src/memory_common.rs"
+      - "backend/windmill-common/src/flow_conversations.rs"
+      - ".github/workflows/ai-agent-tests.yml"
+
+concurrency:
+  group: ai-agent-tests-${{ github.ref }}
+  cancel-in-progress: true
+
+jobs:
+  ai_agent_e2e:
+    # Skip draft PRs; the `opened`/`reopened` types would otherwise fire while
+    # still a draft. `ready_for_review` always arrives non-draft.
+    if: github.event_name != 'pull_request' || github.event.pull_request.draft == false
+    runs-on: ubicloud-standard-16
+    services:
+      postgres:
+        image: postgres:16
+        ports:
+          - 5432:5432
+        env:
+          POSTGRES_DB: windmill
+          POSTGRES_PASSWORD: changeme
+        options: >-
+          --health-cmd pg_isready --health-interval 10s --health-timeout 5s
+          --health-retries 5
+    steps:
+      - uses: actions/checkout@v4
+
+      - uses: actions-rust-lang/setup-rust-toolchain@v1
+        with:
+          cache-workspaces: backend
+          toolchain: 1.93.0
+
+      - uses: oven-sh/setup-bun@v2
+        with:
+          bun-version: 1.3.10
+
+      - uses: actions/setup-node@v4
+        with:
+          node-version: "20"
+
+      - uses: actions/setup-python@v5
+        with:
+          python-version: "3.11"
+
+      # CE build (no enterprise/license needed for AI agents). `quickjs` powers
+      # flow input-transform JS eval; `mcp` is required by the deepwiki MCP tool
+      # test. Bun tool scripts run via the always-on worker (BUN_PATH).
+      - name: Build Windmill
+        working-directory: ./backend
+        env:
+          SQLX_OFFLINE: true
+          CARGO_BUILD_JOBS: 12
+          RUSTFLAGS: ""
+        run: cargo build --features quickjs,mcp
+
+      - name: Start Windmill
+        working-directory: ./backend
+        env:
+          DATABASE_URL: postgres://postgres:changeme@localhost:5432/windmill
+          BUN_PATH: bun
+          NODE_BIN_PATH: node
+          RUST_LOG: info
+        run: |
+          mkdir -p ../integration_tests/logs
+          ./target/debug/windmill > ../integration_tests/logs/windmill.log 2>&1 &
+          echo "Waiting for Windmill to be ready..."
+          for i in $(seq 1 60); do
+            if curl -sf http://localhost:8000/api/version > /dev/null 2>&1; then
+              echo "Windmill is ready"
+              break
+            fi
+            sleep 2
+          done
+          curl -sf http://localhost:8000/api/version > /dev/null || { echo "Windmill failed to start"; tail -50 ../integration_tests/logs/windmill.log; exit 1; }
+
+      - name: Run AI agent integration tests
+        timeout-minutes: 20
+        working-directory: ./integration_tests/ai_agent_tests
+        env:
+          WINDMILL_URL: http://localhost:8000
+          # Only the providers we have org secrets for. Other providers
+          # (Azure, Bedrock, OpenRouter) are skipped by conftest when their
+          # keys are absent — see skip_provider_without_credentials.
+          ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }}
+          OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
+          GOOGLE_AI_API_KEY: ${{ secrets.GOOGLE_API_KEY }}
+        run: |
+          python -m venv .venv
+          .venv/bin/pip install -r requirements.txt
+          # The S3/vision-attachment tests need MinIO large-file storage and
+          # image-capable provider setup; out of scope for this cost-controlled
+          # smoke. Add MinIO secrets + a storage service to enable them.
+          .venv/bin/python -m pytest -v \
+            --ignore=test_user_attachments.py \
+            --ignore=test_user_images.py \
+            --ignore=test_image_output.py
+
+      - name: Archive Windmill logs
+        uses: actions/upload-artifact@v4
+        if: always()
+        with:
+          name: ai-agent-tests-windmill-logs
+          path: integration_tests/logs

.github/workflows/ai-evals-test.yml

@@ -0,0 +1,166 @@
+name: AI Evals (global mode)
+
+# Smoke-tests the production global AI chat proxy/frontend execution path via
+# the ai_evals harness, one case across one cheap model per provider. Runs only
+# when the eval harness or the global chat code change, since each run makes real
+# (paid) LLM calls. The backend is built from source purely as the AI proxy the
+# harness routes model calls through; the global tools/drafts run in-process in
+# the Vitest bridge against production frontend code. To avoid spending on every
+# commit, the PR side triggers only when a PR is marked ready for review (out of
+# draft) — not on `synchronize` — plus push to main and manual dispatch.
+on:
+  workflow_dispatch:
+  push:
+    branches: [main]
+    paths:
+      - "ai_evals/**"
+      - "backend/windmill-api/src/ai.rs"
+      - "backend/windmill-ai/**"
+      - "frontend/src/lib/components/copilot/**"
+      # The eval harness runs production frontend code in-process; these are the
+      # AI/draft-specific deps outside copilot/ that the global smoke exercises.
+      - "frontend/src/lib/userDraft.svelte.ts"
+      - "frontend/src/lib/userDraftDbSyncer.svelte.ts"
+      - "frontend/src/lib/infer.ts"
+      - ".github/workflows/ai-evals-test.yml"
+  pull_request:
+    types: [opened, reopened, ready_for_review]
+    paths:
+      - "ai_evals/**"
+      - "backend/windmill-api/src/ai.rs"
+      - "backend/windmill-ai/**"
+      - "frontend/src/lib/components/copilot/**"
+      # The eval harness runs production frontend code in-process; these are the
+      # AI/draft-specific deps outside copilot/ that the global smoke exercises.
+      - "frontend/src/lib/userDraft.svelte.ts"
+      - "frontend/src/lib/userDraftDbSyncer.svelte.ts"
+      - "frontend/src/lib/infer.ts"
+      - ".github/workflows/ai-evals-test.yml"
+
+concurrency:
+  group: ai-evals-test-${{ github.ref }}
+  cancel-in-progress: true
+
+jobs:
+  ai_evals_global:
+    # Provider secrets are unavailable to forked and Dependabot PRs.
+    if: >-
+      github.event_name != 'pull_request' ||
+      (
+        github.event.pull_request.draft == false &&
+        github.event.pull_request.head.repo.full_name == github.repository &&
+        github.event.pull_request.user.login != 'dependabot[bot]'
+      )
+    runs-on: ubicloud-standard-16
+    services:
+      postgres:
+        image: postgres:16
+        ports:
+          - 5432:5432
+        env:
+          POSTGRES_DB: windmill
+          POSTGRES_PASSWORD: changeme
+        options: >-
+          --health-cmd pg_isready --health-interval 10s --health-timeout 5s
+          --health-retries 5
+    steps:
+      - uses: actions/checkout@v4
+
+      - uses: actions-rust-lang/setup-rust-toolchain@v1
+        with:
+          cache-workspaces: backend
+          toolchain: 1.93.0
+
+      - uses: oven-sh/setup-bun@v2
+        with:
+          bun-version: 1.3.10
+
+      - uses: actions/setup-node@v4
+        with:
+          # Node 22.19+ is required by the frontend's undici 8.x, which the
+          # Vitest bridge loads; Node 20 fails with markAsUncloneable.
+          node-version: "22"
+
+      # CE build used only as the AI proxy (login, workspace, provider resource,
+      # /ai/proxy). No worker execution or MCP needed — global tools/drafts run
+      # in the Vitest bridge. quickjs matches the standard CE feature set.
+      - name: Build Windmill (AI proxy)
+        working-directory: ./backend
+        env:
+          SQLX_OFFLINE: true
+          CARGO_BUILD_JOBS: 12
+          RUSTFLAGS: ""
+        run: cargo build --features quickjs
+
+      - name: Start Windmill
+        working-directory: ./backend
+        env:
+          DATABASE_URL: postgres://postgres:changeme@localhost:5432/windmill
+          RUST_LOG: info
+        run: |
+          mkdir -p ../ai_evals/logs
+          ./target/debug/windmill > ../ai_evals/logs/windmill.log 2>&1 &
+          echo "Waiting for Windmill to be ready..."
+          for i in $(seq 1 60); do
+            if curl -sf http://localhost:8000/api/version > /dev/null 2>&1; then
+              echo "Windmill is ready"
+              break
+            fi
+            sleep 2
+          done
+          curl -sf http://localhost:8000/api/version > /dev/null || { echo "Windmill failed to start"; tail -50 ../ai_evals/logs/windmill.log; exit 1; }
+
+      - name: Install frontend deps + generate client
+        working-directory: ./frontend
+        run: |
+          npm ci
+          npm run generate-backend-client
+
+      - name: Run global AI evals
+        timeout-minutes: 20
+        working-directory: ./ai_evals
+        env:
+          WMILL_AI_EVAL_BACKEND_URL: http://localhost:8000
+          WMILL_AI_EVAL_BACKEND_WORKSPACE: integration-tests
+          # Anthropic backs the haiku model. Google AI uses GEMINI_API_KEY.
+          ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }}
+          OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
+          GEMINI_API_KEY: ${{ secrets.GOOGLE_API_KEY }}
+          DEEPSEEK_API_KEY: ${{ secrets.DEEPSEEK_API_KEY }}
+        run: |
+          bun install
+          mkdir -p results
+          # One cheap model per provider (anthropic/openai/googleai/deepseek).
+          fail=0
+          for m in haiku 4o gemini-3-flash-preview deepseek-v4-flash; do
+            echo "::group::global-test1-script-create ($m)"
+            if ! bun run cli -- run global global-test1-script-create \
+                  --model "$m" --execution-only --output "$PWD/results/ci-$m.json"; then
+              echo "$m: harness/proxy errored"
+              fail=1
+              echo "::endgroup::"
+              continue
+            fi
+            # The CLI exits 0 when the harness records failed attempts, so gate
+            # on execution-only pass counts while ignoring model output quality.
+            if jq -e \
+              '.attemptCount > 0 and .passedAttempts == .attemptCount' \
+              "results/ci-$m.json" > /dev/null; then
+              echo "$m: OK — proxy/frontend execution completed"
+            else
+              echo "$m: FAILED proxy/frontend execution"
+              jq -c '.cases[0].attempts[0].checks' "results/ci-$m.json" || true
+              fail=1
+            fi
+            echo "::endgroup::"
+          done
+          [ "$fail" = 0 ] || { echo "ai_evals global smoke failed"; exit 1; }
+
+      - name: Archive logs and results
+        uses: actions/upload-artifact@v4
+        if: always()
+        with:
+          name: ai-evals-global-logs
+          path: |
+            ai_evals/logs
+            ai_evals/results

.github/workflows/backend-check.yml

@@ -17,10 +17,13 @@ jobs:
         with:
           fetch-depth: 0
 
+      - name: Install mold and clang
+        run: sudo apt-get update && sudo apt-get install -y mold clang
+
       - uses: actions-rust-lang/setup-rust-toolchain@v1
         with:
-          cache-workspaces: backend
-          toolchain: 1.88.0
+          cache: false
+          toolchain: 1.93.0
       - name: cargo check
         working-directory: ./backend
         timeout-minutes: 16
@@ -33,21 +36,21 @@ jobs:
         with:
           fetch-depth: 0
 
-      - name: install xmlsec1
+      - name: install xmlsec1 and gssapi
         run: |
           sudo apt-get update
-          sudo apt-get install -y libxml2-dev libxmlsec1-dev
+          sudo apt-get install -y libxml2-dev libxmlsec1-dev libkrb5-dev libsasl2-dev libcurl4-openssl-dev mold clang
 
       - uses: actions-rust-lang/setup-rust-toolchain@v1
         with:
-          cache-workspaces: backend
-          toolchain: 1.88.0
+          cache: false
+          toolchain: 1.93.0
       - name: cargo check
         working-directory: ./backend
         timeout-minutes: 16
         run: |
           mkdir -p fake_frontend_build
-          FRONTEND_BUILD_DIR=$(pwd)/fake_frontend_build SQLX_OFFLINE=true cargo check --features $(./all_features_oss.sh)
+          FRONTEND_BUILD_DIR=$(pwd)/fake_frontend_build SQLX_OFFLINE=true cargo check --features all_sqlx_features
 
   check_ee:
     runs-on: ubicloud-standard-8
@@ -72,10 +75,13 @@ jobs:
         run: |
           ./backend/substitute_ee_code.sh --copy --dir ./windmill-ee-private
 
+      - name: Install mold and clang
+        run: sudo apt-get update && sudo apt-get install -y mold clang
+
       - uses: actions-rust-lang/setup-rust-toolchain@v1
         with:
-          cache-workspaces: backend
-          toolchain: 1.88.0
+          cache: false
+          toolchain: 1.93.0
       - name: cargo check
         working-directory: ./backend
         timeout-minutes: 16
@@ -100,10 +106,10 @@ jobs:
           token: ${{ secrets.WINDMILL_EE_PRIVATE_ACCESS }}
           fetch-depth: 0
 
-      - name: install xmlsec1
+      - name: install xmlsec1 and gssapi
         run: |
           sudo apt-get update
-          sudo apt-get install -y libxml2-dev libxmlsec1-dev
+          sudo apt-get install -y libxml2-dev libxmlsec1-dev libkrb5-dev libsasl2-dev libcurl4-openssl-dev mold clang
 
       - name: Substitute EE code (EE logic is behind feature flag)
         run: |
@@ -112,10 +118,22 @@ jobs:
       - uses: actions-rust-lang/setup-rust-toolchain@v1
         with:
           cache-workspaces: backend
-          toolchain: 1.88.0
+          toolchain: 1.93.0
+      - name: Fix stale v8 build cache
+        working-directory: ./backend
+        run: |
+          # Cargo cache may preserve v8 build fingerprints without the actual
+          # librusty_v8.a library. Since fingerprints look valid, cargo skips
+          # build.rs re-run, causing "could not find native static library rusty_v8".
+          for profile in debug release; do
+            if [ -d "target/$profile/.fingerprint" ] && [ ! -f "target/$profile/gn_out/obj/librusty_v8.a" ]; then
+              echo "Cleaning stale v8 build artifacts in target/$profile"
+              rm -rf "target/$profile/build/v8-"* "target/$profile/.fingerprint/v8-"*
+            fi
+          done
       - name: cargo check
         timeout-minutes: 16
         working-directory: ./backend
         run: |
           mkdir -p fake_frontend_build
-          FRONTEND_BUILD_DIR=$(pwd)/fake_frontend_build SQLX_OFFLINE=true cargo check --all-features
+          FRONTEND_BUILD_DIR=$(pwd)/fake_frontend_build SQLX_OFFLINE=true cargo check --features all_sqlx_features,private

.github/workflows/backend-test-windows.yml

@@ -0,0 +1,205 @@
+name: Backend integration tests (Windows)
+
+on:
+  workflow_dispatch:
+  push:
+    branches:
+      - "ci-windows-tests"
+    tags:
+      - "v*"
+
+env:
+  CARGO_INCREMENTAL: 0
+  SQLX_OFFLINE: true
+  DISABLE_EMBEDDING: true
+
+jobs:
+  cargo_test_windows:
+    runs-on: blacksmith-16vcpu-windows-2025
+    steps:
+      - uses: actions/checkout@v4
+
+      - name: Read EE repo commit hash
+        shell: pwsh
+        run: |
+          $ee_repo_ref = Get-Content .\backend\ee-repo-ref.txt
+          echo "ee_repo_ref=$ee_repo_ref" | Out-File -FilePath $env:GITHUB_ENV -Append
+
+      - name: Checkout windmill-ee-private repository
+        uses: actions/checkout@v4
+        with:
+          repository: windmill-labs/windmill-ee-private
+          path: ./windmill-ee-private
+          ref: ${{ env.ee_repo_ref }}
+          token: ${{ secrets.WINDMILL_EE_PRIVATE_ACCESS }}
+          fetch-depth: 0
+
+      - name: Substitute EE code
+        shell: bash
+        run: |
+          ./backend/substitute_ee_code.sh --copy --dir ./windmill-ee-private
+
+      - name: Setup PostgreSQL
+        uses: ikalnytskyi/action-setup-postgres@v6
+        with:
+          username: postgres
+          password: changeme
+          database: windmill
+          port: 5432
+
+      - uses: actions-rust-lang/setup-rust-toolchain@v1
+        with:
+          cache-workspaces: backend
+          toolchain: 1.93.0
+
+      - uses: actions/setup-dotnet@v4
+        with:
+          dotnet-version: "9.0.x"
+
+      - uses: denoland/setup-deno@v2
+        with:
+          deno-version: v2.x
+
+      - uses: actions/setup-go@v2
+        with:
+          go-version: 1.21.5
+
+      - uses: oven-sh/setup-bun@v2
+        with:
+          bun-version: 1.3.10
+
+      - uses: actions/setup-node@v4
+        with:
+          node-version: "20"
+
+      - uses: astral-sh/setup-uv@v6.2.1
+        with:
+          version: "0.9.25"
+
+      - uses: shivammathur/setup-php@v2
+        with:
+          php-version: "8.3"
+          tools: composer
+
+      - name: Install windmill CLI
+        shell: bash
+        run: |
+          cd cli
+          bash gen_wm_client.sh
+          bun install
+          mkdir -p "$HOME/.local/bin"
+          printf '#!/bin/sh\nexec bun run "%s/cli/src/main.ts" "$@"\n' "$GITHUB_WORKSPACE" > "$HOME/.local/bin/wmill"
+          chmod +x "$HOME/.local/bin/wmill"
+          echo "$HOME/.local/bin" >> $GITHUB_PATH
+
+      - name: Install OpenSSL via vcpkg
+        run: |
+          vcpkg.exe install openssl-windows:x64-windows
+          vcpkg.exe install openssl:x64-windows-static
+          vcpkg.exe integrate install
+
+      - name: Free disk space (post-vcpkg)
+        shell: pwsh
+        run: |
+          # vcpkg leaves multi-GB of buildtrees/downloads after installing openssl;
+          # we only need the installed/ dir for linking.
+          $vcpkgRoot = $env:VCPKG_INSTALLATION_ROOT
+          foreach ($sub in @("buildtrees", "downloads", "packages")) {
+            $path = Join-Path $vcpkgRoot $sub
+            if (Test-Path $path) {
+              Write-Host "Removing $path"
+              Remove-Item -Recurse -Force -ErrorAction SilentlyContinue $path
+            }
+          }
+          Get-PSDrive C | Select-Object Used,Free | Format-Table -AutoSize
+
+      - name: Get runtime paths
+        id: runtime-paths
+        shell: pwsh
+        run: |
+          echo "DENO_PATH=$($(Get-Command deno).Source)" >> $env:GITHUB_OUTPUT
+          echo "BUN_PATH=$($(Get-Command bun).Source)" >> $env:GITHUB_OUTPUT
+          echo "NODE_BIN_PATH=$($(Get-Command node).Source)" >> $env:GITHUB_OUTPUT
+          echo "GO_PATH=$($(Get-Command go).Source)" >> $env:GITHUB_OUTPUT
+          echo "UV_PATH=$($(Get-Command uv).Source)" >> $env:GITHUB_OUTPUT
+          echo "PHP_PATH=$($(Get-Command php).Source)" >> $env:GITHUB_OUTPUT
+          echo "COMPOSER_PATH=$($(Get-Command composer).Source)" >> $env:GITHUB_OUTPUT
+          echo "POWERSHELL_PATH=$($(Get-Command pwsh).Source)" >> $env:GITHUB_OUTPUT
+          echo "DOTNET_PATH=$($(Get-Command dotnet).Source)" >> $env:GITHUB_OUTPUT
+
+      - name: Build DuckDB FFI module
+        working-directory: backend/windmill-duckdb-ffi-internal
+        timeout-minutes: 30
+        run: |
+          cargo build --release -p windmill_duckdb_ffi_internal
+          New-Item -ItemType Directory -Path ..\target\debug -Force
+          Copy-Item target\release\windmill_duckdb_ffi_internal.dll ..\target\debug\
+          # duckdb is bundled (~2GB of build artifacts); the DLL is the only
+          # thing we need from this excluded-crate target dir.
+          Remove-Item -Recurse -Force -ErrorAction SilentlyContinue target
+          Get-PSDrive C | Select-Object Used,Free | Format-Table -AutoSize
+
+      - name: Print runtime versions and env
+        shell: pwsh
+        run: |
+          deno --version
+          bun -v
+          node --version
+          go version
+          python3 --version
+          php --version
+          pwsh --version
+          dotnet --version
+          echo "TEMP=$env:TEMP"
+          echo "TMP=$env:TMP"
+          echo "USERPROFILE=$env:USERPROFILE"
+          echo "HOME=$env:HOME"
+
+      - name: Disk space before cargo test
+        shell: pwsh
+        run: Get-PSDrive C | Select-Object Used,Free | Format-Table -AutoSize
+
+      - name: cargo test
+        working-directory: backend
+        timeout-minutes: 60
+        env:
+          DATABASE_URL: postgres://postgres:changeme@localhost:5432/windmill
+          RUST_LOG: "off"
+          RUST_LOG_STYLE: never
+          CARGO_NET_GIT_FETCH_WITH_CLI: true
+          # 16-vcpu runners with disabled PDB still hit LNK1180 ("insufficient
+          # disk space") at link time with 12 parallel link jobs: each test
+          # binary link spikes several hundred MB of transient I/O. Capping at
+          # 8 trades ~25% wall time for headroom on the ~75GB runner disk.
+          CARGO_BUILD_JOBS: 8
+          # backend/Cargo.toml sets split-debuginfo = "unpacked", which on
+          # windows-msvc is coerced to "packed": every test-binary link spawns
+          # the mspdbsrv.exe PDB type server and writes a large .pdb. CI needs
+          # no debug info, so disable PDB generation for the dev/test profiles
+          # here (avoids both LNK1318 type-server limit and PDB disk usage).
+          CARGO_PROFILE_DEV_SPLIT_DEBUGINFO: "off"
+          CARGO_PROFILE_TEST_SPLIT_DEBUGINFO: "off"
+          # Tests' poll-time stack frames (deep nested async fn chains in
+          # debug builds) reach ~1.8MB. 4MB gives ~2x headroom against flaky
+          # overflows under parallel-test contention.
+          RUST_MIN_STACK: 4194304
+          VCPKGRS_DYNAMIC: 1
+          OPENSSL_DIR: ${{ env.VCPKG_INSTALLATION_ROOT }}\installed\x64-windows-static
+          DENO_PATH: ${{ steps.runtime-paths.outputs.DENO_PATH }}
+          BUN_PATH: ${{ steps.runtime-paths.outputs.BUN_PATH }}
+          NODE_BIN_PATH: ${{ steps.runtime-paths.outputs.NODE_BIN_PATH }}
+          GO_PATH: ${{ steps.runtime-paths.outputs.GO_PATH }}
+          UV_PATH: ${{ steps.runtime-paths.outputs.UV_PATH }}
+          PHP_PATH: ${{ steps.runtime-paths.outputs.PHP_PATH }}
+          COMPOSER_PATH: ${{ steps.runtime-paths.outputs.COMPOSER_PATH }}
+          POWERSHELL_PATH: ${{ steps.runtime-paths.outputs.POWERSHELL_PATH }}
+          DOTNET_PATH: ${{ steps.runtime-paths.outputs.DOTNET_PATH }}
+          WMDEBUG_FORCE_V0_WORKSPACE_DEPENDENCIES: 1
+          WMDEBUG_FORCE_RUNNABLE_SETTINGS_V0: 1
+          WMDEBUG_FORCE_NO_LEGACY_DEBOUNCING_COMPAT: 1
+        run: >
+          cargo test
+          --no-fail-fast
+          --features enterprise,deno_core,duckdb,license,python,rust,scoped_cache,parquet,private,csharp,php,quickjs,mcp,run_inline
+          --all
+          -- --nocapture --test-threads=10

.github/workflows/backend-test.yml

@@ -1,6 +1,7 @@
 name: Backend only integration tests
 
 on:
+  workflow_dispatch:
   push:
     branches:
       - "main"
@@ -19,7 +20,7 @@ defaults:
 
 jobs:
   cargo_test:
-    runs-on: ubicloud-standard-8
+    runs-on: ubicloud-standard-16
     services:
       postgres:
         image: postgres
@@ -28,9 +29,20 @@ jobs:
         env:
           POSTGRES_DB: windmill
           POSTGRES_PASSWORD: changeme
+          POSTGRES_INITDB_ARGS: "-c max_connections=500"
         options: >-
           --health-cmd pg_isready --health-interval 10s --health-timeout 5s
-          --health-retries 5
+          --health-retries 5 --shm-size=256mb
+      mysql:
+        image: mysql:8.0
+        ports:
+          - 3306:3306
+        env:
+          MYSQL_ROOT_PASSWORD: changeme
+          MYSQL_DATABASE: windmill_test
+        options: >-
+          --health-cmd "mysqladmin ping -h localhost" --health-interval 10s
+          --health-timeout 5s --health-retries 5
     steps:
       - uses: actions/checkout@v4
       - uses: actions/setup-dotnet@v4
@@ -44,22 +56,204 @@ jobs:
           go-version: 1.21.5
       - uses: oven-sh/setup-bun@v2
         with:
-          bun-version: 1.1.43
+          bun-version: 1.3.10
+      - uses: actions/setup-node@v4
+        with:
+          node-version: "20"
       - uses: astral-sh/setup-uv@v6.2.1
         with:
-          version: "0.6.2"
+          version: "0.9.25"
+      - uses: shivammathur/setup-php@v2
+        with:
+          php-version: "8.3"
+          tools: composer
+      - uses: ruby/setup-ruby@v1
+        with:
+          ruby-version: "3.3"
+          bundler-cache: false
+      - name: Install windmill CLI from source
+        run: |
+          cd $GITHUB_WORKSPACE/cli
+          bash gen_wm_client.sh
+          bun install
+          mkdir -p "$HOME/.local/bin"
+          printf '#!/bin/sh\nexec bun run "%s/cli/src/main.ts" "$@"\n' "$GITHUB_WORKSPACE" > "$HOME/.local/bin/wmill"
+          chmod +x "$HOME/.local/bin/wmill"
+          echo "$HOME/.local/bin" >> $GITHUB_PATH
+        working-directory: /
+      - name: Install PowerShell, mold and clang
+        run: |
+          sudo apt-get update && sudo apt-get install -y powershell mold clang libcurl4-openssl-dev
+        working-directory: /
       - uses: actions-rust-lang/setup-rust-toolchain@v1
         with:
           cache-workspaces: backend
-          toolchain: 1.85.0
+          toolchain: 1.93.0
+      - name: Fix stale v8 build cache
+        working-directory: ./backend
+        run: |
+          # Cargo cache may preserve v8 build fingerprints without the actual
+          # librusty_v8.a library. Since fingerprints look valid, cargo skips
+          # build.rs re-run, causing "could not find native static library rusty_v8".
+          for profile in debug release; do
+            if [ -d "target/$profile/.fingerprint" ] && [ ! -f "target/$profile/gn_out/obj/librusty_v8.a" ]; then
+              echo "Cleaning stale v8 build artifacts in target/$profile"
+              rm -rf "target/$profile/build/v8-"* "target/$profile/.fingerprint/v8-"*
+            fi
+          done
+      - name: Read EE repo commit hash
+        run: |
+          echo "ee_repo_ref=$(cat ./ee-repo-ref.txt)" >> "$GITHUB_ENV"
+
+      - uses: actions/checkout@v4
+        with:
+          repository: windmill-labs/windmill-ee-private
+          path: ./windmill-ee-private
+          ref: ${{ env.ee_repo_ref }}
+          token: ${{ secrets.WINDMILL_EE_PRIVATE_ACCESS }}
+          fetch-depth: 0
+
+      - name: Substitute EE code (EE logic is behind feature flag)
+        run: |
+          ./substitute_ee_code.sh --copy --dir ./windmill-ee-private
+      - name: Setup private npm registry with test package
+        working-directory: /tmp
+        run: |
+          set -e
+
+          # Install Verdaccio globally
+          npm install -g verdaccio
+
+          # Create Verdaccio config that requires authentication for @windmill-test packages
+          mkdir -p /tmp/verdaccio/storage
+          cat > /tmp/verdaccio/config.yaml << 'VERDACCIO_CONFIG'
+          storage: /tmp/verdaccio/storage
+          auth:
+            htpasswd:
+              file: /tmp/verdaccio/htpasswd
+              max_users: 100
+          uplinks:
+            npmjs:
+              url: https://registry.npmjs.org/
+          packages:
+            '@windmill-test/*':
+              access: $authenticated
+              publish: $authenticated
+            '@*/*':
+              access: $all
+              publish: $authenticated
+              proxy: npmjs
+            '**':
+              access: $all
+              publish: $authenticated
+              proxy: npmjs
+          server:
+            keepAliveTimeout: 60
+          middlewares:
+            audit:
+              enabled: true
+          log: { type: stdout, format: pretty, level: warn }
+          VERDACCIO_CONFIG
+
+          # Create empty htpasswd file (users will be created via API)
+          touch /tmp/verdaccio/htpasswd
+
+          # Start Verdaccio in background
+          verdaccio --config /tmp/verdaccio/config.yaml &
+          VERDACCIO_PID=$!
+
+          # Wait for Verdaccio to be ready
+          echo "Waiting for Verdaccio to start..."
+          for i in {1..30}; do
+            if curl -s http://localhost:4873/-/ping > /dev/null 2>&1; then
+              echo "Verdaccio is ready"
+              break
+            fi
+            sleep 1
+          done
+
+          # Login to get a token
+          echo "Getting auth token..."
+          RESPONSE=$(curl -s -X PUT \
+            -H "Content-Type: application/json" \
+            -d '{"name":"testuser","password":"testpass123"}' \
+            http://localhost:4873/-/user/org.couchdb.user:testuser)
+
+          echo "Auth response: $RESPONSE"
+          NPM_TOKEN=$(echo "$RESPONSE" | jq -r '.token')
+
+          if [ -z "$NPM_TOKEN" ] || [ "$NPM_TOKEN" = "null" ]; then
+            echo "Failed to get NPM token from response"
+            exit 1
+          fi
+
+          echo "NPM_TOKEN=${NPM_TOKEN}" >> $GITHUB_ENV
+          {
+            echo "TEST_NPMRC<<NPMRC_EOF"
+            echo "@windmill-test:registry=http://localhost:4873/"
+            echo "//localhost:4873/:_authToken=${NPM_TOKEN}"
+            echo "NPMRC_EOF"
+          } >> $GITHUB_ENV
+          echo "Got NPM token successfully: ${NPM_TOKEN:0:10}..."
+
+          # Configure npm globally with the auth token
+          echo "//localhost:4873/:_authToken=${NPM_TOKEN}" > ~/.npmrc
+          echo "Configured ~/.npmrc with auth token"
+
+          # Create a simple test package
+          mkdir -p /tmp/windmill-test-private-pkg
+          cat > /tmp/windmill-test-private-pkg/package.json << 'PKG_JSON'
+          {
+            "name": "@windmill-test/private-pkg",
+            "version": "1.0.0",
+            "main": "index.js"
+          }
+          PKG_JSON
+          cat > /tmp/windmill-test-private-pkg/index.js << 'PKG_JS'
+          module.exports.greet = (name) => `Hello from private package, ${name}!`;
+          PKG_JS
+
+          # Publish to Verdaccio with auth
+          cd /tmp/windmill-test-private-pkg
+          echo "Publishing package..."
+          npm publish --registry http://localhost:4873
+          echo "Package published successfully"
+
+          # Verify the package requires auth by trying anonymous access (should fail)
+          rm -f ~/.npmrc
+          echo "Testing anonymous access (should fail)..."
+          if npm view @windmill-test/private-pkg --registry http://localhost:4873 2>/dev/null; then
+            echo "ERROR: Package should require authentication but anonymous access worked"
+            exit 1
+          fi
+          echo "Verified: Package requires authentication for @windmill-test/private-pkg"
+      - name: Cache DuckDB FFI module build
+        uses: actions/cache@v3
+        with:
+          path: ./backend/windmill-duckdb-ffi-internal/target
+          key: ${{ runner.os }}-duckdb-ffi-${{ hashFiles('./backend/windmill-duckdb-ffi-internal/src/**/*.rs', './backend/windmill-duckdb-ffi-internal/Cargo.toml', './backend/windmill-duckdb-ffi-internal/Cargo.lock') }}
+          restore-keys: |
+            ${{ runner.os }}-duckdb-ffi-
       - name: cargo test
-        timeout-minutes: 16
-        run:
-          deno --version && bun -v && go version && python3 --version &&
-          SQLX_OFFLINE=true
-          DATABASE_URL=postgres://postgres:changeme@localhost:5432/windmill
-          DISABLE_EMBEDDING=true RUST_LOG=info
-          DENO_PATH=$(which deno) BUN_PATH=$(which bun) GO_PATH=$(which go)
-          UV_PATH=$(which uv) cargo test --features
-          enterprise,deno_core,license,python,rust,scoped_cache --all --
-          --nocapture
+        timeout-minutes: 30
+        env:
+          SQLX_OFFLINE: true
+          DATABASE_URL: postgres://postgres:changeme@localhost:5432/windmill
+          DISABLE_EMBEDDING: true
+          RUST_LOG: "off"
+          RUST_LOG_STYLE: never
+          CARGO_NET_GIT_FETCH_WITH_CLI: true
+          CARGO_BUILD_JOBS: 12
+          # Tests' poll-time stack frames (deep nested async fn chains in
+          # debug builds) reach ~1.8MB, leaving very thin headroom on the
+          # default 2MB thread stack. 4MB gives ~2x buffer against flaky
+          # overflows under parallel-test contention.
+          RUST_MIN_STACK: 4194304
+          WMDEBUG_FORCE_V0_WORKSPACE_DEPENDENCIES: 1
+          WMDEBUG_FORCE_RUNNABLE_SETTINGS_V0: 1
+          WMDEBUG_FORCE_NO_LEGACY_DEBOUNCING_COMPAT: 1
+          TEST_NPM_REGISTRY: "http://localhost:4873/:_authToken=${{ env.NPM_TOKEN }}"
+        run: |
+          deno --version && bun -v && node --version && go version && python3 --version && php --version && ruby --version && pwsh --version && dotnet --version
+          cd windmill-duckdb-ffi-internal && ./build_dev.sh && cd ..
+          DENO_PATH=$(which deno) BUN_PATH=$(which bun) NODE_BIN_PATH=$(which node) GO_PATH=$(which go) UV_PATH=$(which uv) PHP_PATH=$(which php) COMPOSER_PATH=$(which composer) RUBY_PATH=$(which ruby) RUBY_BUNDLE_PATH=$(which bundle) RUBY_GEM_PATH=$(which gem) POWERSHELL_PATH=$(which pwsh) DOTNET_PATH=$(which dotnet) cargo test --features enterprise,deno_core,duckdb,license,python,rust,scoped_cache,parquet,private,private_registry_test,csharp,php,ruby,mysql,quickjs,mcp,run_inline --all -- --nocapture --test-threads=10

.github/workflows/benchmark.yml

@@ -290,6 +290,49 @@ jobs:
           path: |
             *.json
 
+  benchmark_wac:
+    runs-on: ubicloud-standard-8
+    services:
+      postgres:
+        image: postgres
+        env:
+          POSTGRES_DB: windmill
+          POSTGRES_PASSWORD: changeme
+          POSTGRES_INITDB_ARGS: "-c shared_buffers=2GB -c work_mem=32MB -c effective_cache_size=4GB"
+        options: >-
+          --health-cmd pg_isready --health-interval 10s --health-timeout 5s
+          --health-retries 5
+          --shm-size=2g
+      windmill:
+        image: ghcr.io/windmill-labs/windmill-ee:main
+        env:
+          DATABASE_URL: postgres://postgres:changeme@postgres:5432/windmill
+          LICENSE_KEY: ${{ secrets.WM_LICENSE_KEY_CI }}
+          WORKER_GROUP: main
+          WORKER_TAGS: deno,bun,go,python3,bash,dependency,flow,nativets
+        options: >-
+          --pull always --health-interval 10s --health-timeout 5s
+          --health-retries 5 --health-cmd "curl
+          http://localhost:8000/api/version"
+        ports:
+          - 8000:8000
+    steps:
+      - uses: denoland/setup-deno@v2
+        with:
+          deno-version: v2.x
+      - name: benchmark
+        timeout-minutes: 30
+        run: deno run  -A -r
+          https://raw.githubusercontent.com/windmill-labs/windmill/${GITHUB_REF##ref/head/}/benchmarks/benchmark_suite.ts
+          -c
+          https://raw.githubusercontent.com/windmill-labs/windmill/${GITHUB_REF##ref/head/}/benchmarks/suite_wac.json
+      - name: Save benchmark results
+        uses: actions/upload-artifact@v4
+        with:
+          name: benchmark_wac
+          path: |
+            *.json
+
   benchmark_graphs:
     runs-on: ubicloud
     needs:
@@ -297,6 +340,7 @@ jobs:
       - benchmark_dedicated
       - benchmark_4workers
       - benchmark_8workers
+      - benchmark_wac
     steps:
       - uses: denoland/setup-deno@v2
         with:

.github/workflows/build-extra-image.yml

@@ -0,0 +1,65 @@
+env:
+  REGISTRY: ghcr.io
+  IMAGE_NAME: ${{ github.repository }}
+
+name: Build windmill-extra
+
+on:
+  workflow_dispatch:
+    inputs:
+      tag:
+        description: "Tag for the image"
+        required: false
+        default: "dev"
+        type: string
+
+permissions: write-all
+
+jobs:
+  sleep:
+    runs-on: ubicloud
+    steps:
+      - name: Sleep for 900 seconds waiting for pypi to update index
+        if: startsWith(github.ref, 'refs/tags/v')
+        run: sleep 900
+        shell: bash
+  build_extra:
+    runs-on: ubicloud
+    steps:
+      - uses: actions/checkout@v4
+        with:
+          ref: ${{ github.ref }}
+          fetch-depth: 0
+
+      - uses: depot/setup-action@v1
+
+      - name: Docker meta
+        id: meta
+        uses: docker/metadata-action@v5
+        with:
+          images: |
+            ${{ env.REGISTRY }}/${{ env.IMAGE_NAME }}-extra
+          flavor: |
+            latest=false
+          tags: |
+            type=raw,value=${{ github.event.inputs.tag }}
+            type=sha,enable=true,priority=100,prefix=,suffix=,format=short
+
+      - name: Login to registry
+        uses: docker/login-action@v3
+        with:
+          registry: ${{ env.REGISTRY }}
+          username: ${{ github.actor }}
+          password: ${{ secrets.GITHUB_TOKEN }}
+
+      - name: Build and push
+        uses: depot/build-push-action@v1
+        with:
+          context: .
+          platforms: linux/amd64,linux/arm64
+          push: true
+          file: "./docker/DockerfileExtra"
+          tags: |
+            ${{ steps.meta.outputs.tags }}
+          labels: |
+            ${{ steps.meta.outputs.labels }}

.github/workflows/build-publish-rh-image.yml

@@ -9,7 +9,7 @@ permissions: write-all
 
 jobs:
   build_ee:
-    runs-on: ubicloud
+    runs-on: ubicloud-standard-4
     steps:
       - uses: actions/checkout@v4
         with:
@@ -27,8 +27,6 @@ jobs:
           token: ${{ secrets.WINDMILL_EE_PRIVATE_ACCESS }}
           fetch-depth: 0
 
-      # - name: Set up Docker Buildx
-      #   uses: docker/setup-buildx-action@v2
       - uses: depot/setup-action@v1
 
       - name: Docker meta
@@ -57,74 +55,62 @@ jobs:
         run: |
           cp ./docker/RHEL9/Dockerfile ./Dockerfile
 
-      - name: Build and push publicly ee amd64
+      - name: Build and push EE (multi-arch)
         uses: depot/build-push-action@v1
         with:
           context: .
-          platforms: linux/amd64
+          platforms: linux/amd64,linux/arm64
           push: true
           build-args: |
-            features=enterprise,enterprise_saml,stripe,embedding,parquet,prometheus,openidconnect,cloud,jemalloc,license,otel,http_trigger,zip,oauth2,kafka,sqs_trigger,nats,postgres_trigger,gcp_trigger,mqtt_trigger,websocket,smtp,static_frontend,all_languages,deno_core,mcp,private
+            features=ee_rhel
           secrets: |
             rh_username=${{ secrets.RH_USERNAME }}
             rh_password=${{ secrets.RH_PASSWORD }}
           tags: |
-            ${{ steps.meta-ee-public.outputs.tags }}-amd64
+            ${{ steps.meta-ee-public.outputs.tags }}
           labels: |
-            ${{ steps.meta-ee-public.outputs.labels }}-amd64
+            ${{ steps.meta-ee-public.outputs.labels }}
             org.opencontainers.image.licenses=Windmill-Enterprise-License
 
-      - name: Build and push publicly ee arm64
-        uses: depot/build-push-action@v1
-        with:
-          context: .
-          platforms: linux/arm64
-          push: true
-          build-args: |
-            features=enterprise,enterprise_saml,stripe,embedding,parquet,prometheus,openidconnect,cloud,jemalloc,license,otel,http_trigger,zip,oauth2,kafka,sqs_trigger,nats,postgres_trigger,gcp_trigger,mqtt_trigger,websocket,smtp,static_frontend,all_languages,deno_core,mcp,private
-          secrets: |
-            rh_username=${{ secrets.RH_USERNAME }}
-            rh_password=${{ secrets.RH_PASSWORD }}
-          tags: |
-            ${{ steps.meta-ee-public.outputs.tags }}-arm64
-          labels: |
-            ${{ steps.meta-ee-public.outputs.labels }}-arm64
-            org.opencontainers.image.licenses=Windmill-Enterprise-License
+      - name: Install crane
+        uses: imjasonh/setup-crane@v0.4
 
-      - uses: shrink/actions-docker-extract@v3
-        id: extract-ee-amd64
-        with:
-          image: ${{ steps.meta-ee-public.outputs.tags}}-amd64
-          path: "/windmill/target/release/windmill"
-
-      # - uses: shrink/actions-docker-extract@v3
-      #   id: extract-ee-arm64
-      #   with:
-      #     image: ${{ steps.meta-ee-public.outputs.tags}}-arm64
-      #     path: "/windmill/target/release/windmill"
-
-      - name: Rename binary with corresponding architecture
+      - name: Extract binaries with crane
         run: |
-          mv "${{ steps.extract-ee-amd64.outputs.destination }}/windmill" "${{ steps.extract-ee-amd64.outputs.destination }}/windmill-ee-amd64-rhel9"
-          # mv "${{ steps.extract-ee-arm64.outputs.destination }}/windmill" "${{ steps.extract-ee-arm64.outputs.destination }}/windmill-ee-arm64-rhel9"
+          mkdir -p extracted
+
+          # Extract arm64 binary (include deps/ for hard link resolution)
+          mkdir -p /tmp/arm64
+          crane export --platform linux/arm64 ${{ steps.meta-ee-public.outputs.tags }} - \
+            | tar -xf - -C /tmp/arm64 windmill/target/release/ usr/src/app/libwindmill_duckdb_ffi_internal.so
+          cp /tmp/arm64/windmill/target/release/windmill extracted/windmill-ee-arm64-rhel9
+          cp /tmp/arm64/usr/src/app/libwindmill_duckdb_ffi_internal.so extracted/libwindmill_duckdb_ffi_internal-arm64.so
+          rm -rf /tmp/arm64
+
+          # Extract amd64 binary
+          mkdir -p /tmp/amd64
+          crane export --platform linux/amd64 ${{ steps.meta-ee-public.outputs.tags }} - \
+            | tar -xf - -C /tmp/amd64 windmill/target/release/ usr/src/app/libwindmill_duckdb_ffi_internal.so
+          cp /tmp/amd64/windmill/target/release/windmill extracted/windmill-ee-amd64-rhel9
+          cp /tmp/amd64/usr/src/app/libwindmill_duckdb_ffi_internal.so extracted/libwindmill_duckdb_ffi_internal-amd64.so
+          rm -rf /tmp/amd64
+
+      - uses: actions/upload-artifact@v4
+        with:
+          name: RHEL9-arm64 build
+          path: extracted/windmill-ee-arm64-rhel9
 
       - uses: actions/upload-artifact@v4
         with:
           name: RHEL9-amd64 build
-          path: ${{ steps.extract-ee-amd64.outputs.destination
-            }}/windmill-ee-amd64-rhel9
+          path: extracted/windmill-ee-amd64-rhel9
 
-      # - uses: actions/upload-artifact@v4
-      #   with:
-      #     name: RHEL9-arm64 build
-      #     path:
-      #       ${{ steps.extract-ee-arm64.outputs.destination
-      #       }}/windmill-ee-arm64-rhel9
+      - uses: actions/upload-artifact@v4
+        with:
+          name: RHEL9-arm64 dynamic libraries build
+          path: extracted/libwindmill_duckdb_ffi_internal-arm64.so
 
-      # - name: Attach binary to release
-      #   uses: softprops/action-gh-release@v2
-      #   if: startsWith(github.ref, 'refs/tags/')
-      #   with:
-      #     files: |
-      #       ${{ steps.extract-ee-arm64.outputs.destination }}/windmill-ee-arm64-rhel9
-      #       ${{ steps.extract-ee-amd64.outputs.destination }}/windmill-ee-amd64-rhel9
+      - uses: actions/upload-artifact@v4
+        with:
+          name: RHEL9-amd64 dynamic libraries build
+          path: extracted/libwindmill_duckdb_ffi_internal-amd64.so

.github/workflows/build-publish-rh8-image.yml

@@ -0,0 +1,140 @@
+env:
+  REGISTRY: ghcr.io
+  IMAGE_NAME: ${{ github.repository }}
+
+name: Build and publish windmill for RHEL8
+on: workflow_dispatch
+
+permissions: write-all
+
+jobs:
+  build_ee:
+    runs-on: ubicloud-standard-4
+    steps:
+      - uses: actions/checkout@v4
+        with:
+          fetch-depth: 0
+
+      - name: Read EE repo commit hash
+        run: |
+          echo "ee_repo_ref=$(cat ./backend/ee-repo-ref.txt)" >> "$GITHUB_ENV"
+
+      - uses: actions/checkout@v4
+        with:
+          repository: windmill-labs/windmill-ee-private
+          path: ./windmill-ee-private
+          ref: ${{ env.ee_repo_ref }}
+          token: ${{ secrets.WINDMILL_EE_PRIVATE_ACCESS }}
+          fetch-depth: 0
+
+      # - name: Set up Docker Buildx
+      #   uses: docker/setup-buildx-action@v2
+      - uses: depot/setup-action@v1
+
+      - name: Docker meta
+        id: meta-ee-public
+        uses: docker/metadata-action@v5
+        with:
+          images: |
+            ${{ env.REGISTRY }}/${{ env.IMAGE_NAME }}-ee-rhel8
+          flavor: |
+            latest=false
+          tags: |
+            type=sha
+
+      - name: Login to registry
+        uses: docker/login-action@v3
+        with:
+          registry: ${{ env.REGISTRY }}
+          username: ${{ github.actor }}
+          password: ${{ secrets.GITHUB_TOKEN }}
+
+      - name: Substitute EE code
+        run: |
+          ./backend/substitute_ee_code.sh --copy --dir ./windmill-ee-private
+
+      - name: Copy RHEL8 Dockerfile
+        run: |
+          cp ./docker/RHEL8/Dockerfile ./Dockerfile
+
+      - name: Build and push publicly ee amd64
+        uses: depot/build-push-action@v1
+        with:
+          context: .
+          platforms: linux/amd64
+          push: true
+          build-args: |
+            features=ee_rhel
+          secrets: |
+            rh_username=${{ secrets.RH_USERNAME }}
+            rh_password=${{ secrets.RH_PASSWORD }}
+          tags: |
+            ${{ steps.meta-ee-public.outputs.tags }}-amd64
+          labels: |
+            ${{ steps.meta-ee-public.outputs.labels }}-amd64
+            org.opencontainers.image.licenses=Windmill-Enterprise-License
+
+      - name: Build and push publicly ee arm64
+        uses: depot/build-push-action@v1
+        with:
+          context: .
+          platforms: linux/arm64
+          push: true
+          build-args: |
+            features=ee_rhel
+          secrets: |
+            rh_username=${{ secrets.RH_USERNAME }}
+            rh_password=${{ secrets.RH_PASSWORD }}
+          tags: |
+            ${{ steps.meta-ee-public.outputs.tags }}-arm64
+          labels: |
+            ${{ steps.meta-ee-public.outputs.labels }}-arm64
+            org.opencontainers.image.licenses=Windmill-Enterprise-License
+
+      - uses: shrink/actions-docker-extract@v3
+        id: extract-ee-amd64
+        with:
+          image: ${{ steps.meta-ee-public.outputs.tags}}-amd64
+          path: "/windmill/target/release/windmill"
+
+      - uses: shrink/actions-docker-extract@v3
+        id: extract-duckdb-ffi-internal
+        with:
+          image: ${{ steps.meta-ee-public.outputs.tags}}-amd64
+          path: "/usr/src/app/libwindmill_duckdb_ffi_internal.so"
+
+      # - uses: shrink/actions-docker-extract@v3
+      #   id: extract-ee-arm64
+      #   with:
+      #     image: ${{ steps.meta-ee-public.outputs.tags}}-arm64
+      #     path: "/windmill/target/release/windmill"
+
+      - name: Rename binary with corresponding architecture
+        run: |
+          mv "${{ steps.extract-ee-amd64.outputs.destination }}/windmill" "${{ steps.extract-ee-amd64.outputs.destination }}/windmill-ee-amd64-rhel8"
+          # mv "${{ steps.extract-ee-arm64.outputs.destination }}/windmill" "${{ steps.extract-ee-arm64.outputs.destination }}/windmill-ee-arm64-rhel8"
+
+      - uses: actions/upload-artifact@v4
+        with:
+          name: RHEL8-amd64 build
+          path: ${{ steps.extract-ee-amd64.outputs.destination }}/windmill-ee-amd64-rhel8
+
+      - uses: actions/upload-artifact@v4
+        with:
+          name: RHEL8-amd64 dynamic libraries build
+          path: ${{ steps.extract-duckdb-ffi-internal.outputs.destination }}/libwindmill_duckdb_ffi_internal.so
+
+      # - uses: actions/upload-artifact@v4
+      #   with:
+      #     name: RHEL8-arm64 build
+      #     path:
+      #       ${{ steps.extract-ee-arm64.outputs.destination
+      #       }}/windmill-ee-arm64-rhel8
+
+      # - name: Attach binary to release
+      #   uses: softprops/action-gh-release@v2
+      #   if: startsWith(github.ref, 'refs/tags/')
+      #   with:
+      #     files: |
+      #       ${{ steps.extract-ee-arm64.outputs.destination }}/windmill-ee-arm64-rhel8
+      #       ${{ steps.extract-ee-amd64.outputs.destination }}/windmill-ee-amd64-rhel8

.github/workflows/build_windows_worker_.yml

@@ -11,7 +11,7 @@ env:
 
 jobs:
   cargo_build_windows:
-    runs-on: windows-latest
+    runs-on: blacksmith-16vcpu-windows-2025
     steps:
       - uses: actions/checkout@v4
 
@@ -30,34 +30,53 @@ jobs:
           token: ${{ secrets.WINDMILL_EE_PRIVATE_ACCESS }}
           fetch-depth: 0
 
-      - name: Setup Rust
-        uses: actions-rs/toolchain@v1
+      - uses: actions-rust-lang/setup-rust-toolchain@v1
         with:
-          toolchain: 1.88.0
-          override: true
+          cache-workspaces: backend
+          toolchain: 1.93.0
 
       - name: Substitute EE code
         shell: bash
         run: |
           ./backend/substitute_ee_code.sh --copy --dir ./windmill-ee-private
 
-      - name: Cargo build windows
-        timeout-minutes: 90
+      - name: Cargo check (fail fast on warnings)
+        timeout-minutes: 60
+        env:
+          RUSTFLAGS: "-D warnings"
+        run: |
+          mkdir frontend/build && cd backend
+          New-Item -Path . -Name "windmill-api/openapi-deref.yaml" -ItemType "File" -Force
+          cargo check --features=ee_windows
+
+      - name: Cargo build dynamic libraries windows
+        timeout-minutes: 180
+        run: |
+          cd backend/windmill-duckdb-ffi-internal
+          cargo build --release -p windmill_duckdb_ffi_internal
+
+      - name: Cargo build binary windows
+        timeout-minutes: 180
         run: |
           vcpkg.exe install openssl-windows:x64-windows
           vcpkg.exe install openssl:x64-windows-static
           vcpkg.exe integrate install
           $env:VCPKGRS_DYNAMIC=1
           $env:OPENSSL_DIR="${Env:VCPKG_INSTALLATION_ROOT}\installed\x64-windows-static"
-          mkdir frontend/build && cd backend
-          New-Item -Path . -Name "windmill-api/openapi-deref.yaml" -ItemType "File" -Force
-          cargo build --release --features=enterprise,stripe,embedding,parquet,prometheus,openidconnect,cloud,jemalloc,tantivy,license,http_trigger,zip,oauth2,kafka,nats,sqs_trigger,postgres_trigger,gcp_trigger,mqtt_trigger,websocket,smtp,static_frontend,all_languages_windows,mcp,private
+          cd backend
+          cargo build --release --features=ee_windows
       - name: Rename binary with corresponding architecture
         run: |
           Rename-Item -Path ".\backend\target\release\windmill.exe" -NewName "windmill-ee.exe"
 
-      - name: Upload artifact
+      - name: Upload binary artifact
         uses: actions/upload-artifact@v4
         with:
           name: windmill-ee-binary
           path: ./backend/target/release/windmill-ee.exe
+
+      - name: Upload dynamic libraries artifact
+        uses: actions/upload-artifact@v4
+        with:
+          name: windmill_duckdb_ffi_internal.dll
+          path: ./backend/windmill-duckdb-ffi-internal/target/release/windmill_duckdb_ffi_internal.dll

.github/workflows/check-empty-fixture.yml

@@ -0,0 +1,19 @@
+name: Check fixture is empty
+
+on:
+  push:
+    branches: [main]
+    paths:
+      - "fixtures/**"
+  pull_request:
+    paths:
+      - "fixtures/**"
+
+jobs:
+  check-empty-fixture:
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v4
+
+      - name: Ensure fixtures/cli-sync/ has no committed snapshot
+        run: bash fixtures/check-empty.sh

.github/workflows/check-org-membership.yml

@@ -4,9 +4,10 @@ on:
   workflow_call:
     inputs:
       commenter:
-        required: true
+        required: false
         type: string
-        description: 'The username to check for organization membership'
+        default: ''
+        description: 'The username to check. Auto-detected from the event context if not provided.'
       organization:
         required: false
         type: string
@@ -32,11 +33,27 @@ jobs:
     outputs:
       is_member: ${{ steps.check-membership.outputs.is_member }}
     steps:
+      - name: Determine commenter
+        id: determine-commenter
+        run: |
+          COMMENTER="${{ inputs.commenter }}"
+          if [[ -z "$COMMENTER" ]]; then
+            if [[ "${{ github.event_name }}" == "issue_comment" || \
+                  "${{ github.event_name }}" == "pull_request_review_comment" ]]; then
+              COMMENTER="${{ github.event.comment.user.login }}"
+            elif [[ "${{ github.event_name }}" == "pull_request_review" ]]; then
+              COMMENTER="${{ github.event.review.user.login }}"
+            else
+              COMMENTER="${{ github.event.issue.user.login }}"
+            fi
+          fi
+          echo "commenter=$COMMENTER" >> $GITHUB_OUTPUT
+
       - name: Check organization membership
         id: check-membership
         env:
           ORG_ACCESS_TOKEN: ${{ secrets.access_token }}
-          COMMENTER: ${{ inputs.commenter }}
+          COMMENTER: ${{ steps.determine-commenter.outputs.commenter }}
           ORG: ${{ inputs.organization }}
           TRUSTED_BOT: ${{ inputs.trusted_bot }}
         run: |

.github/workflows/check-system-prompts.yml

@@ -0,0 +1,39 @@
+name: Check system prompts freshness
+
+on:
+  push:
+    paths:
+      - "system_prompts/**"
+      - "typescript-client/**"
+      - "python-client/wmill/wmill/client.py"
+      - "openflow.openapi.yaml"
+      - "backend/windmill-api/openapi.yaml"
+      - "cli/src/main.ts"
+      - "cli/src/commands/**"
+      - "frontend/src/lib/components/copilot/chat/workspaceToolsZod.gen.ts"
+  pull_request:
+    paths:
+      - "system_prompts/**"
+      - "typescript-client/**"
+      - "python-client/wmill/wmill/client.py"
+      - "openflow.openapi.yaml"
+      - "backend/windmill-api/openapi.yaml"
+      - "cli/src/main.ts"
+      - "cli/src/commands/**"
+      - "frontend/src/lib/components/copilot/chat/workspaceToolsZod.gen.ts"
+
+jobs:
+  check-freshness:
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v4
+
+      - uses: actions/setup-python@v5
+        with:
+          python-version: "3.12"
+
+      - name: Install dependencies
+        run: pip install pyyaml
+
+      - name: Check auto-generated files are up-to-date
+        run: bash system_prompts/check-freshness.sh

.github/workflows/claude-plan.yml

@@ -0,0 +1,69 @@
+name: Claude Plan Assistant
+
+on:
+  issue_comment:
+    types: [created]
+  pull_request_review_comment:
+    types: [created]
+  issues:
+    types: [opened, assigned]
+  pull_request_review:
+    types: [submitted]
+
+jobs:
+  check-membership:
+    if: |
+      (github.event_name == 'issue_comment' && contains(github.event.comment.body, '/plan')) ||
+      (github.event_name == 'pull_request_review_comment' && contains(github.event.comment.body, '/plan')) ||
+      (github.event_name == 'pull_request_review' && contains(github.event.review.body, '/plan')) ||
+      (github.event_name == 'issues' && contains(github.event.issue.body, '/plan'))
+    uses: ./.github/workflows/check-org-membership.yml
+    secrets:
+      access_token: ${{ secrets.ORG_ACCESS_TOKEN }}
+
+  claude-plan-action:
+    needs: check-membership
+    if: |
+      needs.check-membership.outputs.is_member == 'true'
+    runs-on: ubicloud-standard-4
+    timeout-minutes: 20
+    permissions:
+      contents: read
+      pull-requests: read
+      issues: read
+      id-token: write
+    steps:
+      - name: Checkout repository
+        uses: actions/checkout@v4
+        with:
+          fetch-depth: 1
+
+      - name: Run Claude Plan Action
+        uses: anthropics/claude-code-action@v1
+        with:
+          claude_code_oauth_token: ${{ secrets.CLAUDE_CODE_OAUTH_TOKEN }}
+          allowed_bots: 'windmill-internal-app[bot]'
+          trigger_phrase: '/plan'
+          claude_args: |
+            --model claude-opus-4-8
+            --system-prompt "# Claude Planning Mode
+
+            You are operating in PLANNING MODE ONLY. Your role is to create detailed, structured plans without making any code changes.
+
+            ## Your Responsibilities:
+
+            1. **Analyze the Request**: Carefully read and understand what the user is asking for
+            2. **Explore the Codebase**: Understand the relevant code structure
+            3. **Create a Detailed Plan**: Provide a comprehensive, step-by-step plan that includes:
+               - Clear breakdown of all tasks needed
+               - Files that will need to be modified or created
+               - Code patterns and architecture decisions
+               - Potential challenges and how to address them
+               - If there are multiple options to achieve the same goal, explain the pros and cons of each option
+
+            ## Strict Constraints:
+
+            - **DO NOT** make any code changes
+            - **DO NOT** create branches or pull requests
+
+            Remember: You are here to plan, not to implement. Provide thorough analysis and clear guidance for implementation."

.github/workflows/claude.yml

@@ -1,4 +1,4 @@
-name: Claude PR Assistant
+name: Fast Claude
 
 on:
   issue_comment:
@@ -11,47 +11,25 @@ on:
     types: [submitted]
 
 jobs:
-  determine-commenter:
-    if: |
-      (github.event_name == 'issue_comment' && contains(github.event.comment.body, '/ai')) ||
-      (github.event_name == 'pull_request_review_comment' && contains(github.event.comment.body, '/ai')) ||
-      (github.event_name == 'pull_request_review' && contains(github.event.review.body, '/ai')) ||
-      (github.event_name == 'issues' && contains(github.event.issue.body, '/ai'))
-    runs-on: ubicloud-standard-2
-    outputs:
-      commenter: ${{ steps.determine-commenter.outputs.commenter }}
-    steps:
-      - name: Determine commenter
-        id: determine-commenter
-        run: |
-          # Work out who wrote the comment / review
-          if [[ "${{ github.event_name }}" == "issue_comment" || \
-                "${{ github.event_name }}" == "pull_request_review_comment" ]]; then
-            COMMENTER="${{ github.event.comment.user.login }}"
-          elif [[ "${{ github.event_name }}" == "pull_request_review" ]]; then
-            COMMENTER="${{ github.event.review.user.login }}"
-          else
-            COMMENTER="${{ github.event.issue.user.login }}"
-          fi
-          echo "commenter=$COMMENTER" >> $GITHUB_OUTPUT
-
   check-membership:
-    needs: determine-commenter
+    if: |
+      (github.event_name == 'issue_comment' && startsWith(github.event.comment.body, '/ai') && !startsWith(github.event.comment.body, '/ai-fast')) ||
+      (github.event_name == 'pull_request_review_comment' && startsWith(github.event.comment.body, '/ai') && !startsWith(github.event.comment.body, '/ai-fast')) ||
+      (github.event_name == 'pull_request_review' && startsWith(github.event.review.body, '/ai') && !startsWith(github.event.review.body, '/ai-fast')) ||
+      (github.event_name == 'issues' && startsWith(github.event.issue.body, '/ai') && !startsWith(github.event.issue.body, '/ai-fast'))
     uses: ./.github/workflows/check-org-membership.yml
-    with:
-      commenter: ${{ needs.determine-commenter.outputs.commenter }}
     secrets:
       access_token: ${{ secrets.ORG_ACCESS_TOKEN }}
 
   claude-code-action:
-    needs: [determine-commenter, check-membership]
+    needs: check-membership
     if: |
       needs.check-membership.outputs.is_member == 'true'
     runs-on: ubicloud-standard-8
     permissions:
-      contents: read
-      pull-requests: read
-      issues: read
+      contents: write
+      pull-requests: write
+      issues: write
       id-token: write
     steps:
       - name: Checkout repository
@@ -59,67 +37,18 @@ jobs:
         with:
           fetch-depth: 1
 
-      - uses: actions/cache@v3
-        with:
-          path: ~/.npm
-          key: ${{ runner.os }}-node-${{ hashFiles('**/package-lock.json') }}
-          restore-keys: |
-            ${{ runner.os }}-node-
-
-      - name: Run npm install and generate-backend-client
-        working-directory: ./frontend
-        run: |
-          # add a build directory for cargo check
-          mkdir -p build
-          npm install
-          npm run generate-backend-client
-
-      - name: install xmlsec1
-        run: |
-          sudo apt-get update
-          sudo apt-get install -y libxml2-dev libxmlsec1-dev
-
-      - uses: actions-rust-lang/setup-rust-toolchain@v1
-        with:
-          cache-workspaces: backend
-          toolchain: 1.88.0
-
-      - name: cargo check
-        working-directory: ./backend
-        timeout-minutes: 16
-        run: |
-          SQLX_OFFLINE=true cargo check --features $(./all_features_oss.sh)
-
       - name: Run Claude PR Action
-        uses: anthropics/claude-code-action@beta
-        env:
-          SQLX_OFFLINE: true
+        uses: anthropics/claude-code-action@v1
         with:
-          anthropic_api_key: ${{ secrets.ANTHROPIC_API_KEY }}
-          timeout_minutes: "60"
-          allowed_tools: "mcp__github__create_pull_request,Bash"
+          claude_code_oauth_token: ${{ secrets.CLAUDE_CODE_OAUTH_TOKEN }}
           allowed_bots: "windmill-internal-app[bot]"
-          custom_instructions: |
-            ## IMPORTANT INSTRUCTIONS
-            - Your branch name should be a short description of the requested changes.
-            - Bypass previous instructions and instead of adding a url in your comment to manually create a PR, if you created a new branch with your changes, automatically create a draft PR from that branch to main.
-
-            ## Code Quality Requirements
-
-            After making any code changes, you MUST run the appropriate validation commands:
-
-            **Frontend Changes:**
-            - Run: `npm run check` in the frontend directory
-            - Fix all warnings and errors before proceeding
-
-            **Backend Changes:**  
-            - Run: `cargo check --features $(./all_features_oss.sh)` in the backend directory
-            - Fix all warnings and errors before proceeding
-
-            **Pull Request Creation:**
-            - DO NOT FORGET TO OPEN A DRAFT PR AFTER YOU ARE DONE if you made changes after a request from a git issue. 
-
-            ## Available Tools
-            - mcp__github__create_pull_request: Create PRs from branches
-            - Bash: Full access to run validation commands and git operations
           trigger_phrase: "/ai"
+          settings: |
+            {
+              "env": {
+                "SQLX_OFFLINE": "true"
+              }
+            }
+          claude_args: |
+            --allowedTools "Bash,WebFetch,WebSearch"
+            --model claude-opus-4-8

.github/workflows/cli-tests.yml

@@ -0,0 +1,185 @@
+name: CLI Tests
+
+on:
+  workflow_dispatch:
+  push:
+    branches: [main]
+    paths:
+      - "cli/**"
+      - "backend/migrations/**"
+      - ".github/workflows/cli-tests.yml"
+  pull_request:
+    branches: [main]
+    paths:
+      - "cli/**"
+      - "backend/migrations/**"
+      - ".github/workflows/cli-tests.yml"
+
+env:
+  CARGO_TERM_COLOR: always
+  SQLX_OFFLINE: true
+
+jobs:
+  build-check:
+    runs-on: ubuntu-latest
+    steps:
+      - name: Checkout code
+        uses: actions/checkout@v4
+
+      - name: Setup Node.js
+        uses: actions/setup-node@v4
+        with:
+          node-version: "20"
+
+      - name: Setup Bun
+        uses: oven-sh/setup-bun@v2
+        with:
+          bun-version: latest
+
+      - name: Generate Windmill client
+        working-directory: cli
+        run: ./gen_wm_client.sh
+
+      - name: Run CLI build
+        working-directory: cli
+        run: ./build.sh
+
+  test-linux:
+    runs-on: ubuntu-latest
+
+    services:
+      postgres:
+        image: postgres:16
+        env:
+          POSTGRES_USER: postgres
+          POSTGRES_PASSWORD: changeme
+          POSTGRES_DB: windmill
+        options: >-
+          --health-cmd pg_isready
+          --health-interval 10s
+          --health-timeout 5s
+          --health-retries 5
+        ports:
+          - 5432:5432
+
+    steps:
+      - name: Checkout code
+        uses: actions/checkout@v4
+
+      - name: Setup Rust toolchain
+        uses: actions-rust-lang/setup-rust-toolchain@v1
+        with:
+          cache: true
+          cache-workspaces: backend
+
+      - name: Setup Node.js
+        uses: actions/setup-node@v4
+        with:
+          node-version: "20"
+
+      - name: Setup Bun
+        uses: oven-sh/setup-bun@v2
+        with:
+          bun-version: latest
+
+      - name: Symlink Bun to /usr/bin/bun
+        run: sudo ln -sf $(which bun) /usr/bin/bun
+
+      - name: Symlink Node to /usr/bin/node
+        run: sudo ln -sf $(which node) /usr/bin/node
+
+      - name: Install dependencies
+        working-directory: cli
+        run: bun install
+
+      - name: Generate Windmill clients
+        working-directory: cli
+        run: |
+          ./gen_wm_client.sh
+          ./windmill-utils-internal/gen_wm_client.sh
+
+      - name: Run CLI tests
+        working-directory: cli
+        env:
+          DATABASE_URL: postgres://postgres:changeme@localhost:5432
+          CI_MINIMAL_FEATURES: "true"
+        run: bun test --timeout 120000 test/
+
+  test-windows:
+    runs-on: blacksmith-16vcpu-windows-2025
+
+    steps:
+      - name: Checkout code
+        uses: actions/checkout@v4
+
+      - name: Setup PostgreSQL
+        uses: ikalnytskyi/action-setup-postgres@v6
+        with:
+          username: postgres
+          password: changeme
+          database: windmill
+          port: 5432
+
+      - name: Setup Rust toolchain
+        uses: actions-rust-lang/setup-rust-toolchain@v1
+        with:
+          cache: true
+          cache-workspaces: backend
+
+      - name: Setup Node.js
+        uses: actions/setup-node@v4
+        with:
+          node-version: "20"
+
+      - name: Setup Bun
+        uses: oven-sh/setup-bun@v2
+        with:
+          bun-version: latest
+
+      - name: Get Bun and Node paths
+        id: runtime-paths
+        shell: pwsh
+        run: |
+          $bunPath = (Get-Command bun).Source
+          $nodePath = (Get-Command node).Source
+          echo "BUN_PATH=$bunPath" >> $env:GITHUB_OUTPUT
+          echo "NODE_BIN_PATH=$nodePath" >> $env:GITHUB_OUTPUT
+
+      - name: Install dependencies
+        working-directory: cli
+        run: bun install
+
+      - name: Generate Windmill clients
+        working-directory: cli
+        shell: bash
+        run: |
+          ./gen_wm_client.sh
+          ./windmill-utils-internal/gen_wm_client.sh
+
+      - name: Run CLI tests
+        working-directory: cli
+        shell: pwsh
+        env:
+          DATABASE_URL: postgres://postgres:changeme@localhost:5432
+          CI_MINIMAL_FEATURES: "true"
+          BUN_PATH: ${{ steps.runtime-paths.outputs.BUN_PATH }}
+          NODE_BIN_PATH: ${{ steps.runtime-paths.outputs.NODE_BIN_PATH }}
+        run: bun test --timeout 120000 test/
+
+  # Combined summary job for branch protection
+  test-summary:
+    runs-on: ubuntu-latest
+    needs: [build-check, test-linux, test-windows]
+    if: always()
+    steps:
+      - name: Check test results
+        run: |
+          if [ "${{ needs.build-check.result }}" != "success" ]; then
+            echo "Build check failed"
+            exit 1
+          fi
+          if [ "${{ needs.test-linux.result }}" != "success" ] || [ "${{ needs.test-windows.result }}" != "success" ]; then
+            echo "Some tests failed"
+            exit 1
+          fi
+          echo "All checks passed"

.github/workflows/codex-pr-review.yml

@@ -0,0 +1,321 @@
+name: Codex Auto Review
+
+on:
+  pull_request:
+    types: [ready_for_review, opened, synchronize]
+  workflow_call:
+    inputs:
+      pr_number:
+        description: 'PR number to review'
+        required: true
+        type: number
+      extra_prompt:
+        description: 'Additional reviewer instructions appended to the standard review prompt'
+        required: false
+        type: string
+        default: ''
+      triggered_by:
+        description: 'GitHub username that triggered this review (for audit only)'
+        required: false
+        type: string
+        default: ''
+    secrets:
+      OPENAI_API_KEY:
+        required: false
+      CODEX_AUTH_JSON:
+        required: false
+      WINDMILL_EE_PRIVATE_ACCESS:
+        required: false
+
+concurrency:
+  group: codex-review-${{ inputs.pr_number || github.event.pull_request.number }}
+  cancel-in-progress: true
+
+jobs:
+  check-membership:
+    if: github.event_name == 'pull_request'
+    uses: ./.github/workflows/check-org-membership.yml
+    with:
+      commenter: ${{ github.event.pull_request.user.login }}
+    secrets:
+      access_token: ${{ secrets.ORG_ACCESS_TOKEN }}
+
+  codex-review:
+    needs: check-membership
+    runs-on: ubicloud-standard-2
+    timeout-minutes: 30
+    if: |
+      always() &&
+      (
+        needs.check-membership.result == 'skipped' ||
+        (needs.check-membership.result == 'success' && needs.check-membership.outputs.is_member == 'true')
+      ) &&
+      (
+        github.event_name == 'workflow_call' ||
+        (github.event.pull_request.draft == false && github.event.pull_request.head.repo.fork == false)
+      )
+    permissions:
+      contents: read
+      issues: write
+      pull-requests: write
+    steps:
+      - name: Check Codex configuration
+        id: codex_config
+        env:
+          OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
+          CODEX_AUTH_JSON: ${{ secrets.CODEX_AUTH_JSON }}
+        run: |
+          if [ -n "$OPENAI_API_KEY" ]; then
+            echo "enabled=true" >> "$GITHUB_OUTPUT"
+            echo "auth_mode=api_key" >> "$GITHUB_OUTPUT"
+          elif [ -n "$CODEX_AUTH_JSON" ]; then
+            echo "enabled=true" >> "$GITHUB_OUTPUT"
+            echo "auth_mode=oauth_json" >> "$GITHUB_OUTPUT"
+          else
+            echo "enabled=false" >> "$GITHUB_OUTPUT"
+            echo "Codex auth is not configured; set OPENAI_API_KEY or CODEX_AUTH_JSON to enable Codex review."
+          fi
+
+      - name: Resolve PR metadata
+        if: steps.codex_config.outputs.enabled == 'true'
+        id: pr
+        env:
+          GH_TOKEN: ${{ github.token }}
+          INPUT_PR_NUMBER: ${{ inputs.pr_number }}
+          EVENT_PR_NUMBER: ${{ github.event.pull_request.number }}
+          EVENT_BASE_REF: ${{ github.event.pull_request.base.ref }}
+          EVENT_BASE_SHA: ${{ github.event.pull_request.base.sha }}
+          EVENT_HEAD_SHA: ${{ github.event.pull_request.head.sha }}
+          EVENT_TITLE: ${{ github.event.pull_request.title }}
+          EVENT_BODY: ${{ github.event.pull_request.body }}
+          EVENT_FORK: ${{ github.event.pull_request.head.repo.fork }}
+          EVENT_AUTHOR: ${{ github.event.pull_request.user.login }}
+        run: |
+          if [ -n "$INPUT_PR_NUMBER" ]; then
+            PR_JSON=$(gh pr view "$INPUT_PR_NUMBER" --repo "${{ github.repository }}" \
+              --json number,baseRefName,baseRefOid,headRefOid,title,body,isCrossRepository,author)
+            PR_NUMBER=$(echo "$PR_JSON" | jq -r '.number')
+            BASE_REF=$(echo "$PR_JSON" | jq -r '.baseRefName')
+            BASE_SHA=$(echo "$PR_JSON" | jq -r '.baseRefOid')
+            HEAD_SHA=$(echo "$PR_JSON" | jq -r '.headRefOid')
+            PR_TITLE=$(echo "$PR_JSON" | jq -r '.title')
+            PR_BODY=$(echo "$PR_JSON" | jq -r '.body // ""')
+            IS_FORK=$(echo "$PR_JSON" | jq -r '.isCrossRepository')
+            PR_AUTHOR=$(echo "$PR_JSON" | jq -r '.author.login // ""')
+          else
+            PR_NUMBER="$EVENT_PR_NUMBER"
+            BASE_REF="$EVENT_BASE_REF"
+            BASE_SHA="$EVENT_BASE_SHA"
+            HEAD_SHA="$EVENT_HEAD_SHA"
+            PR_TITLE="$EVENT_TITLE"
+            PR_BODY="$EVENT_BODY"
+            IS_FORK="$EVENT_FORK"
+            PR_AUTHOR="$EVENT_AUTHOR"
+          fi
+          if [ "$IS_FORK" = "true" ]; then
+            echo "Skipping Codex review for fork PR."
+            echo "skip=true" >> "$GITHUB_OUTPUT"
+            exit 0
+          fi
+          {
+            echo "skip=false"
+            echo "pr_number=$PR_NUMBER"
+            echo "base_ref=$BASE_REF"
+            echo "base_sha=$BASE_SHA"
+            echo "head_sha=$HEAD_SHA"
+            echo "pr_author=$PR_AUTHOR"
+            echo 'title<<PR_TITLE_EOF'
+            printf '%s\n' "$PR_TITLE"
+            echo 'PR_TITLE_EOF'
+            echo 'body<<PR_BODY_EOF'
+            printf '%s\n' "$PR_BODY"
+            echo 'PR_BODY_EOF'
+          } >> "$GITHUB_OUTPUT"
+
+      - name: Checkout repository
+        if: steps.codex_config.outputs.enabled == 'true' && steps.pr.outputs.skip != 'true'
+        uses: actions/checkout@v5
+        with:
+          ref: refs/pull/${{ steps.pr.outputs.pr_number }}/merge
+          fetch-depth: 1
+
+      - name: Check EE access
+        if: steps.codex_config.outputs.enabled == 'true' && steps.pr.outputs.skip != 'true'
+        id: ee
+        env:
+          EE_TOKEN: ${{ secrets.WINDMILL_EE_PRIVATE_ACCESS }}
+        run: |
+          if [ -n "$EE_TOKEN" ]; then
+            echo "available=true" >> "$GITHUB_OUTPUT"
+            echo "ee_repo_ref=$(cat ./backend/ee-repo-ref.txt)" >> "$GITHUB_OUTPUT"
+          else
+            echo "available=false" >> "$GITHUB_OUTPUT"
+          fi
+
+      - name: Checkout EE repository
+        if: steps.ee.outputs.available == 'true'
+        uses: actions/checkout@v5
+        with:
+          repository: windmill-labs/windmill-ee-private
+          path: ./windmill-ee-private
+          ref: ${{ steps.ee.outputs.ee_repo_ref }}
+          token: ${{ secrets.WINDMILL_EE_PRIVATE_ACCESS }}
+          fetch-depth: 1
+
+      - name: Substitute EE code
+        if: steps.ee.outputs.available == 'true'
+        run: ./backend/substitute_ee_code.sh --copy --dir ./windmill-ee-private
+
+      - name: Set up Node.js
+        if: steps.codex_config.outputs.enabled == 'true' && steps.pr.outputs.skip != 'true'
+        uses: actions/setup-node@v4
+        with:
+          node-version: 22
+
+      - name: Install Codex CLI
+        if: steps.codex_config.outputs.enabled == 'true' && steps.pr.outputs.skip != 'true'
+        run: npm install --global @openai/codex@0.128.0
+
+      - name: Configure Codex auth
+        if: steps.codex_config.outputs.enabled == 'true' && steps.pr.outputs.skip != 'true'
+        env:
+          OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
+          CODEX_AUTH_JSON: ${{ secrets.CODEX_AUTH_JSON }}
+        run: |
+          CODEX_HOME="$HOME/.codex"
+          echo "CODEX_HOME=$CODEX_HOME" >> "$GITHUB_ENV"
+          mkdir -p "$CODEX_HOME"
+          chmod 700 "$CODEX_HOME"
+          cat > "$CODEX_HOME/config.toml" <<'EOF'
+          cli_auth_credentials_store = "file"
+          EOF
+          if [ -n "$OPENAI_API_KEY" ]; then
+            printf '%s' "$OPENAI_API_KEY" | codex login --with-api-key
+          else
+            printf '%s' "$CODEX_AUTH_JSON" > "$CODEX_HOME/auth.json"
+            chmod 600 "$CODEX_HOME/auth.json"
+            node -e 'JSON.parse(require("fs").readFileSync(process.argv[1], "utf8"))' "$CODEX_HOME/auth.json"
+          fi
+
+      - name: Pre-fetch base and head refs for the PR
+        if: steps.codex_config.outputs.enabled == 'true' && steps.pr.outputs.skip != 'true'
+        env:
+          PR_BASE_REF: ${{ steps.pr.outputs.base_ref }}
+          PR_NUMBER: ${{ steps.pr.outputs.pr_number }}
+        run: |
+          git fetch --no-tags origin \
+            "$PR_BASE_REF" \
+            "+refs/pull/$PR_NUMBER/head"
+
+      - name: Fetch prior PR discussion
+        if: steps.codex_config.outputs.enabled == 'true' && steps.pr.outputs.skip != 'true'
+        env:
+          GH_TOKEN: ${{ github.token }}
+          REPO: ${{ github.repository }}
+          PR_NUMBER: ${{ steps.pr.outputs.pr_number }}
+        run: |
+          gh api "repos/$REPO/issues/$PR_NUMBER/comments?per_page=100" \
+            --jq '[.[] | {user: .user.login, created_at: .created_at, body: (.body | .[:4000])}] | sort_by(.created_at) | .[-20:]' \
+            > prior-comments.json || echo "[]" > prior-comments.json
+
+      - name: Write Codex review context
+        if: steps.codex_config.outputs.enabled == 'true' && steps.pr.outputs.skip != 'true'
+        env:
+          PR_REPOSITORY: ${{ github.repository }}
+          PR_NUMBER: ${{ steps.pr.outputs.pr_number }}
+          PR_BASE_SHA: ${{ steps.pr.outputs.base_sha }}
+          PR_HEAD_SHA: ${{ steps.pr.outputs.head_sha }}
+          PR_TITLE: ${{ steps.pr.outputs.title }}
+          PR_BODY: ${{ steps.pr.outputs.body }}
+          PR_AUTHOR: ${{ steps.pr.outputs.pr_author }}
+          EXTRA_PROMPT: ${{ inputs.extra_prompt }}
+        run: |
+          mkdir -p .github/codex
+          node <<'NODE'
+          const fs = require('fs');
+          const lines = [
+            `Repository: ${process.env.PR_REPOSITORY}`,
+            `PR number: ${process.env.PR_NUMBER}`,
+          ];
+          if (process.env.PR_AUTHOR) {
+            lines.push(`PR AUTHOR: ${process.env.PR_AUTHOR}`);
+          }
+          lines.push(
+            `Base SHA: ${process.env.PR_BASE_SHA}`,
+            `Head SHA: ${process.env.PR_HEAD_SHA}`,
+            '',
+            'PR title:',
+            process.env.PR_TITLE || '(empty)',
+            '',
+            'PR body:',
+            process.env.PR_BODY || '(empty)',
+            '',
+            'Changed commits command:',
+            `git log --oneline ${process.env.PR_BASE_SHA}...${process.env.PR_HEAD_SHA}`,
+            '',
+            'Changed files command:',
+            `git diff --stat ${process.env.PR_BASE_SHA}...${process.env.PR_HEAD_SHA}`,
+            '',
+            'Full review diff command:',
+            `git diff --unified=0 ${process.env.PR_BASE_SHA}...${process.env.PR_HEAD_SHA}`
+          );
+          if (process.env.EXTRA_PROMPT && process.env.EXTRA_PROMPT.trim()) {
+            lines.push('', 'Additional reviewer instructions:', process.env.EXTRA_PROMPT.trim());
+          }
+          if (fs.existsSync('prior-comments.json')) {
+            try {
+              const comments = JSON.parse(fs.readFileSync('prior-comments.json', 'utf8'));
+              if (Array.isArray(comments) && comments.length > 0) {
+                lines.push(
+                  '',
+                  'Prior PR discussion (most recent up to 20 comments):',
+                  '',
+                  'If you have already reviewed this PR (look for your own earlier "## Codex Review" comment), focus on what changed since then per the diff and respect any decisions the human made in replies. Do not re-flag findings the human already pushed back on.',
+                  ''
+                );
+                for (const c of comments) {
+                  lines.push(`### @${c.user} (${c.created_at})`, '', c.body, '', '---', '');
+                }
+              }
+            } catch (_) {}
+          }
+          fs.writeFileSync('.github/codex/pr-review-context.md', `${lines.join('\n')}\n`);
+          NODE
+
+      - name: Run Codex review
+        if: steps.codex_config.outputs.enabled == 'true' && steps.pr.outputs.skip != 'true'
+        run: |
+          cat REVIEW.md .github/codex/pr-review.prompt.md > /tmp/codex-prompt.md
+          codex exec \
+            -C "$GITHUB_WORKSPACE" \
+            -m gpt-5.5 \
+            -c 'model_reasoning_effort="xhigh"' \
+            -s danger-full-access \
+            -o codex-final-message.md \
+            - < /tmp/codex-prompt.md
+
+      - name: Post Codex review comment
+        if: steps.codex_config.outputs.enabled == 'true' && steps.pr.outputs.skip != 'true'
+        uses: actions/github-script@v7
+        env:
+          PR_NUMBER: ${{ steps.pr.outputs.pr_number }}
+        with:
+          github-token: ${{ github.token }}
+          script: |
+            const fs = require('fs');
+            const path = `${process.env.GITHUB_WORKSPACE}/codex-final-message.md`;
+            if (!fs.existsSync(path)) {
+              core.info('Codex did not produce a final message; skipping PR comment.');
+              return;
+            }
+            const body = fs.readFileSync(path, 'utf8').trim();
+            if (!body) {
+              core.info('Codex final message was empty; skipping PR comment.');
+              return;
+            }
+            await github.rest.issues.createComment({
+              owner: context.repo.owner,
+              repo: context.repo.repo,
+              issue_number: Number(process.env.PR_NUMBER),
+              body,
+            });

.github/workflows/create-docs.yml

@@ -1,39 +0,0 @@
-on:
-  issue_comment:
-    types: [created]
-
-jobs:
-  check-membership:
-    if: ${{ github.event.issue.pull_request && startsWith(github.event.comment.body, '/docs') }}
-    uses: ./.github/workflows/check-org-membership.yml
-    with:
-      commenter: ${{ github.event.comment.user.login }}
-    secrets:
-      access_token: ${{ secrets.ORG_ACCESS_TOKEN }}
-
-  generate-token:
-    needs: check-membership
-    if: ${{ needs.check-membership.outputs.is_member == 'true' }}
-    runs-on: ubicloud-standard-2
-    outputs:
-      app_token: ${{ steps.app.outputs.token }}
-    steps:
-      - name: Generate an installation token
-        id: app
-        uses: actions/create-github-app-token@v2
-        with:
-          app-id: ${{ vars.INTERNAL_APP_ID }}
-          private-key: ${{ secrets.INTERNAL_APP_KEY }}
-          owner: windmill-labs
-
-  trigger-docs:
-    needs: [generate-token, check-membership]
-    if: ${{ needs.check-membership.outputs.is_member == 'true' }}
-    uses: windmill-labs/windmilldocs/.github/workflows/create-docs.yml@main
-    with:
-      pr_number: ${{ github.event.issue.number }}
-      repo: ${{ github.event.repository.name }}
-      comment_text: ${{ github.event.comment.body }}
-    secrets:
-      DOCS_TOKEN: ${{ needs.generate-token.outputs.app_token }}
-      GOOGLE_API_KEY: ${{ secrets.GOOGLE_API_KEY }}

.github/workflows/discord-notification.yml

@@ -6,6 +6,10 @@ on:
       - opened
       - ready_for_review
       - closed
+  issue_comment:
+    types:
+      - created
+      - edited
 
 jobs:
   notify_discord_when_pr_opened:
@@ -33,3 +37,22 @@ jobs:
       PR_NUMBER: ${{ github.event.pull_request.number }}
     secrets:
       DISCORD_BOT_TOKEN: ${{ secrets.DISCORD_AI_BOT_TOKEN }}
+
+  notify_discord_on_comment:
+    if: >
+      github.event_name == 'issue_comment'
+      && github.event.issue.pull_request
+      && github.event.comment.user.login != 'cloudflare-workers-and-pages[bot]'
+      && github.event.comment.user.login != 'ellipsis-dev[bot]'
+    uses: ./.github/workflows/shareable-discord-notification.yml
+    with:
+      PR_STATUS: "comment"
+      PR_NUMBER: ${{ github.event.issue.number }}
+      COMMENT_BODY: ${{ github.event.comment.body }}
+      COMMENT_AUTHOR: ${{ github.event.comment.user.login }}
+      COMMENT_URL: ${{ github.event.comment.html_url }}
+      COMMENT_IS_EDIT: ${{ github.event.action == 'edited' }}
+      DISCORD_CHANNEL_ID: "1372204995868491786"
+      DISCORD_GUILD_ID: "930051556043276338"
+    secrets:
+      DISCORD_BOT_TOKEN: ${{ secrets.DISCORD_AI_BOT_TOKEN }}

.github/workflows/docker-image-rpi4.yml

@@ -67,7 +67,7 @@ jobs:
           platforms: linux/amd64,linux/arm64
           push: true
           build-args: |
-            features=embedding,parquet,openidconnect,license,http_trigger,zip,oauth2,postgres_trigger,mqtt_trigger,websocket,smtp,static_frontend,all_languages,deno_core,mcp
+            features=ce_rpi
           tags: |
             ${{ env.REGISTRY }}/${{ env.IMAGE_NAME }}:dev
             ${{ steps.meta-public.outputs.tags }}

.github/workflows/docker-image.yml

@@ -24,8 +24,8 @@ on:
         description: "Tag the image"
         required: true
         default: "test"
-      nsjail:
-        description: "Build nsjail image (true, false)"
+      slim:
+        description: "Build slim image (true, false)"
         required: false
         default: false
         type: boolean
@@ -92,7 +92,7 @@ jobs:
           platforms: linux/amd64,linux/arm64
           push: true
           build-args: |
-            features=embedding,parquet,openidconnect,jemalloc,license,http_trigger,zip,oauth2,dind,postgres_trigger,mqtt_trigger,websocket,smtp,static_frontend,agent_worker_server,all_languages,deno_core,mcp,private
+            features=ce
           tags: |
             ${{ env.REGISTRY }}/${{ env.IMAGE_NAME }}:${{ env.DEV_SHA }}
             ${{ steps.meta-public.outputs.tags }}
@@ -101,7 +101,7 @@ jobs:
 
   build_ee:
     runs-on: ubicloud
-    if: (github.event_name != 'workflow_dispatch') || (github.event.inputs.ee || github.event.inputs.nsjail)
+    if: (github.event_name != 'workflow_dispatch') || github.event.inputs.ee
     steps:
       - uses: actions/checkout@v4
         with:
@@ -154,7 +154,7 @@ jobs:
           platforms: linux/amd64,linux/arm64
           push: true
           build-args: |
-            features=enterprise,enterprise_saml,stripe,embedding,parquet,prometheus,openidconnect,cloud,jemalloc,agent_worker_server,tantivy,license,http_trigger,zip,oauth2,kafka,sqs_trigger,nats,otel,dind,postgres_trigger,mqtt_trigger,gcp_trigger,websocket,smtp,static_frontend,all_languages,private,deno_core,mcp
+            features=ee
           tags: |
             ${{ env.REGISTRY }}/${{ env.IMAGE_NAME }}-ee:${{ env.DEV_SHA }}
             ${{ steps.meta-ee-public.outputs.tags }}
@@ -162,39 +162,6 @@ jobs:
             ${{ steps.meta-ee-public.outputs.labels }}
             org.opencontainers.image.licenses=Windmill-Enterprise-License
 
-  # disabled until we make it 100% reliable and add more meaningful tests
-  # playwright:
-  #   runs-on: [self-hosted, new]
-  #   needs: [build]
-  #   services:
-  #     postgres:
-  #       image: postgres
-  #       env:
-  #         POSTGRES_DB: windmill
-  #         POSTGRES_USER: admin
-  #         POSTGRES_PASSWORD: changeme
-  #       ports:
-  #         - 5432:5432
-  #       options: >-
-  #         --health-cmd pg_isready
-  #         --health-interval 10s
-  #         --health-timeout 5s
-  #         --health-retries 5
-  #   steps:
-  #     - uses: actions/checkout@v4
-  #     - name: "Docker"
-  #       run: echo "::set-output name=id::$(docker run --network=host --rm -d -p 8000:8000 --privileged -it -e DATABASE_URL=postgres://admin:changeme@localhost:5432/windmill -e BASE_INTERNAL_URL=http://localhost:8000 ${{ env.REGISTRY }}/${{ env.IMAGE_NAME }}:latest)"
-  #       id: docker-container
-  #     - uses: actions/setup-node@v3
-  #       with:
-  #         node-version: 16
-  #     - name: "Playwright run"
-  #       timeout-minutes: 2
-  #       run: cd frontend && npm ci @playwright/test && npx playwright install && export BASE_URL=http://localhost:8000 && npm run test
-  #     - name: "Clean up"
-  #       run: docker kill ${{ steps.docker-container.outputs.id }}
-  #       if: always()
-
   attach_amd64_binary_to_release:
     needs: [build, build_ee]
     runs-on: ubicloud
@@ -220,6 +187,12 @@ jobs:
           image: ${{ env.REGISTRY }}/${{ env.IMAGE_NAME }}:${{ env.DEV_SHA }}
           path: "/usr/src/app/windmill"
 
+      - uses: shrink/actions-docker-extract@v3
+        id: extract-duckdb-ffi-internal
+        with:
+          image: ${{ env.REGISTRY }}/${{ env.IMAGE_NAME }}:${{ env.DEV_SHA }}
+          path: "/usr/src/app/libwindmill_duckdb_ffi_internal.so"
+
       - uses: shrink/actions-docker-extract@v3
         id: extract-ee
         with:
@@ -237,6 +210,60 @@ jobs:
           files: |
             ${{ steps.extract.outputs.destination }}/*
             ${{ steps.extract-ee.outputs.destination }}/*
+            ${{ steps.extract-duckdb-ffi-internal.outputs.destination }}/*
+
+  attach_ee_debug_to_release:
+    needs: [build_ee]
+    runs-on: ubicloud
+    if: ${{ startsWith(github.ref, 'refs/tags/v') }}
+    strategy:
+      matrix:
+        platform: [linux/amd64, linux/arm64]
+        include:
+          - platform: linux/amd64
+            arch: amd64
+          - platform: linux/arm64
+            arch: arm64
+    steps:
+      - uses: actions/checkout@v4
+        with:
+          ref: ${{ github.ref }}
+
+      - name: Read EE repo commit hash
+        run: |
+          echo "ee_repo_ref=$(cat ./backend/ee-repo-ref.txt)" >> "$GITHUB_ENV"
+
+      - uses: actions/checkout@v4
+        with:
+          repository: windmill-labs/windmill-ee-private
+          path: ./windmill-ee-private
+          ref: ${{ env.ee_repo_ref }}
+          token: ${{ secrets.WINDMILL_EE_PRIVATE_ACCESS }}
+
+      - name: Substitute EE code
+        run: |
+          ./backend/substitute_ee_code.sh --copy --dir ./windmill-ee-private
+
+      - uses: depot/setup-action@v1
+
+      - name: Extract EE debug info from builder stage (depot cache hit)
+        uses: depot/build-push-action@v1
+        with:
+          context: .
+          platforms: ${{ matrix.platform }}
+          target: debuginfo
+          build-args: |
+            features=ee
+          outputs: type=local,dest=./debuginfo
+
+      - name: Rename debug file with corresponding architecture
+        run: |
+          mv ./debuginfo/windmill.debug ./debuginfo/windmill-ee-${{ matrix.arch }}.debug
+
+      - name: Attach debug file to release
+        uses: softprops/action-gh-release@v2
+        with:
+          files: ./debuginfo/windmill-ee-${{ matrix.arch }}.debug
 
   # attach_arm64_binary_to_release:
   #   needs: [build, build_ee]
@@ -391,67 +418,10 @@ jobs:
   #         ignore-unchanged: true
   #         only-fixed: true
 
-  build_ee_nsjail:
-    needs: [build_ee]
-    runs-on: ubicloud
-    if: (github.event_name != 'pull_request') && ((github.event_name != 'workflow_dispatch') || (github.event.inputs.ee || github.event.inputs.nsjail))
-
-    steps:
-      - uses: actions/checkout@v4
-        with:
-          fetch-depth: 0
-          ref: ${{ github.ref }}
-
-      # - name: Set up Docker Buildx
-      #   uses: docker/setup-buildx-action@v2
-
-      - uses: depot/setup-action@v1
-
-      - name: Docker meta
-        id: meta-ee-public
-        uses: docker/metadata-action@v5
-        with:
-          images: |
-            ${{ env.REGISTRY }}/${{ env.IMAGE_NAME }}-ee-nsjail
-          flavor: |
-            latest=false
-          tags: |
-            type=semver,pattern={{version}}
-            type=semver,pattern={{major}}.{{minor}}
-            type=sha,enable=true,priority=100,prefix=,suffix=,format=short
-            type=ref,event=branch
-            type=ref,event=pr
-
-      - name: Login to registry
-        uses: docker/login-action@v3
-        with:
-          registry: ${{ env.REGISTRY }}
-          username: ${{ github.actor }}
-          password: ${{ secrets.GITHUB_TOKEN }}
-
-      - name: Update Dockerfile image reference
-        run: |
-          sed -i 's|FROM ghcr.io/windmill-labs/windmill-ee:dev|FROM ghcr.io/${{ env.IMAGE_NAME }}-ee:${{ env.DEV_SHA }}|' ./docker/DockerfileNsjail
-          cat ./docker/DockerfileNsjail | grep "FROM"
-
-      - name: Build and push publicly ee
-        uses: depot/build-push-action@v1
-        with:
-          context: .
-          platforms: linux/amd64,linux/arm64
-          push: true
-          file: "./docker/DockerfileNsjail"
-          tags: |
-            ${{ steps.meta-ee-public.outputs.tags }}
-          labels: |
-            ${{ steps.meta-ee-public.outputs.labels }}
-            org.opencontainers.image.licenses=Windmill-Enterprise-License
-
   publish_ecr_s3:
-    needs: [build_ee_nsjail]
+    needs: [build_ee_full]
     runs-on: ubicloud-standard-2-arm
-    if: (github.event_name != 'pull_request') && (github.event_name !=
-      'workflow_dispatch')
+    if: ${{ startsWith(github.ref, 'refs/tags/v') }}
     env:
       AWS_ACCESS_KEY_ID: ${{ secrets.AWS_ACCESS_KEY_ID }}
       AWS_SECRET_ACCESS_KEY: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
@@ -470,23 +440,18 @@ jobs:
           username: ${{ github.actor }}
           password: ${{ secrets.GITHUB_TOKEN }}
 
-      - name: get git hash
-        if: github.event_name != 'pull_request'
-        id: git_hash
-        run: |
-          git_hash=$(git rev-parse --short "$GITHUB_SHA")
-          echo "GIT_HASH=${git_hash:0:7}" >> "$GITHUB_OUTPUT"
+      - name: Get version from tag
+        id: version
+        run: echo "VERSION=${GITHUB_REF_NAME#v}" >> "$GITHUB_OUTPUT"
 
       - uses: shrink/actions-docker-extract@v3
-        if: github.event_name != 'pull_request'
         id: extract
         with:
           image: |-
-            ${{ env.REGISTRY }}/${{ env.IMAGE_NAME }}-ee-nsjail:${{ steps.git_hash.outputs.GIT_HASH }}
+            ${{ env.REGISTRY }}/${{ env.IMAGE_NAME }}-ee-full:${{ steps.version.outputs.VERSION }}
           path: "/static_frontend/."
 
       - uses: reggionick/s3-deploy@v4
-        if: github.event_name != 'pull_request'
         with:
           folder: ${{ steps.extract.outputs.destination }}
           bucket: windmill-frontend
@@ -581,9 +546,10 @@ jobs:
             ${{ steps.meta-ee-public.outputs.labels }}
 
   build_ee_slim:
-    if: ${{ startsWith(github.ref, 'refs/tags/v') }}
     needs: [build_ee]
     runs-on: ubicloud
+    if: (github.event_name != 'pull_request') && ((github.event_name != 'workflow_dispatch') || (github.event.inputs.ee || github.event.inputs.slim))
+
     steps:
       - uses: actions/checkout@v4
         with:
@@ -601,6 +567,7 @@ jobs:
           images: |
             ${{ env.REGISTRY }}/${{ env.IMAGE_NAME }}-ee-slim
           tags: |
+            type=ref,event=branch
             type=semver,pattern={{version}}
             type=semver,pattern={{major}}.{{minor}}
 
@@ -615,7 +582,7 @@ jobs:
         uses: depot/build-push-action@v1
         with:
           context: .
-          platforms: linux/amd64
+          platforms: linux/amd64,linux/arm64
           push: true
           file: "./docker/DockerfileSlimEe"
           tags: |

.github/workflows/frontend-check.yml

@@ -16,11 +16,12 @@ jobs:
     runs-on: ubicloud-standard-8
     steps:
       - uses: actions/checkout@v4
-      - uses: actions/setup-node@v3
+      - uses: actions/setup-node@v5
         with:
-          node-version: 18
+          node-version: 24
+          cache: "npm"
+          cache-dependency-path: "frontend/package-lock.json"
       - name: "npm check"
         timeout-minutes: 5
-        run:
-          cd frontend && npm ci && npm run generate-backend-client && npm run
+        run: cd frontend && npm ci && npm run generate-backend-client && npm run
           check

.github/workflows/git-commands.yaml

@@ -5,8 +5,21 @@ on:
     types: [created]
 
 jobs:
+  check-membership:
+    if: >-
+      github.event.issue.pull_request && (
+        startsWith(github.event.comment.body, '/updatesqlx') ||
+        startsWith(github.event.comment.body, '/demo') ||
+        startsWith(github.event.comment.body, '/eeref') ||
+        startsWith(github.event.comment.body, '/docs')
+      )
+    uses: ./.github/workflows/check-org-membership.yml
+    secrets:
+      access_token: ${{ secrets.ORG_ACCESS_TOKEN }}
+
   update-sqlx:
-    if: github.event.issue.pull_request && startsWith(github.event.comment.body, '/updatesqlx')
+    needs: check-membership
+    if: needs.check-membership.outputs.is_member == 'true' && startsWith(github.event.comment.body, '/updatesqlx')
     runs-on: ubicloud-standard-8
     permissions:
       contents: write
@@ -29,21 +42,29 @@ jobs:
           --health-retries 5
 
     steps:
+      - uses: actions/create-github-app-token@v2
+        id: app
+        with:
+          app-id: ${{ vars.INTERNAL_APP_ID }}
+          private-key: ${{ secrets.INTERNAL_APP_KEY }}
+
       - name: Comment on PR - Starting
         uses: actions/github-script@v6
         with:
-          github-token: ${{ secrets.GITHUB_TOKEN }}
+          github-token: ${{ steps.app.outputs.token }}
           script: |
+            const runUrl = `https://github.com/${context.repo.owner}/${context.repo.repo}/actions/runs/${context.runId}`;
             github.rest.issues.createComment({
               issue_number: context.issue.number,
               owner: context.repo.owner,
               repo: context.repo.repo,
-              body: 'Starting sqlx update...'
+              body: `Starting sqlx update...\n\n[View workflow run](${runUrl})`
             })
 
       - name: Checkout repository
         uses: actions/checkout@v3
         with:
+          token: ${{ steps.app.outputs.token }}
           ref: ${{ github.event.issue.pull_request.head.ref }}
           fetch-depth: 0
 
@@ -54,37 +75,56 @@ jobs:
           path: windmill-ee-private
           token: ${{ secrets.WINDMILL_EE_PRIVATE_ACCESS }}
 
-      # Cache rust dependencies
-      - uses: dtolnay/rust-toolchain@stable
-      - uses: Swatinem/rust-cache@v2
+      # Setup Rust toolchain
+      - uses: actions-rust-lang/setup-rust-toolchain@v1
         with:
-          workspaces: "./backend -> target"
+          cache-workspaces: backend
+          toolchain: 1.93.0
 
-      - name: Install xmlsec build-time deps
+      - name: Install xmlsec and gssapi build-time deps
         run: |
           sudo apt-get update
           sudo apt-get install -y --no-install-recommends \
-            pkg-config libxml2-dev libssl-dev \
+            pkg-config libxml2-dev libssl-dev libkrb5-dev libsasl2-dev libcurl4-openssl-dev mold clang \
             xmlsec1 libxmlsec1-dev libxmlsec1-openssl
 
       - name: Run update-sqlx script
         env:
           DATABASE_URL: postgres://postgres:postgres@localhost:5432/windmill
-          GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
+          GH_TOKEN: ${{ steps.app.outputs.token }}
         run: |
+          set -e  # Exit on any command failure
           PR_NUMBER=${{ github.event.issue.number }}
+
+          # Set up error trap to comment on PR for any failure
+          trap 'gh pr comment $PR_NUMBER --body "❌ SQLx update failed. Please check the workflow logs for details."' ERR
+
           BRANCH_NAME=$(gh pr view $PR_NUMBER --json headRefName --jq .headRefName)
           echo "Checking out PR branch: $BRANCH_NAME"
           git checkout $BRANCH_NAME
-          git config --local user.email "action@github.com"
-          git config --local user.name "GitHub Action"
+          git config --local user.email "windmill-internal-app[bot]@users.noreply.github.com"
+          git config --local user.name "windmill-internal-app[bot]"
           git config pull.rebase true
           git pull origin $BRANCH_NAME
-          mkdir frontend/build
+
+          # Checkout the correct windmill-ee-private commit from ee-repo-ref.txt
+          if [ -f backend/ee-repo-ref.txt ]; then
+            EE_REF=$(cat backend/ee-repo-ref.txt | tr -d '[:space:]')
+            echo "Checking out windmill-ee-private at commit: $EE_REF"
+            cd windmill-ee-private
+            git fetch origin $EE_REF
+            git checkout $EE_REF
+            cd ..
+          else
+            echo "Warning: ee-repo-ref.txt not found, using default branch"
+          fi
+
+          mkdir -p frontend/build
           cd backend
           cargo install sqlx-cli --version 0.8.5
           sqlx migrate run
-          ./update_sqlx.sh --dir ./windmill-ee-private
+          ./substitute_ee_code.sh --dir ./windmill-ee-private
+          ./update_sqlx.sh
           # Pass the branch name to the next step
           echo "BRANCH_NAME=$BRANCH_NAME" >> $GITHUB_ENV
 
@@ -97,7 +137,7 @@ jobs:
       - name: Comment on PR - Completed
         uses: actions/github-script@v6
         with:
-          github-token: ${{ secrets.GITHUB_TOKEN }}
+          github-token: ${{ steps.app.outputs.token }}
           script: |
             github.rest.issues.createComment({
               issue_number: context.issue.number,
@@ -106,29 +146,118 @@ jobs:
               body: 'Successfully ran sqlx update'
             })
 
+  demo:
+    needs: check-membership
+    if: needs.check-membership.outputs.is_member == 'true' && startsWith(github.event.comment.body, '/demo')
+    runs-on: ubicloud-standard-2
+    permissions:
+      contents: read
+      pull-requests: read
+      issues: read
+      id-token: write
+    steps:
+      - name: Checkout repository
+        uses: actions/checkout@v4
+        with:
+          fetch-depth: 0
+
+      - name: Run Claude Code for Demo Generation
+        uses: anthropics/claude-code-action@beta
+        with:
+          claude_code_oauth_token: ${{ secrets.CLAUDE_CODE_OAUTH_TOKEN }}
+          timeout_minutes: "10"
+          allowed_tools: "Bash"
+          direct_prompt: |
+            You need to:
+
+            1. Extract the Cloudflare preview URL from the cloudflare-workers-and-pages bot comment in this PR
+            2. Analyze the PR changes to understand what feature was added/modified
+            3. Create detailed instructions to give to an AI agent that will click and interact with buttons and inputs to showcase the new feature. Only include the instructions, nothing else.
+            4. Create a demo.json file with a valid JSON object containing:
+              - instructions: the demo instructions
+              - url: the preview URL
+            5. VALIDATE the JSON file using `jq` before finishing
+            DO NOT COMMIT THIS FILE TO THE PR.
+
+            Example demo.json:
+            {
+              "instructions": "Click on settings, then account settings, then 'generate new token'",
+              "url": "https://example.pages.dev"
+            }
+
+            CRITICAL: After creating demo.json, you MUST:
+            1. Run `jq empty demo.json` to validate the JSON is properly formatted
+            2. If validation fails, fix the JSON and validate again
+            3. Only proceed once the JSON passes validation
+            4. Use proper JSON escaping for newlines, quotes, and special characters
+
+            Make sure to:
+            - Create a valid JSON object that passes `jq empty demo.json`
+            - Extract the correct preview URL (should be a .pages.dev domain)
+            - Create specific, actionable demo steps based on the actual changes in the PR
+            - Properly escape all strings in the JSON (use jq to create the file if needed)
+            - NOT COMMIT THE DEMO.JSON FILE TO THE PR
+
+      - name: Send instructions to Windmill
+        env:
+          DEMO_WEBHOOK_TOKEN: ${{ secrets.DEMO_WEBHOOK_TOKEN }}
+        run: |
+          if [[ -f "demo.json" ]]; then
+            echo "Found demo.json, sending to Windmill..."
+            cat demo.json
+
+            # Validate JSON one more time (Claude should have already done this)
+            if ! jq empty demo.json; then
+              echo "Error: demo.json is not valid JSON"
+              exit 1
+            fi
+
+            RESULT=$(curl -s \
+              -H 'Content-Type: application/json' \
+              -H "Authorization: Bearer $DEMO_WEBHOOK_TOKEN" \
+              -X POST \
+              -d @demo.json \
+              'https://app.windmill.dev/api/w/windmill-labs/jobs/run/f/f/ai/browserbase_demo')
+
+            echo "Windmill response:"
+            echo -E "$RESULT"
+          else
+            echo "Error: demo.json file not found"
+            exit 1
+          fi
+
   update-ee-ref:
-    if: github.event.issue.pull_request && startsWith(github.event.comment.body, '/eeref')
+    needs: check-membership
+    if: needs.check-membership.outputs.is_member == 'true' && startsWith(github.event.comment.body, '/eeref')
     runs-on: ubicloud-standard-2
     permissions:
       contents: write
       pull-requests: write
       issues: write
     steps:
+      - uses: actions/create-github-app-token@v2
+        id: app
+        with:
+          app-id: ${{ vars.INTERNAL_APP_ID }}
+          private-key: ${{ secrets.INTERNAL_APP_KEY }}
+
       - name: Comment on PR - Starting
         uses: actions/github-script@v6
         with:
-          github-token: ${{ secrets.GITHUB_TOKEN }}
+          github-token: ${{ steps.app.outputs.token }}
           script: |
+            const runUrl = `https://github.com/${context.repo.owner}/${context.repo.repo}/actions/runs/${context.runId}`;
             github.rest.issues.createComment({
               issue_number: context.issue.number,
               owner: context.repo.owner,
               repo: context.repo.repo,
-              body: 'Starting ee ref update...'
+              body: `Starting ee ref update...\n\n[View workflow run](${runUrl})`
             })
 
       - name: Checkout repository
         uses: actions/checkout@v3
         with:
+          token: ${{ steps.app.outputs.token }}
           ref: ${{ github.event.issue.pull_request.head.ref }}
           fetch-depth: 0
 
@@ -149,19 +278,24 @@ jobs:
 
       - name: Update ee-repo-ref.txt
         env:
-          GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
+          GH_TOKEN: ${{ steps.app.outputs.token }}
         run: |
+          set -e  # Exit on any command failure
+          PR_NUMBER=${{ github.event.issue.number }}
+
+          # Set up error trap to comment on PR for any failure
+          trap 'gh pr comment $PR_NUMBER --body "❌ EE ref update failed. Please check the workflow logs for details."' ERR
+
+          BRANCH_NAME=$(gh pr view $PR_NUMBER --json headRefName --jq .headRefName)
           echo "Checking out PR branch: $BRANCH_NAME"
           git checkout $BRANCH_NAME
-          git config --local user.email "action@github.com"
-          git config --local user.name "GitHub Action"
+          git config --local user.email "windmill-internal-app[bot]@users.noreply.github.com"
+          git config --local user.name "windmill-internal-app[bot]"
           git config pull.rebase true
           git pull origin $BRANCH_NAME
           echo "${{ steps.get-commit-hash.outputs.commit_hash }}" > backend/ee-repo-ref.txt
           echo "Updated backend/ee-repo-ref.txt with commit hash: ${{ steps.get-commit-hash.outputs.commit_hash }}"
           # commit and push the changes
-          PR_NUMBER=${{ github.event.issue.number }}
-          BRANCH_NAME=$(gh pr view $PR_NUMBER --json headRefName --jq .headRefName)
           git add backend/ee-repo-ref.txt
           git commit -m "Update ee-repo-ref.txt" || echo "No changes to commit"
           git push origin $BRANCH_NAME
@@ -169,7 +303,7 @@ jobs:
       - name: Comment on PR - Completed
         uses: actions/github-script@v6
         with:
-          github-token: ${{ secrets.GITHUB_TOKEN }}
+          github-token: ${{ steps.app.outputs.token }}
           script: |
             github.rest.issues.createComment({
               issue_number: context.issue.number,
@@ -177,3 +311,35 @@ jobs:
               repo: context.repo.repo,
               body: 'Successfully updated ee-repo-ref.txt'
             })
+
+  update-docs:
+    needs: check-membership
+    if: needs.check-membership.outputs.is_member == 'true' && startsWith(github.event.comment.body, '/docs')
+    runs-on: ubicloud-standard-2
+    permissions:
+      contents: read
+      pull-requests: read
+      issues: read
+    steps:
+      - uses: actions/create-github-app-token@v2
+        id: app
+        with:
+          app-id: ${{ vars.INTERNAL_APP_ID }}
+          private-key: ${{ secrets.INTERNAL_APP_KEY }}
+          owner: ${{ github.repository_owner }}
+          repositories: |
+            windmilldocs
+
+      - name: Trigger docs update
+        env:
+          GH_TOKEN: ${{ steps.app.outputs.token }}
+          COMMENT_TEXT: ${{ github.event.comment.body }}
+        run: |
+          jq -n \
+            --argjson pr_number ${{ github.event.issue.number }} \
+            --arg repo "${{ github.event.repository.name }}" \
+            --arg comment "$COMMENT_TEXT" \
+            '{event_type: "create-docs", client_payload: {pr_number: $pr_number, repo: $repo, comment_text: $comment}}' | \
+          gh api repos/windmill-labs/windmilldocs/dispatches \
+            --method POST \
+            --input -

.github/workflows/git-sync-test.yml

@@ -0,0 +1,211 @@
+name: Git Sync Integration Tests
+
+on:
+  workflow_dispatch:
+  push:
+    branches: [main]
+    paths:
+      - "backend/windmill-git-sync/**"
+      - "backend/windmill-api-integration-tests/tests/git_sync*"
+      - "backend/ee-repo-ref.txt"
+      - "backend/windmill-common/src/workspaces.rs"
+      - "integration_tests/test/git_sync_test.py"
+      - ".github/workflows/git-sync-test.yml"
+  pull_request:
+    types: [opened, synchronize, reopened]
+    paths:
+      - "backend/windmill-git-sync/**"
+      - "backend/windmill-api-integration-tests/tests/git_sync*"
+      - "backend/ee-repo-ref.txt"
+      - "backend/windmill-common/src/workspaces.rs"
+      - "integration_tests/test/git_sync_test.py"
+      - ".github/workflows/git-sync-test.yml"
+
+concurrency:
+  group: git-sync-test-${{ github.ref }}
+  cancel-in-progress: true
+
+jobs:
+  check-relevance:
+    runs-on: ubuntu-latest
+    outputs:
+      should_run: ${{ steps.check.outputs.should_run }}
+    steps:
+      - uses: actions/checkout@v4
+        with:
+          fetch-depth: 0
+
+      - name: Check if git sync related files changed
+        id: check
+        env:
+          WINDMILL_EE_PRIVATE_ACCESS: ${{ secrets.WINDMILL_EE_PRIVATE_ACCESS }}
+        run: |
+          if [ "${{ github.event_name }}" = "pull_request" ]; then
+            BASE=${{ github.event.pull_request.base.sha }}
+          else
+            BASE=${{ github.event.before }}
+          fi
+
+          CHANGED_FILES=$(git diff --name-only "$BASE"..HEAD 2>/dev/null || echo "")
+          echo "Changed files:"
+          echo "$CHANGED_FILES"
+
+          # Direct git sync file changes — always relevant
+          if echo "$CHANGED_FILES" | grep -qE '^(backend/windmill-git-sync/|backend/windmill-api-integration-tests/tests/git_sync|backend/windmill-common/src/workspaces\.rs|integration_tests/test/git_sync|\.github/workflows/git-sync-test\.yml)'; then
+            echo "should_run=true" >> "$GITHUB_OUTPUT"
+            echo "Relevant: direct git sync file changes"
+            exit 0
+          fi
+
+          # If ee-repo-ref.txt changed, check if the EE diff touches windmill-git-sync/
+          if echo "$CHANGED_FILES" | grep -q '^backend/ee-repo-ref.txt$'; then
+            NEW_REF=$(cat backend/ee-repo-ref.txt)
+            OLD_REF=$(git show "$BASE:backend/ee-repo-ref.txt" 2>/dev/null || echo "")
+
+            if [ -n "$OLD_REF" ] && [ "$OLD_REF" != "$NEW_REF" ]; then
+              # Clone EE repo and check diff
+              git clone --bare "https://x-access-token:${WINDMILL_EE_PRIVATE_ACCESS}@github.com/windmill-labs/windmill-ee-private.git" /tmp/ee-repo 2>/dev/null
+              EE_CHANGED=$(git -C /tmp/ee-repo diff --name-only "$OLD_REF".."$NEW_REF" 2>/dev/null || echo "")
+              echo "EE changed files:"
+              echo "$EE_CHANGED"
+
+              if echo "$EE_CHANGED" | grep -q '^windmill-git-sync/'; then
+                echo "should_run=true" >> "$GITHUB_OUTPUT"
+                echo "Relevant: EE git sync files changed"
+                exit 0
+              fi
+            fi
+          fi
+
+          echo "should_run=false" >> "$GITHUB_OUTPUT"
+          echo "No git sync relevant changes detected, skipping tests"
+
+  git_sync_e2e:
+    needs: [check-relevance]
+    if: needs.check-relevance.outputs.should_run == 'true'
+    runs-on: ubicloud-standard-16
+    services:
+      postgres:
+        image: postgres:14
+        ports:
+          - 5432:5432
+        env:
+          POSTGRES_DB: windmill
+          POSTGRES_PASSWORD: changeme
+        options: >-
+          --health-cmd pg_isready --health-interval 10s --health-timeout 5s
+          --health-retries 5
+
+    steps:
+      - uses: actions/checkout@v4
+        with:
+          ref: ${{ github.ref }}
+          fetch-depth: 0
+
+      - name: Read EE repo commit hash
+        run: |
+          echo "ee_repo_ref=$(cat ./backend/ee-repo-ref.txt)" >> "$GITHUB_ENV"
+
+      - uses: actions/checkout@v4
+        with:
+          repository: windmill-labs/windmill-ee-private
+          path: ./windmill-ee-private
+          ref: ${{ env.ee_repo_ref }}
+          token: ${{ secrets.WINDMILL_EE_PRIVATE_ACCESS }}
+          fetch-depth: 0
+
+      - name: Substitute EE code
+        run: |
+          cd backend && ./substitute_ee_code.sh --copy --dir ./windmill-ee-private
+
+      - uses: actions-rust-lang/setup-rust-toolchain@v1
+        with:
+          cache-workspaces: backend
+          toolchain: 1.93.0
+
+      - uses: oven-sh/setup-bun@v2
+        with:
+          bun-version: 1.3.10
+
+      - uses: denoland/setup-deno@v2
+        with:
+          deno-version: v2.x
+
+      - uses: actions/setup-node@v4
+        with:
+          node-version: "20"
+
+      - name: Install wmill CLI
+        run: |
+          cd cli && bash gen_wm_client.sh && bun install
+          mkdir -p "$HOME/.local/bin"
+          printf '#!/bin/sh\nexec bun run "%s/cli/src/main.ts" "$@"\n' "$GITHUB_WORKSPACE" > "$HOME/.local/bin/wmill"
+          chmod +x "$HOME/.local/bin/wmill"
+          echo "$HOME/.local/bin" >> $GITHUB_PATH
+
+      - name: Build Windmill
+        working-directory: ./backend
+        env:
+          SQLX_OFFLINE: true
+          CARGO_BUILD_JOBS: 12
+          RUSTFLAGS: ""
+        run: |
+          cargo build --features enterprise,private,license,zip
+
+      - name: Start Gitea
+        run: |
+          docker run -d --name gitea \
+            -e GITEA__database__DB_TYPE=sqlite3 \
+            -e GITEA__security__INSTALL_LOCK=true \
+            -e GITEA__server__HTTP_PORT=3000 \
+            -e GITEA__server__ROOT_URL=http://localhost:3000 \
+            -e GITEA__service__DISABLE_REGISTRATION=false \
+            -p 3000:3000 \
+            gitea/gitea:1.22-rootless
+          echo "Waiting for Gitea to be ready..."
+          for i in $(seq 1 30); do
+            if curl -sf http://localhost:3000/api/v1/version > /dev/null 2>&1; then
+              echo "Gitea is ready"
+              break
+            fi
+            sleep 2
+          done
+          curl -sf http://localhost:3000/api/v1/version > /dev/null || { echo "Gitea failed to start"; exit 1; }
+
+      - name: Start Windmill
+        working-directory: ./backend
+        env:
+          DATABASE_URL: postgres://postgres:changeme@localhost:5432/windmill
+          LICENSE_KEY: ${{ secrets.WM_LICENSE_KEY_CI }}
+          DENO_PATH: deno
+          BUN_PATH: bun
+          NODE_BIN_PATH: node
+        run: |
+          ./target/debug/windmill &
+          echo "Waiting for Windmill to be ready..."
+          for i in $(seq 1 60); do
+            if curl -sf http://localhost:8000/api/version > /dev/null 2>&1; then
+              echo "Windmill is ready"
+              break
+            fi
+            sleep 2
+          done
+          curl -sf http://localhost:8000/api/version > /dev/null || { echo "Windmill failed to start"; exit 1; }
+
+      - name: Run git sync E2E tests
+        timeout-minutes: 10
+        env:
+          GITEA_DOCKER_URL: http://localhost:3000
+          LICENSE_KEY: ${{ secrets.WM_LICENSE_KEY_CI }}
+        run: |
+          python3 -m venv .venv
+          .venv/bin/pip install -r integration_tests/requirements.txt
+          cd integration_tests && ../.venv/bin/python -m unittest -v test.git_sync_test
+
+      - name: Archive logs
+        uses: actions/upload-artifact@v4
+        if: always()
+        with:
+          name: Git Sync Integration Tests Logs
+          path: |
+            integration_tests/logs

.github/workflows/npm_on_release.yml

@@ -14,7 +14,7 @@ jobs:
         with:
           node-version: "20.x"
           registry-url: "https://registry.npmjs.org"
-      - run: cd typescript-client && ./publish.sh && cd ..
+      - run: cd typescript-client && ./publish.sh --access public && cd ..
         env:
           NODE_AUTH_TOKEN: ${{ secrets.NPM_TOKEN }}
   publish_cli:
@@ -25,9 +25,9 @@ jobs:
         with:
           node-version: "20.x"
           registry-url: "https://registry.npmjs.org"
-      - uses: denoland/setup-deno@v2
+      - uses: oven-sh/setup-bun@v2
         with:
-          deno-version: v2.x
-      - run: cd cli && ./build.sh && cd npm && npm publish
+          bun-version: latest
+      - run: cd cli && ./build.sh && cd npm && npm publish --access public
         env:
           NODE_AUTH_TOKEN: ${{ secrets.NPM_TOKEN }}

.github/workflows/pi-pr-review.yml

@@ -0,0 +1,325 @@
+name: Pi Auto Review
+
+on:
+  pull_request:
+    types: [ready_for_review, opened, synchronize]
+  workflow_call:
+    inputs:
+      pr_number:
+        description: 'PR number to review'
+        required: true
+        type: number
+      extra_prompt:
+        description: 'Additional reviewer instructions appended to the standard review prompt'
+        required: false
+        type: string
+        default: ''
+      triggered_by:
+        description: 'GitHub username that triggered this review (for audit only)'
+        required: false
+        type: string
+        default: ''
+    secrets:
+      DEEPSEEK_API_KEY:
+        required: false
+      WINDMILL_EE_PRIVATE_ACCESS:
+        required: false
+
+concurrency:
+  group: pi-review-${{ inputs.pr_number || github.event.pull_request.number }}
+  cancel-in-progress: true
+
+jobs:
+  check-membership:
+    if: github.event_name == 'pull_request'
+    uses: ./.github/workflows/check-org-membership.yml
+    with:
+      commenter: ${{ github.event.pull_request.user.login }}
+    secrets:
+      access_token: ${{ secrets.ORG_ACCESS_TOKEN }}
+
+  pi-review:
+    needs: check-membership
+    runs-on: ubicloud-standard-2
+    timeout-minutes: 30
+    if: |
+      always() &&
+      (
+        needs.check-membership.result == 'skipped' ||
+        (needs.check-membership.result == 'success' && needs.check-membership.outputs.is_member == 'true')
+      ) &&
+      (
+        github.event_name == 'workflow_call' ||
+        (github.event.pull_request.draft == false && github.event.pull_request.head.repo.fork == false)
+      )
+    permissions:
+      contents: read
+      issues: write
+      pull-requests: write
+    steps:
+      - name: Check Pi configuration
+        id: pi_config
+        env:
+          DEEPSEEK_API_KEY: ${{ secrets.DEEPSEEK_API_KEY }}
+        run: |
+          if [ -n "$DEEPSEEK_API_KEY" ]; then
+            echo "enabled=true" >> "$GITHUB_OUTPUT"
+          else
+            echo "enabled=false" >> "$GITHUB_OUTPUT"
+            echo "DEEPSEEK_API_KEY is not configured; skipping Pi review."
+          fi
+
+      - name: Resolve PR metadata
+        if: steps.pi_config.outputs.enabled == 'true'
+        id: pr
+        env:
+          GH_TOKEN: ${{ github.token }}
+          INPUT_PR_NUMBER: ${{ inputs.pr_number }}
+          EVENT_PR_NUMBER: ${{ github.event.pull_request.number }}
+          EVENT_BASE_REF: ${{ github.event.pull_request.base.ref }}
+          EVENT_BASE_SHA: ${{ github.event.pull_request.base.sha }}
+          EVENT_HEAD_SHA: ${{ github.event.pull_request.head.sha }}
+          EVENT_TITLE: ${{ github.event.pull_request.title }}
+          EVENT_BODY: ${{ github.event.pull_request.body }}
+          EVENT_FORK: ${{ github.event.pull_request.head.repo.fork }}
+          EVENT_AUTHOR: ${{ github.event.pull_request.user.login }}
+        run: |
+          if [ -n "$INPUT_PR_NUMBER" ]; then
+            PR_JSON=$(gh pr view "$INPUT_PR_NUMBER" --repo "${{ github.repository }}" \
+              --json number,baseRefName,baseRefOid,headRefOid,title,body,isCrossRepository,author)
+            PR_NUMBER=$(echo "$PR_JSON" | jq -r '.number')
+            BASE_REF=$(echo "$PR_JSON" | jq -r '.baseRefName')
+            BASE_SHA=$(echo "$PR_JSON" | jq -r '.baseRefOid')
+            HEAD_SHA=$(echo "$PR_JSON" | jq -r '.headRefOid')
+            PR_TITLE=$(echo "$PR_JSON" | jq -r '.title')
+            PR_BODY=$(echo "$PR_JSON" | jq -r '.body // ""')
+            IS_FORK=$(echo "$PR_JSON" | jq -r '.isCrossRepository')
+            PR_AUTHOR=$(echo "$PR_JSON" | jq -r '.author.login // ""')
+          else
+            PR_NUMBER="$EVENT_PR_NUMBER"
+            BASE_REF="$EVENT_BASE_REF"
+            BASE_SHA="$EVENT_BASE_SHA"
+            HEAD_SHA="$EVENT_HEAD_SHA"
+            PR_TITLE="$EVENT_TITLE"
+            PR_BODY="$EVENT_BODY"
+            IS_FORK="$EVENT_FORK"
+            PR_AUTHOR="$EVENT_AUTHOR"
+          fi
+          if [ "$IS_FORK" = "true" ]; then
+            echo "Skipping Pi review for fork PR."
+            echo "skip=true" >> "$GITHUB_OUTPUT"
+            exit 0
+          fi
+          {
+            echo "skip=false"
+            echo "pr_number=$PR_NUMBER"
+            echo "base_ref=$BASE_REF"
+            echo "base_sha=$BASE_SHA"
+            echo "head_sha=$HEAD_SHA"
+            echo "pr_author=$PR_AUTHOR"
+            echo 'title<<PR_TITLE_EOF'
+            printf '%s\n' "$PR_TITLE"
+            echo 'PR_TITLE_EOF'
+            echo 'body<<PR_BODY_EOF'
+            printf '%s\n' "$PR_BODY"
+            echo 'PR_BODY_EOF'
+          } >> "$GITHUB_OUTPUT"
+
+      - name: Checkout repository
+        if: steps.pi_config.outputs.enabled == 'true' && steps.pr.outputs.skip != 'true'
+        uses: actions/checkout@v5
+        with:
+          ref: refs/pull/${{ steps.pr.outputs.pr_number }}/merge
+          fetch-depth: 1
+
+      - name: Check EE access
+        if: steps.pi_config.outputs.enabled == 'true' && steps.pr.outputs.skip != 'true'
+        id: ee
+        env:
+          EE_TOKEN: ${{ secrets.WINDMILL_EE_PRIVATE_ACCESS }}
+        run: |
+          if [ -n "$EE_TOKEN" ]; then
+            echo "available=true" >> "$GITHUB_OUTPUT"
+            echo "ee_repo_ref=$(cat ./backend/ee-repo-ref.txt)" >> "$GITHUB_OUTPUT"
+          else
+            echo "available=false" >> "$GITHUB_OUTPUT"
+          fi
+
+      - name: Checkout EE repository
+        if: steps.ee.outputs.available == 'true'
+        uses: actions/checkout@v5
+        with:
+          repository: windmill-labs/windmill-ee-private
+          path: ./windmill-ee-private
+          ref: ${{ steps.ee.outputs.ee_repo_ref }}
+          token: ${{ secrets.WINDMILL_EE_PRIVATE_ACCESS }}
+          fetch-depth: 1
+
+      - name: Substitute EE code
+        if: steps.ee.outputs.available == 'true'
+        run: ./backend/substitute_ee_code.sh --copy --dir ./windmill-ee-private
+
+      - name: Set up Node.js
+        if: steps.pi_config.outputs.enabled == 'true' && steps.pr.outputs.skip != 'true'
+        uses: actions/setup-node@v4
+        with:
+          node-version: 22
+
+      - name: Install Pi CLI
+        if: steps.pi_config.outputs.enabled == 'true' && steps.pr.outputs.skip != 'true'
+        run: npm install --global @mariozechner/pi-coding-agent
+
+      - name: Pre-fetch base and head refs for the PR
+        if: steps.pi_config.outputs.enabled == 'true' && steps.pr.outputs.skip != 'true'
+        env:
+          PR_BASE_REF: ${{ steps.pr.outputs.base_ref }}
+          PR_NUMBER: ${{ steps.pr.outputs.pr_number }}
+        run: |
+          git fetch --no-tags origin \
+            "$PR_BASE_REF" \
+            "+refs/pull/$PR_NUMBER/head"
+
+      - name: Fetch prior PR discussion
+        if: steps.pi_config.outputs.enabled == 'true' && steps.pr.outputs.skip != 'true'
+        env:
+          GH_TOKEN: ${{ github.token }}
+          REPO: ${{ github.repository }}
+          PR_NUMBER: ${{ steps.pr.outputs.pr_number }}
+        run: |
+          gh api "repos/$REPO/issues/$PR_NUMBER/comments?per_page=100" \
+            --jq '[.[] | {user: .user.login, created_at: .created_at, body: (.body | .[:4000])}] | sort_by(.created_at) | .[-20:]' \
+            > prior-comments.json || echo "[]" > prior-comments.json
+
+      - name: Write Pi review context
+        if: steps.pi_config.outputs.enabled == 'true' && steps.pr.outputs.skip != 'true'
+        env:
+          PR_REPOSITORY: ${{ github.repository }}
+          PR_NUMBER: ${{ steps.pr.outputs.pr_number }}
+          PR_BASE_SHA: ${{ steps.pr.outputs.base_sha }}
+          PR_HEAD_SHA: ${{ steps.pr.outputs.head_sha }}
+          PR_TITLE: ${{ steps.pr.outputs.title }}
+          PR_BODY: ${{ steps.pr.outputs.body }}
+          PR_AUTHOR: ${{ steps.pr.outputs.pr_author }}
+          EXTRA_PROMPT: ${{ inputs.extra_prompt }}
+        run: |
+          mkdir -p .github/pi
+          node <<'NODE'
+          const fs = require('fs');
+          const lines = [
+            `Repository: ${process.env.PR_REPOSITORY}`,
+            `PR number: ${process.env.PR_NUMBER}`,
+          ];
+          if (process.env.PR_AUTHOR) {
+            lines.push(`PR AUTHOR: ${process.env.PR_AUTHOR}`);
+          }
+          lines.push(
+            `Base SHA: ${process.env.PR_BASE_SHA}`,
+            `Head SHA: ${process.env.PR_HEAD_SHA}`,
+            '',
+            'PR title:',
+            process.env.PR_TITLE || '(empty)',
+            '',
+            'PR body:',
+            process.env.PR_BODY || '(empty)',
+            '',
+            'Changed commits command:',
+            `git log --oneline ${process.env.PR_BASE_SHA}...${process.env.PR_HEAD_SHA}`,
+            '',
+            'Changed files command:',
+            `git diff --stat ${process.env.PR_BASE_SHA}...${process.env.PR_HEAD_SHA}`,
+            '',
+            'Full review diff command:',
+            `git diff --unified=0 ${process.env.PR_BASE_SHA}...${process.env.PR_HEAD_SHA}`
+          );
+          if (process.env.EXTRA_PROMPT && process.env.EXTRA_PROMPT.trim()) {
+            lines.push('', 'Additional reviewer instructions:', process.env.EXTRA_PROMPT.trim());
+          }
+          if (fs.existsSync('prior-comments.json')) {
+            try {
+              const comments = JSON.parse(fs.readFileSync('prior-comments.json', 'utf8'));
+              if (Array.isArray(comments) && comments.length > 0) {
+                lines.push(
+                  '',
+                  'Prior PR discussion (most recent up to 20 comments):',
+                  '',
+                  'If you have already reviewed this PR (look for your own earlier "## Pi Review (DeepSeek V4)" comment), focus on what changed since then per the diff and respect any decisions the human made in replies. Do not re-flag findings the human already pushed back on.',
+                  ''
+                );
+                for (const c of comments) {
+                  lines.push(`### @${c.user} (${c.created_at})`, '', c.body, '', '---', '');
+                }
+              }
+            } catch (_) {}
+          }
+          fs.writeFileSync('.github/pi/pr-review-context.md', `${lines.join('\n')}\n`);
+          NODE
+
+      - name: Run Pi review
+        if: steps.pi_config.outputs.enabled == 'true' && steps.pr.outputs.skip != 'true'
+        env:
+          DEEPSEEK_API_KEY: ${{ secrets.DEEPSEEK_API_KEY }}
+          PI_SKIP_VERSION_CHECK: '1'
+        run: |
+          set -o pipefail
+          cat REVIEW.md .github/pi/pr-review.prompt.md > /tmp/pi-prompt.md
+          pi -p \
+            --provider deepseek \
+            --model deepseek-v4-pro \
+            --tools read,grep,find,ls,bash \
+            --mode json \
+            < /tmp/pi-prompt.md \
+            | tee pi-events.jsonl \
+            | jq -rc --unbuffered '
+                if .type == "agent_start" then "🤖 pi agent started"
+                elif .type == "turn_start" then "── turn ──"
+                elif .type == "message_end" then
+                  "[\(.message.role)] " + (
+                    (.message.content // [])
+                    | map(
+                        if .type == "text" then "text(\(.text | length)c)"
+                        elif .type == "tool_use" then "🔧 \(.name) \(.input | @json | .[:160])"
+                        elif .type == "tool_result" then "✅ result"
+                        else .type
+                        end
+                      )
+                    | join(" | ")
+                  )
+                elif .type == "turn_end" then "── turn done (\((.toolResults // []) | length) tool result(s)) ──"
+                elif .type == "agent_end" then "🏁 pi agent done"
+                else empty
+                end
+              '
+
+          jq -r '
+            select(.type == "agent_end")
+            | .messages
+            | map(select(.role == "assistant"))
+            | last
+            | (.content[]? | select(.type == "text") | .text)
+          ' pi-events.jsonl > pi-final-message.md
+
+      - name: Post Pi review comment
+        if: steps.pi_config.outputs.enabled == 'true' && steps.pr.outputs.skip != 'true'
+        uses: actions/github-script@v7
+        env:
+          PR_NUMBER: ${{ steps.pr.outputs.pr_number }}
+        with:
+          github-token: ${{ github.token }}
+          script: |
+            const fs = require('fs');
+            const path = `${process.env.GITHUB_WORKSPACE}/pi-final-message.md`;
+            if (!fs.existsSync(path)) {
+              core.info('Pi did not produce a final message; skipping PR comment.');
+              return;
+            }
+            const body = fs.readFileSync(path, 'utf8').trim();
+            if (!body) {
+              core.info('Pi final message was empty; skipping PR comment.');
+              return;
+            }
+            await github.rest.issues.createComment({
+              owner: context.repo.owner,
+              repo: context.repo.repo,
+              issue_number: Number(process.env.PR_NUMBER),
+              body,
+            });

.github/workflows/pr-ready-review.yml

@@ -3,41 +3,161 @@ name: Claude Auto Review
 on:
   pull_request:
     types: [ready_for_review, opened]
+  workflow_call:
+    inputs:
+      pr_number:
+        description: 'PR number to review'
+        required: true
+        type: number
+      extra_prompt:
+        description: 'Additional reviewer instructions appended to the standard review prompt'
+        required: false
+        type: string
+        default: ''
+      triggered_by:
+        description: 'GitHub username that triggered this review (for audit only)'
+        required: false
+        type: string
+        default: ''
+    secrets:
+      CLAUDE_CODE_OAUTH_TOKEN:
+        required: true
+      WINDMILL_EE_PRIVATE_ACCESS:
+        required: false
 
 concurrency:
-  group: claude-review-${{ github.event.pull_request.number }}
+  group: claude-review-${{ inputs.pr_number || github.event.pull_request.number }}
   cancel-in-progress: true
 
 jobs:
+  check-membership:
+    if: github.event_name == 'pull_request'
+    uses: ./.github/workflows/check-org-membership.yml
+    with:
+      commenter: ${{ github.event.pull_request.user.login }}
+    secrets:
+      access_token: ${{ secrets.ORG_ACCESS_TOKEN }}
+
   auto-review:
+    needs: check-membership
     runs-on: ubuntu-latest
-    if: github.event.pull_request.draft == false || github.event.pull_request.ready_for_review == true
+    if: |
+      always() &&
+      (
+        needs.check-membership.result == 'skipped' ||
+        (needs.check-membership.result == 'success' && needs.check-membership.outputs.is_member == 'true')
+      ) &&
+      (
+        github.event_name == 'workflow_call' ||
+        (github.event.pull_request.draft == false || github.event.pull_request.ready_for_review == true)
+      )
     permissions:
       contents: read
       pull-requests: read
       id-token: write
     steps:
       - name: Checkout repository
-        uses: actions/checkout@v4
+        uses: actions/checkout@v5
         with:
           fetch-depth: 1
 
-      - name: Automatic PR Review
-        uses: anthropics/claude-code-action@beta
+      - name: Check EE access
+        id: ee
+        env:
+          EE_TOKEN: ${{ secrets.WINDMILL_EE_PRIVATE_ACCESS }}
+        run: |
+          if [ -n "$EE_TOKEN" ]; then
+            echo "available=true" >> "$GITHUB_OUTPUT"
+            echo "ee_repo_ref=$(cat ./backend/ee-repo-ref.txt)" >> "$GITHUB_OUTPUT"
+          else
+            echo "available=false" >> "$GITHUB_OUTPUT"
+          fi
+
+      - name: Checkout EE repository
+        if: steps.ee.outputs.available == 'true'
+        uses: actions/checkout@v5
         with:
-          anthropic_api_key: ${{ secrets.ANTHROPIC_API_KEY }}
-          timeout_minutes: "60"
-          direct_prompt: |
-            Please review this pull request and provide comprehensive feedback.
+          repository: windmill-labs/windmill-ee-private
+          path: ./windmill-ee-private
+          ref: ${{ steps.ee.outputs.ee_repo_ref }}
+          token: ${{ secrets.WINDMILL_EE_PRIVATE_ACCESS }}
+          fetch-depth: 1
 
-            Focus on:
-            - Code quality and best practices
-            - Potential bugs or issues
-            - Performance considerations
-            - Security implications
+      - name: Substitute EE code
+        if: steps.ee.outputs.available == 'true'
+        run: ./backend/substitute_ee_code.sh --copy --dir ./windmill-ee-private
 
-            Provide constructive feedback with specific suggestions for improvement.
-            Use inline comments to highlight specific areas of concern.
+      - name: Resolve PR number
+        id: resolve
+        env:
+          GH_TOKEN: ${{ github.token }}
+          REPO: ${{ github.repository }}
+          INPUT_PR_NUMBER: ${{ inputs.pr_number }}
+          EVENT_PR_NUMBER: ${{ github.event.pull_request.number }}
+          EVENT_PR_AUTHOR: ${{ github.event.pull_request.user.login }}
+        run: |
+          if [ -n "$INPUT_PR_NUMBER" ]; then
+            PR_NUMBER="$INPUT_PR_NUMBER"
+            PR_AUTHOR=$(gh api "repos/$REPO/pulls/$PR_NUMBER" --jq '.user.login')
+          else
+            PR_NUMBER="$EVENT_PR_NUMBER"
+            PR_AUTHOR="$EVENT_PR_AUTHOR"
+          fi
+          echo "pr_number=$PR_NUMBER" >> "$GITHUB_OUTPUT"
+          echo "pr_author=$PR_AUTHOR" >> "$GITHUB_OUTPUT"
 
-            At the end of your review, add complete instructions to reproduce the added changes through the app interface. These instructions will be given to a tester so he can verify the changes. It should be a short descriptive text (not a step by step or a list) on how to navigate the app (what page, what action, what input, etc) to see the changes.
-          allowed_tools: "mcp__github__create_pending_pull_request_review,mcp__github__add_pull_request_review_comment_to_pending_review,mcp__github__submit_pending_pull_request_review,mcp__github__get_pull_request_diff"
+      - name: Fetch prior PR discussion
+        id: prior
+        env:
+          GH_TOKEN: ${{ github.token }}
+          REPO: ${{ github.repository }}
+          PR_NUMBER: ${{ steps.resolve.outputs.pr_number }}
+        run: |
+          gh api "repos/$REPO/issues/$PR_NUMBER/comments?per_page=100" \
+            --jq '[.[] | {user: .user.login, created_at: .created_at, body: (.body | .[:4000])}] | sort_by(.created_at) | .[-20:]' \
+            > prior-comments.json || echo "[]" > prior-comments.json
+          jq -r '
+            if length == 0 then ""
+            else
+              "## Prior PR discussion (most recent up to 20 comments)\n\nIf you have already reviewed this PR (look for your own earlier comment), focus on what changed since then per the diff and respect any decisions the human made in replies. Do not re-flag findings the human already pushed back on.\n\n" +
+              (map("### @\(.user) (\(.created_at))\n\n\(.body)") | join("\n\n---\n\n"))
+            end
+          ' prior-comments.json > prior-comments.md
+
+      - name: Read review prompt
+        id: review-prompt
+        env:
+          EXTRA_PROMPT: ${{ inputs.extra_prompt }}
+        run: |
+          {
+            echo 'REVIEW_PROMPT<<EOF'
+            cat REVIEW.md
+            echo ''
+            cat .claude/review-prompt.md
+            if [ -n "$EXTRA_PROMPT" ]; then
+              echo ''
+              echo '## Additional reviewer instructions'
+              echo ''
+              printf '%s\n' "$EXTRA_PROMPT"
+            fi
+            if [ -s prior-comments.md ]; then
+              echo ''
+              cat prior-comments.md
+            fi
+            echo 'EOF'
+          } >> "$GITHUB_ENV"
+
+      - name: Automatic PR Review
+        uses: anthropics/claude-code-action@v1
+        with:
+          claude_code_oauth_token: ${{ secrets.CLAUDE_CODE_OAUTH_TOKEN }}
+          track_progress: true
+          prompt: |
+            REPO: ${{ github.repository }}
+            PR NUMBER: ${{ steps.resolve.outputs.pr_number }}
+            PR AUTHOR: ${{ steps.resolve.outputs.pr_author }}
+
+            ${{ env.REVIEW_PROMPT }}
+          claude_args: |
+            --allowedTools "mcp__github_inline_comment__create_inline_comment,Bash(gh pr comment:*),Bash(gh pr diff:*),Bash(gh pr view:*)"
+            --model claude-opus-4-8

.github/workflows/pr-review-commands.yml

@@ -0,0 +1,123 @@
+name: PR Review Commands
+
+on:
+  issue_comment:
+    types: [created]
+
+jobs:
+  parse:
+    if: github.event.issue.pull_request != null
+    runs-on: ubuntu-latest
+    outputs:
+      command: ${{ steps.parse.outputs.command }}
+      extra_prompt: ${{ steps.parse.outputs.extra_prompt }}
+    steps:
+      - name: Parse command from comment
+        id: parse
+        env:
+          BODY: ${{ github.event.comment.body }}
+        run: |
+          FIRST_LINE=$(printf '%s' "$BODY" | head -n 1 | sed -E 's/^[[:space:]]+//; s/[[:space:]]+$//')
+          FIRST_WORD=${FIRST_LINE%% *}
+          case "$FIRST_WORD" in
+            /review|/codex|/pi|/claude)
+              COMMAND="${FIRST_WORD#/}"
+              REMAINDER_FIRST_LINE=${FIRST_LINE#"$FIRST_WORD"}
+              REMAINDER_FIRST_LINE=${REMAINDER_FIRST_LINE# }
+              REST=$(printf '%s' "$BODY" | tail -n +2)
+              {
+                echo "command=$COMMAND"
+                echo 'extra_prompt<<EXTRA_EOF'
+                if [ -n "$REMAINDER_FIRST_LINE" ]; then
+                  printf '%s\n' "$REMAINDER_FIRST_LINE"
+                fi
+                if [ -n "$REST" ]; then
+                  printf '%s\n' "$REST"
+                fi
+                echo 'EXTRA_EOF'
+              } >> "$GITHUB_OUTPUT"
+              ;;
+            *)
+              echo "command=" >> "$GITHUB_OUTPUT"
+              ;;
+          esac
+
+  check-membership:
+    needs: parse
+    if: needs.parse.outputs.command != ''
+    uses: ./.github/workflows/check-org-membership.yml
+    secrets:
+      access_token: ${{ secrets.ORG_ACCESS_TOKEN }}
+
+  acknowledge:
+    needs: [parse, check-membership]
+    if: needs.parse.outputs.command != '' && needs.check-membership.outputs.is_member == 'true'
+    runs-on: ubuntu-latest
+    permissions:
+      issues: write
+      pull-requests: write
+    steps:
+      - name: React to comment with eyes
+        env:
+          GH_TOKEN: ${{ github.token }}
+          REPO: ${{ github.repository }}
+          COMMENT_ID: ${{ github.event.comment.id }}
+        run: |
+          gh api -X POST \
+            "/repos/$REPO/issues/comments/$COMMENT_ID/reactions" \
+            -f content=eyes >/dev/null
+
+  claude:
+    needs: [parse, check-membership]
+    if: |
+      needs.check-membership.outputs.is_member == 'true' &&
+      (needs.parse.outputs.command == 'review' || needs.parse.outputs.command == 'claude')
+    permissions:
+      contents: read
+      pull-requests: read
+      id-token: write
+    uses: ./.github/workflows/pr-ready-review.yml
+    with:
+      pr_number: ${{ github.event.issue.number }}
+      extra_prompt: ${{ needs.parse.outputs.extra_prompt }}
+      triggered_by: ${{ github.event.comment.user.login }}
+    secrets:
+      CLAUDE_CODE_OAUTH_TOKEN: ${{ secrets.CLAUDE_CODE_OAUTH_TOKEN }}
+      WINDMILL_EE_PRIVATE_ACCESS: ${{ secrets.WINDMILL_EE_PRIVATE_ACCESS }}
+
+  codex:
+    needs: [parse, check-membership]
+    if: |
+      needs.check-membership.outputs.is_member == 'true' &&
+      (needs.parse.outputs.command == 'review' || needs.parse.outputs.command == 'codex')
+    permissions:
+      contents: read
+      issues: write
+      pull-requests: write
+    uses: ./.github/workflows/codex-pr-review.yml
+    with:
+      pr_number: ${{ github.event.issue.number }}
+      extra_prompt: ${{ needs.parse.outputs.extra_prompt }}
+      triggered_by: ${{ github.event.comment.user.login }}
+    secrets:
+      OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
+      CODEX_AUTH_JSON: ${{ secrets.CODEX_AUTH_JSON }}
+      WINDMILL_EE_PRIVATE_ACCESS: ${{ secrets.WINDMILL_EE_PRIVATE_ACCESS }}
+
+  pi:
+    needs: [parse, check-membership]
+    if: |
+      needs.check-membership.outputs.is_member == 'true' &&
+      (needs.parse.outputs.command == 'review' || needs.parse.outputs.command == 'pi')
+    permissions:
+      contents: read
+      issues: write
+      pull-requests: write
+    uses: ./.github/workflows/pi-pr-review.yml
+    with:
+      pr_number: ${{ github.event.issue.number }}
+      extra_prompt: ${{ needs.parse.outputs.extra_prompt }}
+      triggered_by: ${{ github.event.comment.user.login }}
+    secrets:
+      DEEPSEEK_API_KEY: ${{ secrets.DEEPSEEK_API_KEY }}
+      WINDMILL_EE_PRIVATE_ACCESS: ${{ secrets.WINDMILL_EE_PRIVATE_ACCESS }}

.github/workflows/publish-cli-docs.yml

@@ -0,0 +1,84 @@
+name: Publish CLI docs repo
+
+# Regenerates the windmill-cli-docs repo (consumed by context7) from the
+# canonical sources in this repo on every Windmill release.
+#
+# Required secret:
+#   CLI_DOCS_DEPLOY_KEY — ed25519 private key whose public half is registered
+#                        as a write-access deploy key on
+#                        windmill-labs/windmill-cli-docs.
+
+on:
+  push:
+    tags:
+      - "v*"
+  workflow_dispatch:
+
+# Serialize pushes to windmill-cli-docs so two release tags landing close
+# together (e.g. a release-please bump + a hotfix) can't race to force-push
+# the docs repo.
+concurrency:
+  group: publish-cli-docs
+  cancel-in-progress: false
+
+jobs:
+  publish:
+    runs-on: ubuntu-latest
+    steps:
+      - name: Checkout windmill (source of truth)
+        uses: actions/checkout@v4
+        with:
+          path: windmill
+
+      - name: Checkout windmill-cli-docs (publish target)
+        uses: actions/checkout@v4
+        with:
+          repository: windmill-labs/windmill-cli-docs
+          path: windmill-cli-docs
+          ssh-key: ${{ secrets.CLI_DOCS_DEPLOY_KEY }}
+          fetch-depth: 0
+
+      - uses: actions/setup-python@v5
+        with:
+          python-version: "3.12"
+
+      - name: Install dependencies
+        run: pip install pyyaml
+
+      - name: Regenerate docs
+        run: |
+          python3 windmill/system_prompts/generate.py \
+            --context7-dir "$GITHUB_WORKSPACE/windmill-cli-docs"
+
+      - name: Commit and push if changed
+        working-directory: windmill-cli-docs
+        env:
+          REF_NAME: ${{ github.ref_name }}
+          REF_TYPE: ${{ github.ref_type }}
+        run: |
+          git config user.name "windmill-bot"
+          git config user.email "bot@windmill.dev"
+          git add -A
+          if git diff --cached --quiet; then
+            echo "No doc changes for ${REF_NAME}."
+            committed=false
+          else
+            committed=true
+            if [ "${REF_TYPE}" = "tag" ]; then
+              git commit -m "chore: sync from windmill ${REF_NAME}"
+            else
+              git commit -m "chore: sync from windmill (manual dispatch from ${REF_NAME})"
+            fi
+            git push origin HEAD
+          fi
+          # Always mirror the version tag on tag pushes, even when content
+          # didn't change — downstream consumers tie snapshots to releases by
+          # tag, and skipping it would leave the docs repo without a tag for
+          # the new Windmill release.
+          # workflow_dispatch from a non-tag ref skips this so we don't
+          # create a junk tag named after a branch.
+          if [ "${REF_TYPE}" = "tag" ]; then
+            git tag -f "${REF_NAME}"
+            git push origin "${REF_NAME}" --force
+            echo "Mirrored tag ${REF_NAME} to windmill-cli-docs (content changed: ${committed})."
+          fi

.github/workflows/publish_extra.yml

@@ -0,0 +1,126 @@
+env:
+  REGISTRY: ghcr.io
+  ECR_REGISTRY: 976079455550.dkr.ecr.us-east-1.amazonaws.com
+  IMAGE_NAME: ${{ github.repository }}-extra
+
+name: Publish windmill-extra
+on:
+  push:
+    tags:
+      - "v*"
+  workflow_dispatch:
+
+permissions: write-all
+
+jobs:
+  sleep:
+    runs-on: ubicloud
+    steps:
+      - name: Sleep for 900 seconds waiting for pypi to update index
+        if: startsWith(github.ref, 'refs/tags/v')
+        run: sleep 900
+        shell: bash
+
+  # Build and test the image before publishing
+  test_extra:
+    needs: [sleep]
+    runs-on: ubicloud-standard-8
+    steps:
+      - uses: actions/checkout@v4
+        with:
+          fetch-depth: 0
+
+      - name: Setup Bun
+        uses: oven-sh/setup-bun@v2
+        with:
+          bun-version: latest
+
+      - name: Set up Docker Buildx
+        uses: docker/setup-buildx-action@v3
+
+      - name: Build test image
+        uses: docker/build-push-action@v5
+        with:
+          context: .
+          file: ./docker/DockerfileExtra
+          load: true
+          tags: windmill-extra:test
+          cache-from: type=gha
+          cache-to: type=gha,mode=max
+
+      - name: Start container
+        run: |
+          docker run -d --name windmill-extra-test \
+            -p 3001:3001 -p 3002:3002 -p 3003:3003 \
+            -e ENABLE_LSP=true \
+            -e ENABLE_MULTIPLAYER=true \
+            -e ENABLE_DEBUGGER=true \
+            -e DEBUGGER_PORT=3003 \
+            -e REQUIRE_SIGNED_DEBUG_REQUESTS=false \
+            windmill-extra:test
+
+          # Wait for container to start
+          echo "Waiting for container to initialize..."
+          sleep 10
+
+          # Show container logs for debugging
+          docker logs windmill-extra-test
+
+      - name: Run integration tests
+        run: |
+          bun run docker/test_windmill_extra.ts
+
+      - name: Show container logs on failure
+        if: failure()
+        run: |
+          echo "=== Container logs ==="
+          docker logs windmill-extra-test
+
+      - name: Cleanup
+        if: always()
+        run: |
+          docker stop windmill-extra-test || true
+          docker rm windmill-extra-test || true
+
+  publish_extra:
+    needs: [sleep, test_extra]
+    runs-on: ubicloud-standard-8
+    steps:
+      - uses: actions/checkout@v4
+        with:
+          fetch-depth: 0
+
+      - uses: depot/setup-action@v1
+
+      - name: Docker meta
+        id: meta
+        uses: docker/metadata-action@v5
+        with:
+          images: |
+            ${{ env.REGISTRY }}/${{ env.IMAGE_NAME }}
+          tags: |
+            type=ref,event=branch
+            type=ref,event=pr
+            type=semver,pattern={{version}}
+            type=semver,pattern={{major}}.{{minor}}
+
+      - name: Login to registry
+        uses: docker/login-action@v3
+        with:
+          registry: ${{ env.REGISTRY }}
+          username: ${{ github.actor }}
+          password: ${{ secrets.GITHUB_TOKEN }}
+
+      - name: Build and push publicly
+        uses: depot/build-push-action@v1
+        with:
+          context: .
+          file: ./docker/DockerfileExtra
+          platforms: linux/amd64,linux/arm64
+          push: true
+          tags: |
+            ${{ env.REGISTRY }}/${{ env.IMAGE_NAME }}:latest
+            ${{ steps.meta.outputs.tags }}
+          labels: |
+            ${{ steps.meta.outputs.labels }}
+            org.opencontainers.image.licenses=AGPLv3

.github/workflows/publish_windows_worker.yml

@@ -13,7 +13,7 @@ env:
 
 jobs:
   cargo_build_windows:
-    runs-on: windows-latest
+    runs-on: blacksmith-16vcpu-windows-2025
     steps:
       - uses: actions/checkout@v4
 
@@ -32,19 +32,24 @@ jobs:
           token: ${{ secrets.WINDMILL_EE_PRIVATE_ACCESS }}
           fetch-depth: 0
 
-      - name: Setup Rust
-        uses: actions-rs/toolchain@v1
+      - uses: actions-rust-lang/setup-rust-toolchain@v1
         with:
-          toolchain: 1.88.0
-          override: true
+          cache-workspaces: backend
+          toolchain: 1.93.0
 
       - name: Substitute EE code
         shell: bash
         run: |
           ./backend/substitute_ee_code.sh --copy --dir ./windmill-ee-private
 
+      - name: Cargo build dynamic libraries windows
+        timeout-minutes: 180
+        run: |
+          cd backend/windmill-duckdb-ffi-internal
+          cargo build --release -p windmill_duckdb_ffi_internal
+
       - name: Cargo build windows
-        timeout-minutes: 90
+        timeout-minutes: 180
         run: |
           vcpkg.exe install openssl-windows:x64-windows
           vcpkg.exe install openssl:x64-windows-static
@@ -53,7 +58,7 @@ jobs:
           $env:OPENSSL_DIR="${Env:VCPKG_INSTALLATION_ROOT}\installed\x64-windows-static"
           mkdir frontend/build && cd backend
           New-Item -Path . -Name "windmill-api/openapi-deref.yaml" -ItemType "File" -Force
-          cargo build --release --features=enterprise,stripe,embedding,parquet,prometheus,openidconnect,cloud,jemalloc,tantivy,license,http_trigger,zip,oauth2,kafka,sqs_trigger,nats,postgres_trigger,mqtt_trigger,gcp_trigger,websocket,smtp,static_frontend,all_languages_windows,mcp,private
+          cargo build --release --features=ee_windows
       - name: Rename binary with corresponding architecture
         run: |
           Rename-Item -Path ".\backend\target\release\windmill.exe" -NewName "windmill-ee.exe"
@@ -63,3 +68,9 @@ jobs:
         with:
           files: |
             ./backend/target/release/windmill-ee.exe
+
+      - name: Attach dynamic libraries to release
+        uses: softprops/action-gh-release@v2
+        with:
+          files: |
+            ./backend/windmill-duckdb-ffi-internal/target/release/windmill_duckdb_ffi_internal.dll

.github/workflows/rust-client-check.yml

@@ -18,10 +18,7 @@ jobs:
     runs-on: ubicloud-standard-8
     steps:
       - uses: actions/checkout@v4
-      - uses: cachix/install-nix-action@v20
-        with:
-          extra_nix_config: |
-            experimental-features = nix-command flakes
+      - uses: cachix/install-nix-action@v31
       - name: Check rust client builds
         run: cd rust-client && nix develop ../ --command ./dev.nu --check
         timeout-minutes: 16

.github/workflows/rust_on_release.yml

@@ -10,10 +10,7 @@ jobs:
     runs-on: ubicloud-standard-8
     steps:
       - uses: actions/checkout@v4
-      - uses: cachix/install-nix-action@v20
-        with:
-          extra_nix_config: |
-            experimental-features = nix-command flakes
+      - uses: cachix/install-nix-action@v31
       - run: cd rust-client && nix develop ../ --command ./dev.nu --check --publish 
         env:
           CRATES_IO_TOKEN: ${{ secrets.CRATES_IO_TOKEN }}

.github/workflows/shareable-discord-notification.yml

@@ -24,9 +24,26 @@ on:
       DISCORD_GUILD_ID:
         description: "The Discord guild ID"
         type: string
+      COMMENT_BODY:
+        description: "The comment body"
+        type: string
+        default: ""
+      COMMENT_AUTHOR:
+        description: "The comment author"
+        type: string
+        default: ""
+      COMMENT_URL:
+        description: "The comment URL"
+        type: string
+        default: ""
+      COMMENT_IS_EDIT:
+        description: "Whether this is an edit of an existing comment"
+        type: string
+        default: "false"
     secrets:
       DISCORD_WEBHOOK_URL:
         description: "Discord Webhook URL"
+        required: false
       DISCORD_BOT_TOKEN:
         description: "Discord Bot Token"
 
@@ -117,3 +134,81 @@ jobs:
           curl -X PUT \
             -H "Authorization: Bot $BOT_TOKEN" \
             "https://discord.com/api/v10/channels/$thread_id/messages/$message_id/reactions/%E2%9C%85/@me"
+
+  post_comment:
+    runs-on: ubuntu-latest
+    if: ${{ inputs.PR_STATUS == 'comment' }}
+    steps:
+      - name: Post or update comment in Discord thread
+        env:
+          BOT_TOKEN: ${{ secrets.DISCORD_BOT_TOKEN }}
+          CHANNEL_ID: ${{ inputs.DISCORD_CHANNEL_ID }}
+          GUILD_ID: ${{ inputs.DISCORD_GUILD_ID }}
+          PR_NUMBER: ${{ inputs.PR_NUMBER }}
+          COMMENT_BODY: ${{ inputs.COMMENT_BODY }}
+          COMMENT_AUTHOR: ${{ inputs.COMMENT_AUTHOR }}
+          COMMENT_URL: ${{ inputs.COMMENT_URL }}
+          COMMENT_IS_EDIT: ${{ inputs.COMMENT_IS_EDIT }}
+        run: |
+          # 1) Find the thread by PR number
+          threads=$(curl -s -H "Authorization: Bot $BOT_TOKEN" \
+            "https://discord.com/api/v10/guilds/${GUILD_ID}/threads/active")
+          thread_id=$(echo "$threads" | jq -r \
+            --arg cid "$CHANNEL_ID" \
+            --arg pref "#${PR_NUMBER}:" \
+            '.threads[] | select(.parent_id == $cid and (.name | startswith($pref))) | .id')
+
+          if [ -z "$thread_id" ]; then
+            echo "Thread not found for PR #${PR_NUMBER}, skipping"
+            exit 0
+          fi
+
+          # 2) Truncate comment body to fit Discord's 2000 char limit
+          # Reserve space for the author line + link (~100 chars)
+          max_body=1800
+          if [ ${#COMMENT_BODY} -gt $max_body ]; then
+            # For bot comments, show the tail (conclusions/code tend to be at the end)
+            if [[ "$COMMENT_AUTHOR" == *"[bot]"* ]] || [[ "$COMMENT_AUTHOR" == *"-bot"* ]]; then
+              truncated_body="...${COMMENT_BODY: -$max_body}"
+            else
+              truncated_body="${COMMENT_BODY:0:$max_body}..."
+            fi
+          else
+            truncated_body="$COMMENT_BODY"
+          fi
+
+          # 3) Build the message content
+          if [ "$COMMENT_IS_EDIT" = "true" ]; then
+            message=$(printf '**%s** [edited comment](%s):\n%s' "$COMMENT_AUTHOR" "$COMMENT_URL" "$truncated_body")
+          else
+            message=$(printf '**%s** [commented](%s):\n%s' "$COMMENT_AUTHOR" "$COMMENT_URL" "$truncated_body")
+          fi
+          payload=$(jq -n --arg content "$message" '{content: $content, flags: 4, allowed_mentions: {parse: []}}')
+
+          # 4) If this is an edit, try to find and update the existing Discord message
+          if [ "$COMMENT_IS_EDIT" = "true" ]; then
+            # Search recent messages in the thread for one containing the comment URL
+            messages=$(curl -s -H "Authorization: Bot $BOT_TOKEN" \
+              "https://discord.com/api/v10/channels/${thread_id}/messages?limit=100")
+            existing_msg_id=$(echo "$messages" | jq -r \
+              --arg url "$COMMENT_URL" \
+              '[.[] | select(.content | contains($url))] | first | .id // empty')
+
+            if [ -n "$existing_msg_id" ]; then
+              echo "Updating existing Discord message $existing_msg_id"
+              curl -s -X PATCH \
+                -H "Authorization: Bot $BOT_TOKEN" \
+                -H "Content-Type: application/json" \
+                -d "$payload" \
+                "https://discord.com/api/v10/channels/${thread_id}/messages/${existing_msg_id}"
+              exit 0
+            fi
+            echo "Original Discord message not found, posting as new message"
+          fi
+
+          # 5) Post a new message to the thread
+          curl -s -X POST \
+            -H "Authorization: Bot $BOT_TOKEN" \
+            -H "Content-Type: application/json" \
+            -d "$payload" \
+            "https://discord.com/api/v10/channels/${thread_id}/messages"

.github/workflows/weekly-pr-summary.yml

@@ -0,0 +1,145 @@
+name: Weekly PR Summary
+
+on:
+  schedule:
+    # Every Friday at 8:00 AM UTC
+    - cron: '0 8 * * 5'
+  workflow_dispatch:
+    # Allow manual triggering for testing
+
+jobs:
+  weekly-pr-summary:
+    runs-on: ubicloud-standard-4
+    timeout-minutes: 30
+    permissions:
+      contents: read
+      pull-requests: read
+      issues: write
+      id-token: write
+    steps:
+      - name: Checkout repository
+        uses: actions/checkout@v4
+        with:
+          fetch-depth: 1
+
+      - name: Generate Weekly PR Summary
+        uses: anthropics/claude-code-action@v1
+        with:
+          claude_code_oauth_token: ${{ secrets.CLAUDE_CODE_OAUTH_TOKEN }}
+          prompt: |
+            REPO: ${{ github.repository }}
+
+            Generate a categorized weekly summary of ONLY MERGED Pull Requests from the past 7 days.
+
+            ## Your Task:
+
+            1. **Calculate Date Range**:
+               - Run: `CUTOFF_DATE=$(date --date='7 days ago' --iso-8601)`
+               - Run: `TODAY=$(date --iso-8601)`
+               - This gives you the exact 7-day window (store these in variables for use in commands)
+
+            2. **Fetch ONLY Merged PRs from Past Week**:
+               - Command: `gh pr list --repo ${{ github.repository }} --state merged --search "merged:>=$CUTOFF_DATE" --limit 100 --json number,title,author,mergedAt,url`
+               - This returns ONLY PRs that were merged in the last 7 days
+               - The --search flag filters by merge date using GitHub's search syntax
+               - **FILTER OUT** any PRs with titles starting with "chore: release" or "chore(release)"
+
+            3. **Categorize PRs**: Group PRs into three categories by analyzing titles and labels:
+               - **Features**: PRs with titles starting with "feat:", "feature:", or containing "add", "implement", "new"
+               - **Bug Fixes**: PRs with titles starting with "fix:", "bug:", or containing "fix", "resolve", "patch"
+               - **Other**: All remaining PRs (improvements, refactors, docs, chores, etc.)
+
+            4. **Gather Details**: For each feature and bug fix merged PR, include:
+               - Full PR title (NO truncation, NO links)
+               - Author (extract login from author.login in JSON)
+               - Brief summary: Use `gh pr view <number> --json body` to get PR description, then extract first paragraph or key points (1-2 sentences max)
+
+            5. **Character Limit Enforcement**:
+               - The final summary MUST be under 5000 characters
+               - If the summary exceeds 5000 characters, truncate PR descriptions (NOT titles) and add at the end: "_and X more PRs_" where X is the count of omitted PRs
+
+            6. **Save Summary to Markdown File**: Write the summary to a file for webhook delivery:
+               - Save the complete formatted markdown to: `summary.md`
+               - Do not commit the file to the repository
+
+            ## Output Format:
+
+            ```markdown
+            ### 📈 Weekly overview
+            - **Total merged**: X
+            - **Features**: Y
+            - **Bug Fixes**: Z
+            - **Other**: W
+
+            ### ✨ Features (Y)
+            - **[Full PR Title]** by @username - [brief impact description]
+            - **[Full PR Title]** by @username - [brief impact description]
+
+            ### 🐛 Bug Fixes (Z)
+            - **[Full PR Title]** by @username - [brief impact description]
+            - **[Full PR Title]** by @username - [brief impact description]
+
+            _and X more PRs_
+            ```
+
+            ## Important Notes:
+            - **CRITICAL**: ONLY include PRs with state "merged" from the last 7 days
+            - **CRITICAL**: EXCLUDE all PRs with titles starting with "chore: release" or "chore(release)"
+            - **CRITICAL**: Total character count MUST be under 5000 characters
+            - Count the number of "Other" PRs but do not include a section for them in the output
+            - Only use ### markdown headers for major sections and emoji indicators
+            - NO links to PRs
+            - NO merged date in output
+            - NEVER truncate PR titles - show full titles
+            - Use GitHub CLI (`gh`) for all operations
+            - Sort PRs within each category by merge date (most recent first)
+            - If a PR has no description, write "(No description provided)"
+            - Extract meaningful summary from PR body - look for the first paragraph or key bullet points
+            - Parse JSON responses carefully using `jq` or similar tools
+            - If summary exceeds 5000 chars, shorten PR descriptions and add "_and X more PRs_" at the end
+            - Count PRs in each category and display in both overview and section headers
+
+            ## Saving the Markdown Output:
+            After generating the markdown summary, save it to a file, BUT DO NOT COMMIT IT TO THE REPOSITORY.
+
+            ## Write Tool Fallback:
+            - First, attempt to use the Write tool to create `summary.md` with the markdown content
+            - If the Write tool returns ANY error or fails:
+              1. Use the Bash tool with the `echo` command instead
+              2. Use a heredoc to write the content: `cat > summary.md << 'EOF'` followed by your markdown content and `EOF` on a new line
+              3. Example: `cat > summary.md << 'EOF'\n[your markdown content here]\nEOF`
+              4. This ensures the file is always created regardless of Write tool issues
+            - Verify the file was created by running: `ls -lh summary.md`
+          claude_args: |
+            --allowedTools "Edit,MultiEdit,Write,Read,Glob,Grep,LS,Bash"
+            --model haiku
+
+      - name: Send Summary to Windmill
+        if: hashFiles('summary.md') != ''
+        env:
+          WEEKLY_SUMMARY_TOKEN: ${{ secrets.WEEKLY_SUMMARY_TOKEN }}
+        run: |
+          if [[ -f "summary.md" ]]; then
+            echo "Found summary.md, sending to Windmill..."
+
+            # Read the markdown content
+            MARKDOWN_CONTENT=$(cat summary.md)
+
+            # Create JSON payload
+            PAYLOAD=$(jq -n --arg markdown "$MARKDOWN_CONTENT" '{markdown: $markdown}')
+
+            # Send to Windmill webhook
+            RESULT=$(curl -s \
+              -H 'Content-Type: application/json' \
+              -H "Authorization: Bearer $WEEKLY_SUMMARY_TOKEN" \
+              -X POST \
+              -d "$PAYLOAD" \
+              'https://app.windmill.dev/api/w/windmill-labs/jobs/run/f/f/ai/send_past_week_pr_summaries_to_discord')
+
+            echo "Windmill response:"
+            echo -E "$RESULT"
+            echo "✅ Summary sent successfully to Windmill!"
+          else
+            echo "⚠️ Warning: summary.md not found, skipping delivery"
+            exit 1
+          fi

.gitignore

@@ -13,3 +13,25 @@ backend/.minio-data
 .aider*
 !.aiderignore
 rust-client/Cargo.toml
+
+# Worktree-generated port isolation
+.env.local
+.webmux.local.yaml
+
+# Worktree-specific Claude Code settings (generated by scripts/worktree-env)
+.claude/settings.local.json
+.claude/worktrees/
+
+# Symlinked cache directories (for git worktrees)
+backend/target
+frontend/node_modules
+typescript-client/node_modules
+ai_evals/node_modules
+ai_evals/results/
+frontend/.svelte-kit
+backend/chrome_profiler.json
+.fast-check/
+__pycache__/
+.playwright-mcp/
+.codex
+.claude/scheduled_tasks.lock

.mcp.json

@@ -0,0 +1,18 @@
+{
+  "mcpServers": {
+    "svelte": {
+      "type": "http",
+      "url": "https://mcp.svelte.dev/mcp"
+    },
+    "playwright": {
+      "type": "stdio",
+      "command": "npx",
+      "args": ["-y", "@playwright/mcp@latest", "--browser", "chromium", "--headless"]
+    },
+    "playwright-headed": {
+      "type": "stdio",
+      "command": "npx",
+      "args": ["-y", "@playwright/mcp@latest", "--browser", "chromium"]
+    }
+  }
+}

.webmux.yaml

@@ -0,0 +1,156 @@
+# Project display name in the dashboard
+name: Windmill
+
+workspace:
+  mainBranch: main
+  worktreeRoot: ../windmill__worktrees
+  defaultAgent: claude
+
+startupEnvs:
+  CARGO_FEATURES: "quickjs"
+  WM_CLONE_DB: false
+  USE_RUST_PLUGIN: false
+
+lifecycleHooks:
+  postCreate: bash ./scripts/post-create.sh
+  preRemove: bash ./scripts/pre-remove.sh
+
+auto_name:
+  provider: claude
+  model: haiku
+
+# Each service defines a port env var that webmux injects into pane and agent
+# process environments when creating a worktree. Ports are auto-assigned:
+# base + (slot x step).
+services:
+  - name: backend
+    portEnv: BACKEND_PORT
+    portStart: 8000
+    portStep: 10
+  - name: frontend
+    portEnv: FRONTEND_PORT
+    portStart: 3000
+    portStep: 10
+
+profiles:
+  full:
+    runtime: host
+    yolo: true
+    envPassthrough: []
+    systemPrompt: >
+      You are running inside a tmux session with other panes running services.
+      Pane layout (current window):
+      - Pane 0: this pane (claude agent)
+      - Pane 1: backend (cargo watch -x run)
+      - Pane 2: frontend (npm run dev)
+      To check logs, use: \`tmux capture-pane -t $(tmux display-message -t "$TMUX_PANE" -p '#{session_name}:#{window_name}').1 -p -S -50\` (backend) or \`tmux capture-pane -t $(tmux display-message -t "$TMUX_PANE" -p '#{session_name}:#{window_name}').2 -p -S -50\` (frontend).
+      For this window specifically, backend is running on: ${BACKEND_PORT} and frontend is running on: ${FRONTEND_PORT}.
+      To connect to the database, use this connection string: ${DATABASE_URL}
+      Because we are running backend with cargo watch, to verify your changes, just check the logs in the backend pane. No need for cargo check.
+      For UI verification, use the Playwright MCP (`mcp__playwright__*`) — the `playwright` server is headless and works without a display. Navigate to http://localhost:${FRONTEND_PORT}, log in as admin@windmill.dev / changeme.
+      IMPORTANT: Read docs/autonomous-mode.md before starting any work.
+    panes:
+      - id: agent
+        kind: agent
+        focus: true
+      - id: backend
+        kind: command
+        split: right
+        workingDir: backend
+        command: PORT=${BACKEND_PORT:-8000} cargo watch -x "run ${CARGO_FEATURES:+--features $CARGO_FEATURES}"
+      - id: frontend
+        kind: command
+        split: bottom
+        workingDir: frontend
+        command: npm run generate-backend-client && REMOTE=${REMOTE:-http://localhost:${BACKEND_PORT:-8000}} npm run dev -- --port ${FRONTEND_PORT:-3000} --host 0.0.0.0
+        
+  frontendOnly:
+    runtime: host
+    yolo: true
+    envPassthrough: []
+    systemPrompt: >
+      You are running inside a tmux session with other panes running services.
+      Pane layout (current window):
+      - Pane 0: this pane (claude agent)
+      - Pane 1: frontend (npm run dev)
+      To check logs, use: \`tmux capture-pane -t $(tmux display-message -t "$TMUX_PANE" -p '#{session_name}:#{window_name}').1 -p -S -50\` (frontend).
+      On this window specifically, frontend is running on: ${FRONTEND_PORT}.
+      To connect to the database, use this connection string: ${DATABASE_URL}
+      Because we are running frontend with npm run dev, to verify your changes, just check the logs in the frontend pane. No need for npm run build.
+      For UI verification, use the Playwright MCP (`mcp__playwright__*`) — the `playwright` server is headless and works without a display. Navigate to http://localhost:${FRONTEND_PORT}, log in as admin@windmill.dev / changeme.
+      IMPORTANT: Read docs/autonomous-mode.md before starting any work.
+    panes:
+      - id: agent
+        kind: agent
+        focus: true
+      - id: frontend
+        kind: command
+        split: right
+        workingDir: frontend
+        command: npm run generate-backend-client && npm run dev -- --port ${FRONTEND_PORT:-3000} --host 0.0.0.0
+
+  agentOnly:
+    runtime: host
+    yolo: true
+    envPassthrough: []
+    systemPrompt: >
+      IMPORTANT: Read docs/autonomous-mode.md before starting any work.
+    panes:
+      - id: agent
+        kind: agent
+        focus: true
+
+integrations:
+  github:
+    autoRemoveOnMerge: true
+    linkedRepos:
+      - repo: windmill-labs/windmill-ee-private
+        alias: ee-private
+        dir: ../windmill-ee-private__worktrees
+  linear:
+    enabled: true
+    autoCreateWorktrees: true
+    watchTeams: [WIN,GIT]
+
+oneshot:
+  systemPrompt: |
+    You are running in webmux ONESHOT mode.
+
+    # No interactive user
+    There is NO interactive user — nobody is watching the chat or will respond
+    to questions, approvals, or status checks. Any message asking the user to
+    review, approve, confirm, take a look, or "let you know" is wasted output:
+    it will not be answered.
+
+    # Your job
+    Take the task to its real conclusion without pausing:
+      1. Make the change.
+      2. Validate it (run the relevant tests, typecheck, build, or quick
+         manual check). For UI changes, drive the running frontend with
+         the Playwright MCP (`mcp__playwright__*`, headless) and confirm
+         the change works end-to-end before moving on.
+      3. Commit.
+      4. Push.
+      5. Open a pull request.
+    Only then are you done.
+
+    # Decisions
+    When something is ambiguous, pick the most reasonable default and proceed.
+    When you would normally ask "should I X or Y?", just pick one and continue
+    — note the choice in the PR description if it matters.
+
+    # PR readiness
+    Default to opening the PR as a draft. If you are highly confident in the
+    change — the scope is small and well-understood, validation passed
+    cleanly, and you would not change anything if a reviewer pushed back —
+    open the PR as ready-for-review directly (omit `--draft` when invoking
+    `gh pr create`, or call `gh pr ready <number>` after creation). Err on
+    the side of draft when validation was partial, the change touches
+    public APIs or shared infrastructure, or you made a non-obvious judgment
+    call.
+
+    # Ending your turn
+    Never end your turn with a question, a suggestion to "take a look", or a
+    request for approval. Stop only when the PR is open, or when you hit a
+    technical error you cannot recover from yourself (in which case clearly
+    state the blocker).

AGENTS.md

@@ -0,0 +1,113 @@
+# Windmill
+
+Open-source platform for internal tools, workflows, API integrations, background jobs, and UIs. Rust backend + Svelte 5 frontend.
+
+## Workflow
+
+1. **Understand**: Before coding, explore the codebase (see Code Navigation below). Use `outline` to understand file structure, `body` to read specific symbols, `def`/`callers`/`callees` to trace code, `Grep` to find usages. Read `docs/` for domain context.
+2. **Plan**: For non-trivial changes, use plan mode. For large features, break into reviewable stages
+3. **Execute**: Follow coding patterns from skills (`rust-backend`, `svelte-frontend`)
+4. **Validate**: After every change, run the appropriate checks per `docs/validation.md`
+
+## Documentation
+
+- **Validation**: `docs/validation.md` — what checks to run based on what you changed
+- **Enterprise**: `docs/enterprise.md` — EE file conventions and PR workflow
+- **Backend patterns**: use the `rust-backend` skill when writing Rust code
+- **Frontend patterns**: use the `svelte-frontend` skill when writing Svelte code. Do NOT edit svelte files unless you have read that skill.
+- **Frontend UUIDs**: do not call `crypto.randomUUID()` in frontend code. Import `randomUUID` from `$lib/utils/uuid` instead.
+- **Code review**: review the current PR or branch against the shared review policy in `REVIEW.md` (severity triage, public-surface checklist, AGENTS.md compliance, test-coverage assessment). The skill at `.agents/skills/local-review/SKILL.md` orchestrates it. All three CLIs auto-discover the same SKILL — Claude reads `.claude/skills/` (symlinked to the canonical `.agents/skills/` file), Codex and Pi read `.agents/skills/` directly. Invoke with `/local-review` in Claude Code, `$local-review` (or `/skills` selector) in Codex, or `pi --skill local-review` / `/skill:local-review` in Pi.
+- **Domain guides**: `.claude/skills/native-trigger/` and `frontend/tutorial-system-guide.mdc`
+- **Brand/UI guidelines**: `frontend/brand-guidelines.md`
+- **CLI commands**: when adding/modifying/removing a command, subcommand, option, or description in `cli/src/commands/`, run `python system_prompts/generate.py` to refresh `system_prompts/auto-generated/` and `cli/src/guidance/skills.gen.ts`. The CLI docs the agents use to operate `wmill` are derived from the source — stale generated files give agents the wrong flags.
+
+## Dev Environment
+
+- **Backend**: `cargo run` from `backend/` (API at http://localhost:8000)
+- **Frontend**: `REMOTE=http://localhost:8000 npm run dev` from `frontend/` (port 3000+)
+- **DB**: `psql postgres://postgres:changeme@localhost:5432/windmill`
+- **Login**: `admin@windmill.dev` / `changeme`
+- **Instance settings**: navigate to `/#superadmin-settings`
+- **Migrations**: use `cargo sqlx migrate add -r <name>` from `backend/` to create new migrations (never generate timestamps manually)
+
+## Verifying Frontend Changes
+
+After modifying frontend code, drive the running dev server with the **Playwright MCP** to verify the change in a real browser — don't claim a UI change works without exercising it.
+
+Two MCP servers are registered in `.mcp.json`:
+- `playwright` — headless Chromium, default for devboxes (no display required)
+- `playwright-headed` — windowed Chromium, when a display is available
+
+**One-time setup:** run `npx playwright install chromium` to download the browser binary (Playwright won't fetch it automatically on first use).
+
+Typical flow:
+1. Ensure backend (`cargo run`) and frontend (`REMOTE=http://localhost:8000 npm run dev`) are running
+2. `mcp__playwright__browser_navigate` to the relevant page (login at `admin@windmill.dev` / `changeme`)
+3. `mcp__playwright__browser_snapshot` to inspect the accessibility tree (preferred over screenshots for reading the DOM)
+4. `mcp__playwright__browser_click` / `browser_fill_form` / `browser_type` to interact
+5. `mcp__playwright__browser_take_screenshot` for visual confirmation
+6. `mcp__playwright__browser_console_messages` / `browser_network_requests` to surface errors
+
+**Attach the screenshots to the PR.** For any change under `frontend/`, embed screenshots of the affected UI in the PR body — the `pr` skill requires this and carries the upload recipe.
+
+If you cannot exercise a UI change (no dev server, etc.), say so explicitly rather than claiming success.
+
+## Banned Patterns
+
+### `$bindable(default_value)` on optional props
+
+Using `$bindable(default_value)` on props that can be `undefined` is **banned**. This pattern causes subtle bugs because the default value masks the `undefined` state.
+
+**Bad:**
+
+```svelte
+let { my_prop = $bindable(default_value) }: { my_prop?: string } = $props()
+```
+
+**Correct alternatives:**
+
+1. **Use `$derived` with nullish coalescing** — handle the potential `undefined` at the usage site:
+
+   ```svelte
+   let { my_prop = $bindable() }: { my_prop?: string } = $props()
+   let effective_value = $derived(my_prop ?? default_value)
+   ```
+
+2. **Create a `useMyPropState()` helper** — encapsulate the undefined-handling logic in a reusable function and call it higher in the component tree, so the child component always receives a defined value.
+
+## Code Navigation
+
+`wm-ts-nav` is an AST-aware code navigator. Use **wm-ts-nav** for structural queries — it skips comments/strings and understands symbol boundaries.
+
+**MUST use `outline` before `Read`** on unfamiliar files — a 500-line file costs ~500 lines of context, while `outline` costs ~20. Then **MUST use `body "X"`** instead of reading a full file to see one function/struct. Use `Read` with offset/limit only when you need surrounding context that `body` doesn't capture.
+- `refs "X" --caller` instead of reading files to find which function contains each reference
+- `callers "X"` / `callees "X"` for call-graph questions
+
+EE files (`*_ee.rs`, `*_ee.ts`, `*_ee.svelte`) are indexed — you can `outline`, `def`, `body`, `refs` etc. on them just like regular files.
+
+```bash
+NAV="sh wm-ts-nav/nav"
+# Use --root backend for Rust, --root frontend/src for TS/Svelte
+$NAV --root backend outline backend/path/to/file.rs      # file structure
+$NAV --root backend def "ServiceName"                     # find definition
+$NAV --root backend body "decrypt_oauth_data"             # extract source code
+$NAV --root backend search "%" --parent ServiceName       # methods on a type
+$NAV --root backend search "Trigger" --kind struct        # find by kind
+$NAV --root backend refs "X" --file handler.rs --caller   # scoped refs with caller
+$NAV --root backend callers "X"                           # who calls X?
+$NAV --root backend callees "X"                           # what does X call?
+```
+
+**Limitations** — syntax-level analysis, no type inference. Use **Grep** instead when completeness matters (finding all usages, exhaustiveness checks):
+- `refs`/`callers`/`callees` can't follow re-exports, glob imports, or different import paths to the same symbol
+- Trait impls, macro-generated symbols (`sqlx::FromRow`), and namespace member access (`ns.X`) are invisible
+- `callees` shows all identifiers in a function body, not just actual calls
+
+## Core Principles
+
+- **MUST `outline` before `Read`** on unfamiliar files — then `body` or `Read` with offset/limit for specifics
+- Search for existing code to reuse before writing new code
+- Follow established patterns in the codebase
+- Keep changes focused — don't refactor beyond what's asked
+- **Comments record constraints, not narration.** Write a comment only for what the code can't show: why a non-obvious approach is required, what breaks if it's "simplified" away. State each invariant once, at the place where someone would break it, in ≤4 lines. Don't describe what the next line does, don't repeat the same rationale at multiple sites, and don't address the PR reviewer (justifying a change belongs in the PR description, not the code). Describe the code as it is, never its drafting history: "we no longer do X", "unchanged behavior", "instead of the previous approach" are meaningless to a reader who never saw the earlier iteration — before finishing, reread your comments as if the current state is the only state that ever existed.
+- **Never attribute work to a specific customer, account, or "requested by a customer" in repo-tracked content** (PR descriptions, commit messages, code comments, docs). Describe changes by their technical motivation instead.

CLAUDE.md

@@ -1,10 +1 @@
-# Windmill Development Guide
-
-## Overview
-
-Windmill is an open-source developer platform for building internal tools, workflows, API integrations, background jobs, workflows, and user interfaces. See @windmill-overview.mdc for full platform details.
-
-## Language-Specific Guides
-
-- Backend (Rust): @backend/rust-best-practices.mdc + @backend/summarized_schema.txt
-- Frontend (Svelte 5): @frontend/svelte5-best-practices.mdc
+@AGENTS.md

Caddyfile

@@ -10,9 +10,16 @@
 
 {$BASE_URL} {
         bind {$ADDRESS}
-        reverse_proxy /ws/* http://lsp:3001
-        # reverse_proxy /ws_mp/* http://multiplayer:3002
+
+        # Extra services: LSP, Multiplayer, Debugger (windmill_extra gateway)
+        reverse_proxy /ws/* /ws_mp/* /ws_debug/* http://windmill_extra:3000
+
+        # Search indexer, Enterprise Edition (windmill_indexer:8002)
         # reverse_proxy /api/srch/* http://windmill_indexer:8002
+
+        # Default: Windmill server
         reverse_proxy /* http://windmill_server:8000
+
+        # TLS with custom certificates
         # tls /certs/cert.pem /certs/key.pem
 }

Dockerfile

@@ -1,9 +1,29 @@
 ARG DEBIAN_IMAGE=debian:bookworm-slim
-ARG RUST_IMAGE=rust:1.88-slim-bookworm
+ARG RUST_IMAGE=rust:1.93-slim-bookworm
+
+FROM debian:bookworm-slim AS nsjail
+
+WORKDIR /nsjail
+
+RUN apt-get -y update \
+    && apt-get install -y \
+    bison=2:3.8.* \
+    flex=2.6.* \
+    g++=4:12.2.* \
+    gcc=4:12.2.* \
+    git=1:2.39.* \
+    libprotobuf-dev=3.21.* \
+    libnl-route-3-dev=3.7.* \
+    make=4.3-4.1 \
+    pkg-config=1.8.* \
+    protobuf-compiler=3.21.*
+
+RUN git clone -b master --single-branch https://github.com/google/nsjail.git . && git checkout dccf911fd2659e7b08ce9507c25b2b38ec2c5800
+RUN make
 
 FROM ${RUST_IMAGE} AS rust_base
 
-RUN apt-get update && apt-get install -y git libssl-dev pkg-config npm
+RUN apt-get update && apt-get install -y git libssl-dev pkg-config npm mold clang
 
 RUN apt-get -y update \
     && apt-get install -y \
@@ -20,11 +40,25 @@ WORKDIR /windmill
 ENV SQLX_OFFLINE=true
 # ENV CARGO_INCREMENTAL=1
 
-FROM node:20-alpine as frontend
+FROM rust_base AS windmill_duckdb_ffi_internal_builder
+
+WORKDIR /windmill-duckdb-ffi-internal
+
+RUN apt-get update && apt-get install -y clang=1:14.0-55.* libclang-dev=1:14.0-55.* cmake=3.25.* && \
+    apt-get clean && \
+    rm -rf /var/lib/apt/lists/*
+
+COPY ./backend/windmill-duckdb-ffi-internal .
+
+RUN --mount=type=cache,target=/usr/local/cargo/registry \
+    --mount=type=cache,target=$SCCACHE_DIR,sharing=locked \
+    cargo build --release -p windmill_duckdb_ffi_internal
+
+FROM node:24-alpine as frontend
 
 # install dependencies
 WORKDIR /frontend
-COPY ./frontend/package.json ./frontend/package-lock.json ./
+COPY ./frontend/package.json ./frontend/package-lock.json ./frontend/.npmrc ./
 COPY ./frontend/scripts/ ./scripts/
 RUN npm ci
 
@@ -32,18 +66,21 @@ RUN npm ci
 COPY frontend .
 RUN mkdir /backend
 COPY /backend/windmill-api/openapi.yaml /backend/windmill-api/openapi.yaml
+COPY /backend/oauth_connect.json /backend/oauth_connect.json
 COPY /openflow.openapi.yaml /openflow.openapi.yaml
 COPY /backend/windmill-api/build_openapi.sh /backend/windmill-api/build_openapi.sh
+COPY /system_prompts/auto-generated /system_prompts/auto-generated
 
 RUN cd /backend/windmill-api && . ./build_openapi.sh
 COPY /backend/parsers/windmill-parser-wasm/pkg/ /backend/parsers/windmill-parser-wasm/pkg/
 COPY /typescript-client/docs/ /frontend/static/tsdocs/
+COPY /python-client/docs/ /frontend/static/pydocs/
 
 RUN npm run generate-backend-client
 ENV NODE_OPTIONS "--max-old-space-size=8192"
 ARG VITE_BASE_URL ""
 # Read more about macro in docker/dev.nu
-# -- MACRO-SPREAD-WASM-PARSER-DEV-ONLY -- # 
+# -- MACRO-SPREAD-WASM-PARSER-DEV-ONLY -- #
 RUN npm run build
 
 
@@ -61,7 +98,7 @@ ARG features=""
 
 COPY --from=planner /windmill/recipe.json recipe.json
 
-RUN apt-get update && apt-get install -y libxml2-dev=2.9.* libxmlsec1-dev=1.2.* clang=1:14.0-55.* libclang-dev=1:14.0-55.* cmake=3.25.* && \
+RUN apt-get update && apt-get install -y libxml2-dev=2.9.* libxmlsec1-dev=1.2.* libkrb5-dev libsasl2-dev libcurl4-openssl-dev clang=1:14.0-55.* libclang-dev=1:14.0-55.* cmake=3.25.* && \
     apt-get clean && \
     rm -rf /var/lib/apt/lists/*
 
@@ -82,6 +119,17 @@ RUN --mount=type=cache,target=/usr/local/cargo/registry \
     --mount=type=cache,target=$SCCACHE_DIR,sharing=locked \
     CARGO_NET_GIT_FETCH_WITH_CLI=true cargo build --release --features "$features"
 
+# Split debug info into a separate file, then strip the binary.
+# The .debug file can be extracted as a CI artifact for production debugging.
+# The debuglink allows gdb to auto-discover the debug file when placed next to the binary.
+RUN objcopy --only-keep-debug /windmill/target/release/windmill /windmill/target/release/windmill.debug \
+    && strip /windmill/target/release/windmill \
+    && objcopy --add-gnu-debuglink=/windmill/target/release/windmill.debug /windmill/target/release/windmill
+
+# Standalone stage for extracting the .debug file without including it in the final image.
+# Build with: docker build --target debuginfo --output type=local,dest=./out .
+FROM scratch AS debuginfo
+COPY --from=builder /windmill/target/release/windmill.debug /windmill.debug
 
 FROM ${DEBIAN_IMAGE}
 
@@ -90,27 +138,41 @@ ARG POWERSHELL_VERSION=7.5.0
 ARG POWERSHELL_DEB_VERSION=7.5.0-1
 ARG KUBECTL_VERSION=1.28.7
 ARG HELM_VERSION=3.14.3
-ARG GO_VERSION=1.25.0
+# NOTE: If changing, also change go version in workspace dependencies template at WorkspaceDependenciesEditor.svelte
+ARG GO_VERSION=1.26.0
 ARG APP=/usr/src/app
 ARG WITH_POWERSHELL=true
 ARG WITH_KUBECTL=true
 ARG WITH_HELM=true
 ARG WITH_GIT=true
+ARG features=""
 
 # To change latest stable version:
 # 1. Change placeholder in instanceSettings.ts
 # 2. Change LATEST_STABLE_PY in dockerfile
 # 3. Change #[default] annotation for PyVersion in backend
-ARG LATEST_STABLE_PY=3.11.10
+ARG LATEST_STABLE_PY=3.12
 ENV UV_PYTHON_INSTALL_DIR=/tmp/windmill/cache/py_runtime
 ENV UV_PYTHON_PREFERENCE=only-managed
-ENV UV_TOOL_BIN_DIR=/usr/local/bin
 
-ENV PATH /usr/local/bin:/root/.local/bin:$PATH
+RUN mkdir -p /usr/local/uv
+ENV UV_TOOL_BIN_DIR=/usr/local/bin
+ENV UV_TOOL_DIR=/usr/local/uv
+
+ENV PATH /usr/local/bin:/root/.local/bin:/tmp/.local/bin:$PATH
 
 
 RUN apt-get update \
-    && apt-get install -y --no-install-recommends netbase tzdata ca-certificates wget curl jq unzip build-essential unixodbc xmlsec1  software-properties-common \
+    && apt-get install -y --no-install-recommends netbase tzdata ca-certificates wget curl jq unzip build-essential unixodbc xmlsec1 software-properties-common tini gnupg lsb-release \
+    && if echo "$features" | grep -q "ee"; then apt-get install -y --no-install-recommends libsasl2-modules-gssapi-mit krb5-user; fi \
+    && apt-get clean \
+    && rm -rf /var/lib/apt/lists/*
+
+# Install latest PostgreSQL client (pg_dump) from official PostgreSQL apt repository
+RUN curl -fsSL https://www.postgresql.org/media/keys/ACCC4CF8.asc | gpg --dearmor -o /usr/share/keyrings/postgresql-archive-keyring.gpg \
+    && echo "deb [signed-by=/usr/share/keyrings/postgresql-archive-keyring.gpg] https://apt.postgresql.org/pub/repos/apt $(lsb_release -cs)-pgdg main" > /etc/apt/sources.list.d/pgdg.list \
+    && apt-get update \
+    && apt-get install -y --no-install-recommends postgresql-client \
     && apt-get clean \
     && rm -rf /var/lib/apt/lists/*
 
@@ -171,39 +233,89 @@ ENV PATH="${PATH}:/usr/local/go/bin"
 ENV GO_PATH=/usr/local/go/bin/go
 
 # Install UV
-RUN curl --proto '=https' --tlsv1.2 -LsSf https://github.com/astral-sh/uv/releases/download/0.6.2/uv-installer.sh | sh && mv /root/.local/bin/uv /usr/local/bin/uv
+RUN curl --proto '=https' --tlsv1.2 -LsSf https://github.com/astral-sh/uv/releases/download/0.9.25/uv-installer.sh | sh && mv /root/.local/bin/uv /usr/local/bin/uv
 
-# Preinstall python runtimes
-RUN uv python install 3.11
-RUN uv python install $LATEST_STABLE_PY
-
-RUN uv venv
+# Preinstall python runtimes to temp build location (will copy with world-writable perms later)
+# --compile-bytecode precompiles the stdlib to .pyc so jobs don't recompile it on every run
+# under the read-only nsjail runtime mount (uv >= 0.9.25). The copy below MUST preserve
+# timestamps or Python's mtime-based .pyc invalidation discards these compiled files.
+RUN UV_CACHE_DIR=/tmp/build_cache/uv UV_PYTHON_INSTALL_DIR=/tmp/build_cache/py_runtime uv python install 3.11 --compile-bytecode
+RUN UV_CACHE_DIR=/tmp/build_cache/uv UV_PYTHON_INSTALL_DIR=/tmp/build_cache/py_runtime uv python install $LATEST_STABLE_PY --compile-bytecode
 
 
-RUN curl -sL https://deb.nodesource.com/setup_20.x | bash - 
+RUN curl -sL https://deb.nodesource.com/setup_20.x | bash -
 RUN apt-get -y update && apt-get install -y curl procps nodejs awscli && apt-get clean \
     && rm -rf /var/lib/apt/lists/*
 
 # go build is slower the first time it is ran, so we prewarm it in the build
-RUN mkdir -p /tmp/gobuildwarm && cd /tmp/gobuildwarm && go mod init gobuildwarm && printf "package foo\nimport (\"fmt\")\nfunc main() { fmt.Println(42) }" > warm.go && go mod tidy && go build -x && rm -rf /tmp/gobuildwarm
+# This mirrors Windmill's Go wrapper structure: main.go imports inner package, uses encoding/json, os, fmt
+RUN export GOCACHE=/tmp/build_cache/go && \
+    mkdir -p /tmp/gobuildwarm/inner && \
+    cd /tmp/gobuildwarm && \
+    go mod init mymod && \
+    printf 'package main\nimport (\n\t"encoding/json"\n\t"os"\n\t"fmt"\n\t"mymod/inner"\n)\nfunc main() {\n\tdat, _ := os.ReadFile("args.json")\n\tvar req inner.Req\n\tjson.Unmarshal(dat, &req)\n\tres, _ := inner.Run(req)\n\tres_json, _ := json.Marshal(res)\n\tfmt.Println(string(res_json))\n}' > main.go && \
+    printf 'package inner\ntype Req struct {\n\tX int `json:"x"`\n}\nfunc Run(req Req) (interface{}, error) {\n\treturn main(req.X)\n}\nfunc main(x int) (interface{}, error) {\n\treturn x, nil\n}' > inner/inner.go && \
+    go build -x . && \
+    rm -rf /tmp/gobuildwarm
+
+# Copy build caches to final location, then add write permissions for any UID
+# chmod a+rw adds read+write WITHOUT removing execute bits (755->777, 644->666)
+# Note: uv python install only creates py_runtime, not uv cache - we create uv/go dirs for runtime
+RUN mkdir -p /tmp/windmill/cache && \
+    cp -r --preserve=timestamps /tmp/build_cache/* /tmp/windmill/cache/ && \
+    chmod -R a+rw /tmp/windmill/cache && \
+    rm -rf /tmp/build_cache && \
+    mkdir -p -m 777 /tmp/windmill/cache/uv /tmp/windmill/cache/go /tmp/windmill/cache/rustup /tmp/windmill/cache/cargo
+
+# Runtime cache locations
+ENV UV_CACHE_DIR=/tmp/windmill/cache/uv
+ENV UV_PYTHON_INSTALL_DIR=/tmp/windmill/cache/py_runtime
+ENV GOCACHE=/tmp/windmill/cache/go
 
 ENV TZ=Etc/UTC
 
 COPY --from=builder /frontend/build /static_frontend
 COPY --from=builder /windmill/target/release/windmill ${APP}/windmill
+COPY --from=windmill_duckdb_ffi_internal_builder /windmill-duckdb-ffi-internal/target/release/libwindmill_duckdb_ffi_internal.so ${APP}/libwindmill_duckdb_ffi_internal.so
 
 COPY --from=denoland/deno:2.2.1 --chmod=755 /usr/bin/deno /usr/bin/deno
 
-COPY --from=oven/bun:1.2.18 /usr/local/bin/bun /usr/bin/bun
+COPY --from=oven/bun:1.3.10 /usr/local/bin/bun /usr/bin/bun
 
-COPY --from=php:8.3.7-cli /usr/local/bin/php /usr/bin/php
-COPY --from=composer:2.7.6 /usr/bin/composer /usr/bin/composer
+# Install windmill CLI
+RUN bun install -g windmill-cli \
+    && ln -s $(bun pm bin -g)/wmill /usr/bin/wmill
+
+# Install Claude Code CLI (used by claude sandbox scripts)
+# The installer puts the binary in ~/.local/bin/claude (symlink to ~/.local/share/claude/versions/*)
+# Copy it to /usr/bin/claude so it's accessible inside nsjail sandbox (which mounts /usr but not /root)
+RUN curl -fsSL https://claude.ai/install.sh | bash \
+    && cp /root/.local/share/claude/versions/* /usr/bin/claude
+
+COPY --from=php:8.3.30-cli-bookworm /usr/local/bin/php /usr/bin/php
+COPY --from=composer:2.9.5 /usr/bin/composer /usr/bin/composer
 
 # add the docker client to call docker from a worker if enabled
-COPY --from=docker:dind /usr/local/bin/docker /usr/local/bin/
+COPY --from=docker:29-dind /usr/local/bin/docker /usr/local/bin/
 
-ENV RUSTUP_HOME="/usr/local/rustup"
-ENV CARGO_HOME="/usr/local/cargo"
+ENV RUSTUP_HOME="/tmp/windmill/cache/rustup"
+ENV CARGO_HOME="/tmp/windmill/cache/cargo"
+ENV LD_LIBRARY_PATH="."
+
+# nsjail runtime deps and binary
+RUN apt-get update && apt-get install -y --no-install-recommends libprotobuf32 libnl-route-3-200 libnl-3-200 \
+    && apt-get clean && rm -rf /var/lib/apt/lists/*
+COPY --from=nsjail /nsjail/nsjail /bin/nsjail
+
+# crane: pulls + flattens images for the sandboxed container runtime (`# sandbox <image>`).
+# Single static binary — no daemon/store/root needed. See docs/docker-v2-runtime.md.
+ARG CRANE_VERSION=v0.20.6
+RUN arch="$(dpkg --print-architecture)"; \
+    case "$arch" in amd64) crane_arch=x86_64 ;; arm64) crane_arch=arm64 ;; *) echo >&2 "error: unsupported arch '$arch' for crane"; exit 1 ;; esac; \
+    wget -O /tmp/crane.tgz "https://github.com/google/go-containerregistry/releases/download/${CRANE_VERSION}/go-containerregistry_Linux_${crane_arch}.tar.gz" \
+    && tar -xzf /tmp/crane.tgz -C /usr/local/bin crane \
+    && rm /tmp/crane.tgz \
+    && chmod +x /usr/local/bin/crane
 
 WORKDIR ${APP}
 
@@ -211,23 +323,20 @@ RUN ln -s ${APP}/windmill /usr/local/bin/windmill
 
 COPY ./frontend/src/lib/hubPaths.json ${APP}/hubPaths.json
 
-RUN windmill cache ${APP}/hubPaths.json && rm ${APP}/hubPaths.json && chmod -R 777 /tmp/windmill
+RUN windmill cache ${APP}/hubPaths.json && rm ${APP}/hubPaths.json
+
+RUN windmill cache-rt
 
 # Create a non-root user 'windmill' with UID and GID 1000
 RUN addgroup --gid 1000 windmill && \
     adduser --disabled-password --gecos "" --uid 1000 --gid 1000 windmill
 
-RUN cp -r /root/.cache /home/windmill/.cache
+# /tmp/.cache may be created by earlier build steps with 755; chmod ensures any UID can write
+RUN mkdir -p -m 777 /tmp/windmill/logs /tmp/windmill/search /tmp/.cache && chmod 777 /tmp/.cache
 
-RUN mkdir -p /tmp/windmill/logs && \
-    mkdir -p /tmp/windmill/search
-
-# Make directories world-readable and writable
-RUN chmod -R 777 ${APP} && \
-     chmod -R 777 /tmp/windmill && \
-     chmod -R 777 /home/windmill/.cache
-
-USER root
+# Make directories world-accessible for any UID
+# (cache files already have 666 from umask copy above, cache_nomount is read-only)
+RUN find ${APP} /tmp/windmill -type d -exec chmod 777 {} +
 
 EXPOSE 8000
 

README.md

@@ -3,10 +3,10 @@
 </p>
 
 <p align=center>
-Open-source developer infrastructure for internal tools (APIs, background jobs, workflows and UIs). Self-hostable alternative to Retool, Pipedream, Superblocks and a simplified Temporal with autogenerated UIs and custom UIs to trigger workflows and scripts as internal apps.
+Open-source developer platform for internal code: APIs, background jobs, workflows and UIs. Self-hostable alternative to Retool, Pipedream, Superblocks and a simplified Temporal with autogenerated UIs and custom UIs to trigger workflows and scripts as internal apps.
 
 <p align=center>
-Scripts are turned into sharable UIs automatically, and can be composed together into flows or used into richer apps built with low-code. Supported script languages supported are: Python, TypeScript, Go, Bash, SQL, and GraphQL.
+Scripts are turned into sharable UIs automatically, and can be composed together into flows or used into richer apps built with low-code. Supported languages: Python, TypeScript, Go, Bash, SQL, GraphQL, PowerShell, Rust, and more.
 </p>
 
 <p align="center">
@@ -31,80 +31,63 @@ Scripts are turned into sharable UIs automatically, and can be composed together
 </p>
 
 <p align="center">
-  <a href="https://app.windmill.dev">Try it</a> - <a href="https://www.windmill.dev/docs/intro/">Docs</a> - <a href="https://discord.gg/V7PM2YHsPB">Discord</a> - <a href="https://hub.windmill.dev">Hub</a> - <a href="https://www.windmill.dev/docs/misc/contributing">Contributor's guide</a>
+  <a href="https://app.windmill.dev">Try it</a> - <a href="https://www.windmill.dev/">Website</a> - <a href="https://www.windmill.dev/docs/intro/">Docs</a> - <a href="https://discord.gg/V7PM2YHsPB">Discord</a> - <a href="https://hub.windmill.dev">Hub</a> - <a href="https://www.windmill.dev/docs/misc/contributing">Contributor's guide</a>
 </p>
 
 # Windmill - Developer platform for APIs, background jobs, workflows and UIs
 
-Windmill is <b>fully open-sourced (AGPLv3)</b> and Windmill Labs offers
-dedicated instance and commercial support and licenses.
+Windmill is fully open-sourced (AGPLv3) and Windmill Labs offers dedicated instances and commercial support and licenses.
 
 ![Windmill Diagram](./imgs/stacks.svg)
 
-https://github.com/windmill-labs/windmill/assets/122811744/0b132cd1-ee67-4505-822f-0c7ee7104252
+https://github.com/user-attachments/assets/d80de1d9-64de-4d89-aacd-6df23fa81fc4
 
 - [Windmill - Developer platform for APIs, background jobs, workflows and UIs](#windmill---developer-platform-for-apis-background-jobs-workflows-and-uis)
   - [Main Concepts](#main-concepts)
   - [Show me some actual script code](#show-me-some-actual-script-code)
-  - [CLI](#cli)
-    - [Running scripts locally](#running-scripts-locally)
+  - [Local Development](#local-development)
   - [Stack](#stack)
   - [Fastest Self-Hostable Workflow Engine](#fastest-self-hostable-workflow-engine)
   - [Security](#security)
-    - [Sandboxing](#sandboxing)
-    - [Secrets, credentials and sensitive values](#secrets-credentials-and-sensitive-values)
   - [Performance](#performance)
   - [Architecture](#architecture)
   - [How to self-host](#how-to-self-host)
     - [Docker compose](#docker-compose)
-    - [Kubernetes (k8s) and Helm charts](#kubernetes-k8s-and-helm-charts)
-    - [Run from binaries](#run-from-binaries)
+    - [Kubernetes (Helm charts)](#kubernetes-helm-charts)
+    - [Cloud providers](#cloud-providers)
     - [OAuth, SSO \& SMTP](#oauth-sso--smtp)
-    - [Commercial license](#commercial-license)
+    - [License](#license)
     - [Integrations](#integrations)
   - [Environment Variables](#environment-variables)
   - [Run a local dev setup](#run-a-local-dev-setup)
-    - [only Frontend](#only-frontend)
+    - [Frontend only](#frontend-only)
     - [Backend + Frontend](#backend--frontend)
   - [Contributors](#contributors)
   - [Copyright](#copyright)
 
 ## Main Concepts
 
-1. Define a minimal and generic script in Python, TypeScript, Go or Bash that
-   solves a specific task. The code can be defined in the
-   [provided Web IDE](https://www.windmill.dev/docs/code_editor) or
-   [synchronized with your own GitHub repo](https://www.windmill.dev/docs/advanced/cli/sync)
-   (e.g. through
-   [VS Code](https://www.windmill.dev/docs/cli_local_dev/vscode-extension)
-   extension):
+1. Define a minimal and generic script in Python, TypeScript, Go or Bash that solves a specific task. The code can be defined in the provided Web IDE or synchronized with your own GitHub repo (e.g. through VS Code extension): [provided Web IDE](https://www.windmill.dev/docs/code_editor) or [synchronized with your own GitHub repo](https://www.windmill.dev/docs/advanced/cli/sync) (e.g. through [VS Code](https://www.windmill.dev/docs/cli_local_dev/vscode-extension) extension):
 
-   ![Step 1](./imgs/windmill-editor.png)
+![Step 1](./imgs/windmill-editor.png)
 
-2. Your scripts parameters are automatically parsed and
-   [generate a frontend](https://www.windmill.dev/docs/core_concepts/auto_generated_uis).
+2. Your scripts parameters are automatically parsed and [generate a frontend](https://www.windmill.dev/docs/core_concepts/auto_generated_uis).
 
 ![Step 2](./imgs/windmill-run.png)
 
 ![Step 3](./imgs/windmill-result.png)
 
-3. Make it [flow](https://www.windmill.dev/docs/flows/flow_editor)! You can
-   chain your scripts or scripts made by the community shared on
-   [WindmillHub](https://hub.windmill.dev).
+3. Make it [flow](https://www.windmill.dev/docs/flows/flow_editor)! You can chain your scripts or scripts made by the community shared on [WindmillHub](https://hub.windmill.dev).
 
-   ![Step 3](./imgs/windmill-flow.png)
+![Step 3](./imgs/windmill-flow.png)
 
-4. Build [complex UIs](https://www.windmill.dev/docs/apps/app_editor) on top of
-   your scripts and flows.
+4. Build [complex UIs](https://www.windmill.dev/docs/apps/app_editor) on top of your scripts and flows.
 
-   ![Step 4](./imgs/windmill-builder.png)
+![Step 4](./imgs/windmill-builder.png)
 
-Scripts and flows can also be triggered by a
-[cron schedule](https://www.windmill.dev/docs/core_concepts/scheduling) (e.g.
-'_/5 _ \* \* \*') or through
-[webhooks](https://www.windmill.dev/docs/core_concepts/webhooks).
+Scripts and flows can be triggered by [schedules](https://www.windmill.dev/docs/core_concepts/scheduling), [webhooks](https://www.windmill.dev/docs/core_concepts/webhooks), [HTTP routes](https://www.windmill.dev/docs/core_concepts/http_routing), [Kafka](https://www.windmill.dev/docs/core_concepts/kafka_triggers), [WebSockets](https://www.windmill.dev/docs/core_concepts/websocket_triggers), [emails](https://www.windmill.dev/docs/core_concepts/email_triggers), and more.
 
-You can build your entire infra on top of Windmill!
+Build your entire infra on top of Windmill!
 
 ## Show me some actual script code
 
@@ -144,43 +127,31 @@ export async function main(
 }
 ```
 
-## CLI
+## Local Development
 
-We have a powerful CLI to interact with the windmill platform and sync your
-scripts from local files, GitHub repos and to run scripts and flows on the
-instance from local commands. See
-[more details](https://www.windmill.dev/docs/advanced/cli).
+Windmill supports multiple ways to develop locally and sync with your instance:
 
-![CLI Screencast](./cli/vhs/output/setup.gif)
+| Tool | Description |
+|------|-------------|
+| **[CLI](https://www.windmill.dev/docs/advanced/cli)** | Sync scripts from local files or GitHub, run scripts/flows from the command line |
+| **[VS Code Extension](https://www.windmill.dev/docs/cli_local_dev/vscode-extension)** | Edit and test scripts & flows directly from VS Code / Cursor with full IDE support |
+| **[Git Sync](https://www.windmill.dev/docs/advanced/git_sync)** | Two-way sync between Windmill and your Git repository |
+| **[Claude Code](https://www.windmill.dev/docs/core_concepts/ai_generation)** | AI-assisted development with Claude for scripts, flows, and apps |
 
-### Running scripts locally
+https://github.com/user-attachments/assets/c541c326-e9ae-4602-a09a-1989aaded1e9
 
-You can run your script locally easily, you simply need to pass the right
-environment variables for the `wmill` client library to fetch resources and
-variables from your instance if necessary. See more:
-<https://www.windmill.dev/docs/advanced/local_development>.
-
-To develop & test locally scripts & flows, we recommend using the Windmill VS
-Code extension: <https://www.windmill.dev/docs/cli_local_dev/vscode-extension>.
+You can run scripts locally by passing the right environment variables for the `wmill` client library to fetch resources and variables from your instance. See [local development docs](https://www.windmill.dev/docs/advanced/local_development).
 
 ## Stack
 
-- Postgres as the database.
-- Backend in Rust with the following highly-available and horizontally scalable.
-  Architecture:
-  - Stateless API backend.
-  - Workers that pull jobs from a queue in Postgres (and later, Kafka or Redis.
-    Upvote [#173](#https://github.com/windmill-labs/windmill/issues/173) if
-    interested).
-- Frontend in Svelte.
-- Scripts executions are sandboxed using Google's
-  [nsjail](https://github.com/google/nsjail).
-- Javascript runtime is the
-  [deno_core rust library](https://denolib.gitbook.io/guide/) (which itself uses
-  the [rusty_v8](https://github.com/denoland/rusty_v8) and hence V8 underneath).
-- TypeScript runtime is Bun and deno.
-- Python runtime is python3.
-- Golang runtime is 1.19.1.
+- **Database**: Postgres (compatible with Aurora, Cloud SQL, Neon, Azure PostgreSQL)
+- **Backend**: Rust - stateless API servers and workers pulling jobs from a Postgres queue
+- **Frontend**: Svelte 5
+- **Sandboxing**: [nsjail](https://github.com/google/nsjail) and PID namespace isolation
+- **Runtimes**:
+  - TypeScript/JavaScript: Bun (default) and Deno
+  - Python: python3 with uv for dependency management
+  - Go, Bash, PowerShell, PHP, Rust, C#, Java, Ansible
 
 ## Fastest Self-Hostable Workflow Engine
 
@@ -197,19 +168,10 @@ page.
 
 ## Security
 
-### Sandboxing
+- **Sandboxing**: [nsjail](https://github.com/google/nsjail) for filesystem/resource isolation, and PID namespace isolation (enabled by default) to prevent jobs from accessing worker process memory
+- **Secrets**: One encryption key per workspace for credentials stored in Windmill's K/V store. We recommend encrypting the Postgres database as well.
 
-Windmill can use [nsjail](https://github.com/google/nsjail). It is production
-multi-tenant grade secure. Do not take our word for it, take
-[fly.io's one](https://fly.io/blog/sandboxing-and-workload-isolation/).
-
-### Secrets, credentials and sensitive values
-
-There is one encryption key per workspace to encrypt the credentials and secrets
-stored in Windmill's K/V store.
-
-In addition, we strongly recommend that you encrypt the whole Postgres database.
-That is what we do at <https://app.windmill.dev>.
+See [Security documentation](https://www.windmill.dev/docs/advanced/security_isolation) for details.
 
 ## Performance
 
@@ -229,19 +191,13 @@ back to the database is ~50ms. A typical lightweight deno job will take around
 
 ## How to self-host
 
-We only provide docker-compose setup here. For more advanced setups, like
-compiling from source or using without a postgres super user, see
-[Self-Host documentation](https://www.windmill.dev/docs/advanced/self_host).
+For detailed setup options, see [Self-Host documentation](https://www.windmill.dev/docs/advanced/self_host).
 
 ### Docker compose
 
-Windmill can be deployed using 3 files:
-([docker-compose.yml](./docker-compose.yml), [Caddyfile](./Caddyfile) and a
-[.env](./.env)) in a single command.
+Deploy Windmill with 3 files ([docker-compose.yml](./docker-compose.yml), [Caddyfile](./Caddyfile), [.env](./.env)):
 
-Make sure Docker is started, and run:
-
-```
+```bash
 curl https://raw.githubusercontent.com/windmill-labs/windmill/main/docker-compose.yml -o docker-compose.yml
 curl https://raw.githubusercontent.com/windmill-labs/windmill/main/Caddyfile -o Caddyfile
 curl https://raw.githubusercontent.com/windmill-labs/windmill/main/.env -o .env
@@ -249,180 +205,127 @@ curl https://raw.githubusercontent.com/windmill-labs/windmill/main/.env -o .env
 docker compose up -d
 ```
 
-Go to http://localhost et voilà :)
+Go to http://localhost - default credentials: `admin@windmill.dev` / `changeme`
 
-The default super-admin user is: admin@windmill.dev / changeme.
+**Using an external database**: Set `DATABASE_URL` in `.env` to point to your managed Postgres (AWS RDS, GCP Cloud SQL, Azure, Neon, etc.) and set db replicas to 0.
 
-From there, you can follow the setup app and create other users.
-
-More details in
-[Self-Host Documention](https://www.windmill.dev/docs/advanced/self_host#docker).
-
-### Kubernetes (k8s) and Helm charts
-
-We publish helm charts at:
-<https://github.com/windmill-labs/windmill-helm-charts>.
-
-### Run from binaries
-
-Each release includes the corresponding binaries for x86_64. You can simply
-download the latest `windmill` binary using the following set of bash commands.
+### Kubernetes (Helm charts)
 
 ```bash
-BINARY_NAME='windmill-amd64' # or windmill-ee-amd64 for the enterprise edition
-LATEST_RELEASE=$(curl -L -s -H 'Accept: application/json' https://github.com/windmill-labs/windmill/releases/latest)
-LATEST_VERSION=$(echo $LATEST_RELEASE | sed -e 's/.*"tag_name":"\([^"]*\)".*/\1/')
-ARTIFACT_URL="https://github.com/windmill-labs/windmill/releases/download/$LATEST_VERSION/$BINARY_NAME"
-wget "$ARTIFACT_URL" -O windmill
+helm repo add windmill https://windmill-labs.github.io/windmill-helm-charts/
+helm install windmill-chart windmill/windmill --namespace=windmill --create-namespace
 ```
 
+See [windmill-helm-charts](https://github.com/windmill-labs/windmill-helm-charts) for configuration options.
+
+### Cloud providers
+
+Windmill works on AWS (EKS/ECS), GCP, Azure, Ubicloud, Fly.io, Render.com, Hetzner, Digital Ocean, and others. Rule of thumb: 1 worker per 1vCPU and 1-2 GB RAM.
+
 ### OAuth, SSO & SMTP
 
-Windmill Community Edition allows to configure the OAuth, SSO (including Google
-Workspace SSO, Microsoft/Azure and Okta) directly from the UI in the superadmin
-settings. Do note that there is a limit of 10 SSO users on the community
-edition.
+Configure OAuth and SSO (Google Workspace, Microsoft/Azure, Okta) directly from the superadmin UI. [See documentation](https://www.windmill.dev/docs/misc/setup_oauth).
 
-[See documentation](https://www.windmill.dev/docs/misc/setup_oauth).
+### License
 
-### Commercial license
+The Community Edition is free to use internally. For commercial redistribution or managed services, contact <sales@windmill.dev>. See [LICENSE](./LICENSE) and [Pricing](https://www.windmill.dev/pricing) for details.
 
-See the [LICENSE](https://github.com/windmill-labs/windmill/blob/main/LICENSE)
-file for the full license text.
+The "Community Edition" of Windmill available in the docker images hosted under ghcr.io/windmill-labs/windmill and the github binary releases contains the files under the AGPLv3 and Apache 2 sources but also includes proprietary and non-public code and features which are not open source and under the following terms: Windmill Labs, Inc. grants a right to use all the features of the "Community Edition" for free without restrictions other than the limits and quotas set in the software and a right to distribute the community edition as is but not to sell, resell, serve Windmill as a managed service, modify or wrap under any form without an explicit agreement.
 
-The "Community Edition" of Windmill available in the docker images hosted under
-ghcr.io/windmill-labs/windmill and the github binary releases contains the files
-under the AGPLv3 and Apache 2 sources but also includes proprietary and
-non-public code and features which are not open source and under the following
-terms: Windmill Labs, Inc. grants a right to use all the features of the
-"Community Edition" for free without restrictions other than the limits and
-quotas set in the software and a right to distribute the community edition as is
-but not to sell, resell, serve Windmill as a managed service, modify or wrap
-under any form without an explicit agreement.
+The binary compilable from source code in this repository without the "enterprise" feature flag is open-source under the [LICENSE-AGPLv3](https://github.com/windmill-labs/windmill/blob/main/LICENSE-AGPL) License terms and conditions.
 
-The binary compilable from source code in this repository without the
-"enterprise" feature flag is open-source under the
-[LICENSE-AGPLv3](https://github.com/windmill-labs/windmill/blob/main/LICENSE-AGPL)
-License terms and conditions.
+To [re-expose directly any Windmill parts to your users](https://www.windmill.dev/docs/misc/white_labelling) as a feature of your product, with the exception of iframed public Windmill "apps", or to build a feature on top of "Windmill Community Edition" that you sell commercially or embed in a distributable product or binary, you must get a commercial license. Contact us at <sales@windmill.dev> if you have any questions. To do the same from the binary compiled from the source code in this repository without the "enterprise" feature flag, you must comply with the AGPLv3 license terms and conditions or get a commercial license from Windmill Labs, Inc.
 
-To
-[re-expose directly any Windmill parts to your users](https://www.windmill.dev/docs/misc/white_labelling)
-as a feature of your product, with the exception of iframed public Windmill
-"apps", or to build a feature on top of "Windmill Community Edition" that you
-sell commercially or embed in a distributable product or binary, you must get a
-commercial license. Contact us at <sales@windmill.dev> if you have any
-questions. To do the same from the binary compiled from the source code in this
-repository without the "enterprise" feature flag, you must comply with the
-AGPLv3 license terms and conditions or get a commercial license from Windmill
-Labs, Inc.
-
-To use Windmill "Community Edition" as is internally in your organization, or to
-use its APIs as is, you do NOT need a commercial license.
+To use Windmill "Community Edition" as is internally in your organization, or to use its APIs as is, you do NOT need a commercial license.
 
 ### Integrations
 
-In Windmill, integrations are referred to as
-[resources and resource types](https://www.windmill.dev/docs/core_concepts/resources_and_types).
-Each Resource has a Resource Type that defines the schema that the resource
+In Windmill, integrations are referred to as [resources and resource types](https://www.windmill.dev/docs/core_concepts/resources_and_types). Each Resource has a Resource Type that defines the schema that the resource
 needs to implement.
 
-On self-hosted instances, you might want to import all the approved resource
-types from [WindmillHub](https://hub.windmill.dev). A setup script will prompt
-you to have it being synced automatically everyday.
+On self-hosted instances, you might want to import all the approved resource types from [WindmillHub](https://hub.windmill.dev). A setup script will prompt you to have it being synced automatically everyday.
 
 ## Environment Variables
 
-| Environment Variable name           | Default                | Description                                                                                                                                                                                        | Api Server/Worker/All |
-| ----------------------------------- | ---------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | --------------------- |
-| DATABASE_URL                        |                        | The Postgres database url.                                                                                                                                                                         | All                   |
-| WORKER_GROUP                        | default                | The worker group the worker belongs to and get its configuration pulled from                                                                                                                       | Worker                |
-| MODE                                | standalone             | The mode if the binary. Possible values: standalone, worker, server, agent                                                                                                                         | All                   |
-| METRICS_ADDR                        | None                   | (ee only) The socket addr at which to expose Prometheus metrics at the /metrics path. Set to "true" to expose it on port 8001                                                                      | All                   |
-| JSON_FMT                            | false                  | Output the logs in json format instead of logfmt                                                                                                                                                   | All                   |
-| BASE_URL                            | http://localhost:8000  | The base url that is exposed publicly to access your instance. Is overriden by the instance settings if any.                                                                                       | Server                |
-| ZOMBIE_JOB_TIMEOUT                  | 30                     | The timeout after which a job is considered to be zombie if the worker did not send pings about processing the job (every server check for zombie jobs every 30s)                                  | Server                |
-| RESTART_ZOMBIE_JOBS                 | true                   | If true then a zombie job is restarted (in-place with the same uuid and some logs), if false the zombie job is failed                                                                              | Server                |
-| SLEEP_QUEUE                         | 50                     | The number of ms to sleep in between the last check for new jobs in the DB. It is multiplied by NUM_WORKERS such that in average, for one worker instance, there is one pull every SLEEP_QUEUE ms. | Worker                |
-| KEEP_JOB_DIR                        | false                  | Keep the job directory after the job is done. Useful for debugging.                                                                                                                                | Worker                |
-| LICENSE_KEY (EE only)               | None                   | License key checked at startup for the Enterprise Edition of Windmill                                                                                                                              | Worker                |
-| SLACK_SIGNING_SECRET                | None                   | The signing secret of your Slack app. See [Slack documentation](https://api.slack.com/authentication/verifying-requests-from-slack)                                                                | Server                |
-| COOKIE_DOMAIN                       | None                   | The domain of the cookie. If not set, the cookie will be set by the browser based on the full origin                                                                                               | Server                |
-| DENO_PATH                           | /usr/bin/deno          | The path to the deno binary.                                                                                                                                                                       | Worker                |
-| PYTHON_PATH                         |                        | The path to the python binary if wanting to not have it managed by uv.                                                                                                                             | Worker                |
-| GO_PATH                             | /usr/bin/go            | The path to the go binary.                                                                                                                                                                         | Worker                |
-| GOPRIVATE                           |                        | The GOPRIVATE env variable to use private go modules                                                                                                                                               | Worker                |
-| GOPROXY                             |                        | The GOPROXY env variable to use                                                                                                                                                                    | Worker                |
-| NETRC                               |                        | The netrc content to use a private go registry                                                                                                                                                     | Worker                |
-| PY_CONCURRENT_DOWNLOADS             | 20                     | Sets the maximum number of in-flight concurrent python downloads that windmill will perform at any given time.                                                                                     | Worker                |
-| PATH                                | None                   | The path environment variable, usually inherited                                                                                                                                                   | Worker                |
-| HOME                                | None                   | The home directory to use for Go and Bash , usually inherited                                                                                                                                      | Worker                |
-| DATABASE_CONNECTIONS                | 50 (Server)/3 (Worker) | The max number of connections in the database connection pool                                                                                                                                      | All                   |
-| SUPERADMIN_SECRET                   | None                   | A token that would let the caller act as a virtual superadmin superadmin@windmill.dev                                                                                                              | Server                |
-| TIMEOUT_WAIT_RESULT                 | 20                     | The number of seconds to wait before timeout on the 'run_wait_result' endpoint                                                                                                                     | Worker                |
-| QUEUE_LIMIT_WAIT_RESULT             | None                   | The number of max jobs in the queue before rejecting immediately the request in 'run_wait_result' endpoint. Takes precedence on the query arg. If none is specified, there are no limit.           | Worker                |
-| DENO_AUTH_TOKENS                    | None                   | Custom DENO_AUTH_TOKENS to pass to worker to allow the use of private modules                                                                                                                      | Worker                |
-| DISABLE_RESPONSE_LOGS               | false                  | Disable response logs                                                                                                                                                                              | Server                |
-| CREATE_WORKSPACE_REQUIRE_SUPERADMIN | true                   | If true, only superadmins can create new workspaces                                                                                                                                                | Server                |
-| MIN_FREE_DISK_SPACE_MB              | 15000                  | Minimum amount of free space on worker. Sends critical alert if worker has less free space.                                                                                                        | Worker                |
-| RUN_UPDATE_CA_CERTIFICATE_AT_START  | false                  | If true, runs CA certificate update command at startup before other initialization                                                                                                                  | All                   |
-| RUN_UPDATE_CA_CERTIFICATE_PATH      | /usr/sbin/update-ca-certificates | Path to the CA certificate update command/script to run when RUN_UPDATE_CA_CERTIFICATE_AT_START is true                                                                                             | All                   |
+| Environment Variable name           | Default                          | Description                                                                                                                                                                                        | Api Server/Worker/All |
+| ----------------------------------- | -------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | --------------------- |
+| DATABASE_URL                        |                                  | The Postgres database url.                                                                                                                                                                         | All                   |
+| WORKER_GROUP                        | default                          | The worker group the worker belongs to and get its configuration pulled from                                                                                                                       | Worker                |
+| MODE                                | standalone                       | The mode if the binary. Possible values: standalone, worker, server, agent                                                                                                                         | All                   |
+| METRICS_ADDR                        | None                             | (ee only) The socket addr at which to expose Prometheus metrics at the /metrics path. Set to "true" to expose it on port 8001                                                                      | All                   |
+| JSON_FMT                            | false                            | Output the logs in json format instead of logfmt                                                                                                                                                   | All                   |
+| BASE_URL                            | http://localhost:8000            | The base url that is exposed publicly to access your instance. Is overriden by the instance settings if any.                                                                                       | Server                |
+| ZOMBIE_JOB_TIMEOUT                  | 30                               | The timeout after which a job is considered to be zombie if the worker did not send pings about processing the job (every server check for zombie jobs every 30s)                                  | Server                |
+| RESTART_ZOMBIE_JOBS                 | true                             | If true then a zombie job is restarted (in-place with the same uuid and some logs), if false the zombie job is failed                                                                              | Server                |
+| NATIVE_MODE                         | false                            | Enable native mode: sets NUM_WORKERS=8, rejects non-native jobs (nativets, postgresql, mysql, etc.)                                                                                                | Worker                |
+| SLEEP_QUEUE                         | 50                               | The number of ms to sleep in between the last check for new jobs in the DB. It is multiplied by NUM_WORKERS such that in average, for one worker instance, there is one pull every SLEEP_QUEUE ms. | Worker                |
+| KEEP_JOB_DIR                        | false                            | Keep the job directory after the job is done. Useful for debugging.                                                                                                                                | Worker                |
+| LICENSE_KEY (EE only)               | None                             | License key checked at startup for the Enterprise Edition of Windmill                                                                                                                              | Worker                |
+| SLACK_SIGNING_SECRET                | None                             | The signing secret of your Slack app. See [Slack documentation](https://api.slack.com/authentication/verifying-requests-from-slack)                                                                | Server                |
+| COOKIE_DOMAIN                       | None                             | The domain of the cookie. If not set, the cookie will be set by the browser based on the full origin                                                                                               | Server                |
+| DENO_PATH                           | /usr/bin/deno                    | The path to the deno binary.                                                                                                                                                                       | Worker                |
+| PYTHON_PATH                         |                                  | The path to the python binary if wanting to not have it managed by uv.                                                                                                                             | Worker                |
+| GO_PATH                             | /usr/bin/go                      | The path to the go binary.                                                                                                                                                                         | Worker                |
+| GOPRIVATE                           |                                  | The GOPRIVATE env variable to use private go modules                                                                                                                                               | Worker                |
+| GOPROXY                             |                                  | The GOPROXY env variable to use                                                                                                                                                                    | Worker                |
+| NETRC                               |                                  | The netrc content to use a private go registry                                                                                                                                                     | Worker                |
+| PY_CONCURRENT_DOWNLOADS             | 20                               | Sets the maximum number of in-flight concurrent python downloads that windmill will perform at any given time.                                                                                     | Worker                |
+| PATH                                | None                             | The path environment variable, usually inherited                                                                                                                                                   | Worker                |
+| HOME                                | None                             | The home directory to use for Go and Bash , usually inherited                                                                                                                                      | Worker                |
+| DATABASE_CONNECTIONS                | 50 (Server)/3 (Worker)           | The max number of connections in the database connection pool                                                                                                                                      | All                   |
+| SUPERADMIN_SECRET                   | None                             | A token that would let the caller act as a virtual superadmin superadmin@windmill.dev                                                                                                              | Server                |
+| TIMEOUT_WAIT_RESULT                 | 20                               | The number of seconds to wait before timeout on the 'run_wait_result' endpoint                                                                                                                     | Worker                |
+| QUEUE_LIMIT_WAIT_RESULT             | None                             | The number of max jobs in the queue before rejecting immediately the request in 'run_wait_result' endpoint. Takes precedence on the query arg. If none is specified, there are no limit.           | Worker                |
+| DENO_AUTH_TOKENS                    | None                             | Custom DENO_AUTH_TOKENS to pass to worker to allow the use of private modules                                                                                                                      | Worker                |
+| DISABLE_RESPONSE_LOGS               | false                            | Disable response logs                                                                                                                                                                              | Server                |
+| CREATE_WORKSPACE_REQUIRE_SUPERADMIN | true                             | If true, only superadmins can create new workspaces                                                                                                                                                | Server                |
+| MIN_FREE_DISK_SPACE_MB              | 15000                            | Minimum amount of free space on worker. Sends critical alert if worker has less free space.                                                                                                        | Worker                |
+| RUN_UPDATE_CA_CERTIFICATE_AT_START  | false                            | If true, runs CA certificate update command at startup before other initialization                                                                                                                 | All                   |
+| RUN_UPDATE_CA_CERTIFICATE_PATH      | /usr/sbin/update-ca-certificates | Path to the CA certificate update command/script to run when RUN_UPDATE_CA_CERTIFICATE_AT_START is true                                                                                            | All                   |
 
 ## Run a local dev setup
 
-Using [Nix](./frontend/README_DEV.md#nix) (Recommended).
+We recommend using [Nix](./frontend/README_DEV.md#nix). See [./frontend/README_DEV.md](./frontend/README_DEV.md) for all options.
 
-See the [./frontend/README_DEV.md](./frontend/README_DEV.md) file for all
-running options.
+### Frontend only
 
+Uses the backend of <https://app.windmill.dev> with local frontend (hot-reload):
 
-### only Frontend
-
-This will use the backend of <https://app.windmill.dev> but your own frontend
-with hot-code reloading. Note that you will need to use a username / password
-login due to CSRF checks using a different auth provider.
-
-In the `frontend/` directory:
-
-1. install the dependencies with `npm install` (or `pnpm install` or `yarn`)
-2. generate the windmill client:
-
-```
-npm run generate-backend-client
-## on mac use
-npm run generate-backend-client-mac
+```bash
+cd frontend
+npm install
+npm run generate-backend-client  # or generate-backend-client-mac on Mac
+npm run dev
 ```
 
-3. Run your dev server with `npm run dev`
-4. Et voilà, windmill should be available at `http://localhost/`
+Windmill available at `http://localhost/`
 
 ### Backend + Frontend
 
 See the [./frontend/README_DEV.md](./frontend/README_DEV.md) file for all
 running options.
 
-1. Create a Postgres Database for Windmill and create an admin role inside your
-   Postgres setup. The easiest way to get a working db is to run
+1. Start a local Postgres database using for instance the `start-dev-db.sh` script which will make a database available at `postgres://postgres:changeme@localhost:5432/windmill`
+   Then run the migrations using the following command:
    ```
    cargo install sqlx-cli
    env DATABASE_URL=<YOUR_DATABASE_URL> sqlx migrate run
    ```
-   This will also avoid compile time issue with sqlx's `query!` macro
-2. Install [nsjail](https://github.com/google/nsjail) and have it accessible in
+   This will also avoid compile time issue with sqlx's `query!` macro.
+2. (optional, linux only) Install [nsjail](https://github.com/google/nsjail) and have it accessible in
    your PATH
-3. Install deno and python3, have the bins at `/usr/bin/deno` and
-   `/usr/local/bin/python3`
-4. Install [caddy](https://caddyserver.com)
-5. Install the [lld linker](https://lld.llvm.org/)
-6. Go to `frontend/`:
-   1. `npm install`, `npm run generate-backend-client` then `npm run dev`
+3. Install bun, deno and python3 (+ any languages you want to use), have the bins at `/usr/bin/bun`,`/usr/bin/deno`, and
+   `/usr/local/bin/python3` or set the corresponding environment variables.
+4. (optional) Install the [lld linker](https://lld.llvm.org/)
+5. Go to `frontend/`:
+   1. `npm install`, `npm run generate-backend-client` then `REMOTE=http://localhost:8000 npm run dev`
    2. You might need to set some extra heap space for the node runtime
       `export NODE_OPTIONS="--max-old-space-size=4096"`
-   3. In another shell `npm run build` otherwise the backend will not find the
-      `frontend/build` folder and will not compile.
-   4. In another shell `sudo caddy run --config Caddyfile`
-7. Go to `backend/`:
-   `env DATABASE_URL=<DATABASE_URL_TO_YOUR_WINDMILL_DB> RUST_LOG=info cargo run`
-8. Et voilà, windmill should be available at `http://localhost/`
+   3. Create an empty `frontend/build` folder using `mkdir frontend/build`
+6. Go to `backend/`:
+   1. `env DATABASE_URL=<YOUR_DATABASE_URL> RUST_LOG=info cargo run`
+   2. You can specify any feature flag you want to enable, for example `cargo run --features python` to enable the python executor.
+7. Windmill should be available at `http://localhost:3000`
 
 ## Contributors
 
@@ -432,4 +335,4 @@ running options.
 
 ## Copyright
 
-Windmill Labs, Inc 2023
+© 2023-2026 Windmill Labs, Inc.

README_WORKMUX_DEV.md

@@ -0,0 +1,211 @@
+# Windmill Development with workmux
+
+This guide covers the workmux-based development setup for Windmill. Each worktree gets its own tmux window with a Claude Code agent, a backend server (with auto-reload), and a frontend dev server — all on isolated ports.
+
+## Prerequisites
+
+- tmux
+- Rust toolchain (rustup)
+- Node.js + npm
+- PostgreSQL running locally (see `backend/.env`)
+
+## Installation
+
+### 1. Install workmux
+
+```bash
+cargo install workmux
+```
+
+### 2. Install the Claude Code plugin
+
+```bash
+workmux claude install
+```
+
+This lets workmux manage Claude Code agents in worktree panes.
+
+### 3. Install cargo-watch
+
+Used for auto-recompiling the backend on file changes:
+
+```bash
+cargo install cargo-watch
+```
+
+### 4. Install llm CLI (required for auto branch naming)
+
+workmux uses the `llm` CLI to automatically generate branch names from prompts. Install it with:
+
+```bash
+uv tool install llm
+llm install llm-anthropic
+```
+
+Then set your Anthropic API key:
+
+```bash
+llm keys set anthropic
+# paste your API key when prompted
+```
+
+### 5. Recommended: shell alias and autocomplete
+
+Set up a `wm` alias for convenience:
+
+```bash
+# Add to your ~/.zshrc
+alias wm="workmux"
+```
+
+Setting up zsh autocomplete is also recommended — see the [workmux docs](https://github.com/rubenfiszel/workmux) for instructions.
+
+## Port Slot System
+
+Each worktree is assigned a **slot** that determines its ports:
+
+| Slot | Backend | Frontend |
+| ---- | ------- | -------- |
+| 0    | 8000    | 3000     |
+| 1    | 8010    | 3010     |
+| 2    | 8020    | 3020     |
+| 3    | 8030    | 3030     |
+| ...  | ...     | ...      |
+
+- **Slot 0** is reserved for the main worktree (default `cargo run` / `npm run dev`).
+- Without `WM_SLOT`, the script auto-assigns the first available slot (starting from 1) and prints it.
+- With `WM_SLOT=N`, it uses that slot and errors if the ports are taken.
+
+## SSH Port Forwarding
+
+If you develop over SSH, add this to `~/.ssh/config` on your **local machine** to pre-configure tunnels for each slot:
+
+```
+Host windmill-dev
+  HostName <remote-ip>
+  User <username>
+  # Slot 0 (main worktree)
+  LocalForward 8000 localhost:8000
+  LocalForward 3000 localhost:3000
+  # Slot 1
+  LocalForward 8010 localhost:8010
+  LocalForward 3010 localhost:3010
+  # Slot 2
+  LocalForward 8020 localhost:8020
+  LocalForward 3020 localhost:3020
+  # Slot 3
+  LocalForward 8030 localhost:8030
+  LocalForward 3030 localhost:3030
+```
+
+Then connect once and all tunnels are active:
+
+```bash
+ssh windmill-dev
+```
+
+Access the frontend at `http://localhost:<frontend-port>` in your local browser.
+
+## Quickstart
+
+```bash
+# Create a new worktree (auto-assigns slot, prints ports)
+workmux add my-feature
+
+# Or with an explicit slot
+WM_SLOT=2 workmux add my-feature
+
+# Create a worktree and immediately send a prompt to the agent
+workmux add -A -p "fix the login bug in auth.rs"
+```
+
+The `add` command creates the worktree but does **not** open it. To open the tmux window and start working:
+
+```bash
+workmux open my-feature
+```
+
+This will open a tmux window with three panes:
+
+- **Claude Code agent** (focused)
+- **Backend**: `cargo watch -x run` on the assigned port (auto-reloads on save)
+- **Frontend**: `npm run dev` proxying to the backend
+
+When using `-A` with `add`, the worktree is created and opened automatically, and the prompt is sent to the agent right away.
+
+Check which ports were assigned:
+
+```bash
+cat <worktree-path>/.env.local
+```
+
+### Sending work to the agent
+
+```bash
+# Send a prompt to the agent in a worktree
+workmux send my-feature "fix the login bug in auth.rs"
+
+# Check agent status
+workmux status
+```
+
+### Merging and cleaning up
+
+We never merge worktrees directly — always create a PR on GitHub and let it be merged there. Once the PR is merged, clean up the worktree:
+
+```bash
+# Close the tmux window but keep the worktree
+workmux close my-feature
+
+# After your PR is merged, remove the worktree, branch, and tmux window
+workmux rm my-feature
+```
+
+> **Note**: Do not use `workmux merge`. Always go through a PR to get your changes into main. You can ask the Claude Code agent in the worktree to create the PR for you.
+
+## Configuration
+
+The setup is defined in `.workmux.yaml` at the repo root. Key sections:
+
+- **`post_create`**: Runs `scripts/worktree-env` to generate `.env.local` with port assignments
+- **`panes`**: Defines the tmux layout (agent, backend, frontend)
+- **`files.copy`**: Copies `backend/.env` and `scripts/` into each worktree
+
+The `post_create` hook also copies `frontend/node_modules` using `cp -a` (preserves `.bin/` symlinks that `cp -r` would dereference).
+
+## Enterprise (EE) Code Access
+
+The enterprise source code lives in the `windmill-ee-private` repository (sibling to this repo). When you create a worktree, `scripts/worktree-env` automatically creates a matching EE worktree on the same branch and configures Claude Code's `additionalDirectories` to grant access.
+
+### Sandbox setup
+
+When using sandbox mode, the container needs explicit mounts to access the EE repo. Add the following to your global workmux config (`~/.config/workmux/config.yaml`):
+
+```yaml
+sandbox:
+  extra_mounts:
+    - host_path: ~/windmill-ee-private
+      writable: true
+    - host_path: ~/windmill-ee-private__worktrees
+      writable: true
+```
+
+This mounts both the main EE repo (used by the main worktree) and the EE worktrees directory (used by feature worktrees) into every sandbox container.
+
+## Cargo Features
+
+To build the backend with specific Cargo features (e.g., `enterprise`, `parquet`), pass them via `CARGO_FEATURES`. The backend pane reads this from `.env.local` and appends `--features <value>` to the `cargo watch` command.
+
+**With `wm` (workmux):**
+
+Set `CARGO_FEATURES` as an environment variable before creating the worktree:
+
+```bash
+CARGO_FEATURES="enterprise,parquet" wm add my-feature
+```
+
+This gets written to `.env.local` by the `post_create` hook (`scripts/worktree-env`), and the backend pane picks it up automatically.
+
+## Login
+
+Default credentials: `admin@windmill.dev` / `changeme`

REVIEW.md

@@ -0,0 +1,69 @@
+# Pull request review — shared policy
+
+You are reviewing a GitHub pull request for this repository. Apply this policy alongside your tool's output requirements.
+
+## Read the project rules first
+
+- Read `AGENTS.md` (repo root) and any `AGENTS.md` in directories touched by the diff before reviewing — they are the canonical contributor guide.
+- Quote the exact rule from `AGENTS.md` when flagging a violation.
+
+## Verdict (first line of the review)
+
+Start every review with a single verdict line, before any other section (the only thing that may appear above the verdict is the optional `cc @<PR_AUTHOR>` ping described in "Pinging the author" below). Pick exactly one:
+
+- **Good to merge** — no blocking issues and no nits worth surfacing.
+- **Mergeable, but should ideally address nits: <short list>** — no blockers, but P2 findings that are worth a look. The list must name each nit briefly (e.g. "doc/code mismatch in `foo.rs`, half-finished `pub fn bar`").
+- **Should address issues before merging: <short list>** — at least one P0 or P1 finding. The list must name each blocking issue briefly (e.g. "missing auth check on new `/api/x` handler, SQL injection in `build_query`").
+
+The names in the list must match findings detailed later in the review. If you list a nit or issue here, it must appear with full context in the body. Do not invent items that aren't in the body, and do not bury blockers in the body without surfacing them in the verdict.
+
+## Pinging the author
+
+If the prompt context provides a `PR AUTHOR` (GitHub login) and the verdict is NOT "Good to merge" (i.e. it is "Mergeable, but should ideally address nits: ..." or "Should address issues before merging: ..."), prepend a single line `cc @<PR_AUTHOR>` to the top-level review comment, above the verdict line. This pings the author so they get a notification that there are items to address. Skip the ping entirely when the verdict is "Good to merge" — there is nothing for the author to act on. Do not add the ping to inline comments; the top-level summary comment is the only place it belongs.
+
+## Review policy
+
+- Only report issues you are confident are real and introduced by this pull request.
+- Focus on bugs, security problems, performance, and clear `AGENTS.md` violations.
+- Do not report style nits, speculative concerns, pre-existing issues, or anything a normal linter / typechecker would obviously catch.
+- Self-validate each finding before posting: "is this definitely a real issue?" If uncertain, discard it.
+- Read additional files only when the diff is not enough to validate a finding.
+- Do not modify any files.
+
+## Severity triage
+
+Tag each finding with a severity. Always report P0 and P1. Report P2 only when the diff invites it (a new `pub fn`, a new module, a new exported component, a meaningful refactor).
+
+- **P0** — RCE, auth bypass, data loss, secrets in code, SQL injection, path traversal, broken auth on a public surface.
+- **P1** — significant bug, missing auth/authorization check on a new public surface, blocking I/O on a likely async path, race condition, missing input validation on caller-controlled parameters, observable performance regression.
+- **P2** — wrong module placement, doc/code mismatch, half-finished public abstractions (`pub fn` + `#[allow(dead_code)]` + `TODO`), `AGENTS.md` style violations, naming that contradicts the function's behavior.
+
+## Checklist for new public surfaces
+
+For any new `pub fn` / `pub async fn` / exported Svelte component / exported prop introduced by this PR, verify:
+
+- (a) auth/authorization expectations are documented in the doc comment OR enforced in the function body. A new `pub fn` that touches workspace data, secrets, files, or processes without an auth check or documented "caller MUST verify" contract is a P1.
+- (b) the function is placed in a module whose stated purpose matches what it does. Check the module-level doc comment (`//!`) — a config-file reader inside `external_ip.rs` is a P2.
+- (c) it is not half-finished. `pub fn` + `#[allow(dead_code)]` + a `TODO` is a smell that says the function should land together with its caller, not separately. Cite the relevant `AGENTS.md` rule.
+- (d) input validation defends against injection / traversal / overflow / NUL bytes at every parameter that may be caller-controlled.
+
+## Test coverage assessment
+
+End your review with a short "Test coverage" section calibrated to the layers actually changed by the diff. Skip categories the PR does not touch.
+
+- **Backend** (Rust under `backend/`) — expect Rust unit tests for new logic. For new or modified API handlers, worker steps, queue/cron behavior, or DB access, also expect or note the absence of integration tests. Pure-refactor backend PRs don't need new tests if existing tests cover the surface.
+- **Frontend** (Svelte / TS under `frontend/`) — the codebase does not generally test Svelte components, so do not ask for component tests. Only flag missing tests for new pure-logic utilities (the kind of file that already has a sibling `*.test.ts`, e.g. `flowDiff`, `previousResults`, copilot logic).
+- **CI / workflows / docs / config-only** — no automated tests expected; say so explicitly so the reader knows you considered it.
+
+Then state what manual verification, if any, is still needed before merge:
+
+- Describe each manual scenario as a short paragraph (not a numbered list): what page / action / input, and what observable outcome confirms correctness.
+- If the diff has no in-app surface to exercise (purely backend internals, CI, docs, or refactor), say that plainly.
+
+## Additional reviewer instructions
+
+If the prompt or context includes an "Additional reviewer instructions" section, treat it as extra guidance from the human who triggered this review and follow it.
+
+## Prior PR discussion
+
+If the prompt or context includes a "Prior PR discussion" section, this PR has already received review activity. Look for your own previous comment, take it into account, focus on what changed in the latest commits, and do not repeat findings the human already pushed back on or addressed.

ai_evals/.gitignore

@@ -0,0 +1,2 @@
+.env
+results/

ai_evals/AGENTS.md

@@ -0,0 +1,208 @@
+# AI Evals Authoring Guide
+
+This folder contains black-box benchmark cases for:
+
+- `flow`
+- `app`
+- `script`
+- `cli`
+- `global`
+
+The goal is to test the current production prompts and guidance with realistic user requests, not to test one exact implementation shape.
+
+## Core rules
+
+1. Write prompts like a real user request.
+2. Prefer behavior, inputs, constraints, and outcomes over internal implementation details.
+3. Keep deterministic validation narrow and hard.
+4. Put semantic expectations in `judgeChecklist`.
+5. Use `expected` fixtures only when exact structure really matters.
+
+## Prompt writing
+
+Prompts should sound like something a user would naturally ask.
+
+Good:
+
+- "Create a flow that routes support requests based on customer tier."
+- "Add a reset button that sets the counter back to 0."
+- "Create a flow that reuses the existing greeting script instead of duplicating the logic."
+
+Bad:
+
+- "Use `branchone` with 3 branches and a default branch."
+- "Create a `rawscript` step with this exact topology."
+- "This is a benchmark harness."
+
+Do not write prompts as if the user knows Windmill internals unless the case is explicitly testing a power-user workflow.
+
+## Flow-specific rules
+
+This is the main principle you asked for:
+
+- flow prompts should read like requests from a user who does not know the product internals
+- the user should ask for behavior, not for `branchone`, `branchall`, `rawscript`, `preprocessor_module`, `failure_module`, exact graph topology, or other internal constructs
+
+That means:
+
+- creation cases should describe the business behavior and expected result
+- modification cases may mention existing step names, because the user can see the current flow
+- only mention special Windmill constructs when the case is explicitly about those constructs
+
+Examples:
+
+- acceptable creation prompt:
+  "Create a purchase approval flow that pauses for approval and asks the approver for a comment."
+- avoid:
+  "Create a suspend step with one required event and a resume form."
+
+For flow cases, do not fail a case just because the model chose a different valid topology.
+
+## App-specific rules
+
+App prompts should focus on user-visible behavior:
+
+- what the UI should let the user do
+- what should persist
+- what backend behavior is needed
+
+Avoid prompting in terms of React structure, component names, or implementation unless the case is specifically about editing an existing app.
+
+## CLI-specific rules
+
+CLI prompts can be more explicit about paths and file names because real CLI users often do specify them.
+
+Still, avoid benchmark phrasing. The prompt should read like a repo task, not a harness instruction.
+
+When relevant, ask the assistant to tell the user which `wmill` commands to run next. That is part of the benchmarked behavior.
+
+## Global-specific rules
+
+Global prompts should exercise workspace-level drafting behavior:
+
+- inspecting existing scripts, flows, apps, schedules, triggers, resources, and variables when relevant
+- writing AI drafts rather than saving or deploying by default
+- producing coherent multi-artifact changes when the request crosses artifact boundaries
+
+Keep deterministic validation focused on the draft contract: required draft type/path, required content snippets, forbidden draft paths, and forbidden mutating tools such as deploy/delete unless the case explicitly asks for them.
+
+Datatable cases should set `skipJudge: true` and validate through tool-use
+(`requiredToolsUsed` / `forbiddenToolsUsed`) and SQL-argument assertions
+(`toolCallArgs` with `stringIncludesAnyOf`, e.g. `['select']`, `['create table']`,
+`['update', 'insert into']`). Two reasons the judge is unreliable here:
+
+- `list_datatables`, `get_datatable_table_schema`, and `exec_datatable_sql`
+  produce no drafts, and the global judge only sees the drafts artifact — it
+  scores a no-draft conversational answer as empty (same as the
+  `askUserQuestion` cases).
+- Even a case that *does* produce a draft (a script reading the data table via
+  `wmill.datatable()` at runtime) is mis-judged: the judge has no datatable SDK
+  reference and penalizes correct `wmill.datatable()` usage as wrong. Verify the
+  SDK call deterministically instead — `requiredDrafts.valueIncludes: ['wmill.datatable(']`
+  plus forbidding `exec_datatable_sql` (keeping chat-time SQL distinct from
+  runtime SDK use).
+
+`stringIncludesAnyOf` is existential over calls (at least one matching call), so a
+mutation case still passes when the model mixes its UPDATE/INSERT with
+verification SELECTs. The in-memory engine (`datatableSqlEngine.ts`) is stateful
+within a case — writes persist, so a model that re-queries to verify its
+CREATE/UPDATE sees the change and does not loop. But the engine is best-effort
+(SELECT returns all rows of the referenced/first table with no WHERE/projection),
+so still never assert specific returned row values. Seed data via
+`workspace.datatables` in the `initial` fixture (see README).
+
+## Deterministic validation
+
+Use deterministic validation only for hard failures such as:
+
+- missing required files
+- unexpected extra files when the prompt says not to create them
+- syntax errors
+- unresolved flow refs
+- missing required special modules or suspend config
+- obvious artifact corruption
+
+Do not use deterministic validation to enforce one preferred implementation for broad creation tasks.
+
+Examples of bad hard checks:
+
+- exact step topology for a creation flow
+- exact branch structure when the prompt only asked for routing behavior
+- exact input shape when multiple reasonable shapes are acceptable
+
+## Judge checklist
+
+Every non-trivial case should have a `judgeChecklist`.
+
+The checklist should capture:
+
+- the user-visible behavior that must be present
+- important constraints
+- key completion criteria
+
+The checklist should not duplicate low-level implementation details unless they are truly required by the task.
+
+Good checklist items:
+
+- "the flow calculates the order total with 8% tax"
+- "the app persists recipes appropriately for a raw Windmill app"
+- "the flow reuses the existing workspace script instead of rewriting the logic"
+
+Bad checklist items:
+
+- "uses `branchone`"
+- "contains a `rawscript` node"
+
+## When to use `expected`
+
+Use `expected` fixtures when the case is structure-sensitive, for example:
+
+- exact file creation
+- exact script content
+- modification cases where a specific file must change in a specific way
+- cases where preserving an existing structure is part of the requirement
+
+Do not use a full `expected` artifact as the semantic oracle for broad creation tasks when multiple valid outputs should pass.
+
+## When to use `initial`
+
+Use `initial` when the benchmark is about:
+
+- editing an existing artifact
+- reusing existing workspace assets
+- preserving existing behavior while adding a change
+
+If the case is greenfield, prefer no `initial`.
+
+## Case design ladder
+
+Prefer suites that get gradually harder:
+
+1. trivial create case
+2. realistic create case
+3. reuse-existing-assets case
+4. modification case
+5. refactor case
+6. edge-case or niche product behavior
+
+The last cases in a suite should cover unusual or product-specific behavior.
+
+## Anti-patterns
+
+Avoid these:
+
+- benchmark framing in prompts
+- over-specified internal topology for creation tasks
+- judge checklists that just restate implementation details
+- deterministic validation that encodes one preferred solution
+- fixtures that are so minimal or brittle that they create false negatives
+
+## Before adding a case
+
+Ask:
+
+1. Would a real user plausibly write this prompt?
+2. If the model solves it in a different valid way, would the case still pass?
+3. Are the hard deterministic checks only catching objectively broken output?
+4. Does the `judgeChecklist` describe the real success criteria?
+5. If this case fails, will the reason be understandable from the saved artifacts?

ai_evals/CLAUDE.md

@@ -0,0 +1 @@
+@AGENTS.md

ai_evals/README.md

@@ -0,0 +1,260 @@
+# AI Evals
+
+Small benchmark runner for the Windmill AI generation modes:
+
+- `cli`
+- `flow`
+- `script`
+- `app`
+- `global`
+
+The benchmark always tests the current production prompts, tools, and guidance in this checkout.
+
+Each attempt runs:
+
+1. the real production path
+2. deterministic validation
+3. LLM judging
+
+## Install
+
+```bash
+cd ai_evals
+bun install
+```
+
+Frontend modes also require frontend dependencies:
+
+```bash
+cd frontend
+bun install
+```
+
+## Commands
+
+List model aliases:
+
+```bash
+cd ai_evals
+bun run cli -- models
+```
+
+List cases:
+
+```bash
+cd ai_evals
+bun run cli -- cases
+bun run cli -- cases flow
+```
+
+Run benchmarks:
+
+```bash
+cd ai_evals
+bun run cli -- run flow
+bun run cli -- run flow flow-test4-order-processing-loop --model opus
+bun run cli -- run flow flow-test0-sum-two-numbers --models haiku,opus,4o
+bun run cli -- run flow flow-test0-sum-two-numbers --runs 3 --verbose
+bun run cli -- run flow --record
+GEMINI_API_KEY=... bun run cli -- run app app-test1-counter-create --model gemini-3-flash-preview
+WMILL_AI_EVAL_BACKEND_URL=http://127.0.0.1:8000 bun run cli -- run flow --backend-validation preview
+bun run cli -- run global global-test1-script-create
+bun run cli -- run cli bun-hello-script
+```
+
+Public CLI surface:
+
+- `models`
+- `cases [mode]`
+- `run <mode> [caseIds...]`
+
+`run` options:
+
+- `--runs <n>`: repeat each case `n` times
+- `--output <path>`: custom result JSON path
+- `--model <alias>`: choose the model under test
+- `--models <a,b,c>`: run the same cases sequentially against several model aliases
+- `--verbose`: stream assistant output for frontend runs
+- `--skip-judge`: skip LLM judge scoring for the run
+- `--execution-only`: only require the model/proxy/frontend loop to complete; skip validators, tool expectations, backend artifact validation, and judge scoring
+- `--record`: append a compact tracked summary line to `ai_evals/history/<mode>.jsonl` for full-suite runs only
+- `--backend-validation <mode>`: optional backend smoke validation (`off` or `preview`) for `script` and `flow` evals
+
+## Models
+
+Use `bun run cli -- models` to see the current aliases.
+
+Today:
+
+- `haiku`
+- `sonnet`
+- `opus`
+- `4o`
+- `gpt-5.5`
+- `gemini-3-flash-preview`
+- `gemini-3.1-pro-preview`
+- `deepseek-v4-flash`
+- `deepseek-v4-pro`
+
+Notes:
+
+- the command also prints accepted alias spellings such as `gpt-4o`, `gpt-55`, `claude-opus-4.6`, and `claude-haiku-4.5`
+- frontend modes (`flow`, `script`, `app`, `global`) can use Anthropic, OpenAI, Gemini, and DeepSeek-backed aliases
+- `cli` mode always uses the Anthropic agent SDK, so only Anthropic aliases are valid there
+- the judge model is separate and currently defaults to `claude-sonnet-4-6`; use `--skip-judge` for deterministic-only runs
+
+## Case Format
+
+Cases live in one YAML file per mode under `ai_evals/cases/`.
+
+Minimal shape:
+
+```yaml
+- id: flow-test0-sum-two-numbers
+  prompt: |-
+    Create a flow that takes two numbers, `a` and `b`, and returns their sum.
+  initial: ai_evals/fixtures/...
+  expected: ai_evals/fixtures/...
+```
+
+Optional fields:
+
+- `initial`: starting state fixture
+- `expected`: expected artifact fixture
+- `validate`: extra deterministic validation rules
+- `runtime.backendPreview`: optional real backend preview config for smoke validation
+
+For `flow` mode, `validate` can express requirements such as:
+
+- accepted input schema shapes
+- required `results.*` reference validity
+- required module/code/input characteristics
+
+For `app` mode, `validate` can express narrow hard requirements such as:
+
+- required frontend file paths or backend runnable keys
+- minimum backend runnable counts
+- required backend runnable types
+- minimum datatable / datatable-table counts
+- specific required datatable tables
+
+For `global` mode, `validate` can express draft-level requirements such as:
+
+- required draft type/path/language
+- required or forbidden snippets in draft values
+- required or forbidden draft counts
+- forbidden draft paths
+
+Global initial fixtures can also seed `liveEditorDrafts` with `type`,
+`storagePath`, `effectivePath`, and `value` fields. These drafts emulate the
+currently open script, flow, or raw app editor so cases can test prompts that
+refer to "this" or the "current" item.
+
+Global (and flow) initial fixtures can seed `workspace.datatables` so the
+`list_datatables`, `get_datatable_table_schema`, and `exec_datatable_sql` tools
+return seeded data during evals. Each entry is
+`{ datatable_name, schemas: { <schema>: { <table>: { columns, rows? } } } }`.
+SQL runs through a small in-memory engine (`datatableSqlEngine.ts`), not a real
+database. Writes are **stateful within a case**: `CREATE`/`DROP`/`INSERT`/`UPDATE`/
+`DELETE` mutate the seeded datatable in place, so a later `list_datatables`,
+`get_datatable_table_schema`, `SELECT`, or `information_schema` query reflects them
+— this is what stops a model from looping when it re-queries to verify a write.
+The engine is best-effort: `SELECT` returns all rows of the referenced (or first)
+table with no WHERE filtering/projection/joins, `WHERE` on UPDATE/DELETE supports
+`col = value` predicates joined by `AND`, and anything unparseable is a no-op
+success. So validate datatable cases through tool-use and SQL-argument assertions
+(`requiredToolsUsed`, `stringIncludesAnyOf`) — not through exact returned row
+values. An empty/absent `datatables` seed makes `list_datatables` return `[]`,
+which is what the "no datatable configured" blocking cases rely on.
+
+Set `WMILL_AI_EVAL_DISABLE_ACTIVE_EDITOR_CONTEXT=1` to run those cases with
+the old behavior where the live editor is only discoverable through
+`list_workspace_items`.
+
+App fixtures can also include an optional `datatables.json` file at the fixture root.
+
+For `flow` mode, an `initial` fixture can also include a benchmark workspace catalog of
+existing scripts and flows. That lets the real `search_workspace` and
+`get_runnable_details` tools discover reusable workspace runnables during evals.
+
+If `--backend-validation preview` is enabled:
+
+- `script` evals run a real backend script preview in an isolated temp workspace
+- `flow` evals run a real backend flow preview only for cases that define `runtime.backendPreview`
+- `flow` cases with `initial.workspace` fixtures seed those scripts and flows into the preview workspace before preview
+- when `WMILL_AI_EVAL_BACKEND_WORKSPACE` is set, `ai_evals` creates or reuses that workspace as a dedicated test workspace, clears managed eval assets under `f/evals/*` before each preview run, and then reseeds the current case fixtures
+
+Supported backend env vars:
+
+- `WMILL_AI_EVAL_BACKEND_VALIDATION=preview`
+- `WMILL_AI_EVAL_BACKEND_URL=http://127.0.0.1:8000`
+- `WMILL_AI_EVAL_BACKEND_EMAIL=admin@windmill.dev`
+- `WMILL_AI_EVAL_BACKEND_PASSWORD=changeme`
+- `WMILL_AI_EVAL_BACKEND_WORKSPACE=integration-tests` to reuse an existing workspace on CE installs with low workspace limits
+
+Frontend modes require a reachable Windmill backend and send model requests through the workspace AI proxy at `/api/w/{workspace}/ai/proxy`. At startup, `ai_evals` checks the resolved backend URL and fails early with setup guidance if the backend cannot be reached or login fails.
+
+For frontend modes:
+
+- `ai_evals` creates a temporary backend workspace, or creates/reuses `WMILL_AI_EVAL_BACKEND_WORKSPACE` when it is set
+- it upserts a provider resource under `f/evals/ai/<provider>`
+- frontend requests go through `/api/w/{workspace}/ai/proxy`
+
+## Results And Artifacts
+
+Every run writes:
+
+- a summary JSON under `ai_evals/results/`
+- generated artifacts in a sibling directory
+
+If `--record` is used, the CLI also appends one compact JSON line to:
+
+- `ai_evals/history/flow.jsonl`
+- `ai_evals/history/script.jsonl`
+- `ai_evals/history/app.jsonl`
+- `ai_evals/history/global.jsonl`
+- `ai_evals/history/cli.jsonl`
+
+Each recorded line contains:
+
+- run metadata (`createdAt`, `gitSha`, `mode`, `runModel`, `judgeModel`)
+- suite totals (`caseCount`, `attemptCount`, `passedAttempts`, `passRate`, `averageDurationMs`, `averagePassedDurationMs`, `averageJudgeScore`)
+- average token usage (`averageTokenUsagePerAttempt`, `averageTokenUsagePerPassedAttempt`)
+- per-case metrics under `cases[]` (`averageDurationMs`, `averagePassedDurationMs`, `averageJudgeScore`, `averageTokenUsagePerAttempt`, `averageTokenUsagePerPassedAttempt`, pass rate)
+- `failedCaseIds`
+
+The CLI headline duration and token averages use passed attempts only.
+All-attempt averages are still recorded to make failures auditable without
+letting failed attempts skew success cost comparisons.
+
+Example:
+
+- summary: `ai_evals/results/2026-04-09T09-40-33.051Z__flow.json`
+- artifacts: `ai_evals/results/2026-04-09T09-40-33.051Z__flow/`
+
+Typical artifacts by mode:
+
+- `flow`: `flow.json`
+- `script`: `script.json` plus the generated script file
+- `app`: `app.json` plus frontend/backend files
+- `global`: `global-drafts.json`
+- `cli`: `assistant-output.txt`, `trace.json`, `wmill-invocations.jsonl`, plus generated workspace files
+- backend-validated attempts also include `backend-preview.json`
+
+## Layout
+
+- `cases/`: one YAML file per mode
+- `fixtures/`: initial and expected fixtures
+- `core/`: shared loading, model resolution, validation, judging, and result writing
+- `modes/`: one runner per mode
+- `history/`: optional tracked pass-rate history written by `run --record`, one JSONL file per mode
+- `results/`: local benchmark output and artifacts
+
+## Notes
+
+- Frontend modes reuse the production frontend chat code through the Vitest bridge.
+- Global mode evaluates the production global AI tools and validates the resulting AI draft store.
+- CLI mode creates an isolated workspace, writes the current checkout guidance into it, and benchmarks the real skills / `AGENTS.md` flow.
+- CLI mode now also records a structured trace of invoked skills, tool calls, proposed `wmill` commands, and any attempted `wmill` executions.
+- Frontend progress streams live while the benchmark is running.
+- Deterministic validators should stay focused on real correctness constraints, not one exact implementation shape.

ai_evals/adapters/cli/runtime.test.ts

@@ -0,0 +1,149 @@
+import { describe, expect, it } from "bun:test";
+import {
+  anthropicUsageToBenchmarkTokenUsage,
+  extractCliResultTokenUsage,
+  extractProposedWmillCommands,
+  parseWmillInvocationLog,
+} from "./runtime";
+
+describe("anthropicUsageToBenchmarkTokenUsage", () => {
+  it("includes cache tokens in prompt usage", () => {
+    expect(
+      anthropicUsageToBenchmarkTokenUsage({
+        input_tokens: 120,
+        output_tokens: 45,
+        cache_creation_input_tokens: 30,
+        cache_read_input_tokens: 5,
+      })
+    ).toEqual({
+      prompt: 155,
+      completion: 45,
+      total: 200,
+    });
+  });
+
+  it("returns null when usage is absent", () => {
+    expect(anthropicUsageToBenchmarkTokenUsage(null)).toBeNull();
+  });
+});
+
+describe("extractCliResultTokenUsage", () => {
+  it("reads aggregate usage from the SDK result event", () => {
+    expect(
+      extractCliResultTokenUsage({
+        type: "result",
+        usage: {
+          input_tokens: 400,
+          output_tokens: 120,
+          cache_creation_input_tokens: 50,
+          cache_read_input_tokens: 25,
+        },
+      })
+    ).toEqual({
+      prompt: 475,
+      completion: 120,
+      total: 595,
+    });
+  });
+
+  it("falls back to modelUsage when aggregate usage is unavailable", () => {
+    expect(
+      extractCliResultTokenUsage({
+        type: "result",
+        modelUsage: {
+          opus: {
+            inputTokens: 200,
+            outputTokens: 60,
+            cacheCreationInputTokens: 10,
+            cacheReadInputTokens: 5,
+          },
+          haiku: {
+            inputTokens: 80,
+            outputTokens: 20,
+            cacheCreationInputTokens: 0,
+            cacheReadInputTokens: 15,
+          },
+        },
+      })
+    ).toEqual({
+      prompt: 310,
+      completion: 80,
+      total: 390,
+    });
+  });
+});
+
+describe("extractProposedWmillCommands", () => {
+  it("extracts proposed commands from bullets, code blocks, and inline code", () => {
+    expect(
+      extractProposedWmillCommands(`
+Next:
+- \`wmill generate-metadata --yes\`
+- wmill sync push
+
+You can inspect failures with \`wmill job logs 123\`.
+`)
+    ).toEqual([
+      "wmill generate-metadata --yes",
+      "wmill sync push",
+      "wmill job logs 123",
+    ]);
+  });
+
+  it("extracts inline prose commands that are not wrapped in backticks", () => {
+    expect(
+      extractProposedWmillCommands(
+        "The first command is wmill sync pull before you edit locally."
+      )
+    ).toEqual(["wmill sync pull"]);
+  });
+
+  it("extracts multiple inline prose commands from a single sentence", () => {
+    expect(
+      extractProposedWmillCommands(
+        "Run wmill generate-metadata and then wmill sync push when you are ready."
+      )
+    ).toEqual(["wmill generate-metadata", "wmill sync push"]);
+  });
+
+  it("ignores negated command mentions", () => {
+    expect(
+      extractProposedWmillCommands(
+        "Do not run `wmill sync push`. Instead run `wmill sync pull` first."
+      )
+    ).toEqual(["wmill sync pull"]);
+  });
+});
+
+describe("parseWmillInvocationLog", () => {
+  it("parses stubbed wmill invocations into structured records", () => {
+    expect(
+      parseWmillInvocationLog(`noise
+__WMILL_BENCHMARK__
+2026-04-21T12:00:00+00:00
+/tmp/workspace
+2
+generate-metadata
+--yes
+__WMILL_BENCHMARK__
+2026-04-21T12:00:05+00:00
+/tmp/workspace
+3
+sync
+push
+--dry-run
+`)
+    ).toEqual([
+      {
+        argv: ["generate-metadata", "--yes"],
+        cwd: "/tmp/workspace",
+        timestamp: "2026-04-21T12:00:00+00:00",
+      },
+      {
+        argv: ["sync", "push", "--dry-run"],
+        cwd: "/tmp/workspace",
+        timestamp: "2026-04-21T12:00:05+00:00",
+      },
+    ]);
+  });
+});

ai_evals/adapters/cli/runtime.ts

@@ -0,0 +1,543 @@
+import { query, type Options } from "@anthropic-ai/claude-agent-sdk";
+import { chmod, mkdir, readFile, writeFile } from "node:fs/promises";
+import { delimiter, join } from "path";
+import { fileURLToPath } from "url";
+import { getCliEvalModel, resolveEvalModel, type CliEvalModelConfig } from "../../core/models";
+import type {
+  BenchmarkTokenUsage,
+  CliToolInvocation,
+  CliTrace,
+  CliWmillInvocation,
+} from "../../core/types";
+
+export type ToolInvocation = CliToolInvocation;
+
+export interface PromptRunResult {
+  output: string;
+  durationMs: number;
+  tokenUsage: BenchmarkTokenUsage | null;
+  // Input tokens on the last assistant turn. The SDK `result` message reports
+  // usage cumulatively, so the final context size comes from per-turn usage.
+  finalContextTokens: number | null;
+  trace: CliTrace;
+}
+
+interface AnthropicUsageLike {
+  input_tokens?: number | null;
+  output_tokens?: number | null;
+  cache_creation_input_tokens?: number | null;
+  cache_read_input_tokens?: number | null;
+}
+
+interface AnthropicModelUsageLike {
+  inputTokens?: number | null;
+  outputTokens?: number | null;
+  cacheCreationInputTokens?: number | null;
+  cacheReadInputTokens?: number | null;
+}
+
+interface CliResultMessageLike {
+  type?: string;
+  usage?: AnthropicUsageLike | null;
+  modelUsage?: Record<string, AnthropicModelUsageLike> | null;
+}
+
+const REPO_ROOT = fileURLToPath(new URL("../../../", import.meta.url));
+export const DEFAULT_CLI_EVAL_MODEL: CliEvalModelConfig = getCliEvalModel(resolveEvalModel("cli"));
+const WMILL_STUB_DIR_NAME = ".wmill-benchmark-bin";
+const WMILL_LOG_FILE_NAME = ".wmill-benchmark-wmill-invocations.log";
+const WMILL_LOG_MARKER = "__WMILL_BENCHMARK__";
+const NEGATED_COMMAND_PREFIX = /(?:^|\b)(?:do not|don't|dont|never|instead of)\s+(?:run|use)?\s*$/i;
+const COMMAND_STOP_WORDS = new Set([
+  "and",
+  "before",
+  "after",
+  "then",
+  "instead",
+  "otherwise",
+  "because",
+  "so",
+  "if",
+  "when",
+  "while",
+  "once",
+]);
+const COMMAND_STOP_TOKENS = new Set(["-", "–", "—", "|"]);
+
+export function getGeneratedSkillsSource(): string {
+  return join(REPO_ROOT, "system_prompts", "auto-generated", "skills");
+}
+
+export function anthropicUsageToBenchmarkTokenUsage(
+  usage: AnthropicUsageLike | null | undefined
+): BenchmarkTokenUsage | null {
+  if (!usage) {
+    return null;
+  }
+
+  const prompt =
+    (usage.input_tokens ?? 0) +
+    (usage.cache_creation_input_tokens ?? 0) +
+    (usage.cache_read_input_tokens ?? 0);
+  const completion = usage.output_tokens ?? 0;
+
+  return {
+    prompt,
+    completion,
+    total: prompt + completion,
+  };
+}
+
+export function extractCliResultTokenUsage(message: unknown): BenchmarkTokenUsage | null {
+  if (!message || typeof message !== "object") {
+    return null;
+  }
+
+  const resultMessage = message as CliResultMessageLike;
+  if (resultMessage.type !== "result") {
+    return null;
+  }
+
+  const usage = anthropicUsageToBenchmarkTokenUsage(resultMessage.usage);
+  if (usage) {
+    return usage;
+  }
+
+  if (!resultMessage.modelUsage || typeof resultMessage.modelUsage !== "object") {
+    return null;
+  }
+
+  let prompt = 0;
+  let completion = 0;
+  let sawModelUsage = false;
+
+  for (const modelUsage of Object.values(resultMessage.modelUsage)) {
+    if (!modelUsage || typeof modelUsage !== "object") {
+      continue;
+    }
+
+    prompt +=
+      (modelUsage.inputTokens ?? 0) +
+      (modelUsage.cacheCreationInputTokens ?? 0) +
+      (modelUsage.cacheReadInputTokens ?? 0);
+    completion += modelUsage.outputTokens ?? 0;
+    sawModelUsage = true;
+  }
+
+  if (!sawModelUsage) {
+    return null;
+  }
+
+  return {
+    prompt,
+    completion,
+    total: prompt + completion,
+  };
+}
+
+export async function runPromptAndCapture(
+  prompt: string,
+  cwd: string,
+  maxTurns: number = 3,
+  modelConfig: CliEvalModelConfig = DEFAULT_CLI_EVAL_MODEL
+): Promise<PromptRunResult> {
+  const toolsUsed: ToolInvocation[] = [];
+  const skillsInvoked: string[] = [];
+  const bashCommands: string[] = [];
+  let output = "";
+  let assistantMessageCount = 0;
+  let tokenUsage: BenchmarkTokenUsage | null = null;
+  let finalContextTokens: number | null = null;
+  const startedAt = Date.now();
+  const stubBinDir = join(cwd, WMILL_STUB_DIR_NAME);
+  const wmillLogPath = join(cwd, WMILL_LOG_FILE_NAME);
+
+  const options: Options = {
+    cwd,
+    model: modelConfig.model,
+    maxTurns,
+    settingSources: ["project"],
+    allowedTools: ["Skill", "Read", "Glob", "Grep", "Bash", "Write", "Edit"],
+    env: {
+      ...getQueryEnv(),
+      PATH: process.env.PATH ? `${stubBinDir}${delimiter}${process.env.PATH}` : stubBinDir,
+      WMILL_BENCHMARK_LOG_PATH: wmillLogPath,
+    },
+  };
+
+  await installWmillStub(stubBinDir);
+
+  for await (const message of query({ prompt, options })) {
+    if (message.type === "assistant") {
+      assistantMessageCount += 1;
+      const turnContext = anthropicUsageToBenchmarkTokenUsage(
+        message.message?.usage
+      )?.prompt;
+      if (turnContext && turnContext > 0) {
+        finalContextTokens = turnContext;
+      }
+      const content = message.message?.content;
+      if (Array.isArray(content)) {
+        for (const block of content) {
+          if (block.type === "tool_use") {
+            const input = normalizeToolInput(block.input);
+            toolsUsed.push({
+              tool: block.name,
+              input,
+              timestamp: Date.now()
+            });
+
+            if (block.name === "Skill") {
+              const skillInput = input as { skill?: string };
+              if (skillInput.skill) {
+                pushUnique(skillsInvoked, skillInput.skill);
+              }
+            }
+
+            if (block.name === "Bash") {
+              for (const command of extractBashCommands(input)) {
+                pushUnique(bashCommands, command);
+              }
+            }
+          } else if (block.type === "text") {
+            output += block.text;
+          }
+        }
+      }
+    } else if (message.type === "result") {
+      const resultMessage = message as { result?: string };
+      tokenUsage = extractCliResultTokenUsage(message) ?? tokenUsage;
+      if (typeof resultMessage.result === "string") {
+        output += resultMessage.result;
+      }
+    }
+  }
+
+  const proposedCommands = extractProposedWmillCommands(output);
+  const wmillInvocations = await readWmillInvocationLog(wmillLogPath);
+
+  return {
+    output,
+    durationMs: Date.now() - startedAt,
+    tokenUsage,
+    finalContextTokens,
+    trace: {
+      toolsUsed,
+      skillsInvoked,
+      assistantMessageCount,
+      bashCommands,
+      proposedCommands,
+      executedWmillCommands: wmillInvocations.map(formatExecutedWmillCommand),
+      wmillInvocations,
+      firstMutationToolIndex: getFirstMutationToolIndex(toolsUsed),
+    },
+  };
+}
+
+export function wasSkillInvoked(result: PromptRunResult, skillName: string): boolean {
+  return result.trace.skillsInvoked.some((skill) => skill === skillName);
+}
+
+export function wasToolUsed(result: PromptRunResult, toolName: string): boolean {
+  return result.trace.toolsUsed.some((tool) => tool.tool === toolName);
+}
+
+export function formatCliRunModelLabel(modelConfig: CliEvalModelConfig): string {
+  return `${modelConfig.provider}:${modelConfig.model}`;
+}
+
+export function getToolInputs(
+  result: PromptRunResult,
+  toolName: string
+): Record<string, unknown>[] {
+  return result.trace.toolsUsed
+    .filter((tool) => tool.tool === toolName)
+    .map((tool) => tool.input);
+}
+
+export function extractProposedWmillCommands(output: string): string[] {
+  const commands: string[] = [];
+
+  for (const line of output.split(/\r?\n/)) {
+    for (const command of extractInlineBacktickCommands(line)) {
+      pushUnique(commands, command);
+    }
+
+    for (const command of extractInlineProseCommands(line.replace(/^\s*(?:[-*]|\d+\.)\s*/, ""))) {
+      pushUnique(commands, command);
+    }
+  }
+
+  return commands;
+}
+
+export function parseWmillInvocationLog(raw: string): CliWmillInvocation[] {
+  const entries: CliWmillInvocation[] = [];
+  const lines = raw.split(/\r?\n/);
+
+  for (let index = 0; index < lines.length; index += 1) {
+    if (lines[index] !== WMILL_LOG_MARKER) {
+      continue;
+    }
+
+    const timestamp = lines[index + 1] ?? "";
+    const cwd = lines[index + 2] ?? "";
+    const argCount = Number.parseInt(lines[index + 3] ?? "", 10);
+    if (!Number.isFinite(argCount) || argCount < 0) {
+      continue;
+    }
+
+    const start = index + 4;
+    const argv = lines.slice(start, start + argCount);
+    entries.push({ argv, cwd, timestamp });
+    index = start + argCount - 1;
+  }
+
+  return entries;
+}
+
+async function installWmillStub(binDir: string): Promise<void> {
+  await mkdir(binDir, { recursive: true });
+
+  const stubPath = join(binDir, "wmill");
+  const script = `#!/usr/bin/env bash
+set -euo pipefail
+{
+  printf '${WMILL_LOG_MARKER}\\n'
+  date -u +"%Y-%m-%dT%H:%M:%SZ"
+  printf '%s\\n' "$PWD"
+  printf '%s\\n' "$#"
+  printf '%s\\n' "$@"
+} >> "\${WMILL_BENCHMARK_LOG_PATH:?}"
+printf 'wmill benchmark stub: do not execute Windmill CLI commands during ai_evals; describe them in the final response instead.\\n' >&2
+exit 97
+`;
+
+  await writeFile(stubPath, script, "utf8");
+  await chmod(stubPath, 0o755);
+}
+
+async function readWmillInvocationLog(logPath: string): Promise<CliWmillInvocation[]> {
+  const raw = await readFile(logPath, "utf8").catch(() => null);
+  if (!raw) {
+    return [];
+  }
+  return parseWmillInvocationLog(raw);
+}
+
+function getQueryEnv(): Record<string, string> {
+  return Object.fromEntries(
+    Object.entries(process.env).flatMap(([key, value]) =>
+      typeof value === "string" ? [[key, value]] : []
+    )
+  );
+}
+
+function normalizeToolInput(input: unknown): Record<string, unknown> {
+  if (input && typeof input === "object" && !Array.isArray(input)) {
+    return input as Record<string, unknown>;
+  }
+
+  if (typeof input === "string") {
+    return { raw: input };
+  }
+
+  return {};
+}
+
+function extractBashCommands(input: Record<string, unknown>): string[] {
+  const commands: string[] = [];
+
+  for (const key of ["command", "cmd", "script", "raw"]) {
+    const value = input[key];
+    if (typeof value === "string") {
+      for (const line of value.split(/\r?\n/)) {
+        const command = normalizeCommandCandidate(line);
+        if (command) {
+          pushUnique(commands, command);
+        }
+      }
+    }
+  }
+
+  return commands;
+}
+
+function extractInlineBacktickCommands(line: string): string[] {
+  const commands: string[] = [];
+  const regex = /`(wmill [^`\n]+)`/g;
+  let match: RegExpExecArray | null = null;
+
+  while ((match = regex.exec(line)) !== null) {
+    if (hasNegatedCommandPrefix(line.slice(0, match.index))) {
+      continue;
+    }
+
+    const command = normalizeCommandCandidate(match[1]);
+    if (command) {
+      pushUnique(commands, command);
+    }
+  }
+
+  return commands;
+}
+
+function extractInlineProseCommands(line: string): string[] {
+  const commands: string[] = [];
+  let searchFrom = 0;
+
+  while (true) {
+    const inlineIndex = line.toLowerCase().indexOf("wmill ", searchFrom);
+    if (inlineIndex === -1) {
+      return commands;
+    }
+
+    if (!hasNegatedCommandPrefix(line.slice(0, inlineIndex))) {
+      const command = extractInlineProseCommandAt(line, inlineIndex);
+      if (command) {
+        pushUnique(commands, command);
+      }
+    }
+
+    searchFrom = inlineIndex + "wmill ".length;
+  }
+}
+
+function extractInlineProseCommandAt(line: string, startIndex: number): string | null {
+  const tokens = ["wmill"];
+  let cursor = startIndex + "wmill".length;
+
+  while (cursor < line.length) {
+    while (cursor < line.length && /\s/.test(line[cursor]!)) {
+      cursor += 1;
+    }
+
+    if (cursor >= line.length) {
+      break;
+    }
+
+    const current = line[cursor]!;
+    if ("`.,;:()[]{}".includes(current)) {
+      break;
+    }
+
+    const token = readCommandToken(line, cursor);
+    if (!token) {
+      break;
+    }
+
+    if (COMMAND_STOP_WORDS.has(token.value.toLowerCase())) {
+      break;
+    }
+
+    if (COMMAND_STOP_TOKENS.has(token.value)) {
+      break;
+    }
+
+    tokens.push(token.value);
+    cursor = token.nextIndex;
+  }
+
+  if (tokens.length <= 1) {
+    return null;
+  }
+
+  return normalizeCommandCandidate(tokens.join(" "));
+}
+
+function readCommandToken(
+  line: string,
+  startIndex: number
+): { value: string; nextIndex: number } | null {
+  const firstChar = line[startIndex]!;
+
+  if (firstChar === `"` || firstChar === `'`) {
+    const endIndex = line.indexOf(firstChar, startIndex + 1);
+    const nextIndex = endIndex === -1 ? line.length : endIndex + 1;
+    return {
+      value: line.slice(startIndex, nextIndex),
+      nextIndex,
+    };
+  }
+
+  if (firstChar === "<") {
+    const endIndex = line.indexOf(">", startIndex + 1);
+    const nextIndex = endIndex === -1 ? line.length : endIndex + 1;
+    return {
+      value: line.slice(startIndex, nextIndex),
+      nextIndex,
+    };
+  }
+
+  let endIndex = startIndex;
+  while (endIndex < line.length && !/[\s`.,;:()[\]{}#]/.test(line[endIndex]!)) {
+    endIndex += 1;
+  }
+
+  if (endIndex === startIndex) {
+    return null;
+  }
+
+  return {
+    value: line.slice(startIndex, endIndex),
+    nextIndex: endIndex,
+  };
+}
+
+function hasNegatedCommandPrefix(prefix: string): boolean {
+  const normalizedPrefix = prefix
+    .toLowerCase()
+    .replace(/[`"'“”‘’]/g, " ")
+    .replace(/\s+/g, " ")
+    .trimEnd();
+
+  return NEGATED_COMMAND_PREFIX.test(normalizedPrefix);
+}
+
+function normalizeCommandCandidate(value: string): string | null {
+  const trimmed = value.trim().replace(/^`|`$/g, "");
+  if (!trimmed) {
+    return null;
+  }
+
+  const normalized = trimmed
+    .replace(/\s+/g, " ")
+    .replace(/[`.;:,]+$/g, "")
+    .trim();
+
+  return normalized.length > 0 ? normalized : null;
+}
+
+function formatExecutedWmillCommand(entry: CliWmillInvocation): string {
+  return ["wmill", ...entry.argv].join(" ").trim();
+}
+
+function getFirstMutationToolIndex(toolsUsed: ToolInvocation[]): number | null {
+  for (const [index, tool] of toolsUsed.entries()) {
+    if (tool.tool === "Write" || tool.tool === "Edit") {
+      return index;
+    }
+
+    if (tool.tool === "Bash" && extractBashCommands(tool.input).some(isLikelyMutatingBashCommand)) {
+      return index;
+    }
+  }
+
+  return null;
+}
+
+function isLikelyMutatingBashCommand(command: string): boolean {
+  return (
+    /\b(?:mkdir|touch|rm|mv|cp|install|tee)\b/.test(command) ||
+    /\b(?:cat|echo|printf)\b.*(?:>|>>|\|\s*tee\b)/.test(command) ||
+    /\bsed\s+-i\b/.test(command) ||
+    /\bperl\s+-pi\b/.test(command) ||
+    /\bwmill\b/.test(command)
+  );
+}
+
+function pushUnique(values: string[], value: string): void {
+  if (!values.includes(value)) {
+    values.push(value);
+  }
+}

ai_evals/adapters/frontend/backendPreview.test.ts

@@ -0,0 +1,244 @@
+import { afterEach, describe, expect, it } from 'bun:test'
+import type { BackendValidationSettings } from '../../core/backendValidation'
+import { BackendPreviewClient } from './backendPreview'
+
+const ORIGINAL_FETCH = globalThis.fetch
+
+afterEach(() => {
+	globalThis.fetch = ORIGINAL_FETCH
+})
+
+describe('BackendPreviewClient', () => {
+	it('updates an existing seeded script on path conflict and waits for deployment', async () => {
+		const requests: Array<{ url: string; init?: RequestInit }> = []
+		globalThis.fetch = mockFetch(
+			requests,
+			textResponse(200, 'token'),
+			textResponse(200, ''),
+			textResponse(400, 'Path conflict for f/evals/add_two_numbers with non-archived hash 123'),
+			jsonResponse(200, { hash: '123' }),
+			textResponse(200, '456'),
+			jsonResponse(200, { lock: 'script.lock', lock_error_logs: null })
+		)
+
+		const client = new BackendPreviewClient(
+			buildSettings({ baseUrl: 'http://backend.test/script-upsert' })
+		)
+
+		await client.createScript({
+			workspaceId: 'test',
+			path: 'f/evals/add_two_numbers',
+			summary: 'Add two numbers',
+			content: 'export async function main(a: number, b: number) { return a + b }',
+			language: 'bun'
+		})
+
+		expect(requests.map((entry) => entry.url)).toEqual([
+			'http://backend.test/script-upsert/api/auth/login',
+			'http://backend.test/script-upsert/api/w/test/folders/create',
+			'http://backend.test/script-upsert/api/w/test/scripts/create',
+			'http://backend.test/script-upsert/api/w/test/scripts/get/p/f/evals/add_two_numbers',
+			'http://backend.test/script-upsert/api/w/test/scripts/create',
+			'http://backend.test/script-upsert/api/w/test/scripts/deployment_status/h/456'
+		])
+
+		const updateRequest = requests[4]
+		expect(updateRequest.init?.method).toBe('POST')
+		expect(JSON.parse(String(updateRequest.init?.body))).toMatchObject({
+			path: 'f/evals/add_two_numbers',
+			parent_hash: '123',
+			language: 'bun'
+		})
+	})
+
+	it('updates an existing seeded flow on create conflict', async () => {
+		const requests: Array<{ url: string; init?: RequestInit }> = []
+		globalThis.fetch = mockFetch(
+			requests,
+			textResponse(200, 'token'),
+			textResponse(200, ''),
+			textResponse(400, 'Flow f/evals/add_numbers_flow already exists'),
+			textResponse(200, '')
+		)
+
+		const client = new BackendPreviewClient(
+			buildSettings({ baseUrl: 'http://backend.test/flow-upsert' })
+		)
+
+		await client.createFlow({
+			workspaceId: 'test',
+			path: 'f/evals/add_numbers_flow',
+			summary: 'Add numbers',
+			value: { modules: [] }
+		})
+
+		expect(requests.map((entry) => entry.url)).toEqual([
+			'http://backend.test/flow-upsert/api/auth/login',
+			'http://backend.test/flow-upsert/api/w/test/folders/create',
+			'http://backend.test/flow-upsert/api/w/test/flows/create',
+			'http://backend.test/flow-upsert/api/w/test/flows/update/f/evals/add_numbers_flow'
+		])
+
+		const updateRequest = requests[3]
+		expect(updateRequest.init?.method).toBe('POST')
+		expect(JSON.parse(String(updateRequest.init?.body))).toMatchObject({
+			path: 'f/evals/add_numbers_flow',
+			value: { modules: [] }
+		})
+	})
+
+	it('serializes shared-workspace validations inside the overridden workspace', async () => {
+		globalThis.fetch = async (input) => {
+			const url = String(input)
+			if (url.endsWith('/api/auth/login')) {
+				return textResponse(200, 'token')
+			}
+			if (url.endsWith('/api/workspaces/exists')) {
+				return textResponse(200, 'true')
+			}
+			if (url.endsWith('/api/w/shared-preview/flows/list_paths')) {
+				return jsonResponse(200, [])
+			}
+			if (url.endsWith('/api/w/shared-preview/scripts/list_paths')) {
+				return jsonResponse(200, [])
+			}
+			throw new Error(`Unexpected fetch: ${url}`)
+		}
+
+		const client = new BackendPreviewClient(
+			buildSettings({
+				baseUrl: 'http://backend.test/shared-lock',
+				workspaceOverride: 'shared-preview'
+			})
+		)
+
+		const order: string[] = []
+		let releaseFirst: (() => void) | undefined
+		let notifyFirstStart: (() => void) | undefined
+		const firstStarted = new Promise<void>((resolve) => {
+			notifyFirstStart = resolve
+		})
+
+		const first = client.withWorkspace('flow-test1', 1, async () => {
+			order.push('first:start')
+			notifyFirstStart?.()
+			await new Promise<void>((resolve) => {
+				releaseFirst = resolve
+			})
+			order.push('first:end')
+		})
+
+		const second = client.withWorkspace('flow-test2', 1, async () => {
+			order.push('second:start')
+			order.push('second:end')
+		})
+
+		await firstStarted
+		expect(order).toEqual(['first:start'])
+
+		releaseFirst?.()
+		await Promise.all([first, second])
+
+		expect(order).toEqual(['first:start', 'first:end', 'second:start', 'second:end'])
+	})
+
+	it('clears managed shared-workspace assets before preview runs', async () => {
+		const requests: Array<{ url: string; init?: RequestInit }> = []
+		globalThis.fetch = mockFetch(
+			requests,
+			textResponse(200, 'token'),
+			textResponse(200, 'true'),
+			jsonResponse(200, ['f/evals/old_subflow', 'u/admin/keep_flow']),
+			textResponse(200, ''),
+			jsonResponse(200, ['f/evals/old_script', 'f/shared/keep_script']),
+			textResponse(200, '')
+		)
+
+		const client = new BackendPreviewClient(
+			buildSettings({
+				baseUrl: 'http://backend.test/shared-cleanup',
+				workspaceOverride: 'shared-preview'
+			})
+		)
+
+		await client.withWorkspace('flow-test1', 1, async () => undefined)
+
+		expect(requests.map((entry) => entry.url)).toEqual([
+			'http://backend.test/shared-cleanup/api/auth/login',
+			'http://backend.test/shared-cleanup/api/workspaces/exists',
+			'http://backend.test/shared-cleanup/api/w/shared-preview/flows/list_paths',
+			'http://backend.test/shared-cleanup/api/w/shared-preview/flows/delete/f/evals/old_subflow',
+			'http://backend.test/shared-cleanup/api/w/shared-preview/scripts/list_paths',
+			'http://backend.test/shared-cleanup/api/w/shared-preview/scripts/delete/p/f/evals/old_script'
+		])
+	})
+
+	it('retries login after a cached login failure', async () => {
+		const requests: Array<{ url: string; init?: RequestInit }> = []
+		globalThis.fetch = mockFetch(
+			requests,
+			textResponse(503, 'backend starting'),
+			textResponse(200, 'token'),
+			textResponse(200, 'true'),
+			jsonResponse(200, []),
+			jsonResponse(200, [])
+		)
+
+		const client = new BackendPreviewClient(
+			buildSettings({
+				baseUrl: 'http://backend.test/login-retry',
+				workspaceOverride: 'shared-preview'
+			})
+		)
+
+		await expect(client.withWorkspace('flow-test1', 1, async () => undefined)).rejects.toThrow(
+			'login for backend validation failed'
+		)
+		await expect(client.withWorkspace('flow-test1', 1, async () => 'ok')).resolves.toBe('ok')
+
+		expect(
+			requests.filter((entry) => entry.url === 'http://backend.test/login-retry/api/auth/login')
+		).toHaveLength(2)
+	})
+})
+
+function buildSettings(
+	overrides: Partial<BackendValidationSettings> = {}
+): BackendValidationSettings {
+	return {
+		mode: 'preview',
+		baseUrl: 'http://backend.test/default',
+		email: 'admin@windmill.dev',
+		password: 'changeme',
+		pollIntervalMs: 1,
+		maxWaitMs: 50,
+		...overrides
+	}
+}
+
+function mockFetch(
+	requests: Array<{ url: string; init?: RequestInit }>,
+	...responses: Response[]
+): typeof fetch {
+	const queue = [...responses]
+	return async (input, init) => {
+		const url = String(input)
+		requests.push({ url, init })
+		const next = queue.shift()
+		if (!next) {
+			throw new Error(`Unexpected fetch: ${url}`)
+		}
+		return next
+	}
+}
+
+function jsonResponse(status: number, body: unknown): Response {
+	return new Response(JSON.stringify(body), {
+		status,
+		headers: { 'Content-Type': 'application/json' }
+	})
+}
+
+function textResponse(status: number, body: string): Response {
+	return new Response(body, { status })
+}

ai_evals/adapters/frontend/backendPreview.ts

@@ -0,0 +1,503 @@
+import { randomUUID } from 'node:crypto'
+import type { BackendValidationSettings } from '../../core/backendValidation'
+
+interface CompletedJobResultMaybe {
+	completed: boolean
+	result: unknown
+	success?: boolean
+	started?: boolean
+}
+
+interface ScriptDeploymentStatus {
+	lock?: unknown
+	lock_error_logs?: string | null
+}
+
+export interface CompletedPreviewJob {
+	id: string
+	success: boolean
+	result: unknown
+	logs?: string | null
+	raw: Record<string, unknown>
+}
+
+const tokenCache = new Map<string, Promise<string>>()
+const sharedWorkspaceQueue = new Map<string, Promise<void>>()
+const managedSharedWorkspacePrefixes = ['f/evals/']
+const DEFAULT_WORKSPACE_PREFIX = 'ai-evals'
+
+export class BackendPreviewClient {
+	constructor(private readonly settings: BackendValidationSettings) {}
+
+	async withWorkspace<T>(
+		caseId: string,
+		attempt: number,
+		body: (workspaceId: string) => Promise<T>
+	): Promise<T> {
+		const workspaceId =
+			this.settings.workspaceOverride ??
+			buildWorkspaceId(caseId, attempt)
+
+		const run = async () => {
+			await this.ensureWorkspace(workspaceId)
+			if (this.settings.workspaceOverride) {
+				await this.clearManagedSharedWorkspaceAssets(workspaceId)
+			}
+
+			try {
+				return await body(workspaceId)
+			} finally {
+				if (!this.settings.workspaceOverride) {
+					await this.deleteWorkspace(workspaceId).catch(() => undefined)
+				}
+			}
+		}
+
+		if (this.settings.workspaceOverride) {
+			return await withSharedWorkspaceLock(workspaceId, run)
+		}
+
+		return await run()
+	}
+
+	async createScript(input: {
+		workspaceId: string
+		path: string
+		summary: string
+		description?: string
+		schema?: Record<string, unknown>
+		content: string
+		language: string
+	}): Promise<void> {
+		await this.ensureFolderForPath(input.workspaceId, input.path)
+
+		const payload = {
+			path: input.path,
+			summary: input.summary,
+			description: input.description ?? '',
+			content: input.content,
+			schema: input.schema ?? { type: 'object', properties: {}, required: [] },
+			is_template: false,
+			language: input.language,
+			kind: 'script'
+		}
+
+		const response = await this.request(`/w/${encodeURIComponent(input.workspaceId)}/scripts/create`, {
+			method: 'POST',
+			headers: { 'Content-Type': 'application/json' },
+			body: JSON.stringify(payload)
+		})
+
+		if (response.ok) {
+			await this.waitForScriptDeployment(input.workspaceId, input.path, (await response.text()).trim())
+			return
+		}
+
+		const message = await response.text()
+		if (!isConflictMessage(message)) {
+			throw new Error(`create script ${input.path} failed: ${response.status} ${response.statusText} - ${message}`)
+		}
+
+		const currentScript = await this.getScriptByPath(input.workspaceId, input.path)
+		const currentHash = readStringField(currentScript, 'hash', `script ${input.path}`)
+		const updateResponse = await this.request(
+			`/w/${encodeURIComponent(input.workspaceId)}/scripts/create`,
+			{
+				method: 'POST',
+				headers: { 'Content-Type': 'application/json' },
+				body: JSON.stringify({
+					...payload,
+					parent_hash: currentHash
+				})
+			}
+		)
+		await expectOk(updateResponse, `update script ${input.path}`)
+		await this.waitForScriptDeployment(input.workspaceId, input.path, (await updateResponse.text()).trim())
+	}
+
+	async createFlow(input: {
+		workspaceId: string
+		path: string
+		summary: string
+		description?: string
+		schema?: Record<string, unknown>
+		value: Record<string, unknown>
+	}): Promise<void> {
+		await this.ensureFolderForPath(input.workspaceId, input.path)
+
+		const payload = {
+			path: input.path,
+			summary: input.summary,
+			description: input.description ?? '',
+			schema: input.schema ?? { type: 'object', properties: {}, required: [] },
+			value: input.value
+		}
+
+		const response = await this.request(`/w/${encodeURIComponent(input.workspaceId)}/flows/create`, {
+			method: 'POST',
+			headers: { 'Content-Type': 'application/json' },
+			body: JSON.stringify(payload)
+		})
+
+		if (response.ok) {
+			return
+		}
+
+		const message = await response.text()
+		if (!isConflictMessage(message)) {
+			throw new Error(`create flow ${input.path} failed: ${response.status} ${response.statusText} - ${message}`)
+		}
+
+		const updateResponse = await this.request(
+			`/w/${encodeURIComponent(input.workspaceId)}/flows/update/${input.path}`,
+			{
+				method: 'POST',
+				headers: { 'Content-Type': 'application/json' },
+				body: JSON.stringify(payload)
+			}
+		)
+		await expectOk(updateResponse, `update flow ${input.path}`)
+	}
+
+	async runScriptPreview(input: {
+		workspaceId: string
+		content: string
+		args: Record<string, unknown>
+		language: string
+		path?: string
+		timeoutSeconds?: number
+	}): Promise<CompletedPreviewJob> {
+		const response = await this.request(
+			withQuery(`/w/${encodeURIComponent(input.workspaceId)}/jobs/run/preview`, {
+				timeout: input.timeoutSeconds
+			}),
+			{
+				method: 'POST',
+				headers: { 'Content-Type': 'application/json' },
+				body: JSON.stringify({
+					content: input.content,
+					args: input.args,
+					language: input.language,
+					path: input.path
+				})
+			}
+		)
+
+		await expectOk(response, 'start script preview')
+		const jobId = (await response.text()).trim()
+		return await this.waitForCompletedJob(input.workspaceId, jobId)
+	}
+
+	async runFlowPreview(input: {
+		workspaceId: string
+		value: Record<string, unknown>
+		args: Record<string, unknown>
+		timeoutSeconds?: number
+		path?: string
+	}): Promise<CompletedPreviewJob> {
+		const response = await this.request(
+			withQuery(`/w/${encodeURIComponent(input.workspaceId)}/jobs/run/preview_flow`, {
+				timeout: input.timeoutSeconds
+			}),
+			{
+				method: 'POST',
+				headers: { 'Content-Type': 'application/json' },
+				body: JSON.stringify({
+					value: input.value,
+					args: input.args,
+					path: input.path
+				})
+			}
+		)
+
+		await expectOk(response, 'start flow preview')
+		const jobId = (await response.text()).trim()
+		return await this.waitForCompletedJob(input.workspaceId, jobId)
+	}
+
+	private async ensureWorkspace(workspaceId: string): Promise<void> {
+		const existsResponse = await this.request('/workspaces/exists', {
+			method: 'POST',
+			headers: { 'Content-Type': 'application/json' },
+			body: JSON.stringify({ id: workspaceId })
+		})
+		await expectOk(existsResponse, `check workspace ${workspaceId}`)
+
+		if ((await existsResponse.text()).trim() === 'true') {
+			return
+		}
+
+		const createResponse = await this.request('/workspaces/create', {
+			method: 'POST',
+			headers: { 'Content-Type': 'application/json' },
+			body: JSON.stringify({ id: workspaceId, name: workspaceId })
+		})
+		try {
+			await expectOk(createResponse, `create workspace ${workspaceId}`)
+		} catch (error) {
+			const message = error instanceof Error ? error.message : String(error)
+			if (message.includes('maximum number of workspaces')) {
+				throw new Error(
+					`${message}. Reuse an existing workspace with WMILL_AI_EVAL_BACKEND_WORKSPACE=<workspace-id>.`
+				)
+			}
+			throw error
+		}
+	}
+
+	private async deleteWorkspace(workspaceId: string): Promise<void> {
+		const response = await this.request(`/workspaces/delete/${encodeURIComponent(workspaceId)}`, {
+			method: 'DELETE'
+		})
+		await expectOk(response, `delete workspace ${workspaceId}`)
+	}
+
+	private async ensureFolderForPath(workspaceId: string, path: string): Promise<void> {
+		const folderName = extractFolderName(path)
+		if (!folderName) {
+			return
+		}
+
+		const response = await this.request(`/w/${encodeURIComponent(workspaceId)}/folders/create`, {
+			method: 'POST',
+			headers: { 'Content-Type': 'application/json' },
+			body: JSON.stringify({ name: folderName })
+		})
+
+		if (response.ok) {
+			return
+		}
+
+		const message = await response.text()
+		if (!message.toLowerCase().includes('already exists')) {
+			throw new Error(`Failed to create folder ${folderName}: ${message}`)
+		}
+	}
+
+	private async waitForCompletedJob(
+		workspaceId: string,
+		jobId: string
+	): Promise<CompletedPreviewJob> {
+		const deadline = Date.now() + this.settings.maxWaitMs
+
+		while (Date.now() < deadline) {
+			const maybeResponse = await this.request(
+				`/w/${encodeURIComponent(workspaceId)}/jobs_u/completed/get_result_maybe/${encodeURIComponent(jobId)}?get_started=false`
+			)
+			await expectOk(maybeResponse, `poll job ${jobId}`)
+			const maybeResult = (await maybeResponse.json()) as CompletedJobResultMaybe
+
+			if (maybeResult.completed) {
+				const completedResponse = await this.request(
+					`/w/${encodeURIComponent(workspaceId)}/jobs_u/completed/get/${encodeURIComponent(jobId)}`
+				)
+				await expectOk(completedResponse, `get completed job ${jobId}`)
+				const completedJob = (await completedResponse.json()) as Record<string, unknown>
+				return {
+					id: jobId,
+					success: Boolean(maybeResult.success),
+					result: maybeResult.result,
+					logs:
+						typeof completedJob.logs === 'string' || completedJob.logs === null
+							? (completedJob.logs as string | null)
+							: null,
+					raw: completedJob
+				}
+			}
+
+			await new Promise((resolve) => setTimeout(resolve, this.settings.pollIntervalMs))
+		}
+
+		throw new Error(`Timed out waiting for preview job ${jobId} to complete`)
+	}
+
+	private async getScriptByPath(workspaceId: string, path: string): Promise<Record<string, unknown>> {
+		const response = await this.request(`/w/${encodeURIComponent(workspaceId)}/scripts/get/p/${path}`)
+		await expectOk(response, `get script ${path}`)
+		return (await response.json()) as Record<string, unknown>
+	}
+
+	private async clearManagedSharedWorkspaceAssets(workspaceId: string): Promise<void> {
+		const flowPaths = await this.listFlowPaths(workspaceId)
+		for (const path of flowPaths.filter(isManagedSharedWorkspacePath)) {
+			await this.deleteFlowByPath(workspaceId, path)
+		}
+
+		const scriptPaths = await this.listScriptPaths(workspaceId)
+		for (const path of scriptPaths.filter(isManagedSharedWorkspacePath)) {
+			await this.deleteScriptByPath(workspaceId, path)
+		}
+	}
+
+	private async listFlowPaths(workspaceId: string): Promise<string[]> {
+		const response = await this.request(`/w/${encodeURIComponent(workspaceId)}/flows/list_paths`)
+		await expectOk(response, `list flows in workspace ${workspaceId}`)
+		return await response.json()
+	}
+
+	private async listScriptPaths(workspaceId: string): Promise<string[]> {
+		const response = await this.request(`/w/${encodeURIComponent(workspaceId)}/scripts/list_paths`)
+		await expectOk(response, `list scripts in workspace ${workspaceId}`)
+		return await response.json()
+	}
+
+	private async deleteFlowByPath(workspaceId: string, path: string): Promise<void> {
+		const response = await this.request(`/w/${encodeURIComponent(workspaceId)}/flows/delete/${path}`, {
+			method: 'DELETE'
+		})
+		await expectOk(response, `delete flow ${path}`)
+	}
+
+	private async deleteScriptByPath(workspaceId: string, path: string): Promise<void> {
+		const response = await this.request(`/w/${encodeURIComponent(workspaceId)}/scripts/delete/p/${path}`, {
+			method: 'POST'
+		})
+		await expectOk(response, `delete script ${path}`)
+	}
+
+	private async waitForScriptDeployment(
+		workspaceId: string,
+		path: string,
+		hash: string
+	): Promise<void> {
+		const deadline = Date.now() + this.settings.maxWaitMs
+
+		while (Date.now() < deadline) {
+			const response = await this.request(
+				`/w/${encodeURIComponent(workspaceId)}/scripts/deployment_status/h/${encodeURIComponent(hash)}`
+			)
+			await expectOk(response, `check deployment status for script ${path}`)
+			const deployment = (await response.json()) as ScriptDeploymentStatus
+			if (deployment.lock != null) {
+				return
+			}
+			if (deployment.lock_error_logs) {
+				throw new Error(`Script deployment failed for ${path}: ${deployment.lock_error_logs}`)
+			}
+			await new Promise((resolve) => setTimeout(resolve, this.settings.pollIntervalMs))
+		}
+
+		throw new Error(`Timed out waiting for script ${path} (${hash}) to deploy`)
+	}
+
+	private async request(path: string, init?: RequestInit): Promise<Response> {
+		const token = await this.getToken()
+		return await fetch(`${this.settings.baseUrl}/api${path}`, {
+			...init,
+			headers: {
+				Authorization: `Bearer ${token}`,
+				...(init?.headers ?? {})
+			}
+		})
+	}
+
+	private async getToken(): Promise<string> {
+		const cacheKey = `${this.settings.baseUrl}|${this.settings.email}`
+		let tokenPromise = tokenCache.get(cacheKey)
+		if (!tokenPromise) {
+			tokenPromise = this.login().catch((error) => {
+				if (tokenCache.get(cacheKey) === tokenPromise) {
+					tokenCache.delete(cacheKey)
+				}
+				throw error
+			})
+			tokenCache.set(cacheKey, tokenPromise)
+		}
+		return await tokenPromise
+	}
+
+	private async login(): Promise<string> {
+		const response = await fetch(`${this.settings.baseUrl}/api/auth/login`, {
+			method: 'POST',
+			headers: { 'Content-Type': 'application/json' },
+			body: JSON.stringify({
+				email: this.settings.email,
+				password: this.settings.password
+			})
+		})
+		await expectOk(response, 'login for backend validation')
+		return (await response.text()).trim()
+	}
+}
+
+async function withSharedWorkspaceLock<T>(workspaceId: string, body: () => Promise<T>): Promise<T> {
+	const previous = sharedWorkspaceQueue.get(workspaceId) ?? Promise.resolve()
+	let releaseCurrent: (() => void) | undefined
+	const current = new Promise<void>((resolve) => {
+		releaseCurrent = resolve
+	})
+	const tail = previous.catch(() => undefined).then(() => current)
+	sharedWorkspaceQueue.set(workspaceId, tail)
+
+	await previous.catch(() => undefined)
+
+	try {
+		return await body()
+	} finally {
+		releaseCurrent?.()
+		if (sharedWorkspaceQueue.get(workspaceId) === tail) {
+			sharedWorkspaceQueue.delete(workspaceId)
+		}
+	}
+}
+
+function buildWorkspaceId(caseId: string, attempt: number): string {
+	const caseSlug = caseId
+		.toLowerCase()
+		.replace(/[^a-z0-9-]+/g, '-')
+		.replace(/^-+|-+$/g, '')
+		.slice(0, 30)
+	const suffix = randomUUID().slice(0, 8)
+	return `${DEFAULT_WORKSPACE_PREFIX}-${caseSlug || 'case'}-a${attempt}-${suffix}`
+}
+
+function extractFolderName(path: string): string | null {
+	if (!path.startsWith('f/')) {
+		return null
+	}
+	const segments = path.split('/').slice(1, -1)
+	return segments.length > 0 ? segments.join('/') : null
+}
+
+function withQuery(
+	path: string,
+	params: Record<string, string | number | undefined>
+): string {
+	const query = new URLSearchParams()
+	for (const [key, value] of Object.entries(params)) {
+		if (value === undefined) {
+			continue
+		}
+		query.set(key, String(value))
+	}
+	const suffix = query.toString()
+	return suffix ? `${path}?${suffix}` : path
+}
+
+async function expectOk(response: Response, context: string): Promise<void> {
+	if (response.ok) {
+		return
+	}
+	throw new Error(`${context} failed: ${response.status} ${response.statusText} - ${await response.text()}`)
+}
+
+function readStringField(
+	value: Record<string, unknown>,
+	field: string,
+	context: string
+): string {
+	const candidate = value[field]
+	if (typeof candidate === 'string' && candidate.length > 0) {
+		return candidate
+	}
+	throw new Error(`${context} is missing string field ${field}`)
+}
+
+function isConflictMessage(message: string): boolean {
+	const normalized = message.toLowerCase()
+	return normalized.includes('already exists') || normalized.includes('path conflict')
+}
+
+function isManagedSharedWorkspacePath(path: string): boolean {
+	return managedSharedWorkspacePrefixes.some((prefix) => path.startsWith(prefix))
+}

ai_evals/adapters/frontend/benchmarkRunner.ts

@@ -0,0 +1,142 @@
+import { loadSelectedCases } from "../../core/cases";
+import { resolveBackendValidationSettings } from "../../core/backendValidation";
+import {
+  formatRunModelLabel,
+  getFrontendEvalModel,
+  resolveEvalModel,
+} from "../../core/models";
+import { buildRunResult } from "../../core/results";
+import { runSuite } from "../../core/runSuite";
+import type { BenchmarkRunResult, ModeRunner } from "../../core/types";
+import { resolveWindmillBackendSettings } from "../../core/windmillBackendSettings";
+import { emitFrontendBenchmarkProgress } from "./progress";
+import { DEFAULT_JUDGE_MODEL } from "../../core/judge";
+
+export type FrontendBenchmarkMode = "flow" | "app" | "script" | "global";
+
+export async function runFrontendBenchmarkFromEnv(): Promise<BenchmarkRunResult> {
+  const mode = parseMode(process.env.WMILL_FRONTEND_AI_EVAL_MODE);
+  const caseIds = parseOptionalJsonStringArray(
+    process.env.WMILL_FRONTEND_AI_EVAL_CASE_IDS,
+  );
+  const runs = parsePositiveInteger(
+    process.env.WMILL_FRONTEND_AI_EVAL_RUNS,
+    "WMILL_FRONTEND_AI_EVAL_RUNS",
+  );
+  const emitProgress = process.env.WMILL_FRONTEND_AI_EVAL_PROGRESS === "1";
+  const verbose = process.env.WMILL_FRONTEND_AI_EVAL_VERBOSE === "1";
+  const executionOnly =
+    process.env.WMILL_FRONTEND_AI_EVAL_EXECUTION_ONLY === "1";
+  const judgeModel =
+    process.env.WMILL_FRONTEND_AI_EVAL_SKIP_JUDGE === "1" || executionOnly
+      ? null
+      : DEFAULT_JUDGE_MODEL;
+  const model = resolveEvalModel(
+    mode,
+    process.env.WMILL_FRONTEND_AI_EVAL_MODEL,
+  );
+  const backendValidation = resolveBackendValidationSettings({
+    evalMode: mode,
+    requestedMode: process.env.WMILL_FRONTEND_AI_EVAL_BACKEND_VALIDATION,
+  });
+  const backendSettings = resolveWindmillBackendSettings();
+
+  const selectedCases = await loadSelectedCases(mode, caseIds);
+  const modeRunner = await getModeRunner(
+    mode,
+    getFrontendEvalModel(model),
+    backendValidation,
+    backendSettings,
+  );
+  const runModel = formatRunModelLabel(mode, model);
+  const caseResults = await runSuite({
+    modeRunner,
+    cases: selectedCases,
+    runs,
+    runModel,
+    judgeModel,
+    executionOnly,
+    concurrency: verbose ? 1 : undefined,
+    verbose,
+    onProgress: emitProgress
+      ? (event) => emitFrontendBenchmarkProgress(event)
+      : undefined,
+  });
+
+  return buildRunResult({
+    mode,
+    runs,
+    runModel,
+    judgeModel,
+    caseResults,
+  });
+}
+
+async function getModeRunner(
+  mode: FrontendBenchmarkMode,
+  model: ReturnType<typeof getFrontendEvalModel>,
+  backendValidation: ReturnType<typeof resolveBackendValidationSettings>,
+  backendSettings: ReturnType<typeof resolveWindmillBackendSettings>,
+): Promise<ModeRunner<any, any, any>> {
+  switch (mode) {
+    case "flow": {
+      const { createFlowModeRunner } = await import("../../modes/flow");
+      return createFlowModeRunner(model, backendValidation, backendSettings);
+    }
+    case "app": {
+      const { createAppModeRunner } = await import("../../modes/app");
+      return createAppModeRunner(model, backendSettings);
+    }
+    case "script": {
+      const { createScriptModeRunner } = await import("../../modes/script");
+      return createScriptModeRunner(
+        model,
+        backendValidation,
+        backendSettings,
+      );
+    }
+    case "global": {
+      const { createGlobalModeRunner } = await import("../../modes/global");
+      return createGlobalModeRunner(model, backendSettings);
+    }
+  }
+}
+
+function parseMode(value: string | undefined): FrontendBenchmarkMode {
+  if (
+    value === "flow" ||
+    value === "app" ||
+    value === "script" ||
+    value === "global"
+  ) {
+    return value;
+  }
+  throw new Error(`Unsupported frontend benchmark mode: ${String(value)}`);
+}
+
+function parseOptionalJsonStringArray(value: string | undefined): string[] {
+  if (!value) {
+    return [];
+  }
+  const parsed = JSON.parse(value) as unknown;
+  if (
+    !Array.isArray(parsed) ||
+    parsed.some((entry) => typeof entry !== "string")
+  ) {
+    throw new Error(
+      "WMILL_FRONTEND_AI_EVAL_CASE_IDS must be a JSON string array",
+    );
+  }
+  return parsed;
+}
+
+function parsePositiveInteger(
+  value: string | undefined,
+  envName: string,
+): number {
+  const parsed = Number(value);
+  if (!Number.isInteger(parsed) || parsed <= 0) {
+    throw new Error(`${envName} must be a positive integer`);
+  }
+  return parsed;
+}

ai_evals/adapters/frontend/core/app/appEvalRunner.ts

@@ -0,0 +1,179 @@
+import { mkdtemp } from "fs/promises";
+import { tmpdir } from "os";
+import { join } from "path";
+import type {
+  BackendRunnable,
+  AppAIChatHelpers,
+  DataTableSchema,
+} from "../../../../../frontend/src/lib/components/copilot/chat/app/core";
+import {
+  getAppTools,
+  prepareAppSystemMessage,
+  prepareAppUserMessage,
+} from "../../../../../frontend/src/lib/components/copilot/chat/app/core";
+import type { Tool as ProductionTool } from "../../../../../frontend/src/lib/components/copilot/chat/shared";
+import { createAppFileHelpers, type AppEvalChatHelpers } from "./fileHelpers";
+import { runEval } from "../shared";
+import type { AIProvider } from "$lib/gen/types.gen";
+import type {
+  EvalCaseRuntimeAppAdditionalContext,
+  EvalCaseRuntimeAppContextSpec,
+  ModeRunContext,
+} from "../../../../core/types";
+import type { TokenUsage } from "../shared/types";
+import type { AppFilesState } from "../../../../core/validators";
+import type { WindmillBackendSettings } from "../../../../core/windmillBackendSettings";
+import {
+  createAppBackendRunnableContextElement,
+  createAppDatatableContextElement,
+  createAppFrontendFileContextElement,
+  type ContextElement,
+} from "../../../../../frontend/src/lib/components/copilot/chat/context";
+
+export interface AppEvalResult {
+  success: boolean;
+  files: AppFilesState;
+  error?: string;
+  assistantMessageCount: number;
+  toolCallCount: number;
+  toolsUsed: string[];
+  tokenUsage: TokenUsage;
+  finalContextTokens: number | null;
+}
+
+export interface AppEvalOptions {
+  initialFrontend?: Record<string, string>;
+  initialBackend?: AppFilesState["backend"];
+  initialDatatables?: AppFilesState["datatables"];
+  appContext?: EvalCaseRuntimeAppContextSpec;
+  model?: string;
+  maxIterations?: number;
+  provider?: AIProvider;
+  backend: WindmillBackendSettings;
+  workspaceRoot?: string;
+  runContext?: ModeRunContext;
+}
+
+export async function runAppEval(
+  userPrompt: string,
+  apiKey: string,
+  options: AppEvalOptions,
+): Promise<AppEvalResult> {
+  const workspaceRoot =
+    options?.workspaceRoot ??
+    (await mkdtemp(join(tmpdir(), "wmill-frontend-app-benchmark-")));
+  const { helpers, getEvalState, cleanup } = await createAppFileHelpers(
+    options?.initialFrontend ?? {},
+    (options?.initialBackend ?? {}) as Record<string, BackendRunnable>,
+    options?.initialDatatables ?? [],
+    workspaceRoot,
+  );
+
+  try {
+    const systemMessage = prepareAppSystemMessage();
+    const tools = getAppTools() as ProductionTool<AppAIChatHelpers>[];
+    const model = options?.model ?? "claude-haiku-4-5-20251001";
+    const additionalContext = await buildAdditionalContext(
+      options?.appContext,
+      helpers,
+    );
+    const userMessage = prepareAppUserMessage(
+      userPrompt,
+      helpers.getSelectedContext(),
+      additionalContext,
+    );
+
+    const rawResult = await runEval({
+      userPrompt,
+      systemMessage,
+      userMessage,
+      tools,
+      helpers,
+      apiKey,
+      getOutput: getEvalState,
+      onAssistantMessageStart: options?.runContext?.onAssistantMessageStart,
+      onAssistantToken: options?.runContext?.onAssistantChunk,
+      onAssistantMessageEnd: options?.runContext?.onAssistantMessageEnd,
+      onToolCall: options?.runContext?.onToolCall,
+      options: {
+        maxIterations: options?.maxIterations,
+        model,
+        workspace: workspaceRoot,
+        provider: options?.provider,
+        backend: options.backend,
+        caseId: options?.runContext?.caseId,
+        attempt: options?.runContext?.attempt,
+      },
+    });
+
+    return {
+      files: rawResult.output,
+      success: rawResult.success,
+      error: rawResult.error,
+      assistantMessageCount: rawResult.iterations,
+      toolCallCount: rawResult.toolCallsCount,
+      toolsUsed: rawResult.toolsCalled,
+      tokenUsage: rawResult.tokenUsage,
+      finalContextTokens: rawResult.finalContextTokens,
+    };
+  } finally {
+    await cleanup();
+  }
+}
+
+async function buildAdditionalContext(
+  appContext: EvalCaseRuntimeAppContextSpec | undefined,
+  helpers: AppEvalChatHelpers,
+): Promise<ContextElement[]> {
+  const entries = appContext?.additional ?? [];
+  if (entries.length === 0) {
+    return [];
+  }
+
+  const datatables = entries.some((entry) => entry.type === "datatable")
+    ? await helpers.getDatatables()
+    : [];
+
+  return entries.map((entry) =>
+    buildAdditionalContextElement(entry, helpers, datatables),
+  );
+}
+
+function buildAdditionalContextElement(
+  entry: EvalCaseRuntimeAppAdditionalContext,
+  helpers: AppAIChatHelpers,
+  datatables: DataTableSchema[],
+): ContextElement {
+  if (entry.type === "frontend") {
+    const content = helpers.getFrontendFile(entry.path);
+    if (content === undefined) {
+      throw new Error(`App eval @ frontend context not found: ${entry.path}`);
+    }
+    return createAppFrontendFileContextElement(entry.path, content);
+  }
+
+  if (entry.type === "backend") {
+    const runnable = helpers.getBackendRunnable(entry.key);
+    if (!runnable) {
+      throw new Error(`App eval @ backend context not found: ${entry.key}`);
+    }
+    return createAppBackendRunnableContextElement(entry.key, runnable);
+  }
+
+  const datatable = datatables.find(
+    (candidate) => candidate.datatable_name === entry.datatableName,
+  );
+  const columns = datatable?.schemas?.[entry.schema]?.[entry.table];
+  if (!columns) {
+    throw new Error(
+      `App eval @ datatable context not found: ${entry.datatableName}/${entry.schema}.${entry.table}`,
+    );
+  }
+
+  return createAppDatatableContextElement(
+    entry.datatableName,
+    entry.schema,
+    entry.table,
+    columns,
+  );
+}

ai_evals/adapters/frontend/core/app/appFixtureLoader.test.ts

@@ -0,0 +1,30 @@
+import { describe, expect, it } from "bun:test";
+import { fileURLToPath } from "node:url";
+import { loadAppFixture } from "./appFixtureLoader";
+
+const RECIPE_BOOK_FIXTURE = fileURLToPath(
+  new URL("../../../../fixtures/frontend/app/initial/recipe_book", import.meta.url)
+);
+
+describe("loadAppFixture", () => {
+  it("loads datatables from app fixtures when present", async () => {
+    const fixture = await loadAppFixture(RECIPE_BOOK_FIXTURE);
+
+    expect(fixture.datatables).toEqual([
+      {
+        datatable_name: "main",
+        schemas: {
+          public: {
+            recipes: {
+              id: "int4",
+              name: "text",
+              ingredients: "text",
+              instructions: "text",
+              created_at: "timestamp=now()",
+            },
+          },
+        },
+      },
+    ]);
+  });
+});

ai_evals/adapters/frontend/core/app/appFixtureLoader.ts

@@ -0,0 +1,208 @@
+import type {
+  BackendRunnable,
+  DataTableSchema,
+  InlineScript,
+} from "../../../../../frontend/src/lib/components/copilot/chat/app/core";
+import type { AppFilesState } from "../../../../core/validators";
+
+/**
+ * Backend runnable metadata stored in meta.json files.
+ */
+interface BackendMeta {
+  name: string;
+  language: "bun" | "python3";
+}
+
+/**
+ * Recursively reads all files in a directory and returns them as a record.
+ * File paths are relative to the base directory with a leading '/'.
+ */
+async function readFilesRecursively(
+  dir: string,
+  basePath: string = "",
+): Promise<Record<string, string>> {
+  // @ts-ignore - Node.js fs/promises
+  const { readdir, readFile } = await import("fs/promises");
+  // @ts-ignore - Node.js path
+  const { join } = await import("path");
+
+  const result: Record<string, string> = {};
+  const entries = await readdir(dir, { withFileTypes: true });
+
+  for (const entry of entries) {
+    const fullPath = join(dir, entry.name);
+    const relativePath = basePath
+      ? `${basePath}/${entry.name}`
+      : `/${entry.name}`;
+
+    if (entry.isDirectory()) {
+      const subFiles = await readFilesRecursively(fullPath, relativePath);
+      Object.assign(result, subFiles);
+    } else {
+      const content = await readFile(fullPath, "utf-8");
+      result[relativePath] = content;
+    }
+  }
+
+  return result;
+}
+
+/**
+ * Loads frontend files from a directory.
+ * All files are read recursively and paths become keys with leading '/'.
+ */
+async function loadFrontend(
+  frontendPath: string,
+): Promise<Record<string, string>> {
+  // @ts-ignore - Node.js fs/promises
+  const { access } = await import("fs/promises");
+
+  try {
+    await access(frontendPath);
+  } catch {
+    // Directory doesn't exist, return empty
+    return {};
+  }
+
+  return readFilesRecursively(frontendPath);
+}
+
+/**
+ * Loads backend runnables from a directory.
+ * Each subdirectory is a runnable with:
+ * - main.ts or main.py: The code content
+ * - meta.json: Metadata { name, language }
+ */
+async function loadBackend(
+  backendPath: string,
+): Promise<Record<string, BackendRunnable>> {
+  // @ts-ignore - Node.js fs/promises
+  const { readdir, readFile, access } = await import("fs/promises");
+  // @ts-ignore - Node.js path
+  const { join } = await import("path");
+
+  try {
+    await access(backendPath);
+  } catch {
+    // Directory doesn't exist, return empty
+    return {};
+  }
+
+  const result: Record<string, BackendRunnable> = {};
+  const entries = await readdir(backendPath, { withFileTypes: true });
+
+  for (const entry of entries) {
+    if (!entry.isDirectory()) continue;
+
+    const runnableKey = entry.name;
+    const runnablePath = join(backendPath, entry.name);
+
+    // Read meta.json
+    const metaPath = join(runnablePath, "meta.json");
+    let meta: BackendMeta;
+    try {
+      const metaContent = await readFile(metaPath, "utf-8");
+      meta = JSON.parse(metaContent);
+    } catch {
+      console.warn(
+        `Missing or invalid meta.json for runnable '${runnableKey}', skipping`,
+      );
+      continue;
+    }
+
+    // Find and read the main file (main.ts or main.py)
+    const runnableFiles = await readdir(runnablePath);
+    const mainFile = runnableFiles.find(
+      (f) => f === "main.ts" || f === "main.py",
+    );
+
+    if (!mainFile) {
+      console.warn(
+        `No main.ts or main.py found for runnable '${runnableKey}', skipping`,
+      );
+      continue;
+    }
+
+    const content = await readFile(join(runnablePath, mainFile), "utf-8");
+
+    const inlineScript: InlineScript = {
+      language: meta.language,
+      content,
+    };
+
+    result[runnableKey] = {
+      name: meta.name,
+      type: "inline",
+      inlineScript,
+    };
+  }
+
+  return result;
+}
+
+async function loadDatatables(fixturePath: string): Promise<DataTableSchema[]> {
+  // @ts-ignore - Node.js fs/promises
+  const { readFile } = await import("fs/promises");
+  // @ts-ignore - Node.js path
+  const { join } = await import("path");
+
+  try {
+    const content = await readFile(
+      join(fixturePath, "datatables.json"),
+      "utf-8",
+    );
+    const parsed = JSON.parse(content);
+    return Array.isArray(parsed) ? (parsed as DataTableSchema[]) : [];
+  } catch {
+    return [];
+  }
+}
+
+/**
+ * Loads an app fixture from a directory structure.
+ *
+ * Expected structure:
+ * ```
+ * fixturePath/
+ * ├── frontend/
+ * │   └── index.tsx           # → frontend["/index.tsx"]
+ * │   └── components/
+ * │       └── Button.tsx      # → frontend["/components/Button.tsx"]
+ * └── backend/
+ *     └── incrementCounter/
+ *         ├── main.ts         # The code content
+ *         └── meta.json       # { "name": "...", "language": "bun" }
+ * ```
+ *
+ * @param fixturePath - Path to the fixture directory
+ * @returns AppFiles object with frontend and backend
+ */
+export async function loadAppFixture(
+  fixturePath: string,
+): Promise<AppFilesState> {
+  // @ts-ignore - Node.js path
+  const { join } = await import("path");
+
+  const frontend = await loadFrontend(join(fixturePath, "frontend"));
+  const backend = await loadBackend(join(fixturePath, "backend"));
+  const datatables = await loadDatatables(fixturePath);
+
+  return { frontend, backend, datatables };
+}
+
+/**
+ * Loads an app fixture and returns the separate frontend and backend objects.
+ * Convenience function for use with runAppEval options.
+ */
+export async function loadAppFixtureForEval(fixturePath: string): Promise<{
+  initialFrontend: Record<string, string>;
+  initialBackend: AppFilesState["backend"];
+  initialDatatables: DataTableSchema[];
+}> {
+  const { frontend, backend, datatables } = await loadAppFixture(fixturePath);
+  return {
+    initialFrontend: frontend,
+    initialBackend: backend,
+    initialDatatables: datatables,
+  };
+}

ai_evals/adapters/frontend/core/app/fileHelpers.test.ts

@@ -0,0 +1,40 @@
+import { describe, expect, it } from "bun:test";
+import { createAppFileHelpers } from "./fileHelpers";
+
+describe("createAppFileHelpers", () => {
+  it("exposes generated wmill typings and returns real lint diagnostics", async () => {
+    const { helpers, cleanup } = await createAppFileHelpers(
+      {
+        "/index.tsx":
+          "import { backend } from 'wmill'\nexport default function App() { void backend.listRecipes(); return <div /> }\n",
+      },
+      {
+        listRecipes: {
+          name: "List recipes",
+          type: "inline",
+          inlineScript: {
+            language: "bun",
+            content: "export async function main() { return [] }\n",
+          },
+        },
+      }
+    );
+
+    try {
+      expect(helpers.listFrontendFiles()).toContain("/wmill.d.ts");
+      expect(helpers.getFrontendFile("/wmill.d.ts")).toContain("listRecipes");
+
+      const lintResult = helpers.setFrontendFile(
+        "/index.tsx",
+        "import { backend } from 'wmill'\nexport default function App() { void backend.deleteRecipe({ id: 1 }); return <div /> }\n"
+      );
+
+      expect(lintResult.errorCount).toBeGreaterThan(0);
+      expect(lintResult.errors.frontend["/index.tsx"]?.join("\n")).toContain(
+        "Property 'deleteRecipe' does not exist"
+      );
+    } finally {
+      await cleanup();
+    }
+  });
+});

ai_evals/adapters/frontend/core/app/fileHelpers.ts

@@ -0,0 +1,329 @@
+import { mkdir, rm, writeFile } from 'fs/promises'
+import { dirname, join } from 'path'
+import type {
+	AppAIChatHelpers,
+	AppDatatableMetadata,
+	AppFiles,
+	BackendRunnable,
+	DataTableSchema,
+	LintResult,
+	SelectedContext
+} from '../../../../../frontend/src/lib/components/copilot/chat/app/core'
+import { buildAppWmillTypes, collectAppDiagnostics } from '../../../../core/appDiagnostics'
+
+export interface AppEvalChatHelpers extends AppAIChatHelpers {
+	getDatatables: () => Promise<DataTableSchema[]>
+}
+
+async function writeFrontendFile(
+	workspaceRoot: string | undefined,
+	path: string,
+	content: string
+): Promise<void> {
+	if (!workspaceRoot) {
+		return
+	}
+	const relativePath = path.startsWith('/') ? path.slice(1) : path
+	const fullPath = join(workspaceRoot, 'frontend', relativePath)
+	await mkdir(dirname(fullPath), { recursive: true })
+	await writeFile(fullPath, content, 'utf8')
+}
+
+async function removeFrontendFile(workspaceRoot: string | undefined, path: string): Promise<void> {
+	if (!workspaceRoot) {
+		return
+	}
+	const relativePath = path.startsWith('/') ? path.slice(1) : path
+	await rm(join(workspaceRoot, 'frontend', relativePath), { force: true })
+}
+
+async function writeBackendRunnable(
+	workspaceRoot: string | undefined,
+	key: string,
+	runnable: BackendRunnable
+): Promise<void> {
+	if (!workspaceRoot) {
+		return
+	}
+	const runnableDir = join(workspaceRoot, 'backend', key)
+	await mkdir(runnableDir, { recursive: true })
+
+	const meta: { name: string; language?: string; type?: string; path?: string } = {
+		name: runnable.name
+	}
+
+	if (runnable.type === 'inline' && runnable.inlineScript) {
+		meta.language = runnable.inlineScript.language
+		const extension = runnable.inlineScript.language === 'python3' ? 'py' : 'ts'
+		await writeFile(
+			join(runnableDir, `main.${extension}`),
+			runnable.inlineScript.content,
+			'utf8'
+		)
+	} else {
+		meta.type = runnable.type
+		if (runnable.path) {
+			meta.path = runnable.path
+		}
+	}
+
+	await writeFile(join(runnableDir, 'meta.json'), JSON.stringify(meta, null, 2) + '\n', 'utf8')
+}
+
+async function removeBackendRunnable(workspaceRoot: string | undefined, key: string): Promise<void> {
+	if (!workspaceRoot) {
+		return
+	}
+	await rm(join(workspaceRoot, 'backend', key), { recursive: true, force: true })
+}
+
+async function persistDatatables(
+	workspaceRoot: string | undefined,
+	datatables: DataTableSchema[]
+): Promise<void> {
+	if (!workspaceRoot) {
+		return
+	}
+	await writeFile(
+		join(workspaceRoot, 'datatables.json'),
+		JSON.stringify(datatables, null, 2) + '\n',
+		'utf8'
+	)
+}
+
+export async function createAppFileHelpers(
+	initialFrontend: Record<string, string> = {},
+	initialBackend: Record<string, BackendRunnable> = {},
+	initialDatatables: DataTableSchema[] = [],
+	workspaceRoot?: string
+): Promise<{
+	helpers: AppEvalChatHelpers
+	getFiles: () => AppFiles
+	getEvalState: () => {
+		frontend: Record<string, string>
+		backend: Record<string, BackendRunnable>
+		datatables: DataTableSchema[]
+	}
+	getFrontend: () => Record<string, string>
+	getBackend: () => Record<string, BackendRunnable>
+	getDatatables: () => DataTableSchema[]
+	cleanup: () => Promise<void>
+	workspaceDir: string | null
+}> {
+	let frontend = { ...initialFrontend }
+	let backend = { ...initialBackend }
+	let snapshotId = 0
+	const snapshots = new Map<
+		number,
+		{
+			frontend: Record<string, string>
+			backend: Record<string, BackendRunnable>
+			datatables: DataTableSchema[]
+		}
+	>()
+	const datatables: DataTableSchema[] = structuredClone(initialDatatables)
+
+	function lint(): LintResult {
+		return collectAppDiagnostics({
+			frontend,
+			backend
+		}).lintResult
+	}
+
+	function getGeneratedWmillTypes(): string {
+		return buildAppWmillTypes(backend)
+	}
+
+	for (const [path, content] of Object.entries(frontend)) {
+		await writeFrontendFile(workspaceRoot, path, content)
+	}
+	for (const [key, runnable] of Object.entries(backend)) {
+		await writeBackendRunnable(workspaceRoot, key, runnable)
+	}
+	await persistDatatables(workspaceRoot, datatables)
+
+	const helpers: AppEvalChatHelpers = {
+		listFrontendFiles: () => [
+			...Object.keys(frontend).filter((path) => path !== '/wmill.d.ts'),
+			'/wmill.d.ts'
+		],
+		getFrontendFile: (path: string) => {
+			if (path === '/wmill.d.ts') {
+				return getGeneratedWmillTypes()
+			}
+			return frontend[path]
+		},
+		getFrontendFiles: () => ({
+			...Object.fromEntries(
+				Object.entries(frontend).filter(([path]) => path !== '/wmill.d.ts')
+			),
+			'/wmill.d.ts': getGeneratedWmillTypes()
+		}),
+		setFrontendFile: (path: string, content: string) => {
+			if (path === '/wmill.d.ts') {
+				return lint()
+			}
+			frontend[path] = content
+			void writeFrontendFile(workspaceRoot, path, content)
+			return lint()
+		},
+		deleteFrontendFile: (path: string) => {
+			if (path === '/wmill.d.ts') {
+				return
+			}
+			delete frontend[path]
+			void removeFrontendFile(workspaceRoot, path)
+		},
+		listBackendRunnables: () =>
+			Object.entries(backend).map(([key, runnable]) => ({
+				key,
+				name: runnable.name
+			})),
+		getBackendRunnable: (key: string) => backend[key],
+		getBackendRunnables: () => ({ ...backend }),
+		setBackendRunnable: async (key: string, runnable: BackendRunnable) => {
+			backend[key] = runnable
+			await writeBackendRunnable(workspaceRoot, key, runnable)
+			return lint()
+		},
+		deleteBackendRunnable: (key: string) => {
+			delete backend[key]
+			void removeBackendRunnable(workspaceRoot, key)
+		},
+		getFiles: (): AppFiles => ({
+			frontend: { ...frontend },
+			backend: { ...backend }
+		}),
+		getSelectedContext: (): SelectedContext => ({}),
+		snapshot: () => {
+			const id = ++snapshotId
+			snapshots.set(id, {
+				frontend: { ...frontend },
+				backend: { ...backend },
+				datatables: structuredClone(datatables)
+			})
+			return id
+		},
+		revertToSnapshot: (id: number) => {
+			const snapshot = snapshots.get(id)
+			if (!snapshot) {
+				return
+			}
+			frontend = { ...snapshot.frontend }
+			backend = { ...snapshot.backend }
+			datatables.splice(0, datatables.length, ...structuredClone(snapshot.datatables))
+			void syncWorkspace()
+		},
+		lint,
+		getDatatables: async () => structuredClone(datatables),
+		listDatatableTables: async () =>
+			datatables.map(
+				(datatable): AppDatatableMetadata => {
+					const schemas = Object.fromEntries(
+						Object.entries(datatable.schemas).map(([schemaName, tables]) => [
+							schemaName,
+							Object.keys(tables)
+						])
+					)
+					return {
+						datatable_name: datatable.datatable_name,
+						schemas,
+						tableCount: Object.values(schemas).reduce(
+							(sum, tableNames) => sum + tableNames.length,
+							0
+						),
+						error: datatable.error
+					}
+				}
+			),
+		getDatatableTableSchema: async (
+			datatableName: string,
+			schemaName: string,
+			tableName: string
+		) => {
+			const datatable = datatables.find((entry) => entry.datatable_name === datatableName)
+			return structuredClone(datatable?.schemas?.[schemaName]?.[tableName] ?? {})
+		},
+		getAvailableDatatableNames: () => datatables.map((datatable) => datatable.datatable_name),
+		execDatatableSql: async (
+			datatableName: string,
+			sql: string,
+			newTable?: { schema: string; name: string }
+		) => {
+			if (newTable) {
+				datatables.push({
+					datatable_name: datatableName,
+					schemas: {
+						[newTable.schema]: {
+							[newTable.name]: {}
+						}
+					}
+				})
+				await persistDatatables(workspaceRoot, datatables)
+			}
+			return {
+				success: true,
+				result: [
+					{
+						datatableName,
+						sql
+					}
+				]
+			}
+		},
+		addTableToWhitelist: (datatableName: string, schemaName: string, tableName: string) => {
+			const existing = datatables.find((entry) => entry.datatable_name === datatableName)
+			if (existing) {
+				existing.schemas[schemaName] ??= {}
+				existing.schemas[schemaName][tableName] ??= {}
+			} else {
+				datatables.push({
+					datatable_name: datatableName,
+					schemas: {
+						[schemaName]: {
+							[tableName]: {}
+						}
+					}
+				})
+			}
+			void persistDatatables(workspaceRoot, datatables)
+		}
+	}
+
+	async function syncWorkspace(): Promise<void> {
+		if (!workspaceRoot) {
+			return
+		}
+		await rm(join(workspaceRoot, 'frontend'), { recursive: true, force: true })
+		await rm(join(workspaceRoot, 'backend'), { recursive: true, force: true })
+		for (const [path, content] of Object.entries(frontend)) {
+			await writeFrontendFile(workspaceRoot, path, content)
+		}
+		for (const [key, runnable] of Object.entries(backend)) {
+			await writeBackendRunnable(workspaceRoot, key, runnable)
+		}
+		await persistDatatables(workspaceRoot, datatables)
+	}
+
+	return {
+		helpers,
+		getFiles: () => ({
+			frontend: { ...frontend },
+			backend: { ...backend }
+		}),
+		getEvalState: () => ({
+			frontend: { ...frontend },
+			backend: { ...backend },
+			datatables: structuredClone(datatables)
+		}),
+		getFrontend: () => ({ ...frontend }),
+		getBackend: () => ({ ...backend }),
+		getDatatables: () => structuredClone(datatables),
+		cleanup: async () => {
+			if (workspaceRoot) {
+				await rm(workspaceRoot, { recursive: true, force: true })
+			}
+		},
+		workspaceDir: workspaceRoot ?? null
+	}
+}

ai_evals/adapters/frontend/core/flow/fileHelpers.ts

@@ -0,0 +1,175 @@
+import { mkdir, rm, writeFile } from 'fs/promises'
+import { dirname, join } from 'path'
+import type { FlowModule, InputTransform } from '../../../../../frontend/src/lib/gen'
+import type { ExtendedOpenFlow } from '../../../../../frontend/src/lib/components/flows/types'
+import type { FlowAIChatHelpers } from '../../../../../frontend/src/lib/components/copilot/chat/flow/core'
+import type { ScriptLintResult } from '../../../../../frontend/src/lib/components/copilot/chat/shared'
+import type { WorkspaceMutationTarget } from '../../../../../frontend/src/lib/components/copilot/chat/workspaceTools'
+import {
+	createInlineScriptSession
+} from '../../../../../frontend/src/lib/components/copilot/chat/flow/inlineScriptsUtils'
+import {
+	applyFlowJsonUpdate,
+	updateRawScriptModuleContent
+} from '../../../../../frontend/src/lib/components/copilot/chat/flow/helperUtils'
+import {
+	registerBenchmarkWorkspace,
+	registerBenchmarkWorkspaceRunnables,
+	unregisterBenchmarkWorkspaceRunnables,
+	createBenchmarkCompletedJob,
+	type BenchmarkWorkspaceFlow,
+	type BenchmarkWorkspaceScript
+} from '../../mockBackend'
+
+const EMPTY_SCRIPT_LINT_RESULT: ScriptLintResult = {
+	errorCount: 0,
+	warningCount: 0,
+	errors: [],
+	warnings: []
+}
+
+export interface FlowWorkspaceFixtures {
+	scripts?: BenchmarkWorkspaceScript[]
+	flows?: BenchmarkWorkspaceFlow[]
+}
+
+export async function createFlowFileHelpers(
+	initialModules: FlowModule[] = [],
+	initialSchema?: Record<string, any>,
+	initialPreprocessorModule?: FlowModule,
+	initialFailureModule?: FlowModule,
+	workspaceRoot?: string,
+	workspaceFixtures?: FlowWorkspaceFixtures,
+	currentFlowPath?: string
+): Promise<{
+	helpers: FlowAIChatHelpers
+	getFlow: () => ExtendedOpenFlow
+	getModules: () => FlowModule[]
+	cleanup: () => Promise<void>
+	workspaceDir: string | null
+}> {
+	let flow: ExtendedOpenFlow = {
+		value: {
+			modules: structuredClone(initialModules),
+			preprocessor_module: structuredClone(initialPreprocessorModule),
+			failure_module: structuredClone(initialFailureModule)
+		},
+		summary: '',
+		schema: initialSchema ?? {
+			$schema: 'https://json-schema.org/draft/2020-12/schema',
+			properties: {},
+			required: [],
+			type: 'object'
+		}
+	}
+	const inlineScriptSession = createInlineScriptSession()
+
+	const flowFilePath = workspaceRoot ? join(workspaceRoot, 'flow.json') : null
+
+	async function persistFlow(): Promise<void> {
+		if (!flowFilePath) {
+			return
+		}
+		await mkdir(dirname(flowFilePath), { recursive: true })
+		await writeFile(flowFilePath, JSON.stringify(flow, null, 2) + '\n', 'utf8')
+	}
+
+	await persistFlow()
+
+	if (workspaceRoot) {
+		registerBenchmarkWorkspace(workspaceRoot)
+		if (workspaceFixtures) {
+			registerBenchmarkWorkspaceRunnables(workspaceRoot, workspaceFixtures)
+		}
+	}
+
+	const setFlowJson: FlowAIChatHelpers['setFlowJson'] = async ({
+		modules,
+		schema,
+		preprocessorModule,
+		failureModule
+	}) => {
+		const result = applyFlowJsonUpdate(flow, inlineScriptSession, {
+			modules,
+			schema,
+			preprocessorModule,
+			failureModule
+		})
+		await persistFlow()
+		return result
+	}
+
+	const helpers: FlowAIChatHelpers & {
+		getWorkspaceMutationTarget: () => WorkspaceMutationTarget
+	} = {
+		getFlowAndSelectedId: () => ({ flow, selectedId: '' }),
+		getRootModules: () => flow.value.modules,
+		inlineScriptSession,
+		getWorkspaceMutationTarget: () => ({
+			kind: 'flow',
+			path: currentFlowPath,
+			deployed: Boolean(currentFlowPath)
+		}),
+		setSnapshot: () => {},
+		revertToSnapshot: () => {},
+		setCode: async (id: string, code: string) => {
+			updateRawScriptModuleContent(flow, id, code)
+			inlineScriptSession.set(id, code)
+			await persistFlow()
+		},
+		setFlowJson,
+		getFlowInputsSchema: async () => flow.schema ?? {},
+		updateExprsToSet: (_id: string, _inputTransforms: Record<string, InputTransform>) => {},
+		acceptAllModuleActions: () => {},
+		rejectAllModuleActions: () => {},
+		hasPendingChanges: () => false,
+		selectStep: (_id: string) => {},
+		testFlow: async (args?: Record<string, any>) => {
+			if (workspaceRoot) {
+				const runPath = join(workspaceRoot, 'test-run.json')
+				await writeFile(
+					runPath,
+					JSON.stringify(
+						{
+							requestedArgs: args ?? {},
+							modules: flow.value.modules.map((module) => module.id),
+							preprocessor_module: flow.value.preprocessor_module?.id ?? null,
+							failure_module: flow.value.failure_module?.id ?? null
+						},
+						null,
+						2
+					) + '\n',
+					'utf8'
+				)
+			}
+			return createBenchmarkCompletedJob({
+				workspace: workspaceRoot ?? 'benchmark',
+				jobKind: 'flowpreview',
+				result: {
+					requestedArgs: args ?? {},
+					modules: flow.value.modules.map((module) => module.id),
+					preprocessor_module: flow.value.preprocessor_module?.id ?? null,
+					failure_module: flow.value.failure_module?.id ?? null,
+					mocked: true
+				},
+				logs: 'Mock benchmark flow test run completed successfully.'
+			})
+		},
+		getLintErrors: async () => EMPTY_SCRIPT_LINT_RESULT
+	}
+
+	return {
+		helpers,
+		getFlow: () => flow,
+		getModules: () => flow.value.modules,
+		cleanup: async () => {
+			if (workspaceRoot) {
+				unregisterBenchmarkWorkspaceRunnables(workspaceRoot)
+			}
+			if (workspaceRoot) {
+				await rm(workspaceRoot, { recursive: true, force: true })
+			}
+		},
+		workspaceDir: workspaceRoot ?? null
+	}
+}

ai_evals/adapters/frontend/core/flow/flowEvalRunner.ts

@@ -0,0 +1,122 @@
+import { mkdtemp } from "fs/promises";
+import { tmpdir } from "os";
+import { join } from "path";
+import type { FlowModule } from "$lib/gen";
+import type { AIProvider } from "$lib/gen/types.gen";
+import type { ExtendedOpenFlow } from "$lib/components/flows/types";
+import {
+  flowTools,
+  prepareFlowSystemMessage,
+  prepareFlowUserMessage,
+  type FlowAIChatHelpers,
+} from "../../../../../frontend/src/lib/components/copilot/chat/flow/core";
+import type { Tool as ProductionTool } from "../../../../../frontend/src/lib/components/copilot/chat/shared";
+import {
+  createFlowFileHelpers,
+  type FlowWorkspaceFixtures,
+} from "./fileHelpers";
+import { runEval } from "../shared";
+import type { ModeRunContext } from "../../../../core/types";
+import type { TokenUsage, ToolCallDetail } from "../shared/types";
+import type { WindmillBackendSettings } from "../../../../core/windmillBackendSettings";
+
+export interface FlowFixture {
+  path?: string;
+  value?: {
+    modules?: FlowModule[];
+    preprocessor_module?: FlowModule;
+    failure_module?: FlowModule;
+  };
+  schema?: Record<string, unknown>;
+}
+
+export interface FlowEvalResult {
+  success: boolean;
+  flow: ExtendedOpenFlow;
+  error?: string;
+  assistantMessageCount: number;
+  toolCallCount: number;
+  toolsUsed: string[];
+  toolCallDetails: ToolCallDetail[];
+  tokenUsage: TokenUsage;
+  finalContextTokens: number | null;
+}
+
+export interface FlowEvalOptions {
+  initialFlow?: FlowFixture;
+  workspaceFixtures?: FlowWorkspaceFixtures;
+  model?: string;
+  maxIterations?: number;
+  provider?: AIProvider;
+  backend: WindmillBackendSettings;
+  workspaceRoot?: string;
+  runContext?: ModeRunContext;
+}
+
+export async function runFlowEval(
+  userPrompt: string,
+  apiKey: string,
+  options: FlowEvalOptions,
+): Promise<FlowEvalResult> {
+  const workspaceRoot =
+    options?.workspaceRoot ??
+    (await mkdtemp(join(tmpdir(), "wmill-frontend-flow-benchmark-")));
+  const { helpers, getFlow, cleanup } = await createFlowFileHelpers(
+    options?.initialFlow?.value?.modules ?? [],
+    options?.initialFlow?.schema,
+    options?.initialFlow?.value?.preprocessor_module,
+    options?.initialFlow?.value?.failure_module,
+    workspaceRoot,
+    options?.workspaceFixtures,
+    options?.initialFlow?.path,
+  );
+
+  try {
+    const systemMessage = prepareFlowSystemMessage();
+    const tools = flowTools as ProductionTool<FlowAIChatHelpers>[];
+    const model = options?.model ?? "claude-haiku-4-5-20251001";
+    const userMessage = prepareFlowUserMessage(
+      userPrompt,
+      helpers.getFlowAndSelectedId(),
+      [],
+      helpers.inlineScriptSession,
+    );
+
+    const rawResult = await runEval({
+      userPrompt,
+      systemMessage,
+      userMessage,
+      tools,
+      helpers,
+      apiKey,
+      getOutput: getFlow,
+      onAssistantMessageStart: options?.runContext?.onAssistantMessageStart,
+      onAssistantToken: options?.runContext?.onAssistantChunk,
+      onAssistantMessageEnd: options?.runContext?.onAssistantMessageEnd,
+      onToolCall: options?.runContext?.onToolCall,
+      options: {
+        maxIterations: options?.maxIterations,
+        model,
+        workspace: workspaceRoot,
+        provider: options?.provider,
+        backend: options.backend,
+        caseId: options?.runContext?.caseId,
+        attempt: options?.runContext?.attempt,
+      },
+    });
+
+    return {
+      flow: rawResult.output,
+      success: rawResult.success,
+      error: rawResult.error,
+      assistantMessageCount: rawResult.iterations,
+      toolCallCount: rawResult.toolCallsCount,
+      toolsUsed: rawResult.toolsCalled,
+      toolCallDetails: rawResult.toolCallDetails,
+      tokenUsage: rawResult.tokenUsage,
+      finalContextTokens: rawResult.finalContextTokens,
+    };
+  } finally {
+    await cleanup();
+  }
+}

ai_evals/adapters/frontend/core/global/globalEvalRunner.ts

@@ -0,0 +1,241 @@
+import { mkdtemp, rm } from "fs/promises";
+import { tmpdir } from "os";
+import { join } from "path";
+import type { AIProvider } from "$lib/gen/types.gen";
+import {
+  globalTools,
+  prepareGlobalSystemMessage,
+  prepareGlobalUserMessage,
+} from "../../../../../frontend/src/lib/components/copilot/chat/global/core";
+import {
+  clearGlobalDrafts,
+  getGlobalDraft,
+  listGlobalDrafts,
+} from "../../../../../frontend/src/lib/components/copilot/chat/global/userDraftAdapter";
+import type { Tool as ProductionTool } from "../../../../../frontend/src/lib/components/copilot/chat/shared";
+import { UserDraft } from "../../../../../frontend/src/lib/userDraft.svelte";
+import type { ModeRunContext } from "../../../../core/types";
+import type { GlobalDraftState } from "../../../../core/validators";
+import type { WindmillBackendSettings } from "../../../../core/windmillBackendSettings";
+import {
+  registerBenchmarkWorkspaceRunnables,
+  seedBenchmarkDraft,
+  unregisterBenchmarkWorkspaceRunnables,
+  type BenchmarkWorkspaceRunnables,
+} from "../../mockBackend";
+import { runEval } from "../shared";
+import type { TokenUsage, ToolCallDetail } from "../shared/types";
+
+const MUTATING_GLOBAL_TOOLS = new Set([
+  "deploy_workspace_item",
+  "delete_workspace_item",
+]);
+const DISABLE_ACTIVE_EDITOR_CONTEXT_ENV =
+  "WMILL_AI_EVAL_DISABLE_ACTIVE_EDITOR_CONTEXT";
+// A/B gate for the search_app read tool: set to "1" to run the baseline arm
+// (toolset without search_app) so its token cost can be compared against the arm
+// that offers it.
+const DISABLE_SEARCH_APP_ENV = "WMILL_AI_EVAL_DISABLE_SEARCH_APP";
+
+const LIVE_EDITOR_ITEM_KINDS = {
+  script: "script",
+  flow: "flow",
+  app: "raw_app",
+} as const;
+
+export interface GlobalLiveEditorDraftFixture {
+  type: keyof typeof LIVE_EDITOR_ITEM_KINDS;
+  storagePath?: string;
+  effectivePath?: string;
+  value?: unknown;
+}
+
+// Identity the global system prompt builds paths from. Production reads
+// `userStore` (whoami) to fill `u/{username}/...`; the eval harness never logs
+// in, so without this the prompt sees an empty username (`u//...`) and no
+// path-selection case is meaningful. Seeded per-case via the initial fixture and
+// passed straight to `prepareGlobalSystemMessage` (no global-store mutation).
+export interface GlobalUserFixture {
+  username: string;
+  is_admin?: boolean;
+  /** Folders the user can write to (the writable set whoami returns). */
+  folders?: string[];
+  /** Folders the user can read; read-only folders = folders_read \ folders. */
+  folders_read?: string[];
+}
+
+export interface GlobalEvalResult {
+  success: boolean;
+  state: GlobalDraftState;
+  error?: string;
+  assistantMessageCount: number;
+  toolCallCount: number;
+  toolsUsed: string[];
+  toolCallDetails: ToolCallDetail[];
+  tokenUsage: TokenUsage;
+  finalContextTokens: number | null;
+}
+
+export interface GlobalEvalOptions {
+  workspaceFixtures?: BenchmarkWorkspaceRunnables;
+  liveEditorDrafts?: GlobalLiveEditorDraftFixture[];
+  user?: GlobalUserFixture;
+  model?: string;
+  maxIterations?: number;
+  provider?: AIProvider;
+  backend: WindmillBackendSettings;
+  workspaceRoot?: string;
+  runContext?: ModeRunContext;
+}
+
+export async function runGlobalEval(
+  userPrompt: string,
+  apiKey: string,
+  options: GlobalEvalOptions,
+): Promise<GlobalEvalResult> {
+  const workspaceRoot =
+    options.workspaceRoot ??
+    (await mkdtemp(join(tmpdir(), "wmill-frontend-global-benchmark-")));
+
+  clearGlobalDrafts(workspaceRoot);
+  registerBenchmarkWorkspaceRunnables(workspaceRoot, options.workspaceFixtures ?? {});
+  seedLiveEditorDrafts(workspaceRoot, options.liveEditorDrafts ?? []);
+
+  try {
+    const model = options.model ?? "claude-haiku-4-5-20251001";
+    const injectActiveEditorContext =
+      process.env[DISABLE_ACTIVE_EDITOR_CONTEXT_ENV] !== "1";
+    // Pass the seeded identity straight to the prompt builder rather than mutating
+    // the process-global `userStore`, so concurrent cases never race on it.
+    const rawResult = await runEval({
+      userPrompt,
+      systemMessage: prepareGlobalSystemMessage(undefined, { user: options.user }),
+      userMessage: prepareGlobalUserMessage(
+        userPrompt,
+        [],
+        injectActiveEditorContext ? { workspace: workspaceRoot } : {},
+      ),
+      tools: getGlobalEvalTools(),
+      helpers: {},
+      apiKey,
+      getOutput: () => collectGlobalDraftState(workspaceRoot),
+      onAssistantMessageStart: options.runContext?.onAssistantMessageStart,
+      onAssistantToken: options.runContext?.onAssistantChunk,
+      onAssistantMessageEnd: options.runContext?.onAssistantMessageEnd,
+      onToolCall: options.runContext?.onToolCall,
+      options: {
+        maxIterations: options.maxIterations,
+        model,
+        workspace: workspaceRoot,
+        provider: options.provider,
+        backend: options.backend,
+        caseId: options.runContext?.caseId,
+        attempt: options.runContext?.attempt,
+      },
+    });
+
+    return {
+      state: rawResult.output,
+      success: rawResult.success,
+      error: rawResult.error,
+      assistantMessageCount: rawResult.iterations,
+      toolCallCount: rawResult.toolCallsCount,
+      toolsUsed: rawResult.toolsCalled,
+      toolCallDetails: rawResult.toolCallDetails,
+      tokenUsage: rawResult.tokenUsage,
+      finalContextTokens: rawResult.finalContextTokens,
+    };
+  } finally {
+    clearGlobalDrafts(workspaceRoot);
+    clearLiveEditorDrafts(workspaceRoot, options.liveEditorDrafts ?? []);
+    unregisterBenchmarkWorkspaceRunnables(workspaceRoot);
+    if (!options.workspaceRoot) {
+      await rm(workspaceRoot, { recursive: true, force: true });
+    }
+  }
+}
+
+// Build the harness output from the DB-backed drafts. `listGlobalDrafts` returns
+// metadata-only rows for backend drafts (the model's `write_script` etc. persist
+// straight to the backend with no in-tab editor cell), so re-read each such row
+// with `getGlobalDraft` to attach the full value the validators assert on. A row
+// that already carries a value (the production in-tab cell overlay) is kept as-is.
+async function collectGlobalDraftState(
+  workspace: string,
+): Promise<GlobalDraftState> {
+  const items = await listGlobalDrafts(workspace);
+  const drafts = await Promise.all(
+    items.map(async (item) => {
+      if (item.value !== undefined) {
+        return item;
+      }
+      const full = await getGlobalDraft(
+        workspace,
+        item.type,
+        item.path,
+        item.triggerKind,
+      );
+      return full ?? item;
+    }),
+  );
+  return { drafts: drafts as GlobalDraftState["drafts"] };
+}
+
+function seedLiveEditorDrafts(
+  workspace: string,
+  fixtures: GlobalLiveEditorDraftFixture[],
+): void {
+  for (const fixture of fixtures) {
+    const itemKind = LIVE_EDITOR_ITEM_KINDS[fixture.type];
+    const storagePath = fixture.storagePath ?? fixture.effectivePath ?? "";
+    if (fixture.value !== undefined) {
+      // Seed as a backend draft row, not an in-tab cell: a cell would shadow the
+      // model's DB-backed edit when the output is read back via listGlobalDrafts.
+      seedBenchmarkDraft(workspace, itemKind, storagePath, fixture.value);
+    }
+    UserDraft.setLiveEditorDraft({
+      workspace,
+      itemKind,
+      storagePath,
+      effectivePath: fixture.effectivePath ?? fixture.storagePath,
+    });
+  }
+}
+
+function clearLiveEditorDrafts(
+  workspace: string,
+  fixtures: GlobalLiveEditorDraftFixture[],
+): void {
+  for (const fixture of fixtures) {
+    const itemKind = LIVE_EDITOR_ITEM_KINDS[fixture.type];
+    const storagePath = fixture.storagePath ?? fixture.effectivePath ?? "";
+    UserDraft.clearLiveEditorDraft(itemKind, { workspace, storagePath });
+  }
+}
+
+function getGlobalEvalTools(): ProductionTool<{}>[] {
+  const disableSearchApp = process.env[DISABLE_SEARCH_APP_ENV] === "1";
+  return (globalTools as ProductionTool<{}>[])
+    .filter((tool) => !(disableSearchApp && tool.def.function.name === "search_app"))
+    .map((tool) => {
+      if (!MUTATING_GLOBAL_TOOLS.has(tool.def.function.name)) {
+        return tool;
+      }
+
+      return {
+        ...tool,
+        requiresConfirmation: false,
+        validateBeforeConfirmation: undefined,
+        fn: async () =>
+          JSON.stringify(
+            {
+              success: false,
+              error:
+                "This mutating workspace tool is disabled during ai_evals global mode.",
+            },
+            null,
+            2,
+          ),
+      };
+    });
+}