fix: serialize concurrent python wheel-cache installs to prevent flaky ModuleNotFoundError

The wheel cache (ROOT_CACHE_DIR/python_<v>/<pkg>==<ver>) is shared by every job on a worker, and uv installs into it with `--reinstall --target <dir>`, which transiently empties the directory mid-run. Two jobs installing the same package concurrently (e.g. several WAC tests all importing wmill) clobber each other's files, and a job that imports from the dir during a concurrent reinstall sees a partially populated or wiped package -> flaky ModuleNotFoundError (httpx/wmill). Serialize installs per target dir with a process-wide keyed lock and re-check the `.valid.windmill` marker once the lock is held, so a waiter reuses the freshly populated cache instead of reinstalling over a dir other jobs may be importing from. Reinstall now only runs when no valid install exists, so no concurrent importer can observe a half-populated dir. Reproduced with a cold-wheel-cache loop over the wmill-importing python tests: 8/10 iterations failed before, 0/12 after. Adds a unit test guarding the per-target lock keying. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
feat: bounded-cascade selective execution for pipelines (UI + CLI) (#9695 )
2026-09-13 16:05:00 +00:00 · 2026-06-25 14:03:11 +00:00 · 2026-06-25 12:08:05 +02:00 · 2026-06-25 12:07:06 +02:00 · 2026-06-25 12:05:21 +02:00 · 2026-06-25 12:02:35 +02:00
200 changed files with 12343 additions and 2327 deletions
@@ -0,0 +1,40 @@
+---
+name: ai-chat
+description: Guidance for improving the Windmill AI chat (copilot), especially global mode — tools, prompts, and context-window discipline. Use when editing chat tools, system prompts, or tool-result shapes under frontend/src/lib/components/copilot/chat, or when changing how the chat manages its context window.
+---
+
+## Always benchmark before and after
+
+No context or behavior change ships without an `ai_evals` A/B on the affected mode.
+Add or adjust cases for exactly what you changed — see the `ai-evals` skill for
+authoring and the full run reference.
+
+Run the affected mode **before** your change and **after**, same model(s), same cases.
+
+## Measure the window first, and cumulative second
+
+Optimize **`finalContextTokens`** (window occupancy — what drives overflow and
+compaction), then cumulative prompt tokens.
+
+## Context discipline
+
+The dominant fixed cost is per-iteration overhead: the system prompt **plus every
+tool schema** is re-sent on every loop iteration. So:
+
+- **Every tool and every parameter is a permanent tax.** Justify each one and measure
+  it; an extra "locate" round-trip can cost more than the reads it saves. Strip dead
+  params rather than leaving them in the schema.
+- **Tool results return the minimum.** Never echo content the model already has. The
+  canonical mistake: a write tool that returns the whole edited artifact right after
+  the model authored it — return `{ success, message }` instead. When you touch a
+  *shared* write helper (e.g. `finishAppDraftWrite` in `global/core.ts`), re-check
+  this invariant for **all** the write tools routing through it — the echo has
+  regressed before via a shared refactor.
+
+## Prompts and tool descriptions are part of the surface
+
+The system prompt and tool descriptions steer behavior as much as the tools
+themselves, and are benchmarkable the same way. A description that advertises
+truncation makes the model self-limit; the path-conventions block changes where
+drafts land. Treat prompt/description edits as real changes and A/B them — a
+pure-prompt change is a legitimate, measurable improvement.
@@ -0,0 +1,87 @@
+---
+name: ai-evals
+description: Author and run black-box benchmark cases for the Windmill AI generation modes (flow/app/script/cli/global) in ai_evals/. Use when adding or changing eval cases, or when running before/after benchmarks for AI chat / copilot changes.
+---
+
+# AI evals — authoring and running benchmark cases
+
+`ai_evals/` is a black-box benchmark runner for the Windmill AI generation modes:
+`flow`, `app`, `script`, `cli`, `global`. It always tests the **current** production
+prompts, tools, and guidance in this checkout. Each attempt runs the real production
+path, deterministic validation, then LLM judging.
+
+The goal is to test current production guidance with realistic user requests — **not**
+to pin one exact implementation shape.
+
+## Running benchmarks
+
+```bash
+cd ai_evals
+bun install                       # first time; frontend modes also need `cd frontend && bun install`
+bun run cli -- models             # list model aliases
+bun run cli -- cases global       # list cases for a mode
+bun run cli -- run global global-test1-script-create --model sonnet
+```
+
+Frontend modes (`flow`/`script`/`app`/`global`) route model calls through a Windmill
+backend's `/api/w/<ws>/ai/proxy`, so you need **any** reachable backend:
+
+```bash
+WMILL_AI_EVAL_BACKEND_URL=http://127.0.0.1:<port> WMILL_AI_EVAL_BACKEND_WORKSPACE=integration-tests \
+  bun run cli -- run global <caseIds...> --models sonnet,gpt-5.5,gemini-3.1-pro-preview
+```
+
+- **Reuse an existing workspace.** CE builds cap workspaces, so temp-workspace
+  creation 400s ("reached workspace limit"). Always set
+  `WMILL_AI_EVAL_BACKEND_WORKSPACE=integration-tests` (or any existing workspace) to
+  reuse one. The only side effect of a run is upserting an `f/evals/ai/<provider>`
+  resource there.
+- Provider keys live in `ai_evals/.env` and are auto-loaded by bun. The judge is a
+  separate Anthropic call (default `claude-sonnet-4-6`) regardless of the model under
+  test.
+
+## Authoring core rules
+
+1. Write prompts like a real user request.
+2. Prefer behavior, inputs, constraints, and outcomes over internal implementation.
+3. Keep deterministic validation narrow and hard.
+4. Put semantic expectations in `judgeChecklist`.
+5. Use `expected` fixtures only when exact structure really matters.
+
+### Prompt writing
+
+Prompts should sound like something a user would naturally ask. Do not write prompts
+as if the user knows Windmill internals unless the case explicitly tests a power-user
+workflow.
+
+Good:
+- "Create a flow that routes support requests based on customer tier."
+- "Add a reset button that sets the counter back to 0."
+- "Create a flow that reuses the existing greeting script instead of duplicating the logic."
+
+Bad:
+- "Use `branchone` with 3 branches and a default branch."
+- "Create a `rawscript` step with this exact topology."
+- "This is a benchmark harness."
+
+### Deterministic validation
+
+Use deterministic checks only for hard failures: missing required files; unexpected
+extra files when the prompt says not to create them; syntax errors; unresolved flow
+refs; missing required special modules or suspend config; obvious corruption.
+
+Do **not** encode one preferred implementation. Bad hard checks: exact step topology
+for a creation flow; exact branch structure when the prompt only asked for routing;
+exact input shape when multiple reasonable shapes are acceptable.
+
+### Judge checklist
+
+Every non-trivial case should have a `judgeChecklist` capturing user-visible behavior
+that must be present, important constraints, and key completion criteria — not
+low-level implementation details unless truly required.
+
+Good: "the flow calculates the order total with 8% tax"; "the flow reuses the existing
+workspace script instead of rewriting the logic". Bad: "uses `branchone`"; "contains a
+`rawscript` node".
+
+See `ai_evals/README.md` for the full case format, fields, and fixture details.
@@ -0,0 +1 @@
+../../../.agents/skills/ai-chat/SKILL.md
@@ -0,0 +1 @@
+../../../.agents/skills/ai-evals/SKILL.md
@@ -28,7 +28,7 @@ ENV PATH="${PATH}:/usr/local/go/bin"
 ENV GO_PATH=/usr/local/go/bin/go

 # UV
-RUN curl --proto '=https' --tlsv1.2 -LsSf https://github.com/astral-sh/uv/releases/download/0.9.25/uv-installer.sh | sh && mv /usr/local/cargo/bin/uv /usr/local/bin/uv
+RUN curl --proto '=https' --tlsv1.2 -LsSf https://github.com/astral-sh/uv/releases/download/0.11.24/uv-installer.sh | sh && mv /usr/local/cargo/bin/uv /usr/local/bin/uv

 ENV TZ=Etc/UTC

@@ -74,7 +74,7 @@ jobs:

      - uses: astral-sh/setup-uv@v6.2.1
        with:
-          version: "0.9.25"
+          version: "0.11.24"

      - uses: shivammathur/setup-php@v2
        with:
@@ -62,7 +62,7 @@ jobs:
          node-version: "20"
      - uses: astral-sh/setup-uv@v6.2.1
        with:
-          version: "0.9.25"
+          version: "0.11.24"
      - uses: shivammathur/setup-php@v2
        with:
          php-version: "8.3"
@@ -1,5 +1,64 @@
 # Changelog

+## [1.739.0](https://github.com/windmill-labs/windmill/compare/v1.738.0...v1.739.0) (2026-06-24)
+
+
+### Features
+
+* add /compact session chat command ([#9764](https://github.com/windmill-labs/windmill/issues/9764)) ([83cc553](https://github.com/windmill-labs/windmill/commit/83cc5533ee92e59356a117eefeaf42dad23287f6))
+* add session chat slash commands ([#9748](https://github.com/windmill-labs/windmill/issues/9748)) ([24b95e9](https://github.com/windmill-labs/windmill/commit/24b95e9fe12ba4abdfe1ff6e9f9fe42cb2ded011))
+* **ai-chat:** add /clear session command to start a fresh conversation ([#9769](https://github.com/windmill-labs/windmill/issues/9769)) ([3fafac2](https://github.com/windmill-labs/windmill/commit/3fafac275d2100a6f89924040650cf959945d209))
+* **ai-chat:** context usage gauge + unified model settings menu ([#9763](https://github.com/windmill-labs/windmill/issues/9763)) ([2e020b2](https://github.com/windmill-labs/windmill/commit/2e020b2ccc7a649a5923bff72a98f07d4fc85381))
+* **apps:** show raw-app fork diffs as per-file tree items ([#9491](https://github.com/windmill-labs/windmill/issues/9491)) ([e98df38](https://github.com/windmill-labs/windmill/commit/e98df38ac43823ee85209a4b09cd70690469302d))
+* **frontend:** add filter submenu to collapsed AI sessions popover ([#9757](https://github.com/windmill-labs/windmill/issues/9757)) ([3d48ba7](https://github.com/windmill-labs/windmill/commit/3d48ba7738c3d3356539b5fc44a871f6b7f9d548))
+* **frontend:** restore raw app 'open preview in separate window' ([#9765](https://github.com/windmill-labs/windmill/issues/9765)) ([a116715](https://github.com/windmill-labs/windmill/commit/a116715c418c39d48a91e6c0b4484a31537dff38))
+* **frontend:** show approval wait as a distinct segment in flow timeline ([#9756](https://github.com/windmill-labs/windmill/issues/9756)) ([2a70ccc](https://github.com/windmill-labs/windmill/commit/2a70ccc38675c7c2353807a4f85764a8a35224e2))
+* scope AI sessions per workspace root with lifecycle reconcile ([#9734](https://github.com/windmill-labs/windmill/issues/9734)) ([42c5e7a](https://github.com/windmill-labs/windmill/commit/42c5e7a3fc9b74256de8806ec0d7b62e8bbf029c))
+
+
+### Bug Fixes
+
+* **ai-chat:** strip unclosed &lt;summary&gt; tag leaking into compaction summary ([#9750](https://github.com/windmill-labs/windmill/issues/9750)) ([250a05f](https://github.com/windmill-labs/windmill/commit/250a05f544ae397bb91af5fc83bf408cfe1c554d))
+* **apps:** realign legacy raw-app drafts to raw_app draft kind ([#9761](https://github.com/windmill-labs/windmill/issues/9761)) ([288318a](https://github.com/windmill-labs/windmill/commit/288318ac269714fc03b15622dbb86b1c28268a36))
+* **backend:** resolve folder_labels search_path on non-public (PG_SCHEMA) schemas ([#9758](https://github.com/windmill-labs/windmill/issues/9758)) ([f582878](https://github.com/windmill-labs/windmill/commit/f5828780fd6a8be070b2933ebd41ee6dff98a9e1))
+* forbid superadmin job tokens from global user and token management ([#9715](https://github.com/windmill-labs/windmill/issues/9715)) ([043c2c0](https://github.com/windmill-labs/windmill/commit/043c2c05b7678c49faca0ccb28e5f6393567ba4d))
+* **frontend:** highlight the runtime-chosen branch in flow graph viewer ([#9755](https://github.com/windmill-labs/windmill/issues/9755)) ([de6192b](https://github.com/windmill-labs/windmill/commit/de6192bec1695883a07452f7db2fb51c94dbfd43))
+* **frontend:** keep #content portal target present on AI-session route ([#9754](https://github.com/windmill-labs/windmill/issues/9754)) ([5e09c50](https://github.com/windmill-labs/windmill/commit/5e09c501713ebbe05b28ce0084eca641f0dbe95c))
+* **frontend:** show AI skills settings only when global mode enabled ([#9747](https://github.com/windmill-labs/windmill/issues/9747)) ([c017f7f](https://github.com/windmill-labs/windmill/commit/c017f7f8919a51292ddf01574961d1774bc1ba23))
+* **frontend:** stop flow step id generation from being poisoned by non-canonical keys ([#9766](https://github.com/windmill-labs/windmill/issues/9766)) ([4dbf873](https://github.com/windmill-labs/windmill/commit/4dbf8737238ccc4dc2c67365e6d43f04f46c75b5))
+* persist on-behalf-of user across app deploy paths ([#9773](https://github.com/windmill-labs/windmill/issues/9773)) ([f99781c](https://github.com/windmill-labs/windmill/commit/f99781ca5f77248206c951935cc44acfa5f072eb))
+* reject symlink traversal in job-dir path validation ([#9713](https://github.com/windmill-labs/windmill/issues/9713)) ([b5bd824](https://github.com/windmill-labs/windmill/commit/b5bd8245d81b84fc14d3ea955bf1e66ac576bf37))
+
+
+### Performance Improvements
+
+* **audit:** adaptive timestamp floor for S3 audit-log export ([#9752](https://github.com/windmill-labs/windmill/issues/9752)) ([55bed4a](https://github.com/windmill-labs/windmill/commit/55bed4abcfce2a611b16054573980d2eb613ccb3))
+* **monitor:** vacuum job_perms/job_result_stream right after each orphan sweep ([#9753](https://github.com/windmill-labs/windmill/issues/9753)) ([8912e21](https://github.com/windmill-labs/windmill/commit/8912e21d1571e57b5cf21b7d4d9520e20a28e70d))
+
+## [1.738.0](https://github.com/windmill-labs/windmill/compare/v1.737.0...v1.738.0) (2026-06-23)
+
+
+### Features
+
+* add resource and infrastructure telemetry ([#9737](https://github.com/windmill-labs/windmill/issues/9737)) ([9793d01](https://github.com/windmill-labs/windmill/commit/9793d01575415963a89609a1baf2cd64f0d050cc))
+* render mermaid diagrams in chat code blocks ([#9738](https://github.com/windmill-labs/windmill/issues/9738)) ([cfb9f1d](https://github.com/windmill-labs/windmill/commit/cfb9f1dbc23110ecf8f91bb3c8c81fc6e35dc09b))
+
+
+### Bug Fixes
+
+* **ai-chat:** Fix incorrect editor edits from ai chat [#1](https://github.com/windmill-labs/windmill/issues/1) ([#9741](https://github.com/windmill-labs/windmill/issues/9741)) ([fc797a3](https://github.com/windmill-labs/windmill/commit/fc797a35fe7885630c81453df0fc94769e73873a))
+* allow object storage test for non-super-admins, harden on cloud ([#9739](https://github.com/windmill-labs/windmill/issues/9739)) ([24446e8](https://github.com/windmill-labs/windmill/commit/24446e80093ade349f7fbf65063d2d1cb5551c1e))
+* **frontend:** debounce external code→Monaco sync in Editor ([#9743](https://github.com/windmill-labs/windmill/issues/9743)) ([29c67ce](https://github.com/windmill-labs/windmill/commit/29c67ced97bf2919584986f9d9eceb4337c34ad9))
+* **frontend:** preserve editor content when closing instance settings drawer ([#9740](https://github.com/windmill-labs/windmill/issues/9740)) ([11d0e65](https://github.com/windmill-labs/windmill/commit/11d0e65f3af9a048bc1921bbdd3d676a07483a57))
+* pipeline annotation false-positives from body comments ([#9736](https://github.com/windmill-labs/windmill/issues/9736)) ([984ea72](https://github.com/windmill-labs/windmill/commit/984ea728d98649b66b1cae899bdab9af3176caa7))
+* preserve fork parent linkage on workspace id change ([#9716](https://github.com/windmill-labs/windmill/issues/9716)) ([cbf54d4](https://github.com/windmill-labs/windmill/commit/cbf54d4eb432638e27f67c4c8b879cbcc0291da3))
+* prevent variable push from corrupting is_secret variables ([#9705](https://github.com/windmill-labs/windmill/issues/9705)) ([ba4b368](https://github.com/windmill-labs/windmill/commit/ba4b368706e95e22f346a10e5fe145b0795ac3f6))
+
+
+### Performance Improvements
+
+* **monitor:** skip protected prefix in retention delete via cross-batch watermark (WIN-2088) ([#9744](https://github.com/windmill-labs/windmill/issues/9744)) ([e90b2be](https://github.com/windmill-labs/windmill/commit/e90b2be8fade1eb78cd685890291f5a4553a6a10))
+
 ## [1.737.0](https://github.com/windmill-labs/windmill/compare/v1.736.0...v1.737.0) (2026-06-23)


@@ -163,14 +163,14 @@ ENV PATH /usr/local/bin:/root/.local/bin:/tmp/.local/bin:$PATH


 RUN apt-get update \
-    && apt-get install -y --no-install-recommends netbase tzdata ca-certificates wget curl jq unzip build-essential unixodbc xmlsec1 software-properties-common tini gnupg lsb-release \
+    && apt-get install -y --no-install-recommends netbase tzdata ca-certificates wget curl jq unzip build-essential unixodbc xmlsec1 tini gnupg \
    && if echo "$features" | grep -q "ee"; then apt-get install -y --no-install-recommends libsasl2-modules-gssapi-mit krb5-user; fi \
    && apt-get clean \
    && rm -rf /var/lib/apt/lists/*

 # Install latest PostgreSQL client (pg_dump) from official PostgreSQL apt repository
 RUN curl -fsSL https://www.postgresql.org/media/keys/ACCC4CF8.asc | gpg --dearmor -o /usr/share/keyrings/postgresql-archive-keyring.gpg \
-    && echo "deb [signed-by=/usr/share/keyrings/postgresql-archive-keyring.gpg] https://apt.postgresql.org/pub/repos/apt $(lsb_release -cs)-pgdg main" > /etc/apt/sources.list.d/pgdg.list \
+    && echo "deb [signed-by=/usr/share/keyrings/postgresql-archive-keyring.gpg] https://apt.postgresql.org/pub/repos/apt $(. /etc/os-release; echo "$VERSION_CODENAME")-pgdg main" > /etc/apt/sources.list.d/pgdg.list \
    && apt-get update \
    && apt-get install -y --no-install-recommends postgresql-client \
    && apt-get clean \
@@ -184,11 +184,11 @@ RUN if [ "$WITH_GIT" = "true" ]; then \
    else echo 'Building the image without git'; fi;

 RUN if [ "$WITH_POWERSHELL" = "true" ]; then \
-    if [ "$TARGETPLATFORM" = "linux/amd64" ]; then apt-get update -y && apt install libicu-dev -y && wget -O 'pwsh.deb' "https://github.com/PowerShell/PowerShell/releases/download/v${POWERSHELL_VERSION}/powershell_${POWERSHELL_DEB_VERSION}.deb_amd64.deb" && apt-get clean \
+    if [ "$TARGETPLATFORM" = "linux/amd64" ]; then apt-get update -y && apt install libicu72 -y && wget -O 'pwsh.deb' "https://github.com/PowerShell/PowerShell/releases/download/v${POWERSHELL_VERSION}/powershell_${POWERSHELL_DEB_VERSION}.deb_amd64.deb" && apt-get clean \
    && rm -rf /var/lib/apt/lists/* && \
    dpkg --install 'pwsh.deb' && \
    rm 'pwsh.deb'; \
-    elif [ "$TARGETPLATFORM" = "linux/arm64" ]; then apt-get update -y && apt install libicu-dev -y && wget -O powershell.tar.gz "https://github.com/PowerShell/PowerShell/releases/download/v${POWERSHELL_VERSION}/powershell-${POWERSHELL_VERSION}-linux-arm64.tar.gz" && apt-get clean \
+    elif [ "$TARGETPLATFORM" = "linux/arm64" ]; then apt-get update -y && apt install libicu72 -y && wget -O powershell.tar.gz "https://github.com/PowerShell/PowerShell/releases/download/v${POWERSHELL_VERSION}/powershell-${POWERSHELL_VERSION}-linux-arm64.tar.gz" && apt-get clean \
    && rm -rf /var/lib/apt/lists/* && \
    mkdir -p /opt/microsoft/powershell/7 && \
    tar zxf powershell.tar.gz -C /opt/microsoft/powershell/7 && \
@@ -233,7 +233,7 @@ ENV PATH="${PATH}:/usr/local/go/bin"
 ENV GO_PATH=/usr/local/go/bin/go

 # Install UV
-RUN curl --proto '=https' --tlsv1.2 -LsSf https://github.com/astral-sh/uv/releases/download/0.9.25/uv-installer.sh | sh && mv /root/.local/bin/uv /usr/local/bin/uv
+RUN curl --proto '=https' --tlsv1.2 -LsSf https://github.com/astral-sh/uv/releases/download/0.11.24/uv-installer.sh | sh && mv /root/.local/bin/uv /usr/local/bin/uv

 # Preinstall python runtimes to temp build location (will copy with world-writable perms later)
 # --compile-bytecode precompiles the stdlib to .pyc so jobs don't recompile it on every run
@@ -1,208 +1,14 @@
-# AI Evals Authoring Guide
+# AI Evals

-This folder contains black-box benchmark cases for:
+Black-box benchmark cases for the Windmill AI generation modes (`flow`, `app`,
+`script`, `cli`, `global`).

- `flow`
- `app`
- `script`
- `cli`
- `global`
+**Authoring and running cases is documented in the `ai-evals` skill** — load it
+before adding/changing a case or running a benchmark. Claude Code reads
+`.claude/skills/ai-evals/SKILL.md`; Codex and Pi read
+`.agents/skills/ai-evals/SKILL.md` (same canonical file). Invoke with `/ai-evals` in
+Claude Code, `$ai-evals` in Codex, or `pi --skill ai-evals`.

-The goal is to test the current production prompts and guidance with realistic user requests, not to test one exact implementation shape.
+For AI chat / copilot changes that these evals measure, see the `ai-chat` skill.

-## Core rules
-
-1. Write prompts like a real user request.
-2. Prefer behavior, inputs, constraints, and outcomes over internal implementation details.
-3. Keep deterministic validation narrow and hard.
-4. Put semantic expectations in `judgeChecklist`.
-5. Use `expected` fixtures only when exact structure really matters.
-
-## Prompt writing
-
-Prompts should sound like something a user would naturally ask.
-
-Good:
-
- "Create a flow that routes support requests based on customer tier."
- "Add a reset button that sets the counter back to 0."
- "Create a flow that reuses the existing greeting script instead of duplicating the logic."
-
-Bad:
-
- "Use `branchone` with 3 branches and a default branch."
- "Create a `rawscript` step with this exact topology."
- "This is a benchmark harness."
-
-Do not write prompts as if the user knows Windmill internals unless the case is explicitly testing a power-user workflow.
-
-## Flow-specific rules
-
-This is the main principle you asked for:
-
- flow prompts should read like requests from a user who does not know the product internals
- the user should ask for behavior, not for `branchone`, `branchall`, `rawscript`, `preprocessor_module`, `failure_module`, exact graph topology, or other internal constructs
-
-That means:
-
- creation cases should describe the business behavior and expected result
- modification cases may mention existing step names, because the user can see the current flow
- only mention special Windmill constructs when the case is explicitly about those constructs
-
-Examples:
-
- acceptable creation prompt:
-  "Create a purchase approval flow that pauses for approval and asks the approver for a comment."
- avoid:
-  "Create a suspend step with one required event and a resume form."
-
-For flow cases, do not fail a case just because the model chose a different valid topology.
-
-## App-specific rules
-
-App prompts should focus on user-visible behavior:
-
- what the UI should let the user do
- what should persist
- what backend behavior is needed
-
-Avoid prompting in terms of React structure, component names, or implementation unless the case is specifically about editing an existing app.
-
-## CLI-specific rules
-
-CLI prompts can be more explicit about paths and file names because real CLI users often do specify them.
-
-Still, avoid benchmark phrasing. The prompt should read like a repo task, not a harness instruction.
-
-When relevant, ask the assistant to tell the user which `wmill` commands to run next. That is part of the benchmarked behavior.
-
-## Global-specific rules
-
-Global prompts should exercise workspace-level drafting behavior:
-
- inspecting existing scripts, flows, apps, schedules, triggers, resources, and variables when relevant
- writing AI drafts rather than saving or deploying by default
- producing coherent multi-artifact changes when the request crosses artifact boundaries
-
-Keep deterministic validation focused on the draft contract: required draft type/path, required content snippets, forbidden draft paths, and forbidden mutating tools such as deploy/delete unless the case explicitly asks for them.
-
-Datatable cases should set `skipJudge: true` and validate through tool-use
-(`requiredToolsUsed` / `forbiddenToolsUsed`) and SQL-argument assertions
-(`toolCallArgs` with `stringIncludesAnyOf`, e.g. `['select']`, `['create table']`,
-`['update', 'insert into']`). Two reasons the judge is unreliable here:
-
- `list_datatables`, `get_datatable_table_schema`, and `exec_datatable_sql`
-  produce no drafts, and the global judge only sees the drafts artifact — it
-  scores a no-draft conversational answer as empty (same as the
-  `askUserQuestion` cases).
- Even a case that *does* produce a draft (a script reading the data table via
-  `wmill.datatable()` at runtime) is mis-judged: the judge has no datatable SDK
-  reference and penalizes correct `wmill.datatable()` usage as wrong. Verify the
-  SDK call deterministically instead — `requiredDrafts.valueIncludes: ['wmill.datatable(']`
-  plus forbidding `exec_datatable_sql` (keeping chat-time SQL distinct from
-  runtime SDK use).
-
-`stringIncludesAnyOf` is existential over calls (at least one matching call), so a
-mutation case still passes when the model mixes its UPDATE/INSERT with
-verification SELECTs. The in-memory engine (`datatableSqlEngine.ts`) is stateful
-within a case — writes persist, so a model that re-queries to verify its
-CREATE/UPDATE sees the change and does not loop. But the engine is best-effort
-(SELECT returns all rows of the referenced/first table with no WHERE/projection),
-so still never assert specific returned row values. Seed data via
-`workspace.datatables` in the `initial` fixture (see README).
-
-## Deterministic validation
-
-Use deterministic validation only for hard failures such as:
-
- missing required files
- unexpected extra files when the prompt says not to create them
- syntax errors
- unresolved flow refs
- missing required special modules or suspend config
- obvious artifact corruption
-
-Do not use deterministic validation to enforce one preferred implementation for broad creation tasks.
-
-Examples of bad hard checks:
-
- exact step topology for a creation flow
- exact branch structure when the prompt only asked for routing behavior
- exact input shape when multiple reasonable shapes are acceptable
-
-## Judge checklist
-
-Every non-trivial case should have a `judgeChecklist`.
-
-The checklist should capture:
-
- the user-visible behavior that must be present
- important constraints
- key completion criteria
-
-The checklist should not duplicate low-level implementation details unless they are truly required by the task.
-
-Good checklist items:
-
- "the flow calculates the order total with 8% tax"
- "the app persists recipes appropriately for a raw Windmill app"
- "the flow reuses the existing workspace script instead of rewriting the logic"
-
-Bad checklist items:
-
- "uses `branchone`"
- "contains a `rawscript` node"
-
-## When to use `expected`
-
-Use `expected` fixtures when the case is structure-sensitive, for example:
-
- exact file creation
- exact script content
- modification cases where a specific file must change in a specific way
- cases where preserving an existing structure is part of the requirement
-
-Do not use a full `expected` artifact as the semantic oracle for broad creation tasks when multiple valid outputs should pass.
-
-## When to use `initial`
-
-Use `initial` when the benchmark is about:
-
- editing an existing artifact
- reusing existing workspace assets
- preserving existing behavior while adding a change
-
-If the case is greenfield, prefer no `initial`.
-
-## Case design ladder
-
-Prefer suites that get gradually harder:
-
-1. trivial create case
-2. realistic create case
-3. reuse-existing-assets case
-4. modification case
-5. refactor case
-6. edge-case or niche product behavior
-
-The last cases in a suite should cover unusual or product-specific behavior.
-
-## Anti-patterns
-
-Avoid these:
-
- benchmark framing in prompts
- over-specified internal topology for creation tasks
- judge checklists that just restate implementation details
- deterministic validation that encodes one preferred solution
- fixtures that are so minimal or brittle that they create false negatives
-
-## Before adding a case
-
-Ask:
-
-1. Would a real user plausibly write this prompt?
-2. If the model solves it in a different valid way, would the case still pass?
-3. Are the hard deterministic checks only catching objectively broken output?
-4. Does the `judgeChecklist` describe the real success criteria?
-5. If this case fails, will the reason be understandable from the saved artifacts?
+The full case format, fields, and fixture details remain in `ai_evals/README.md`.
@@ -3,6 +3,7 @@
    Create a draft Bun script at `f/evals/global/greet_user`.
    It should take a string `name` input and return `Hello, ${name}!`.
    Leave it as an AI draft only; do not deploy or save it.
+  initial: ai_evals/fixtures/frontend/global/initial/user_admin_evals_folder.json
  runtime:
    maxTurns: 10
  validate:
@@ -212,6 +212,9 @@ describe("loadCases", () => {
        },
      ],
    });
+    expect(caseEntry?.initialPath).toContain(
+      "ai_evals/fixtures/frontend/global/initial/user_admin_evals_folder.json"
+    );
    expect(caseEntry?.toolExpect).toMatchObject({
      requiredToolsUsed: ["write_script"],
      forbiddenToolsUsed: ["deploy_workspace_item", "delete_workspace_item"],
@@ -0,0 +1,8 @@
+{
+  "user": {
+    "username": "admin",
+    "is_admin": true,
+    "folders": ["evals"],
+    "folders_read": ["evals"]
+  }
+}
@@ -0,0 +1,20 @@
+{
+  "db_name": "PostgreSQL",
+  "query": "SELECT SUM(pg_database_size(datname))::BIGINT AS \"v!\" FROM pg_database",
+  "describe": {
+    "columns": [
+      {
+        "ordinal": 0,
+        "name": "v!",
+        "type_info": "Int8"
+      }
+    ],
+    "parameters": {
+      "Left": []
+    },
+    "nullable": [
+      null
+    ]
+  },
+  "hash": "074dd26f6427f4ff97e92c35163ad042144e656adcca56a4936a2eb196d3f48c"
+}
@@ -0,0 +1,58 @@
+{
+  "db_name": "PostgreSQL",
+  "query": "\n        SELECT language AS \"language!: _\", COUNT(*)::BIGINT AS \"count!\"\n        FROM script\n        WHERE archived = false AND deleted = false AND kind = 'script'\n          AND (auto_kind IS NULL OR auto_kind <> 'wac')\n        GROUP BY language\n        ",
+  "describe": {
+    "columns": [
+      {
+        "ordinal": 0,
+        "name": "language!: _",
+        "type_info": {
+          "Custom": {
+            "name": "script_lang",
+            "kind": {
+              "Enum": [
+                "python3",
+                "deno",
+                "go",
+                "bash",
+                "postgresql",
+                "nativets",
+                "bun",
+                "mysql",
+                "bigquery",
+                "snowflake",
+                "graphql",
+                "powershell",
+                "mssql",
+                "php",
+                "bunnative",
+                "rust",
+                "ansible",
+                "csharp",
+                "oracledb",
+                "nu",
+                "java",
+                "duckdb",
+                "ruby",
+                "rlang"
+              ]
+            }
+          }
+        }
+      },
+      {
+        "ordinal": 1,
+        "name": "count!",
+        "type_info": "Int8"
+      }
+    ],
+    "parameters": {
+      "Left": []
+    },
+    "nullable": [
+      false,
+      null
+    ]
+  },
+  "hash": "11813108dbf6b104eba968c3609c74ac5a589542d3e76b1e756a82fb19d49ee8"
+}
@@ -0,0 +1,29 @@
+{
+  "db_name": "PostgreSQL",
+  "query": "\n        SELECT DISTINCT ON (path) path AS \"path!\", content AS \"content!\"\n        FROM script\n        WHERE workspace_id = $1\n          AND auto_kind = 'pipeline'\n          AND archived = false\n          AND deleted = false\n          AND ($2::text IS NULL OR path LIKE $2)\n        ORDER BY path, created_at DESC\n        ",
+  "describe": {
+    "columns": [
+      {
+        "ordinal": 0,
+        "name": "path!",
+        "type_info": "Varchar"
+      },
+      {
+        "ordinal": 1,
+        "name": "content!",
+        "type_info": "Text"
+      }
+    ],
+    "parameters": {
+      "Left": [
+        "Text",
+        "Text"
+      ]
+    },
+    "nullable": [
+      false,
+      false
+    ]
+  },
+  "hash": "394e2598880aff8a7f4ee05c3fe748be58b6381f5fae5619d8376daefc3b21db"
+}
@@ -0,0 +1,20 @@
+{
+  "db_name": "PostgreSQL",
+  "query": "SELECT COUNT(*)::INT AS \"v!\" FROM pg_stat_activity",
+  "describe": {
+    "columns": [
+      {
+        "ordinal": 0,
+        "name": "v!",
+        "type_info": "Int4"
+      }
+    ],
+    "parameters": {
+      "Left": []
+    },
+    "nullable": [
+      null
+    ]
+  },
+  "hash": "3b439ae7af0fcbb9df8e19faf84abf590e5e94898954711d57a602e6fd8a2f84"
+}
@@ -0,0 +1,20 @@
+{
+  "db_name": "PostgreSQL",
+  "query": "SELECT COUNT(*)::BIGINT AS \"count!\" FROM script WHERE archived = false AND deleted = false AND auto_kind = 'wac'",
+  "describe": {
+    "columns": [
+      {
+        "ordinal": 0,
+        "name": "count!",
+        "type_info": "Int8"
+      }
+    ],
+    "parameters": {
+      "Left": []
+    },
+    "nullable": [
+      null
+    ]
+  },
+  "hash": "3db1c61295c284725eef9e74a8aa2bc7822d263605a445f1f4a76e58e76a3e79"
+}
@@ -0,0 +1,15 @@
+{
+  "db_name": "PostgreSQL",
+  "query": "UPDATE workspace SET parent_workspace_id = $1 WHERE parent_workspace_id = $2",
+  "describe": {
+    "columns": [],
+    "parameters": {
+      "Left": [
+        "Varchar",
+        "Text"
+      ]
+    },
+    "nullable": []
+  },
+  "hash": "40a8cf5e87bb489fd172689e9a6f0f1075b878f9916145929b3cd3b1a53b777e"
+}
@@ -0,0 +1,28 @@
+{
+  "db_name": "PostgreSQL",
+  "query": "SELECT p.id AS \"id!\", p.deleted AS \"deleted!\"\n         FROM workspace f\n         JOIN workspace p ON p.id = f.parent_workspace_id\n         WHERE f.id = $1",
+  "describe": {
+    "columns": [
+      {
+        "ordinal": 0,
+        "name": "id!",
+        "type_info": "Varchar"
+      },
+      {
+        "ordinal": 1,
+        "name": "deleted!",
+        "type_info": "Bool"
+      }
+    ],
+    "parameters": {
+      "Left": [
+        "Text"
+      ]
+    },
+    "nullable": [
+      false,
+      false
+    ]
+  },
+  "hash": "42322020ff9cc7dd7ebafc1cb4122ba3d670cc36bdbc6451f29b8f22f8cff688"
+}
@@ -0,0 +1,20 @@
+{
+  "db_name": "PostgreSQL",
+  "query": "SELECT pg_database_size(current_database())::BIGINT AS \"v!\"",
+  "describe": {
+    "columns": [
+      {
+        "ordinal": 0,
+        "name": "v!",
+        "type_info": "Int8"
+      }
+    ],
+    "parameters": {
+      "Left": []
+    },
+    "nullable": [
+      null
+    ]
+  },
+  "hash": "48a5df355a2bca557a3a541cf66c8e75790b7c3dd7845359019ff645a1f7c8bf"
+}
@@ -0,0 +1,20 @@
+{
+  "db_name": "PostgreSQL",
+  "query": "SELECT current_setting('max_connections')::INT AS \"v!\"",
+  "describe": {
+    "columns": [
+      {
+        "ordinal": 0,
+        "name": "v!",
+        "type_info": "Int4"
+      }
+    ],
+    "parameters": {
+      "Left": []
+    },
+    "nullable": [
+      null
+    ]
+  },
+  "hash": "49e1f5663eed128ed956c9a50bc73a85256c0a3e5a701cc13c944e66f6402617"
+}
@@ -0,0 +1,20 @@
+{
+  "db_name": "PostgreSQL",
+  "query": "SELECT COUNT(*)::BIGINT AS \"count!\" FROM flow WHERE archived = false",
+  "describe": {
+    "columns": [
+      {
+        "ordinal": 0,
+        "name": "count!",
+        "type_info": "Int8"
+      }
+    ],
+    "parameters": {
+      "Left": []
+    },
+    "nullable": [
+      null
+    ]
+  },
+  "hash": "50490ff42fb1f2d78864d7b374d299bf8290c3b969b576cb185c8b5b0abb0265"
+}
@@ -0,0 +1,20 @@
+{
+  "db_name": "PostgreSQL",
+  "query": "SELECT current_setting('server_version_num')::INT AS \"v!\"",
+  "describe": {
+    "columns": [
+      {
+        "ordinal": 0,
+        "name": "v!",
+        "type_info": "Int4"
+      }
+    ],
+    "parameters": {
+      "Left": []
+    },
+    "nullable": [
+      null
+    ]
+  },
+  "hash": "56a98a07a2f6af4d694db05d57acbe0b55cc39d64f73a3c90c250ea670f9cdee"
+}
@@ -0,0 +1,20 @@
+{
+  "db_name": "PostgreSQL",
+  "query": "SELECT CASE WHEN (current_setting('is_superuser') = 'on'\n                                  OR pg_has_role(current_user, 'pg_read_all_stats', 'USAGE'))\n                             AND NOT EXISTS (SELECT 1 FROM pg_prepared_xacts)\n                        THEN (SELECT min(xact_start) FROM pg_stat_activity WHERE xact_start IS NOT NULL)\n                        ELSE NULL END AS \"x\"",
+  "describe": {
+    "columns": [
+      {
+        "ordinal": 0,
+        "name": "x",
+        "type_info": "Timestamptz"
+      }
+    ],
+    "parameters": {
+      "Left": []
+    },
+    "nullable": [
+      null
+    ]
+  },
+  "hash": "5d84f1ede2fbe09923a36d80d7c01699bf9968a02675f9f598c97a19c6df089b"
+}
@@ -0,0 +1,26 @@
+{
+  "db_name": "PostgreSQL",
+  "query": "\n        SELECT\n            COUNT(*) FILTER (WHERE av.raw_app = false)::BIGINT AS \"low_code!\",\n            COUNT(*) FILTER (WHERE av.raw_app = true)::BIGINT AS \"raw!\"\n        FROM app a\n        JOIN app_version av ON av.id = a.versions[array_upper(a.versions, 1)]\n        ",
+  "describe": {
+    "columns": [
+      {
+        "ordinal": 0,
+        "name": "low_code!",
+        "type_info": "Int8"
+      },
+      {
+        "ordinal": 1,
+        "name": "raw!",
+        "type_info": "Int8"
+      }
+    ],
+    "parameters": {
+      "Left": []
+    },
+    "nullable": [
+      null,
+      null
+    ]
+  },
+  "hash": "60dc0f1fa17bd2946ba7ddf0c41fe58b8a53d071cdc83eca3092194b4a9c9174"
+}
@@ -0,0 +1,20 @@
+{
+  "db_name": "PostgreSQL",
+  "query": "SELECT COUNT(*)::BIGINT AS \"count!\" FROM workspace WHERE deleted = false AND id NOT LIKE 'wm-fork%'",
+  "describe": {
+    "columns": [
+      {
+        "ordinal": 0,
+        "name": "count!",
+        "type_info": "Int8"
+      }
+    ],
+    "parameters": {
+      "Left": []
+    },
+    "nullable": [
+      null
+    ]
+  },
+  "hash": "9170a350e1da0b33a421a119d4a5b86575c1be761de158ad664670e981524cbf"
+}
@@ -0,0 +1,30 @@
+{
+  "db_name": "PostgreSQL",
+  "query": "DELETE FROM v2_job_completed\n             WHERE id IN (\n                 SELECT id FROM v2_job_completed\n                 WHERE completed_at <= now() - ($1::bigint::text || ' s')::interval\n                   AND ($3::timestamptz IS NULL OR completed_at >= $3)\n                 ORDER BY completed_at ASC\n                 LIMIT $2\n                 FOR UPDATE SKIP LOCKED\n             )\n             RETURNING id, completed_at",
+  "describe": {
+    "columns": [
+      {
+        "ordinal": 0,
+        "name": "id",
+        "type_info": "Uuid"
+      },
+      {
+        "ordinal": 1,
+        "name": "completed_at",
+        "type_info": "Timestamptz"
+      }
+    ],
+    "parameters": {
+      "Left": [
+        "Int8",
+        "Int8",
+        "Timestamptz"
+      ]
+    },
+    "nullable": [
+      false,
+      false
+    ]
+  },
+  "hash": "a2d4a8aedb15e9faf0a2512fa4241a9e9f2d5a56e12a9d6805ca58ff40f61614"
+}
@@ -0,0 +1,17 @@
+{
+  "db_name": "PostgreSQL",
+  "query": "INSERT INTO workspace (id, name, owner, deleted, premium, parent_workspace_id)\n         SELECT $1, $2, owner, false, premium,\n                CASE WHEN $4 THEN parent_workspace_id ELSE NULL END\n         FROM workspace WHERE id = $3",
+  "describe": {
+    "columns": [],
+    "parameters": {
+      "Left": [
+        "Varchar",
+        "Varchar",
+        "Text",
+        "Bool"
+      ]
+    },
+    "nullable": []
+  },
+  "hash": "a54efa4a7466e61fd54d8fe293cb775225dcb430026cebe15ba4994ac636514d"
+}
@@ -0,0 +1,38 @@
+{
+  "db_name": "PostgreSQL",
+  "query": "\n        SELECT\n            EXISTS(SELECT 1 FROM pg_proc WHERE proname = 'aurora_version') AS \"aurora!\",\n            EXISTS(SELECT 1 FROM pg_roles WHERE rolname = 'rds_superuser') AS \"rds!\",\n            EXISTS(SELECT 1 FROM pg_roles WHERE rolname = 'cloudsqlsuperuser') AS \"cloudsql!\",\n            EXISTS(SELECT 1 FROM pg_roles WHERE rolname IN ('azure_pg_admin', 'azuresu')) AS \"azure!\"\n        ",
+  "describe": {
+    "columns": [
+      {
+        "ordinal": 0,
+        "name": "aurora!",
+        "type_info": "Bool"
+      },
+      {
+        "ordinal": 1,
+        "name": "rds!",
+        "type_info": "Bool"
+      },
+      {
+        "ordinal": 2,
+        "name": "cloudsql!",
+        "type_info": "Bool"
+      },
+      {
+        "ordinal": 3,
+        "name": "azure!",
+        "type_info": "Bool"
+      }
+    ],
+    "parameters": {
+      "Left": []
+    },
+    "nullable": [
+      null,
+      null,
+      null,
+      null
+    ]
+  },
+  "hash": "a90e3a1d7c7c0dfb422f44b0ed599f681f5630c024ae9a437a301f149636b0db"
+}
@@ -0,0 +1,31 @@
+{
+  "db_name": "PostgreSQL",
+  "query": "DELETE FROM v2_job_completed\n             WHERE id IN (\n                 SELECT jc.id FROM v2_job_completed jc\n                 LEFT JOIN v2_job j ON j.id = jc.id\n                 WHERE jc.completed_at <= now() - ($1::bigint::text || ' s')::interval\n                   AND ($4::timestamptz IS NULL OR jc.completed_at >= $4)\n                   AND COALESCE(j.root_job, j.flow_innermost_root_job, jc.id) NOT IN (\n                       SELECT u FROM unnest($3::uuid[]) AS u WHERE u IS NOT NULL\n                   )\n                 ORDER BY jc.completed_at ASC\n                 LIMIT $2\n                 FOR UPDATE OF jc SKIP LOCKED\n             )\n             RETURNING id, completed_at",
+  "describe": {
+    "columns": [
+      {
+        "ordinal": 0,
+        "name": "id",
+        "type_info": "Uuid"
+      },
+      {
+        "ordinal": 1,
+        "name": "completed_at",
+        "type_info": "Timestamptz"
+      }
+    ],
+    "parameters": {
+      "Left": [
+        "Int8",
+        "Int8",
+        "UuidArray",
+        "Timestamptz"
+      ]
+    },
+    "nullable": [
+      false,
+      false
+    ]
+  },
+  "hash": "c033a690fde04da79745e850b72fa7cfd861f1dcad88c7ea75a0a8b014ec1f75"
+}
@@ -1,24 +0,0 @@
-{
-  "db_name": "PostgreSQL",
-  "query": "DELETE FROM v2_job_completed\n         WHERE id IN (\n             SELECT jc.id FROM v2_job_completed jc\n             LEFT JOIN v2_job j ON j.id = jc.id\n             WHERE jc.completed_at <= now() - ($1::bigint::text || ' s')::interval\n               AND COALESCE(j.root_job, j.flow_innermost_root_job, jc.id) NOT IN (\n                   SELECT u FROM unnest($3::uuid[]) AS u WHERE u IS NOT NULL\n               )\n             ORDER BY jc.completed_at ASC\n             LIMIT $2\n             FOR UPDATE OF jc SKIP LOCKED\n         )\n         RETURNING id",
-  "describe": {
-    "columns": [
-      {
-        "ordinal": 0,
-        "name": "id",
-        "type_info": "Uuid"
-      }
-    ],
-    "parameters": {
-      "Left": [
-        "Int8",
-        "Int8",
-        "UuidArray"
-      ]
-    },
-    "nullable": [
-      false
-    ]
-  },
-  "hash": "fbe3a876efd1253d2ef086b03366b2bd117ceb6bc152d2abcd45850ff6aecff9"
-}
@@ -0,0 +1,29 @@
+{
+  "db_name": "PostgreSQL",
+  "query": "SELECT req.id AS \"id!\",\n                (CASE\n                    WHEN usr.email IS NULL THEN 'deleted'\n                    WHEN workspace.deleted THEN 'archived'\n                    ELSE 'active'\n                END) AS \"status!\"\n         FROM unnest($1::text[]) AS req(id)\n         LEFT JOIN workspace ON workspace.id = req.id\n         LEFT JOIN usr ON usr.workspace_id = workspace.id AND usr.email = $2",
+  "describe": {
+    "columns": [
+      {
+        "ordinal": 0,
+        "name": "id!",
+        "type_info": "Text"
+      },
+      {
+        "ordinal": 1,
+        "name": "status!",
+        "type_info": "Text"
+      }
+    ],
+    "parameters": {
+      "Left": [
+        "TextArray",
+        "Text"
+      ]
+    },
+    "nullable": [
+      null,
+      null
+    ]
+  },
+  "hash": "fc4583d1570f3a2a428bb28390ca72e61719fae68aa4b42730f9076f3bd97441"
+}
@@ -2101,9 +2101,9 @@ checksum = "613afe47fcd5fac7ccf1db93babcb082c5994d996f20b8b159f2ad1658eb5724"

 [[package]]
 name = "chacha20"
-version = "0.10.0"
+version = "0.10.1"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "6f8d983286843e49675a4b7a2d174efe136dc93a18d69130dd18198a6c167601"
+checksum = "d524456ba66e72eb8b115ff89e01e497f8e6d11d78b70b1aa13c0fbd97540a81"
 dependencies = [
 "cfg-if",
 "cpufeatures 0.3.0",
@@ -5712,9 +5712,9 @@ dependencies = [

 [[package]]
 name = "hyper-http-proxy"
-version = "1.1.0"
+version = "1.1.1"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "7ad4b0a1e37510028bc4ba81d0e38d239c39671b0f0ce9e02dfa93a8133f7c08"
+checksum = "8021e0ae20c08eadc94d0bdafdeda66d4f0858541c146ae6e46b219bfe58497e"
 dependencies = [
 "bytes",
 "futures-util",
@@ -5726,7 +5726,6 @@ dependencies = [
 "hyper-util",
 "native-tls",
 "pin-project-lite",
- "rustls-native-certs 0.7.3",
 "tokio",
 "tokio-native-tls",
 "tokio-rustls 0.26.4",
@@ -13735,7 +13734,7 @@ dependencies = [

 [[package]]
 name = "windmill"
-version = "1.737.0"
+version = "1.739.0"
 dependencies = [
 "anyhow",
 "async-nats",
@@ -13817,7 +13816,7 @@ dependencies = [

 [[package]]
 name = "windmill-ai"
-version = "1.737.0"
+version = "1.739.0"
 dependencies = [
 "async-stream",
 "async-trait",
@@ -13850,7 +13849,7 @@ dependencies = [

 [[package]]
 name = "windmill-alerting"
-version = "1.737.0"
+version = "1.739.0"
 dependencies = [
 "axum 0.8.9",
 "chrono",
@@ -13863,7 +13862,7 @@ dependencies = [

 [[package]]
 name = "windmill-api"
-version = "1.737.0"
+version = "1.739.0"
 dependencies = [
 "anyhow",
 "argon2",
@@ -14001,7 +14000,7 @@ dependencies = [

 [[package]]
 name = "windmill-api-agent-workers"
-version = "1.737.0"
+version = "1.739.0"
 dependencies = [
 "axum 0.8.9",
 "chrono",
@@ -14024,7 +14023,7 @@ dependencies = [

 [[package]]
 name = "windmill-api-assets"
-version = "1.737.0"
+version = "1.739.0"
 dependencies = [
 "axum 0.8.9",
 "chrono",
@@ -14037,7 +14036,7 @@ dependencies = [

 [[package]]
 name = "windmill-api-auth"
-version = "1.737.0"
+version = "1.739.0"
 dependencies = [
 "anyhow",
 "axum 0.8.9",
@@ -14063,7 +14062,7 @@ dependencies = [

 [[package]]
 name = "windmill-api-client"
-version = "1.737.0"
+version = "1.739.0"
 dependencies = [
 "reqwest 0.12.28",
 "serde",
@@ -14073,7 +14072,7 @@ dependencies = [

 [[package]]
 name = "windmill-api-configs"
-version = "1.737.0"
+version = "1.739.0"
 dependencies = [
 "axum 0.8.9",
 "chrono",
@@ -14090,7 +14089,7 @@ dependencies = [

 [[package]]
 name = "windmill-api-debug"
-version = "1.737.0"
+version = "1.739.0"
 dependencies = [
 "axum 0.8.9",
 "base64 0.22.1",
@@ -14112,7 +14111,7 @@ dependencies = [

 [[package]]
 name = "windmill-api-embeddings"
-version = "1.737.0"
+version = "1.739.0"
 dependencies = [
 "anyhow",
 "axum 0.8.9",
@@ -14135,7 +14134,7 @@ dependencies = [

 [[package]]
 name = "windmill-api-flow-conversations"
-version = "1.737.0"
+version = "1.739.0"
 dependencies = [
 "axum 0.8.9",
 "chrono",
@@ -14151,7 +14150,7 @@ dependencies = [

 [[package]]
 name = "windmill-api-flows"
-version = "1.737.0"
+version = "1.739.0"
 dependencies = [
 "axum 0.8.9",
 "chrono",
@@ -14172,7 +14171,7 @@ dependencies = [

 [[package]]
 name = "windmill-api-groups"
-version = "1.737.0"
+version = "1.739.0"
 dependencies = [
 "axum 0.8.9",
 "chrono",
@@ -14193,7 +14192,7 @@ dependencies = [

 [[package]]
 name = "windmill-api-inputs"
-version = "1.737.0"
+version = "1.739.0"
 dependencies = [
 "axum 0.8.9",
 "chrono",
@@ -14207,7 +14206,7 @@ dependencies = [

 [[package]]
 name = "windmill-api-integration-tests"
-version = "1.737.0"
+version = "1.739.0"
 dependencies = [
 "anyhow",
 "async-nats",
@@ -14242,7 +14241,7 @@ dependencies = [

 [[package]]
 name = "windmill-api-jobs"
-version = "1.737.0"
+version = "1.739.0"
 dependencies = [
 "anyhow",
 "axum 0.8.9",
@@ -14267,7 +14266,7 @@ dependencies = [

 [[package]]
 name = "windmill-api-npm-proxy"
-version = "1.737.0"
+version = "1.739.0"
 dependencies = [
 "axum 0.8.9",
 "flate2",
@@ -14285,7 +14284,7 @@ dependencies = [

 [[package]]
 name = "windmill-api-openapi"
-version = "1.737.0"
+version = "1.739.0"
 dependencies = [
 "anyhow",
 "axum 0.8.9",
@@ -14307,7 +14306,7 @@ dependencies = [

 [[package]]
 name = "windmill-api-schedule"
-version = "1.737.0"
+version = "1.739.0"
 dependencies = [
 "axum 0.8.9",
 "chrono",
@@ -14327,7 +14326,7 @@ dependencies = [

 [[package]]
 name = "windmill-api-scripts"
-version = "1.737.0"
+version = "1.739.0"
 dependencies = [
 "axum 0.8.9",
 "chrono",
@@ -14364,7 +14363,7 @@ dependencies = [

 [[package]]
 name = "windmill-api-settings"
-version = "1.737.0"
+version = "1.739.0"
 dependencies = [
 "anyhow",
 "axum 0.8.9",
@@ -14392,7 +14391,7 @@ dependencies = [

 [[package]]
 name = "windmill-api-sse"
-version = "1.737.0"
+version = "1.739.0"
 dependencies = [
 "lazy_static",
 "serde",
@@ -14404,7 +14403,7 @@ dependencies = [

 [[package]]
 name = "windmill-api-users"
-version = "1.737.0"
+version = "1.739.0"
 dependencies = [
 "argon2",
 "axum 0.8.9",
@@ -14429,7 +14428,7 @@ dependencies = [

 [[package]]
 name = "windmill-api-workers"
-version = "1.737.0"
+version = "1.739.0"
 dependencies = [
 "axum 0.8.9",
 "chrono",
@@ -14443,7 +14442,7 @@ dependencies = [

 [[package]]
 name = "windmill-api-workspaces"
-version = "1.737.0"
+version = "1.739.0"
 dependencies = [
 "axum 0.8.9",
 "chrono",
@@ -14476,7 +14475,7 @@ dependencies = [

 [[package]]
 name = "windmill-audit"
-version = "1.737.0"
+version = "1.739.0"
 dependencies = [
 "chrono",
 "lazy_static",
@@ -14490,7 +14489,7 @@ dependencies = [

 [[package]]
 name = "windmill-autoscaling"
-version = "1.737.0"
+version = "1.739.0"
 dependencies = [
 "anyhow",
 "axum 0.8.9",
@@ -14509,7 +14508,7 @@ dependencies = [

 [[package]]
 name = "windmill-common"
-version = "1.737.0"
+version = "1.739.0"
 dependencies = [
 "aes-gcm",
 "aho-corasick",
@@ -14611,7 +14610,7 @@ dependencies = [

 [[package]]
 name = "windmill-dep-map"
-version = "1.737.0"
+version = "1.739.0"
 dependencies = [
 "chrono",
 "itertools 0.14.0",
@@ -14630,7 +14629,7 @@ dependencies = [

 [[package]]
 name = "windmill-git-sync"
-version = "1.737.0"
+version = "1.739.0"
 dependencies = [
 "regex",
 "serde",
@@ -14645,7 +14644,7 @@ dependencies = [

 [[package]]
 name = "windmill-indexer"
-version = "1.737.0"
+version = "1.739.0"
 dependencies = [
 "anyhow",
 "astral-tokio-tar",
@@ -14669,7 +14668,7 @@ dependencies = [

 [[package]]
 name = "windmill-jseval"
-version = "1.737.0"
+version = "1.739.0"
 dependencies = [
 "anyhow",
 "futures",
@@ -14686,7 +14685,7 @@ dependencies = [

 [[package]]
 name = "windmill-macros"
-version = "1.737.0"
+version = "1.739.0"
 dependencies = [
 "itertools 0.14.0",
 "lazy_static",
@@ -14702,7 +14701,7 @@ dependencies = [

 [[package]]
 name = "windmill-mcp"
-version = "1.737.0"
+version = "1.739.0"
 dependencies = [
 "anyhow",
 "async-trait",
@@ -14723,7 +14722,7 @@ dependencies = [

 [[package]]
 name = "windmill-native-triggers"
-version = "1.737.0"
+version = "1.739.0"
 dependencies = [
 "anyhow",
 "async-trait",
@@ -14754,7 +14753,7 @@ dependencies = [

 [[package]]
 name = "windmill-oauth"
-version = "1.737.0"
+version = "1.739.0"
 dependencies = [
 "anyhow",
 "arc-swap",
@@ -14779,7 +14778,7 @@ dependencies = [

 [[package]]
 name = "windmill-object-store"
-version = "1.737.0"
+version = "1.739.0"
 dependencies = [
 "anyhow",
 "async-stream",
@@ -14813,7 +14812,7 @@ dependencies = [

 [[package]]
 name = "windmill-operator"
-version = "1.737.0"
+version = "1.739.0"
 dependencies = [
 "anyhow",
 "futures",
@@ -14831,7 +14830,7 @@ dependencies = [

 [[package]]
 name = "windmill-parser"
-version = "1.737.0"
+version = "1.739.0"
 dependencies = [
 "convert_case 0.6.0",
 "serde",
@@ -14840,7 +14839,7 @@ dependencies = [

 [[package]]
 name = "windmill-parser-bash"
-version = "1.737.0"
+version = "1.739.0"
 dependencies = [
 "anyhow",
 "lazy_static",
@@ -14852,7 +14851,7 @@ dependencies = [

 [[package]]
 name = "windmill-parser-csharp"
-version = "1.737.0"
+version = "1.739.0"
 dependencies = [
 "anyhow",
 "serde_json",
@@ -14864,7 +14863,7 @@ dependencies = [

 [[package]]
 name = "windmill-parser-go"
-version = "1.737.0"
+version = "1.739.0"
 dependencies = [
 "anyhow",
 "gosyn",
@@ -14876,7 +14875,7 @@ dependencies = [

 [[package]]
 name = "windmill-parser-graphql"
-version = "1.737.0"
+version = "1.739.0"
 dependencies = [
 "anyhow",
 "lazy_static",
@@ -14888,7 +14887,7 @@ dependencies = [

 [[package]]
 name = "windmill-parser-java"
-version = "1.737.0"
+version = "1.739.0"
 dependencies = [
 "anyhow",
 "serde_json",
@@ -14900,7 +14899,7 @@ dependencies = [

 [[package]]
 name = "windmill-parser-nu"
-version = "1.737.0"
+version = "1.739.0"
 dependencies = [
 "anyhow",
 "nu-parser",
@@ -14911,7 +14910,7 @@ dependencies = [

 [[package]]
 name = "windmill-parser-php"
-version = "1.737.0"
+version = "1.739.0"
 dependencies = [
 "anyhow",
 "itertools 0.14.0",
@@ -14922,7 +14921,7 @@ dependencies = [

 [[package]]
 name = "windmill-parser-py"
-version = "1.737.0"
+version = "1.739.0"
 dependencies = [
 "anyhow",
 "itertools 0.14.0",
@@ -14934,7 +14933,7 @@ dependencies = [

 [[package]]
 name = "windmill-parser-py-asset"
-version = "1.737.0"
+version = "1.739.0"
 dependencies = [
 "anyhow",
 "rustpython-ast",
@@ -14945,7 +14944,7 @@ dependencies = [

 [[package]]
 name = "windmill-parser-py-imports"
-version = "1.737.0"
+version = "1.739.0"
 dependencies = [
 "anyhow",
 "async-recursion",
@@ -14967,7 +14966,7 @@ dependencies = [

 [[package]]
 name = "windmill-parser-r"
-version = "1.737.0"
+version = "1.739.0"
 dependencies = [
 "anyhow",
 "serde_json",
@@ -14979,7 +14978,7 @@ dependencies = [

 [[package]]
 name = "windmill-parser-ruby"
-version = "1.737.0"
+version = "1.739.0"
 dependencies = [
 "anyhow",
 "lazy_static",
@@ -14993,7 +14992,7 @@ dependencies = [

 [[package]]
 name = "windmill-parser-rust"
-version = "1.737.0"
+version = "1.739.0"
 dependencies = [
 "anyhow",
 "convert_case 0.6.0",
@@ -15010,7 +15009,7 @@ dependencies = [

 [[package]]
 name = "windmill-parser-sql"
-version = "1.737.0"
+version = "1.739.0"
 dependencies = [
 "anyhow",
 "lazy_static",
@@ -15023,7 +15022,7 @@ dependencies = [

 [[package]]
 name = "windmill-parser-sql-asset"
-version = "1.737.0"
+version = "1.739.0"
 dependencies = [
 "anyhow",
 "serde",
@@ -15035,7 +15034,7 @@ dependencies = [

 [[package]]
 name = "windmill-parser-ts"
-version = "1.737.0"
+version = "1.739.0"
 dependencies = [
 "anyhow",
 "lazy_static",
@@ -15053,7 +15052,7 @@ dependencies = [

 [[package]]
 name = "windmill-parser-ts-asset"
-version = "1.737.0"
+version = "1.739.0"
 dependencies = [
 "anyhow",
 "serde-wasm-bindgen",
@@ -15069,7 +15068,7 @@ dependencies = [

 [[package]]
 name = "windmill-parser-wac"
-version = "1.737.0"
+version = "1.739.0"
 dependencies = [
 "anyhow",
 "rustpython-ast",
@@ -15085,7 +15084,7 @@ dependencies = [

 [[package]]
 name = "windmill-parser-yaml"
-version = "1.737.0"
+version = "1.739.0"
 dependencies = [
 "anyhow",
 "serde",
@@ -15096,7 +15095,7 @@ dependencies = [

 [[package]]
 name = "windmill-queue"
-version = "1.737.0"
+version = "1.739.0"
 dependencies = [
 "anyhow",
 "async-recursion",
@@ -15134,7 +15133,7 @@ dependencies = [

 [[package]]
 name = "windmill-runtime-nativets"
-version = "1.737.0"
+version = "1.739.0"
 dependencies = [
 "anyhow",
 "const_format",
@@ -15173,7 +15172,7 @@ dependencies = [

 [[package]]
 name = "windmill-sql-datatype-parser-wasm"
-version = "1.737.0"
+version = "1.739.0"
 dependencies = [
 "getrandom 0.3.4",
 "wasm-bindgen",
@@ -15184,7 +15183,7 @@ dependencies = [

 [[package]]
 name = "windmill-store"
-version = "1.737.0"
+version = "1.739.0"
 dependencies = [
 "anyhow",
 "async-recursion",
@@ -15218,7 +15217,7 @@ dependencies = [

 [[package]]
 name = "windmill-test-utils"
-version = "1.737.0"
+version = "1.739.0"
 dependencies = [
 "anyhow",
 "async-trait",
@@ -15242,7 +15241,7 @@ dependencies = [

 [[package]]
 name = "windmill-trigger"
-version = "1.737.0"
+version = "1.739.0"
 dependencies = [
 "anyhow",
 "async-trait",
@@ -15275,7 +15274,7 @@ dependencies = [

 [[package]]
 name = "windmill-trigger-azure"
-version = "1.737.0"
+version = "1.739.0"
 dependencies = [
 "anyhow",
 "async-trait",
@@ -15308,7 +15307,7 @@ dependencies = [

 [[package]]
 name = "windmill-trigger-email"
-version = "1.737.0"
+version = "1.739.0"
 dependencies = [
 "anyhow",
 "async-trait",
@@ -15328,7 +15327,7 @@ dependencies = [

 [[package]]
 name = "windmill-trigger-gcp"
-version = "1.737.0"
+version = "1.739.0"
 dependencies = [
 "anyhow",
 "async-trait",
@@ -15362,7 +15361,7 @@ dependencies = [

 [[package]]
 name = "windmill-trigger-http"
-version = "1.737.0"
+version = "1.739.0"
 dependencies = [
 "anyhow",
 "async-trait",
@@ -15398,7 +15397,7 @@ dependencies = [

 [[package]]
 name = "windmill-trigger-kafka"
-version = "1.737.0"
+version = "1.739.0"
 dependencies = [
 "anyhow",
 "async-trait",
@@ -15421,7 +15420,7 @@ dependencies = [

 [[package]]
 name = "windmill-trigger-mqtt"
-version = "1.737.0"
+version = "1.739.0"
 dependencies = [
 "anyhow",
 "async-trait",
@@ -15445,7 +15444,7 @@ dependencies = [

 [[package]]
 name = "windmill-trigger-nats"
-version = "1.737.0"
+version = "1.739.0"
 dependencies = [
 "anyhow",
 "async-nats",
@@ -15469,7 +15468,7 @@ dependencies = [

 [[package]]
 name = "windmill-trigger-postgres"
-version = "1.737.0"
+version = "1.739.0"
 dependencies = [
 "anyhow",
 "async-trait",
@@ -15504,7 +15503,7 @@ dependencies = [

 [[package]]
 name = "windmill-trigger-sqs"
-version = "1.737.0"
+version = "1.739.0"
 dependencies = [
 "anyhow",
 "async-trait",
@@ -15532,7 +15531,7 @@ dependencies = [

 [[package]]
 name = "windmill-trigger-websocket"
-version = "1.737.0"
+version = "1.739.0"
 dependencies = [
 "anyhow",
 "async-trait",
@@ -15557,7 +15556,7 @@ dependencies = [

 [[package]]
 name = "windmill-types"
-version = "1.737.0"
+version = "1.739.0"
 dependencies = [
 "anyhow",
 "bitflags 2.13.0",
@@ -15576,7 +15575,7 @@ dependencies = [

 [[package]]
 name = "windmill-worker"
-version = "1.737.0"
+version = "1.739.0"
 dependencies = [
 "anyhow",
 "async-once-cell",
@@ -15686,7 +15685,7 @@ dependencies = [

 [[package]]
 name = "windmill-worker-volumes"
-version = "1.737.0"
+version = "1.739.0"
 dependencies = [
 "bytes",
 "futures",
@@ -1,6 +1,6 @@
 [package]
 name = "windmill"
-version = "1.737.0"
+version = "1.739.0"
 authors.workspace = true
 edition.workspace = true

@@ -87,7 +87,7 @@ members = [
 exclude = ["./windmill-duckdb-ffi-internal", "./parsers/windmill-parser-wasm"]

 [workspace.package]
-version = "1.737.0"
+version = "1.739.0"
 authors = ["Ruben Fiszel <ruben@windmill.dev>"]
 edition = "2021"

@@ -1 +1 @@
-ac1f6f666f36141cb6ba6f8eaa614821a90464ad
+ed89574be9117cda5e2d7d9de02cb5db066e93e3
@@ -0,0 +1,3 @@
+-- No-op: this migration only re-pins the function's search_path. Reverting would
+-- mean restoring the hardcoded `SET search_path = public`, which is the very bug
+-- this repairs, so there is nothing to undo.
@@ -0,0 +1,22 @@
+-- Repair instances that already applied the folder-labels migrations while the
+-- function hardcoded `SET search_path = public`. On a non-public schema (PG_SCHEMA)
+-- the function was pinned to `public`, so at runtime it read the wrong `folder`
+-- table (or a stray public.folder) instead of the workspace's real one.
+--
+-- `FROM CURRENT` snapshots the migration connection's search_path (the actual
+-- Windmill schema) into the function, keeping the SECURITY DEFINER injection
+-- hardening. On public-schema installs this re-pins to `public`, i.e. a no-op.
+-- Idempotent: redefining with the same body is harmless on already-correct installs.
+CREATE OR REPLACE FUNCTION folder_labels(w_id text, item_path text) RETURNS text[]
+LANGUAGE sql STABLE SECURITY DEFINER SET search_path FROM CURRENT AS $$
+    SELECT (
+        SELECT array_agg(l ORDER BY first_ord)
+        FROM (
+            SELECT u.l, min(u.ord) AS first_ord
+            FROM unnest(f.labels) WITH ORDINALITY AS u(l, ord)
+            GROUP BY u.l
+        ) deduped
+    )
+    FROM folder f
+    WHERE f.workspace_id = w_id AND item_path LIKE 'f/%' AND f.name = split_part(item_path, '/', 2)
+$$;
@@ -0,0 +1,3 @@
+-- Irreversible data backfill: once a raw app's draft is retyped to 'raw_app' it
+-- is indistinguishable from one saved as 'raw_app' by the per-kind code, so the
+-- original typ='app' state cannot be reconstructed. No-op on revert.
@@ -0,0 +1,35 @@
+-- The pre-per-user `DRAFT_TYPE` enum had only ('script','flow','app'): a raw
+-- app's draft was therefore stored as typ='app'. The new model splits app vs
+-- raw_app into distinct draft kinds chosen from the deployed app's `raw_app`
+-- flag, so a raw app's pre-migration draft is invisible to the per-kind lookups
+-- (editor overlay, migrate-legacy, get-for-user), which all query typ='raw_app'.
+-- Realign every such draft (any owner, including the legacy NULL-email row) to
+-- 'raw_app' when the deployed app at that path is a raw app.
+
+-- Drop, don't retype, a stale 'app' row when a 'raw_app' draft already exists
+-- for the same owner (the newer 'raw_app' row, saved with the per-kind code, is
+-- authoritative) — retyping would collide on the draft_pkey_with_user /
+-- draft_pkey_legacy partial unique indexes over (workspace_id, path, typ, email).
+DELETE FROM draft d
+USING app a
+JOIN app_version av ON av.id = a.versions[array_upper(a.versions, 1)]
+WHERE d.typ = 'app'
+  AND a.workspace_id = d.workspace_id
+  AND a.path = d.path
+  AND av.raw_app IS TRUE
+  AND EXISTS (
+    SELECT 1 FROM draft d2
+    WHERE d2.workspace_id = d.workspace_id
+      AND d2.path = d.path
+      AND d2.typ = 'raw_app'
+      AND d2.email IS NOT DISTINCT FROM d.email
+  );
+
+UPDATE draft d
+SET typ = 'raw_app'
+FROM app a
+JOIN app_version av ON av.id = a.versions[array_upper(a.versions, 1)]
+WHERE d.typ = 'app'
+  AND a.workspace_id = d.workspace_id
+  AND a.path = d.path
+  AND av.raw_app IS TRUE;
@@ -6191,7 +6191,7 @@ checksum = "712e227841d057c1ee1cd2fb22fa7e5a5461ae8e48fa2ca79ec42cfc1931183f"

 [[package]]
 name = "windmill-common"
-version = "1.737.0"
+version = "1.739.0"
 dependencies = [
 "aho-corasick",
 "anyhow",
@@ -6272,7 +6272,7 @@ dependencies = [

 [[package]]
 name = "windmill-macros"
-version = "1.737.0"
+version = "1.739.0"
 dependencies = [
 "proc-macro2",
 "quote",
@@ -6284,7 +6284,7 @@ dependencies = [

 [[package]]
 name = "windmill-parser"
-version = "1.737.0"
+version = "1.739.0"
 dependencies = [
 "convert_case",
 "serde",
@@ -6293,7 +6293,7 @@ dependencies = [

 [[package]]
 name = "windmill-parser-bash"
-version = "1.737.0"
+version = "1.739.0"
 dependencies = [
 "anyhow",
 "lazy_static",
@@ -6305,7 +6305,7 @@ dependencies = [

 [[package]]
 name = "windmill-parser-csharp"
-version = "1.737.0"
+version = "1.739.0"
 dependencies = [
 "anyhow",
 "serde_json",
@@ -6317,7 +6317,7 @@ dependencies = [

 [[package]]
 name = "windmill-parser-go"
-version = "1.737.0"
+version = "1.739.0"
 dependencies = [
 "anyhow",
 "gosyn",
@@ -6329,7 +6329,7 @@ dependencies = [

 [[package]]
 name = "windmill-parser-graphql"
-version = "1.737.0"
+version = "1.739.0"
 dependencies = [
 "anyhow",
 "lazy_static",
@@ -6341,7 +6341,7 @@ dependencies = [

 [[package]]
 name = "windmill-parser-java"
-version = "1.737.0"
+version = "1.739.0"
 dependencies = [
 "anyhow",
 "serde_json",
@@ -6353,7 +6353,7 @@ dependencies = [

 [[package]]
 name = "windmill-parser-nu"
-version = "1.737.0"
+version = "1.739.0"
 dependencies = [
 "anyhow",
 "nu-parser",
@@ -6364,7 +6364,7 @@ dependencies = [

 [[package]]
 name = "windmill-parser-php"
-version = "1.737.0"
+version = "1.739.0"
 dependencies = [
 "anyhow",
 "itertools 0.14.0",
@@ -6375,7 +6375,7 @@ dependencies = [

 [[package]]
 name = "windmill-parser-py"
-version = "1.737.0"
+version = "1.739.0"
 dependencies = [
 "anyhow",
 "itertools 0.14.0",
@@ -6387,7 +6387,7 @@ dependencies = [

 [[package]]
 name = "windmill-parser-py-asset"
-version = "1.737.0"
+version = "1.739.0"
 dependencies = [
 "anyhow",
 "rustpython-ast",
@@ -6398,7 +6398,7 @@ dependencies = [

 [[package]]
 name = "windmill-parser-py-imports"
-version = "1.737.0"
+version = "1.739.0"
 dependencies = [
 "anyhow",
 "async-recursion",
@@ -6420,7 +6420,7 @@ dependencies = [

 [[package]]
 name = "windmill-parser-r"
-version = "1.737.0"
+version = "1.739.0"
 dependencies = [
 "anyhow",
 "serde_json",
@@ -6432,7 +6432,7 @@ dependencies = [

 [[package]]
 name = "windmill-parser-ruby"
-version = "1.737.0"
+version = "1.739.0"
 dependencies = [
 "anyhow",
 "lazy_static",
@@ -6446,7 +6446,7 @@ dependencies = [

 [[package]]
 name = "windmill-parser-rust"
-version = "1.737.0"
+version = "1.739.0"
 dependencies = [
 "anyhow",
 "convert_case",
@@ -6463,7 +6463,7 @@ dependencies = [

 [[package]]
 name = "windmill-parser-sql"
-version = "1.737.0"
+version = "1.739.0"
 dependencies = [
 "anyhow",
 "lazy_static",
@@ -6476,7 +6476,7 @@ dependencies = [

 [[package]]
 name = "windmill-parser-sql-asset"
-version = "1.737.0"
+version = "1.739.0"
 dependencies = [
 "anyhow",
 "serde",
@@ -6488,7 +6488,7 @@ dependencies = [

 [[package]]
 name = "windmill-parser-ts"
-version = "1.737.0"
+version = "1.739.0"
 dependencies = [
 "anyhow",
 "lazy_static",
@@ -6506,7 +6506,7 @@ dependencies = [

 [[package]]
 name = "windmill-parser-ts-asset"
-version = "1.737.0"
+version = "1.739.0"
 dependencies = [
 "anyhow",
 "serde-wasm-bindgen",
@@ -6522,7 +6522,7 @@ dependencies = [

 [[package]]
 name = "windmill-parser-wac"
-version = "1.737.0"
+version = "1.739.0"
 dependencies = [
 "anyhow",
 "rustpython-ast",
@@ -6538,7 +6538,7 @@ dependencies = [

 [[package]]
 name = "windmill-parser-wasm"
-version = "1.737.0"
+version = "1.739.0"
 dependencies = [
 "anyhow",
 "getrandom 0.2.17",
@@ -6570,7 +6570,7 @@ dependencies = [

 [[package]]
 name = "windmill-parser-yaml"
-version = "1.737.0"
+version = "1.739.0"
 dependencies = [
 "anyhow",
 "serde",
@@ -6581,7 +6581,7 @@ dependencies = [

 [[package]]
 name = "windmill-types"
-version = "1.737.0"
+version = "1.739.0"
 dependencies = [
 "anyhow",
 "bitflags",
@@ -12,7 +12,7 @@ resolver = "2"
 members = ["."]

 [workspace.package]
-version = "1.737.0"
+version = "1.739.0"
 edition = "2021"
 authors = ["Ruben Fiszel <ruben@windmill.dev>"]

@@ -112,6 +112,11 @@ pub struct ParseAssetsOutput {
    // Drives the worker's write-strategy + snapshot capture.
    #[serde(skip_serializing_if = "Option::is_none", default)]
    pub materialize: Option<MaterializeSpec>,
+    // `// data_test <kind> …` — data-quality assertions run against the
+    // materialized asset after the write commits. Accumulating (multiple
+    // lines allowed). Drives the worker's post-materialize verifier probes.
+    #[serde(skip_serializing_if = "Vec::is_empty", default)]
+    pub data_tests: Vec<DataTest>,
 }

 #[derive(Serialize, Debug, PartialEq, Clone)]
@@ -235,6 +240,41 @@ pub struct MaterializeSpec {
    pub unique_key: Option<String>,
 }

+// `// data_test <kind> …` — a data-quality assertion run against the
+// freshly-materialized asset (post DELETE+INSERT), failing the run on
+// violation. The first extensible annotation family: the parser turns a
+// `data_test` line into one of a known *vocabulary* of checks, and the
+// runtime turns each check into a SQL "verifier" probe. A sibling annotation
+// family (e.g. column-lineage) follows the same shape — a keyword head
+// selecting a variant, the rest parsed per-variant — rather than growing a
+// new closed list. See `docs/ducklake-materialization.md` §"Extensible
+// annotations". Multiple `// data_test` lines accumulate (unlike the
+// single-value annotations above, which are first-write-wins).
+//
+// Built-ins mirror dbt's generic data tests; `Custom` is the escape hatch
+// (dbt's singular test): a DuckDB script path whose SELECT returns the
+// violating rows. The keyword is `data_test` — NOT `test` — to stay clear
+// of the unrelated `// test:` CI-test annotation (see
+// `windmill_common::schema::parse_ci_test_annotation`), matching dbt 1.8's
+// own `tests:` → `data_tests:` rename.
+#[derive(Serialize, Debug, PartialEq, Clone)]
+#[serde(tag = "type", rename_all = "snake_case")]
+pub enum DataTest {
+    // `// data_test unique <col>` — no two non-NULL rows share `column`.
+    Unique { column: String },
+    // `// data_test not_null <col>` — `column` is never NULL.
+    NotNull { column: String },
+    // `// data_test accepted_values <col> = a,b,c` — every non-NULL value of
+    // `column` is one of `values` (comma-separated; surrounding quotes stripped).
+    AcceptedValues { column: String, values: Vec<String> },
+    // `// data_test relationships <col> -> <asset>.<refcol>` — referential
+    // integrity: every non-NULL `column` value exists in `to_path`'s `to_column`.
+    Relationships { column: String, to_kind: AssetKind, to_path: String, to_column: String },
+    // `// data_test <script_path>` — escape hatch: a deployed DuckDB script
+    // whose trailing SELECT returns the violating rows (non-empty ⇒ fail).
+    Custom { path: String },
+}
+
 // `// trigger any` (default) vs `// trigger all`. `Any` = OR: any trigger
 // firing runs the script (current behaviour). `All` = AND: the script
 // runs only once every partition-bearing input has materialized at the
@@ -266,6 +306,7 @@ pub struct PipelineAnnotations {
    pub tag: Option<String>,
    pub retry: Option<RetrySpec>,
    pub materialize: Option<MaterializeSpec>,
+    pub data_tests: Vec<DataTest>,
 }

 impl ParseAssetsOutput {
@@ -290,6 +331,7 @@ impl ParseAssetsOutput {
            tag: pipeline.tag,
            retry: pipeline.retry,
            materialize: pipeline.materialize,
+            data_tests: pipeline.data_tests,
        }
    }
 }
@@ -487,9 +529,13 @@ fn parse_kv_opts(s: &str) -> BTreeMap<String, String> {
    out
 }

-// Scan raw source for pipeline annotations. Language-agnostic: any line
-// whose first non-whitespace tokens are a comment prefix (`//`, `#`, or
-// `--`) followed by one of the recognized keywords:
+// Scan the leading comment header for pipeline annotations. Only the
+// contiguous block of comment lines at the top of the file is considered
+// (blank lines tolerated, scan stops at the first line of actual code) so
+// that ordinary comments in the body can't false-positive as annotations.
+// Language-agnostic: any header line whose first non-whitespace tokens are
+// a comment prefix (`//`, `#`, or `--`) followed by one of the recognized
+// keywords:
 //   - `pipeline`                 → opt-in marker (must be alone on the line)
 //   - `on <trigger-spec>`        → asset / native trigger edge (including
 //                                  the marker-only `on schedule` form)
@@ -527,6 +573,9 @@ pub fn parse_pipeline_annotations(code: &str) -> PipelineAnnotations {

    for raw_line in code.lines() {
        let line = raw_line.trim_start();
+        if line.is_empty() {
+            continue;
+        }
        let rest = if let Some(r) = line.strip_prefix("//") {
            r
        } else if let Some(r) = line.strip_prefix("--") {
@@ -534,7 +583,11 @@ pub fn parse_pipeline_annotations(code: &str) -> PipelineAnnotations {
        } else if let Some(r) = line.strip_prefix('#') {
            r
        } else {
-            continue;
+            // Annotations live in the leading comment header. Stop at the first
+            // line of actual code so comments inside the body (e.g. a regular
+            // `# tag ...` prose comment) can't false-positive as annotations.
+            // Mirrors BashAnnotations::sandbox_image / ssh_target.
+            break;
        };
        let rest = rest.trim_start();

@@ -584,7 +637,14 @@ pub fn parse_pipeline_annotations(code: &str) -> PipelineAnnotations {

        if let Some(after_kw) = consume_keyword(rest, "tag") {
            let name = after_kw.trim();
-            if !name.is_empty() && out.tag.is_none() {
+            // Worker tags are single-word identifiers (e.g. `heavy`, `gpu`).
+            // A value with whitespace or beyond the `script.tag` column width
+            // is almost certainly a regular comment starting with "# tag ...".
+            if !name.is_empty()
+                && !name.contains(char::is_whitespace)
+                && name.len() <= 50
+                && out.tag.is_none()
+            {
                out.tag = Some(name.to_string());
            }
            continue;
@@ -608,6 +668,18 @@ pub fn parse_pipeline_annotations(code: &str) -> PipelineAnnotations {
            continue;
        }

+        // `data_test` is checked before `on`/asset shorthands and is a complete
+        // word (so it never collides with the `// test:` CI annotation, which
+        // has no whitespace after `test`). Accumulates — every well-formed line
+        // adds a check; malformed lines are dropped (fail-safe, the missing
+        // check is then simply absent from the graph + run).
+        if let Some(after_kw) = consume_keyword(rest, "data_test") {
+            if let Some(spec) = parse_data_test_spec(after_kw.trim()) {
+                out.data_tests.push(spec);
+            }
+            continue;
+        }
+
        if let Some(after_kw) = consume_keyword(rest, "on") {
            let spec_text = after_kw.trim();
            if spec_text.is_empty() {
@@ -684,6 +756,91 @@ fn parse_materialize_spec(s: &str) -> Option<MaterializeSpec> {
    Some(MaterializeSpec { target_kind, target_path: path.to_string(), manual, append, unique_key })
 }

+// Parse a `// data_test <kind> …` right-hand side into one `DataTest`. The
+// leading token selects the variant; the remainder is parsed per-variant.
+// Anything not matching a built-in keyword is the `Custom` escape hatch — a
+// single script-path token. Returns `None` for malformed input so a typo
+// fails safe (the check is dropped, never silently mis-parsed).
+//
+// This is the extension seam: a new built-in is one match arm + its parser;
+// a sibling annotation family (column-lineage) reuses the same head-keyword
+// dispatch shape rather than adding a parallel closed list.
+fn parse_data_test_spec(s: &str) -> Option<DataTest> {
+    let s = s.trim();
+    if s.is_empty() {
+        return None;
+    }
+    let mut it = s.splitn(2, char::is_whitespace);
+    let head = it.next()?;
+    let rest = it.next().unwrap_or("").trim();
+    match head {
+        "unique" => Some(DataTest::Unique { column: single_ident(rest)? }),
+        "not_null" => Some(DataTest::NotNull { column: single_ident(rest)? }),
+        "accepted_values" => parse_accepted_values(rest),
+        "relationships" => parse_relationships(rest),
+        // Custom escape hatch: the whole right-hand side must be one path token
+        // (`head` with no trailing content). Trailing content after a
+        // non-built-in head is a malformed built-in (e.g. `uniq order_id`) and
+        // is rejected rather than misread as a path.
+        _ if rest.is_empty() => Some(DataTest::Custom { path: head.to_string() }),
+        _ => None,
+    }
+}
+
+// A single bare identifier token (column name). Rejects empty / multi-token
+// input. The identifier is double-quoted + escaped at codegen, so any
+// character is safe here; we only enforce "exactly one token".
+fn single_ident(s: &str) -> Option<String> {
+    let s = s.trim();
+    if s.is_empty() || s.split_whitespace().count() != 1 {
+        return None;
+    }
+    Some(s.to_string())
+}
+
+// Strip one layer of matching surrounding single or double quotes.
+fn unquote(s: &str) -> &str {
+    let b = s.as_bytes();
+    if b.len() >= 2 && (b[0] == b'"' || b[0] == b'\'') && b[b.len() - 1] == b[0] {
+        &s[1..s.len() - 1]
+    } else {
+        s
+    }
+}
+
+// `<col> = a,b,c` — column, then `=`, then a comma-separated value list.
+// Surrounding quotes are stripped per value; empty values are dropped; a
+// value may not itself contain a comma (v1 limitation).
+fn parse_accepted_values(s: &str) -> Option<DataTest> {
+    let (col, vals) = s.split_once('=')?;
+    let column = single_ident(col)?;
+    let values: Vec<String> = vals
+        .split(',')
+        .map(|v| unquote(v.trim()).to_string())
+        .filter(|v| !v.is_empty())
+        .collect();
+    if values.is_empty() {
+        return None;
+    }
+    Some(DataTest::AcceptedValues { column, values })
+}
+
+// `<col> -> <asset-uri>.<refcol>` — referential integrity. The referenced
+// column is the segment after the final `.`; everything before it is the
+// asset URI (default-syntax shorthands enabled, like `// materialize`).
+fn parse_relationships(s: &str) -> Option<DataTest> {
+    let (col, target) = s.split_once("->")?;
+    let column = single_ident(col)?;
+    let target = target.trim();
+    let (asset_uri, ref_col) = target.rsplit_once('.')?;
+    let to_column = single_ident(ref_col)?;
+    let (to_kind, to_path) = parse_asset_syntax(asset_uri.trim(), true)?;
+    if to_path.is_empty() {
+        return None;
+    }
+    Some(DataTest::Relationships { column, to_kind, to_path: to_path.to_string(), to_column })
+}
+
 // Parse a `// partitioned <kind> [opts]` right-hand side. Recognized kinds:
 // `daily`, `hourly`, `weekly`, `monthly` (with optional tz/format/start),
 // and `dynamic key="<jsonpath>"` (plus optional format).
@@ -1074,6 +1231,56 @@ mod pipeline_annotation_tests {
        assert!(out.tag.is_none());
    }

+    #[test]
+    fn tag_with_whitespace_is_skipped() {
+        // A regular English comment starting with "# tag " must not be
+        // mistaken for a worker-tag annotation (worker tags are single words).
+        let out =
+            parse_pipeline_annotations("# tag this function so we remember to refactor it later");
+        assert!(out.tag.is_none());
+    }
+
+    #[test]
+    fn tag_too_long_is_skipped() {
+        let long = "x".repeat(51);
+        let out = parse_pipeline_annotations(&format!("// tag {long}"));
+        assert!(out.tag.is_none());
+    }
+
+    #[test]
+    fn annotations_in_body_are_ignored() {
+        // Only the leading comment header is scanned. A regular `# tag ...`
+        // prose comment buried in the body — the WIN-2090 false-positive that
+        // crashed the `script.tag` INSERT — must not be treated as an
+        // annotation once real code has started.
+        let code = concat!(
+            "import pandas as pd\n",
+            "\n",
+            "def main():\n",
+            "    # tag each row with its source so downstream steps can filter\n",
+            "    # on s3://should/not/parse\n",
+            "    return pd.DataFrame()\n",
+        );
+        let out = parse_pipeline_annotations(code);
+        assert!(out.tag.is_none());
+        assert!(out.triggers.is_empty());
+    }
+
+    #[test]
+    fn header_allows_blank_lines_before_code() {
+        // Blank lines (e.g. after a shebang) don't end the header; the first
+        // line of real code does.
+        let code = concat!(
+            "#!/usr/bin/env python\n",
+            "\n",
+            "# tag heavy\n",
+            "import os\n",
+            "# tag light\n",
+        );
+        let out = parse_pipeline_annotations(code);
+        assert_eq!(out.tag.as_deref(), Some("heavy"));
+    }
+
    #[test]
    fn retry_count_only() {
        let out = parse_pipeline_annotations("// retry 3");
@@ -1240,4 +1447,102 @@ mod pipeline_annotation_tests {
        assert_eq!(m.get("b").unwrap(), "fine");
        assert!(m.get("garbage").is_none());
    }
+
+    #[test]
+    fn data_test_builtins() {
+        let code = concat!(
+            "// data_test unique order_id\n",
+            "// data_test not_null user_id\n",
+            "// data_test accepted_values status = paid,pending,refunded\n",
+            "// data_test relationships user_id -> datatable://prod/users.id\n",
+        );
+        let out = parse_pipeline_annotations(code);
+        assert_eq!(
+            out.data_tests,
+            vec![
+                DataTest::Unique { column: "order_id".to_string() },
+                DataTest::NotNull { column: "user_id".to_string() },
+                DataTest::AcceptedValues {
+                    column: "status".to_string(),
+                    values: vec![
+                        "paid".to_string(),
+                        "pending".to_string(),
+                        "refunded".to_string()
+                    ],
+                },
+                DataTest::Relationships {
+                    column: "user_id".to_string(),
+                    to_kind: AssetKind::DataTable,
+                    to_path: "prod/users".to_string(),
+                    to_column: "id".to_string(),
+                },
+            ]
+        );
+    }
+
+    #[test]
+    fn data_test_accepts_quotes_and_spacing() {
+        let out = parse_pipeline_annotations("// data_test accepted_values kind = \"a b\", 'c' ,d");
+        assert_eq!(
+            out.data_tests,
+            vec![DataTest::AcceptedValues {
+                column: "kind".to_string(),
+                values: vec!["a b".to_string(), "c".to_string(), "d".to_string()],
+            }]
+        );
+    }
+
+    #[test]
+    fn data_test_custom_escape_hatch() {
+        // A non-built-in single token is a custom script path; default-syntax
+        // asset shorthands are NOT triggered here (a path is just a path).
+        let out = parse_pipeline_annotations("// data_test f/tests/orders_amount_sane");
+        assert_eq!(
+            out.data_tests,
+            vec![DataTest::Custom { path: "f/tests/orders_amount_sane".to_string() }]
+        );
+    }
+
+    #[test]
+    fn data_test_relationships_ducklake_shorthand() {
+        let out = parse_pipeline_annotations(
+            "// data_test relationships sku -> ducklake://warehouse/dim_products.sku",
+        );
+        assert_eq!(
+            out.data_tests,
+            vec![DataTest::Relationships {
+                column: "sku".to_string(),
+                to_kind: AssetKind::Ducklake,
+                to_path: "warehouse/dim_products".to_string(),
+                to_column: "sku".to_string(),
+            }]
+        );
+    }
+
+    #[test]
+    fn data_test_malformed_dropped_fail_safe() {
+        // A misspelled built-in with trailing content is not a valid path token
+        // → dropped, not misread as a custom test. An empty value list, a
+        // missing arrow target, and a bare keyword are all dropped too.
+        let out = parse_pipeline_annotations(concat!(
+            "// data_test uniq order_id\n",        // typo'd built-in + arg
+            "// data_test accepted_values s =\n",  // no values
+            "// data_test relationships a -> b\n", // no `.refcol`
+            "// data_test unique\n",               // missing column
+            "// data_test\n",                      // bare keyword
+        ));
+        assert!(out.data_tests.is_empty());
+    }
+
+    #[test]
+    fn data_test_not_confused_with_ci_test_annotation() {
+        // `// test:` is the unrelated CI-test annotation — it must NOT be
+        // parsed as a data test (no whitespace after `test`, and the keyword
+        // is `data_test` anyway).
+        let out = parse_pipeline_annotations("// test: f/foo/bar\n// data_test unique id");
+        assert_eq!(
+            out.data_tests,
+            vec![DataTest::Unique { column: "id".to_string() }]
+        );
+    }
 }
@@ -482,7 +482,8 @@ pub fn build_wrap_blocks(
    partition_value_sql: &str,
    partitioned: bool,
    strategy: MaterializeStrategy,
-) -> Vec<String> {
+    tests: &[DataTestResolved],
+) -> Result<Vec<String>, String> {
    let target_qualified = format!("{TARGET_ALIAS}.{target_table}");
    let cg = MaterializeCodegen {
        target_qualified: &target_qualified,
@@ -492,6 +493,14 @@ pub fn build_wrap_blocks(
        partitioned,
        strategy,
    };
+    let ctx = DataTestCtx {
+        target_qualified: &target_qualified,
+        asset_path,
+        partition_col,
+        partition_value_sql,
+        partitioned,
+    };
+    let test_sql = build_data_test_checks(tests, &ctx)?;
    let mut blocks: Vec<String> = Vec::new();
    // Setup blocks come from the splitter with their `;` stripped — re-terminate
    // each so that when the executor re-joins and re-splits the assembled query,
@@ -499,15 +508,20 @@ pub fn build_wrap_blocks(
    // don't merge into one malformed statement.
    blocks.extend(plan.setup.iter().map(|s| terminate(s)));
    blocks.push(target_attach.to_string());
+    // Referenced-asset ATTACHes (relationships tests) — read-only, before the
+    // write and the summary that probes them.
+    blocks.extend(test_sql.attaches);
    blocks.extend(cg.statements());
+    // The summary read carries the per-test breakdown (when any tests apply).
    blocks.push(materialize_result_sql(
        &target_qualified,
        asset_path,
        partition_col,
        partition_value_sql,
        partitioned,
+        &test_sql.checks,
    ));
-    blocks
+    Ok(blocks)
 }

 /// The trailing one-row summary the materialize run returns: the asset it
@@ -520,6 +534,7 @@ pub fn materialize_result_sql(
    partition_col: &str,
    partition_value_sql: &str,
    partitioned: bool,
+    checks: &[DataTestCheck],
 ) -> String {
    let (count_expr, partition_sel) = if partitioned {
        // Row count is the slice this run wrote (the partition); `partition`
@@ -536,10 +551,38 @@ pub fn materialize_result_sql(
            String::new(),
        )
    };
-    format!(
-        "SELECT 'ducklake://{asset_path}' AS materialized, \
+    let base_cols = format!(
+        "'ducklake://{asset_path}' AS materialized, \
         {partition_sel}{count_expr} AS rows, \
-         (SELECT max(snapshot_id) FROM ducklake_snapshots('{TARGET_ALIAS}')) AS snapshot_id;"
+         (SELECT max(snapshot_id) FROM ducklake_snapshots('{TARGET_ALIAS}')) AS snapshot_id"
+    );
+    if checks.is_empty() {
+        return format!("SELECT {base_cols};");
+    }
+    // Per-test breakdown. Each check's violating-count is computed once as a CTE
+    // column (`c0`, `c1`, …); the `data_tests` list-of-struct then references
+    // those columns — DuckDB rejects scalar subqueries *inside* a struct/list
+    // literal, hence the CTE. Names are single-quote-escaped. The result row
+    // carries the whole breakdown so the worker runs every test (no
+    // abort-on-first) and decides pass/fail itself.
+    let cte_cols = checks
+        .iter()
+        .enumerate()
+        .map(|(i, c)| format!("{} AS c{i}", c.violating))
+        .collect::<Vec<_>>()
+        .join(", ");
+    let list_items = checks
+        .iter()
+        .enumerate()
+        .map(|(i, c)| {
+            let name = c.name.replace('\'', "''");
+            format!("{{'test': '{name}', 'violating': c{i}}}")
+        })
+        .collect::<Vec<_>>()
+        .join(", ");
+    format!(
+        "WITH _wm_tr AS (SELECT {cte_cols}) \
+         SELECT {base_cols}, [{list_items}] AS data_tests FROM _wm_tr;"
    )
 }

@@ -553,6 +596,271 @@ fn terminate(stmt: &str) -> String {
    }
 }

+// ---------------------------------------------------------------------------
+// Data tests (`// data_test`)
+// ---------------------------------------------------------------------------
+//
+// A data test is the FIRST extensible annotation: the parser yields a
+// `DataTest` from a known vocabulary, and this module turns each into a
+// *check* — a `(name, violating-row-count query)` pair — that runs against the
+// freshly-materialized target after the write commits. The materialize summary
+// query embeds every check's count in one `data_tests` column, so all tests
+// run in a single pass (no abort-on-first) and the worker, not the SQL,
+// decides pass/fail and reports the full per-test breakdown.
+//
+// The pattern is deliberately open: a verifier is just `(name, count query)`.
+// Built-ins differ only in their count query; the `Custom` escape hatch
+// supplies its own (a user SELECT returning the violating rows). A sibling
+// annotation family (column-lineage) can emit its own checks through the same
+// `push_check` shape rather than bolting on a parallel mechanism. See
+// `docs/ducklake-materialization.md`.
+
+use crate::asset_parser::{AssetKind, DataTest};
+
+/// Target context a data-test probe runs against — the materialized table and
+/// the partition slice (when partitioned, tests are scoped to the slice just
+/// written, so a rerun/backfill is independent of other partitions' data).
+#[derive(Debug, Clone)]
+pub struct DataTestCtx<'a> {
+    /// Fully-qualified materialized target, e.g. `_wm_target.orders`.
+    pub target_qualified: &'a str,
+    /// `<name>/<table>` of the target, for human-readable probe messages.
+    pub asset_path: &'a str,
+    /// Physical partition column on the managed table.
+    pub partition_col: &'a str,
+    /// SQL literal/expression for the current partition value (already escaped).
+    pub partition_value_sql: &'a str,
+    /// Whether the target is partitioned (scopes probes to the slice).
+    pub partitioned: bool,
+}
+
+/// A data test resolved enough to generate SQL. Built-ins carry only their
+/// parsed `DataTest`; `Custom` additionally carries the fetched script body
+/// (the parser crate can't fetch it — the worker does and passes it in).
+#[derive(Debug, Clone)]
+pub enum DataTestResolved {
+    BuiltIn(DataTest),
+    Custom { path: String, body: String },
+}
+
+/// One compiled data-test check: a human-readable `name` and a scalar SQL
+/// expression (`violating`) yielding the number of rows that violate it (0 =
+/// pass). The materialize summary query embeds every check's count so the
+/// worker gets the whole breakdown in one result — all tests run (no
+/// abort-on-first) and the worker, not the SQL, decides pass/fail.
+#[derive(Debug, Clone, PartialEq, Eq)]
+pub struct DataTestCheck {
+    pub name: String,
+    /// Scalar subquery yielding the violating-row count, e.g.
+    /// `(SELECT count(*) AS v FROM (…))`.
+    pub violating: String,
+}
+
+/// The SQL a set of data tests compiles to: referenced-asset `ATTACH`
+/// statements (resolved by the executor's ATTACH-transform pass) and the
+/// per-test checks, both in declaration order.
+#[derive(Debug, Clone, Default, PartialEq, Eq)]
+pub struct DataTestChecks {
+    pub attaches: Vec<String>,
+    pub checks: Vec<DataTestCheck>,
+}
+
+/// Alias prefix for a relationships test's referenced asset, attached
+/// read-only alongside the target. `_wm_ref_<n>` so it never collides with the
+/// user's aliases or the reserved `_wm_target`.
+const REF_ALIAS_PREFIX: &str = "_wm_ref_";
+
+// Double-quote a SQL identifier, escaping embedded quotes — so an arbitrary
+// column name from an annotation can't break out of the identifier.
+fn quote_ident(id: &str) -> String {
+    format!("\"{}\"", id.replace('"', "\"\""))
+}
+
+// Quote a possibly schema-qualified table reference (`schema.table`) by quoting
+// each dotted segment independently: `main.dim_products` → `"main"."dim_products"`.
+// Quoting the whole thing would make DuckDB read it as one table name containing
+// a literal dot, querying the wrong table.
+fn quote_qualified(name: &str) -> String {
+    name.split('.')
+        .map(quote_ident)
+        .collect::<Vec<_>>()
+        .join(".")
+}
+
+// Single-quote a SQL string literal, escaping embedded quotes.
+fn quote_lit(s: &str) -> String {
+    format!("'{}'", s.replace('\'', "''"))
+}
+
+// The `WHERE`/`AND` fragment scoping a probe to the current partition, or
+// empty when unpartitioned. `prefix` is `WHERE ` or `AND ` per call site.
+fn partition_scope(ctx: &DataTestCtx, prefix: &str, table_alias: Option<&str>) -> String {
+    if !ctx.partitioned {
+        return String::new();
+    }
+    let col = match table_alias {
+        Some(a) => format!("{a}.{}", quote_ident(ctx.partition_col)),
+        None => quote_ident(ctx.partition_col),
+    };
+    format!("{prefix}{col} = {}", ctx.partition_value_sql)
+}
+
+// Record one check: its display `name` plus `count_query` (which yields a
+// single-column violating-row count) wrapped as a scalar subquery.
+fn push_check(out: &mut DataTestChecks, name: String, count_query: String) {
+    out.checks
+        .push(DataTestCheck { name, violating: format!("({count_query})") });
+}
+
+/// Compile resolved data tests into ATTACH statements + per-test checks for
+/// `ctx`'s target. Pure: returns SQL text, executes nothing. Errors carry an
+/// actionable message (e.g. a relationships target that isn't an attachable
+/// table).
+pub fn build_data_test_checks(
+    tests: &[DataTestResolved],
+    ctx: &DataTestCtx,
+) -> Result<DataTestChecks, String> {
+    let t = ctx.target_qualified;
+    let mut out = DataTestChecks::default();
+    // Dedup ref attaches by (kind, name): a database can't be attached twice,
+    // so multiple relationships into the same db share one alias.
+    let mut ref_aliases: Vec<(AssetKind, String, String)> = Vec::new();
+
+    for resolved in tests {
+        match resolved {
+            DataTestResolved::BuiltIn(DataTest::Unique { column }) => {
+                let c = quote_ident(column);
+                let scope = partition_scope(ctx, " AND ", None);
+                let q = format!(
+                    "SELECT count(*) AS v FROM (SELECT {c} FROM {t} WHERE {c} IS NOT NULL{scope} \
+                     GROUP BY {c} HAVING count(*) > 1)"
+                );
+                push_check(&mut out, format!("unique({column})"), q);
+            }
+            DataTestResolved::BuiltIn(DataTest::NotNull { column }) => {
+                let c = quote_ident(column);
+                let scope = partition_scope(ctx, " AND ", None);
+                let q = format!("SELECT count(*) AS v FROM {t} WHERE {c} IS NULL{scope}");
+                push_check(&mut out, format!("not_null({column})"), q);
+            }
+            DataTestResolved::BuiltIn(DataTest::AcceptedValues { column, values }) => {
+                let c = quote_ident(column);
+                let scope = partition_scope(ctx, " AND ", None);
+                let list = values
+                    .iter()
+                    .map(|v| quote_lit(v))
+                    .collect::<Vec<_>>()
+                    .join(", ");
+                let q = format!(
+                    "SELECT count(*) AS v FROM {t} WHERE {c} IS NOT NULL AND {c} NOT IN ({list}){scope}"
+                );
+                push_check(&mut out, format!("accepted_values({column})"), q);
+            }
+            DataTestResolved::BuiltIn(DataTest::Relationships {
+                column,
+                to_kind,
+                to_path,
+                to_column,
+            }) => {
+                let (ref_name, ref_table) = to_path.split_once('/').ok_or_else(|| {
+                    format!("data_test relationships: target `{to_path}` must be `<name>/<table>`")
+                })?;
+                if ref_table.is_empty() {
+                    return Err(format!(
+                        "data_test relationships: target `{to_path}` has no table"
+                    ));
+                }
+                let scheme = match to_kind {
+                    AssetKind::Ducklake => "ducklake",
+                    AssetKind::DataTable => "datatable",
+                    other => {
+                        return Err(format!(
+                            "data_test relationships: target kind {other:?} is not an attachable \
+                             table (use ducklake:// or datatable://)"
+                        ))
+                    }
+                };
+                // The materialize target's ducklake is already attached as
+                // `_wm_target`; a reference into that same lake must reuse it
+                // rather than ATTACH the same database again under a fresh alias
+                // (DuckDB forbids attaching one database twice). `asset_path` is
+                // the target's `<lake>/<table>`, so its lake is the part before
+                // the first `/`.
+                let target_lake = ctx.asset_path.split('/').next().unwrap_or("");
+                let alias = if *to_kind == AssetKind::Ducklake && ref_name == target_lake {
+                    TARGET_ALIAS.to_string()
+                } else {
+                    // Reuse an existing alias for the same (kind, name), else mint one.
+                    match ref_aliases
+                        .iter()
+                        .find(|(k, n, _)| k == to_kind && n == ref_name)
+                    {
+                        Some((_, _, a)) => a.clone(),
+                        None => {
+                            let a = format!("{REF_ALIAS_PREFIX}{}", ref_aliases.len());
+                            // Escape the name — it is interpolated into a
+                            // single-quoted DuckDB literal (defense-in-depth: the
+                            // name is deploy-time annotation content, but the parser
+                            // places no character restriction on asset paths).
+                            let esc_name = ref_name.replace('\'', "''");
+                            out.attaches
+                                .push(format!("ATTACH '{scheme}://{esc_name}' AS {a};"));
+                            ref_aliases.push((*to_kind, ref_name.to_string(), a.clone()));
+                            a
+                        }
+                    }
+                };
+                let c = quote_ident(column);
+                let rc = quote_ident(to_column);
+                // `ref_table` may be schema-qualified (`schema.table`); quote each
+                // segment so the dot stays a schema separator, not a literal.
+                let rt = quote_qualified(ref_table);
+                let scope = partition_scope(ctx, " AND ", Some("_wm_src"));
+                let q = format!(
+                    "SELECT count(*) AS v FROM {t} _wm_src \
+                     WHERE _wm_src.{c} IS NOT NULL{scope} \
+                     AND NOT EXISTS (SELECT 1 FROM {alias}.{rt} _wm_ref \
+                     WHERE _wm_ref.{rc} = _wm_src.{c})"
+                );
+                push_check(
+                    &mut out,
+                    format!("relationships({column} -> {to_path}.{to_column})"),
+                    q,
+                );
+            }
+            // A parsed Custom must be resolved (body fetched) before codegen.
+            DataTestResolved::BuiltIn(DataTest::Custom { path }) => {
+                return Err(format!(
+                    "data_test custom `{path}`: body not resolved before codegen (internal)"
+                ));
+            }
+            DataTestResolved::Custom { path, body } => {
+                // dbt singular-test convention: the body is a *single* SELECT
+                // (or CTE) returning the violating rows. It is embedded as a
+                // subquery (`FROM (<body>)`), so a multi-statement body would
+                // produce invalid SQL — validate up front with an actionable
+                // error. It runs in the target's connection (can read
+                // `_wm_target` + the user's attaches); partition substitution is
+                // already applied by the worker.
+                let stmts = split_statements(body);
+                if stmts.is_empty() {
+                    return Err(format!("data_test custom `{path}`: empty test body"));
+                }
+                if stmts.len() > 1 {
+                    return Err(format!(
+                        "data_test custom `{path}`: must be a single SELECT returning the \
+                         violating rows (found {} statements)",
+                        stmts.len()
+                    ));
+                }
+                let q = format!("SELECT count(*) AS v FROM ({})", stmts[0]);
+                push_check(&mut out, format!("custom({path})"), q);
+            }
+        }
+    }
+    Ok(out)
+}
+
 #[cfg(test)]
 mod tests {
    use super::*;
@@ -792,7 +1100,9 @@ mod tests {
            "'2026-06-19'",
            true,
            MaterializeStrategy::Replace,
-        );
+            &[],
+        )
+        .unwrap();
        // setup block first, then the target ATTACH, then codegen, then result.
        assert!(blocks[0].starts_with("ATTACH 'ducklake://main' AS dl"));
        // every setup block must be `;`-terminated so re-splitting can't merge it
@@ -814,4 +1124,247 @@ mod tests {
        assert!(last.contains("WHERE _wm_partition = '2026-06-19') AS rows"));
        assert!(last.contains("ducklake_snapshots('_wm_target')"));
    }
+
+    // -- data tests ---------------------------------------------------------
+
+    fn ctx_partitioned() -> DataTestCtx<'static> {
+        DataTestCtx {
+            target_qualified: "_wm_target.orders",
+            asset_path: "analytics/orders",
+            partition_col: "_wm_partition",
+            partition_value_sql: "'2026-06-19'",
+            partitioned: true,
+        }
+    }
+    fn ctx_unpartitioned() -> DataTestCtx<'static> {
+        DataTestCtx { partitioned: false, ..ctx_partitioned() }
+    }
+
+    #[test]
+    fn data_test_unique_and_not_null_partition_scoped() {
+        let tests = vec![
+            DataTestResolved::BuiltIn(DataTest::Unique { column: "order_id".into() }),
+            DataTestResolved::BuiltIn(DataTest::NotNull { column: "user_id".into() }),
+        ];
+        let sql = build_data_test_checks(&tests, &ctx_partitioned()).unwrap();
+        assert!(sql.attaches.is_empty());
+        assert_eq!(sql.checks.len(), 2);
+        // short, asset-free names (the asset is shown once by the breakdown).
+        assert_eq!(sql.checks[0].name, "unique(order_id)");
+        assert_eq!(sql.checks[1].name, "not_null(user_id)");
+        // each `violating` is a scalar count subquery.
+        assert!(sql.checks[0]
+            .violating
+            .starts_with("(SELECT count(*) AS v FROM"));
+        // unique: groups non-null keys within the slice, having count>1
+        assert!(sql.checks[0]
+            .violating
+            .contains("GROUP BY \"order_id\" HAVING count(*) > 1"));
+        assert!(sql.checks[0]
+            .violating
+            .contains("\"order_id\" IS NOT NULL AND \"_wm_partition\" = '2026-06-19'"));
+        // not_null: null rows in the slice
+        assert!(sql.checks[1]
+            .violating
+            .contains("WHERE \"user_id\" IS NULL AND \"_wm_partition\" = '2026-06-19'"));
+    }
+
+    #[test]
+    fn data_test_unpartitioned_has_no_partition_scope() {
+        let tests = vec![DataTestResolved::BuiltIn(DataTest::NotNull {
+            column: "id".into(),
+        })];
+        let sql = build_data_test_checks(&tests, &ctx_unpartitioned()).unwrap();
+        assert!(sql.checks[0].violating.contains("WHERE \"id\" IS NULL"));
+        assert!(!sql.checks[0].violating.contains("_wm_partition"));
+    }
+
+    #[test]
+    fn data_test_accepted_values_escapes_literals() {
+        let tests = vec![DataTestResolved::BuiltIn(DataTest::AcceptedValues {
+            column: "status".into(),
+            values: vec!["paid".into(), "o'brien".into()],
+        })];
+        let sql = build_data_test_checks(&tests, &ctx_unpartitioned()).unwrap();
+        assert!(sql.checks[0]
+            .violating
+            .contains("NOT IN ('paid', 'o''brien')"));
+        assert!(sql.checks[0].violating.contains("\"status\" IS NOT NULL"));
+    }
+
+    #[test]
+    fn data_test_relationships_attaches_ref_and_dedups() {
+        let tests = vec![
+            DataTestResolved::BuiltIn(DataTest::Relationships {
+                column: "user_id".into(),
+                to_kind: AssetKind::DataTable,
+                to_path: "prod/users".into(),
+                to_column: "id".into(),
+            }),
+            // second relationship into the SAME db reuses the alias (no 2nd attach)
+            DataTestResolved::BuiltIn(DataTest::Relationships {
+                column: "buyer_id".into(),
+                to_kind: AssetKind::DataTable,
+                to_path: "prod/buyers".into(),
+                to_column: "id".into(),
+            }),
+        ];
+        let sql = build_data_test_checks(&tests, &ctx_partitioned()).unwrap();
+        assert_eq!(sql.attaches.len(), 1, "same db attached once");
+        assert_eq!(sql.attaches[0], "ATTACH 'datatable://prod' AS _wm_ref_0;");
+        assert!(sql.checks[0]
+            .violating
+            .contains("NOT EXISTS (SELECT 1 FROM _wm_ref_0.\"users\""));
+        assert!(sql.checks[1]
+            .violating
+            .contains("NOT EXISTS (SELECT 1 FROM _wm_ref_0.\"buyers\""));
+        assert!(sql.checks[0]
+            .violating
+            .contains("_wm_src.\"_wm_partition\" = '2026-06-19'"));
+        assert_eq!(
+            sql.checks[0].name,
+            "relationships(user_id -> prod/users.id)"
+        );
+    }
+
+    #[test]
+    fn data_test_relationships_escapes_ref_name_in_attach() {
+        let tests = vec![DataTestResolved::BuiltIn(DataTest::Relationships {
+            column: "k".into(),
+            to_kind: AssetKind::DataTable,
+            to_path: "ev'il/users".into(),
+            to_column: "id".into(),
+        })];
+        let sql = build_data_test_checks(&tests, &ctx_unpartitioned()).unwrap();
+        // single quote in the name is doubled so it can't break out of the literal
+        assert_eq!(sql.attaches[0], "ATTACH 'datatable://ev''il' AS _wm_ref_0;");
+    }
+
+    #[test]
+    fn data_test_relationships_same_lake_reuses_target() {
+        // A relationship into the SAME ducklake as the materialize target
+        // (asset_path = "analytics/orders") must NOT re-ATTACH it — _wm_target
+        // already holds that catalog; reuse it.
+        let tests = vec![DataTestResolved::BuiltIn(DataTest::Relationships {
+            column: "user_id".into(),
+            to_kind: AssetKind::Ducklake,
+            to_path: "analytics/users".into(),
+            to_column: "id".into(),
+        })];
+        let sql = build_data_test_checks(&tests, &ctx_unpartitioned()).unwrap();
+        assert!(
+            sql.attaches.is_empty(),
+            "same-lake ref must not ATTACH again"
+        );
+        assert!(sql.checks[0]
+            .violating
+            .contains("NOT EXISTS (SELECT 1 FROM _wm_target.\"users\""));
+    }
+
+    #[test]
+    fn data_test_relationships_schema_qualified_target() {
+        // `<lake>/<schema>.<table>` — the schema-qualified table must quote each
+        // segment so the dot stays a separator, not part of one identifier.
+        let tests = vec![DataTestResolved::BuiltIn(DataTest::Relationships {
+            column: "sku".into(),
+            to_kind: AssetKind::Ducklake,
+            to_path: "warehouse/main.dim_products".into(),
+            to_column: "sku".into(),
+        })];
+        let sql = build_data_test_checks(&tests, &ctx_unpartitioned()).unwrap();
+        assert_eq!(
+            sql.attaches[0],
+            "ATTACH 'ducklake://warehouse' AS _wm_ref_0;"
+        );
+        assert!(
+            sql.checks[0]
+                .violating
+                .contains("FROM _wm_ref_0.\"main\".\"dim_products\""),
+            "schema-qualified target should be quoted per segment: {}",
+            sql.checks[0].violating
+        );
+    }
+
+    #[test]
+    fn data_test_relationships_rejects_non_attachable_kind() {
+        let tests = vec![DataTestResolved::BuiltIn(DataTest::Relationships {
+            column: "k".into(),
+            to_kind: AssetKind::S3Object,
+            to_path: "bucket/file".into(),
+            to_column: "c".into(),
+        })];
+        assert!(build_data_test_checks(&tests, &ctx_unpartitioned()).is_err());
+    }
+
+    #[test]
+    fn data_test_custom_wraps_body() {
+        let tests = vec![DataTestResolved::Custom {
+            path: "f/tests/amount".into(),
+            body: "SELECT * FROM _wm_target.orders WHERE amount < 0;".into(),
+        }];
+        let sql = build_data_test_checks(&tests, &ctx_unpartitioned()).unwrap();
+        // trailing ; stripped, wrapped as a count subquery
+        assert!(sql.checks[0].violating.contains(
+            "SELECT count(*) AS v FROM (SELECT * FROM _wm_target.orders WHERE amount < 0)"
+        ));
+        assert_eq!(sql.checks[0].name, "custom(f/tests/amount)");
+    }
+
+    #[test]
+    fn data_test_custom_rejects_multi_statement_body() {
+        // The body is embedded as a subquery, so a setup-then-SELECT body would
+        // produce invalid SQL — reject it up front with an actionable error.
+        let tests = vec![DataTestResolved::Custom {
+            path: "f/tests/amount".into(),
+            body: "SET threads = 1; SELECT * FROM _wm_target.orders WHERE amount < 0".into(),
+        }];
+        let err = build_data_test_checks(&tests, &ctx_unpartitioned()).unwrap_err();
+        assert!(err.contains("single SELECT"), "unexpected error: {err}");
+    }
+
+    #[test]
+    fn data_test_unresolved_custom_is_internal_error() {
+        let tests = vec![DataTestResolved::BuiltIn(DataTest::Custom {
+            path: "f/x".into(),
+        })];
+        assert!(build_data_test_checks(&tests, &ctx_unpartitioned()).is_err());
+    }
+
+    #[test]
+    fn materialize_result_sql_embeds_data_tests_breakdown() {
+        let checks = vec![
+            DataTestCheck {
+                name: "unique(order_id)".into(),
+                violating: "(SELECT count(*) AS v FROM q0)".into(),
+            },
+            DataTestCheck {
+                name: "custom(f/t)".into(),
+                violating: "(SELECT count(*) AS v FROM q1)".into(),
+            },
+        ];
+        let sql = materialize_result_sql(
+            "_wm_target.orders",
+            "analytics/orders",
+            "_wm_partition",
+            "'2026-06-19'",
+            false,
+            &checks,
+        );
+        // counts computed once in a CTE, referenced by the list-of-struct.
+        assert!(sql.starts_with("WITH _wm_tr AS (SELECT (SELECT count(*) AS v FROM q0) AS c0,"));
+        assert!(sql.contains("[{'test': 'unique(order_id)', 'violating': c0}, "));
+        assert!(sql.contains("{'test': 'custom(f/t)', 'violating': c1}] AS data_tests"));
+        assert!(sql.contains("FROM _wm_tr;"));
+        // no tests -> plain summary, no CTE / data_tests column.
+        let plain = materialize_result_sql(
+            "_wm_target.orders",
+            "analytics/orders",
+            "_wm_partition",
+            "'x'",
+            false,
+            &[],
+        );
+        assert!(plain.starts_with("SELECT 'ducklake://analytics/orders' AS materialized"));
+        assert!(!plain.contains("data_tests"));
+    }
 }
@@ -301,5 +301,119 @@
 			"tag": null,
 			"retry": null
 		}
+	},
+	{
+		"name": "data_test built-ins accumulate in order",
+		"code": "-- pipeline\n-- materialize ducklake://analytics/orders key=order_id\n-- data_test unique order_id\n-- data_test not_null user_id\n-- data_test accepted_values status = paid,pending,refunded\n-- data_test relationships user_id -> datatable://prod/users.id\nSELECT 1;",
+		"expected": {
+			"in_pipeline": true,
+			"asset_triggers": [],
+			"native_triggers": [],
+			"partition": null,
+			"freshness": null,
+			"tag": null,
+			"retry": null,
+			"materialize": {
+				"target_kind": "ducklake",
+				"target_path": "analytics/orders",
+				"unique_key": "order_id"
+			},
+			"data_tests": [
+				{ "type": "unique", "column": "order_id" },
+				{ "type": "not_null", "column": "user_id" },
+				{
+					"type": "accepted_values",
+					"column": "status",
+					"values": ["paid", "pending", "refunded"]
+				},
+				{
+					"type": "relationships",
+					"column": "user_id",
+					"to_kind": "datatable",
+					"to_path": "prod/users",
+					"to_column": "id"
+				}
+			]
+		}
+	},
+	{
+		"name": "data_test accepted_values strips quotes and spacing",
+		"code": "# data_test accepted_values kind = \"a b\", 'c' ,d\nprint(1)",
+		"expected": {
+			"in_pipeline": false,
+			"asset_triggers": [],
+			"native_triggers": [],
+			"partition": null,
+			"freshness": null,
+			"tag": null,
+			"retry": null,
+			"data_tests": [
+				{ "type": "accepted_values", "column": "kind", "values": ["a b", "c", "d"] }
+			]
+		}
+	},
+	{
+		"name": "data_test custom escape hatch is a script path",
+		"code": "// data_test f/tests/orders_amount_sane\nexport function main() {}",
+		"expected": {
+			"in_pipeline": false,
+			"asset_triggers": [],
+			"native_triggers": [],
+			"partition": null,
+			"freshness": null,
+			"tag": null,
+			"retry": null,
+			"data_tests": [{ "type": "custom", "path": "f/tests/orders_amount_sane" }]
+		}
+	},
+	{
+		"name": "data_test relationships with ducklake shorthand target",
+		"code": "// data_test relationships sku -> ducklake://warehouse/dim_products.sku\nSELECT 1;",
+		"expected": {
+			"in_pipeline": false,
+			"asset_triggers": [],
+			"native_triggers": [],
+			"partition": null,
+			"freshness": null,
+			"tag": null,
+			"retry": null,
+			"data_tests": [
+				{
+					"type": "relationships",
+					"column": "sku",
+					"to_kind": "ducklake",
+					"to_path": "warehouse/dim_products",
+					"to_column": "sku"
+				}
+			]
+		}
+	},
+	{
+		"name": "malformed data_test lines are dropped fail-safe",
+		"code": "// data_test uniq order_id\n// data_test accepted_values s =\n// data_test relationships a -> b\n// data_test unique\n// data_test\n// data_test unique id\nSELECT 1;",
+		"expected": {
+			"in_pipeline": false,
+			"asset_triggers": [],
+			"native_triggers": [],
+			"partition": null,
+			"freshness": null,
+			"tag": null,
+			"retry": null,
+			"data_tests": [{ "type": "unique", "column": "id" }]
+		}
+	},
+	{
+		"name": "ci test annotation is not a data test",
+		"code": "// test: f/foo/bar\n// data_test unique id\nSELECT 1;",
+		"expected": {
+			"in_pipeline": false,
+			"asset_triggers": [],
+			"native_triggers": [],
+			"partition": null,
+			"freshness": null,
+			"tag": null,
+			"retry": null,
+			"data_tests": [{ "type": "unique", "column": "id" }]
+		}
 	}
 ]
@@ -38,6 +38,10 @@ struct Expected {
    // deserializing; only fixtures exercising materialization set it.
    #[serde(default)]
    materialize: Option<ExpectedMaterialize>,
+    // Snake_case `DataTest` serde shape (e.g. {"type":"unique","column":"x"}),
+    // compared against `serde_json::to_value(got.data_tests)`. Absent === [].
+    #[serde(default)]
+    data_tests: Vec<serde_json::Value>,
 }

 #[derive(Deserialize)]
@@ -189,5 +193,12 @@ fn pipeline_annotation_fixtures_match() {
                want.is_some()
            ),
        }
+
+        let got_tests = serde_json::to_value(&got.data_tests).expect("data_tests serialize");
+        assert_eq!(
+            got_tests,
+            serde_json::Value::Array(f.expected.data_tests.clone()),
+            "{ctx}: data tests"
+        );
    }
 }
@@ -1324,6 +1324,9 @@ pub async fn delete_expired_items(db: &DB) -> () {
        let cleanup_start = Instant::now();
        let mut total_deleted = 0u64;
        let mut batch_num = 0i32;
+        // Watermark carried across batches so each one resumes after the rows the previous batch
+        // already processed instead of re-scanning the (potentially undeletable) oldest prefix.
+        let mut completed_at_floor: Option<DateTime<Utc>> = None;

        // Process batches until no more expired jobs or max batches reached
        loop {
@@ -1336,14 +1339,17 @@ pub async fn delete_expired_items(db: &DB) -> () {
            }

            // Each batch runs in its own transaction to avoid long-running locks
-            let batch_result = delete_expired_jobs_batch(db, job_retention_secs, batch_size).await;
+            let batch_result =
+                delete_expired_jobs_batch(db, job_retention_secs, batch_size, completed_at_floor)
+                    .await;

            match batch_result {
-                Ok(deleted_count) => {
+                Ok((deleted_count, max_completed_at)) => {
                    if deleted_count == 0 {
                        // No more expired jobs to delete
                        break;
                    }
+                    completed_at_floor = max_completed_at.or(completed_at_floor);
                    total_deleted += deleted_count as u64;
                    batch_num += 1;
                }
@@ -1510,12 +1516,20 @@ pub async fn check_expiring_tokens(db: &DB) {

 /// Delete a batch of expired jobs with LIMIT and SKIP LOCKED for high-scale environments.
 /// Uses a single transaction per batch to minimize lock duration.
-/// Returns the number of jobs deleted in this batch.
+///
+/// `completed_at_floor` is the watermark from the previous batch in the same cleanup run (the
+/// max `completed_at` it deleted); pass `None` for the first batch. It is re-applied as
+/// `completed_at >= floor` so the scan resumes past the rows already processed instead of
+/// re-walking them (see the inline comment on the DELETE for why this matters).
+///
+/// Returns `(jobs deleted in this batch, max completed_at deleted)`. The caller feeds the
+/// returned watermark back in as `completed_at_floor` for the next batch.
 async fn delete_expired_jobs_batch(
    db: &DB,
    job_retention_secs: i64,
    batch_size: i64,
-) -> error::Result<usize> {
+    completed_at_floor: Option<DateTime<Utc>>,
+) -> error::Result<(usize, Option<DateTime<Utc>>)> {
    let mut tx = db.begin().await?;

    // Fetch active ROOT job IDs that started before the retention period. We only care about
@@ -1531,34 +1545,70 @@ async fn delete_expired_jobs_batch(
    .fetch_all(&mut *tx)
    .await?;

-    // Use FOR UPDATE SKIP LOCKED to avoid contention between replicas
-    // ORDER BY completed_at ensures we delete oldest jobs first.
-    // Active-root exclusion uses `NOT IN (SELECT ... unnest($3))` rather than
-    // `!= ALL($3)`: the subquery form lets the planner build a one-time hashed
-    // SubPlan and apply it as a filter on the ordered index scan, giving O(1)
-    // membership per candidate instead of a per-row linear array scan (which
-    // degrades sharply when many root jobs are active). The `u IS NOT NULL` guard
-    // sidesteps NOT IN's null-trap semantics ($3 holds non-null PK ids).
-    let deleted_jobs: Vec<Uuid> = sqlx::query_scalar!(
-        "DELETE FROM v2_job_completed
-         WHERE id IN (
-             SELECT jc.id FROM v2_job_completed jc
-             LEFT JOIN v2_job j ON j.id = jc.id
-             WHERE jc.completed_at <= now() - ($1::bigint::text || ' s')::interval
-               AND COALESCE(j.root_job, j.flow_innermost_root_job, jc.id) NOT IN (
-                   SELECT u FROM unnest($3::uuid[]) AS u WHERE u IS NOT NULL
-               )
-             ORDER BY jc.completed_at ASC
-             LIMIT $2
-             FOR UPDATE OF jc SKIP LOCKED
-         )
-         RETURNING id",
-        job_retention_secs,
-        batch_size,
-        &active_root_job_ids
-    )
-    .fetch_all(&mut *tx)
-    .await?;
+    // `completed_at_floor` is a watermark carried across batches within a cleanup run: it is the
+    // max(completed_at) deleted by the previous batch. Re-applying it as `completed_at >= floor`
+    // lets each batch resume after the rows the previous batch already processed instead of
+    // re-scanning them. This matters when the oldest rows are undeletable (children of a
+    // still-active root flow): without the floor the `ORDER BY completed_at ASC` scan walks that
+    // same protected prefix on every batch, turning a cleanup run quadratic in prefix size.
+    // Floor only ever skips rows the current run already deleted, was protecting, or skip-locked —
+    // all correctly deferred to the next run, identical to the unbounded scan's semantics.
+    //
+    // Use FOR UPDATE SKIP LOCKED to avoid contention between replicas; ORDER BY completed_at
+    // deletes oldest jobs first.
+    let (deleted_jobs, max_completed_at) = if active_root_job_ids.is_empty() {
+        // Common case: no old root flow is still running, so nothing is protected and the
+        // v2_job join (a PK lookup per candidate) is pure overhead — skip it entirely.
+        let rows = sqlx::query!(
+            "DELETE FROM v2_job_completed
+             WHERE id IN (
+                 SELECT id FROM v2_job_completed
+                 WHERE completed_at <= now() - ($1::bigint::text || ' s')::interval
+                   AND ($3::timestamptz IS NULL OR completed_at >= $3)
+                 ORDER BY completed_at ASC
+                 LIMIT $2
+                 FOR UPDATE SKIP LOCKED
+             )
+             RETURNING id, completed_at",
+            job_retention_secs,
+            batch_size,
+            completed_at_floor,
+        )
+        .fetch_all(&mut *tx)
+        .await?;
+        let max = rows.iter().map(|r| r.completed_at).max();
+        (rows.into_iter().map(|r| r.id).collect::<Vec<Uuid>>(), max)
+    } else {
+        // Active-root exclusion uses `NOT IN (SELECT ... unnest($3))` rather than `!= ALL($3)`:
+        // the subquery form lets the planner build a one-time hashed SubPlan and apply it as a
+        // filter on the ordered index scan, giving O(1) membership per candidate instead of a
+        // per-row linear array scan (which degrades sharply when many root jobs are active). The
+        // `u IS NOT NULL` guard sidesteps NOT IN's null-trap semantics ($3 holds non-null PK ids).
+        let rows = sqlx::query!(
+            "DELETE FROM v2_job_completed
+             WHERE id IN (
+                 SELECT jc.id FROM v2_job_completed jc
+                 LEFT JOIN v2_job j ON j.id = jc.id
+                 WHERE jc.completed_at <= now() - ($1::bigint::text || ' s')::interval
+                   AND ($4::timestamptz IS NULL OR jc.completed_at >= $4)
+                   AND COALESCE(j.root_job, j.flow_innermost_root_job, jc.id) NOT IN (
+                       SELECT u FROM unnest($3::uuid[]) AS u WHERE u IS NOT NULL
+                   )
+                 ORDER BY jc.completed_at ASC
+                 LIMIT $2
+                 FOR UPDATE OF jc SKIP LOCKED
+             )
+             RETURNING id, completed_at",
+            job_retention_secs,
+            batch_size,
+            &active_root_job_ids,
+            completed_at_floor,
+        )
+        .fetch_all(&mut *tx)
+        .await?;
+        let max = rows.iter().map(|r| r.completed_at).max();
+        (rows.into_iter().map(|r| r.id).collect::<Vec<Uuid>>(), max)
+    };

    let deleted_count = deleted_jobs.len();

@@ -1618,7 +1668,7 @@ async fn delete_expired_jobs_batch(

    tx.commit().await?;

-    Ok(deleted_count)
+    Ok((deleted_count, max_completed_at))
 }

 async fn delete_log_files_from_disk_and_store(
@@ -4379,9 +4429,32 @@ RETURNING key,job_id
 // Per-statement cap keeps each delete short and lock-light; the per-cycle batch
 // cap bounds total work per monitor iteration so monitor_db stays responsive.
 // A large backlog drains across several iterations rather than one long delete.
+//
+// These sweeps anti-join the whole table to find orphans, so their cost tracks the heap's
+// physical size. job_perms / job_result_stream_v2 are high-churn (one row per job, deleted
+// here), so their bloat — not the query shape — is what makes the sweep slow. These sweeps run
+// every monitor cycle, but the bulk vacuuming_tables() runs only ~hourly, so dead tuples pile
+// up between bulk vacuums; each sweep VACUUMs its own table right after deleting (see below) to
+// keep the heap near the live working set. The outer `ctid IN (SELECT ... LIMIT)` is
+// deliberate: a `job_id IN (...)` rewrite adds a second scan/probe for the delete and
+// benchmarks slower, so don't "simplify" it.
 const ORPHAN_CLEANUP_BATCH_SIZE: u64 = 100_000;
 const ORPHAN_CLEANUP_MAX_BATCHES: usize = 10;

+// Reclaim the dead tuples a sweep just created so the next sweep's anti-join scans a lean heap
+// instead of a bloated one. Plain VACUUM (not FULL) only takes SHARE UPDATE EXCLUSIVE, so
+// concurrent reads/writes (every job create touches job_perms) keep running, and the visibility
+// map lets it skip unchanged pages so repeated runs are cheap. SKIP_LOCKED means HA replicas
+// don't pile up: one vacuums, the rest skip rather than queue behind it.
+async fn vacuum_after_sweep(db: &DB, table: &str) {
+    if let Err(e) = sqlx::query(&format!("VACUUM (SKIP_LOCKED) {table}"))
+        .execute(db)
+        .await
+    {
+        tracing::warn!("Error vacuuming {table} after orphan cleanup: {e:?}");
+    }
+}
+
 async fn cleanup_job_perms_orphaned(db: &DB) -> error::Result<()> {
    let mut total: u64 = 0;
    for _ in 0..ORPHAN_CLEANUP_MAX_BATCHES {
@@ -4404,6 +4477,7 @@ async fn cleanup_job_perms_orphaned(db: &DB) -> error::Result<()> {

    if total > 0 {
        tracing::info!("Cleaned up {total} orphaned job_perms rows");
+        vacuum_after_sweep(db, "job_perms").await;
    }
    Ok(())
 }
@@ -4435,6 +4509,7 @@ async fn cleanup_job_result_stream_orphaned_jobs(db: &DB) -> error::Result<()> {

    if total > 0 {
        tracing::info!("Cleaned up {total} orphaned job_result_stream_v2 rows");
+        vacuum_after_sweep(db, "job_result_stream_v2").await;
    }
    Ok(())
 }
@@ -0,0 +1,206 @@
+//! A WM_TOKEN (job JWT) running as a superadmin must not be able to perform
+//! global user/token management — promotion, password reset, user creation,
+//! token creation/impersonation, offboarding, or exporting the user table.
+//! A non-admin `wm_deployers` member can mint
+//! such a token implicitly via an app/flow `on_behalf_of`, so trusting it would
+//! let them establish *persistent* superadmin. A real superadmin who needs this
+//! from a script must use a dedicated superadmin API token (which only a real
+//! superadmin can create), not `$WM_TOKEN`.
+//!
+//! The fixture provides `test@windmill.dev` (instance superadmin, token
+//! `SECRET_TOKEN`) and `test2@windmill.dev` (non-superadmin, `SECRET_TOKEN_2`).
+
+use serde_json::json;
+use sqlx::{Pool, Postgres};
+use windmill_common::auth::create_jwt_token;
+use windmill_common::db::Authed;
+use windmill_test_utils::*;
+
+fn client() -> reqwest::Client {
+    reqwest::Client::new()
+}
+
+fn authed(builder: reqwest::RequestBuilder, token: &str) -> reqwest::RequestBuilder {
+    builder.header("Authorization", format!("Bearer {}", token))
+}
+
+/// Mint a WM_TOKEN: an internally-signed job JWT (note the `job_id` claim) for
+/// `email`, exactly as a running app/flow job is issued.
+async fn wm_token(email: &str, is_admin: bool) -> String {
+    let authed = Authed {
+        email: email.to_string(),
+        username: "runner".to_string(),
+        is_admin,
+        is_operator: false,
+        groups: vec![],
+        folders: vec![],
+        scopes: None,
+        token_prefix: None,
+    };
+    create_jwt_token(
+        authed,
+        "test-workspace",
+        3600,
+        Some(uuid::Uuid::new_v4()),
+        Some("app".to_string()),
+        None,
+        None,
+    )
+    .await
+    .expect("mint wm_token")
+}
+
+#[sqlx::test(fixtures("preserve_on_behalf_of"))]
+async fn test_wm_token_cannot_manage_superadmin_users(db: Pool<Postgres>) -> anyhow::Result<()> {
+    initialize_tracing().await;
+    // The server decodes WM_TOKENs with the same in-process JWT secret, so
+    // setting it once lets us mint a valid one below.
+    set_jwt_secret().await;
+
+    let server = ApiServer::start(db.clone()).await?;
+    let port = server.addr.port();
+    let base = format!("http://localhost:{port}/api/users");
+
+    // A superadmin-capable WM_TOKEN — the exact thing a deployer obtains via an
+    // app on_behalf_of pointed at a superadmin.
+    let sa_wm = wm_token("test@windmill.dev", true).await;
+
+    // 1. Cannot mint a (superadmin) token.
+    let resp = authed(client().post(format!("{base}/tokens/create")), &sa_wm)
+        .json(&json!({}))
+        .send()
+        .await?;
+    assert_eq!(
+        resp.status(),
+        401,
+        "superadmin WM_TOKEN must not create tokens: {}",
+        resp.text().await?
+    );
+
+    // 2. Cannot impersonate (mint a token as another user).
+    let resp = authed(client().post(format!("{base}/tokens/impersonate")), &sa_wm)
+        .json(&json!({ "impersonate_email": "test2@windmill.dev" }))
+        .send()
+        .await?;
+    assert_eq!(
+        resp.status(),
+        401,
+        "superadmin WM_TOKEN must not impersonate: {}",
+        resp.text().await?
+    );
+
+    // 3. Cannot promote a user to superadmin.
+    let resp = authed(
+        client().post(format!("{base}/update/test2@windmill.dev")),
+        &sa_wm,
+    )
+    .json(&json!({ "is_super_admin": true }))
+    .send()
+    .await?;
+    assert_eq!(
+        resp.status(),
+        401,
+        "superadmin WM_TOKEN must not promote users: {}",
+        resp.text().await?
+    );
+
+    // 4. Cannot reset its own (the superadmin's) password.
+    let resp = authed(client().post(format!("{base}/setpassword")), &sa_wm)
+        .json(&json!({ "password": "hunter2" }))
+        .send()
+        .await?;
+    assert_eq!(
+        resp.status(),
+        401,
+        "superadmin WM_TOKEN must not reset passwords: {}",
+        resp.text().await?
+    );
+
+    // 4b. Cannot delete a user.
+    let resp = authed(
+        client().delete(format!("{base}/delete/test2@windmill.dev")),
+        &sa_wm,
+    )
+    .send()
+    .await?;
+    assert_eq!(
+        resp.status(),
+        401,
+        "superadmin WM_TOKEN must not delete users: {}",
+        resp.text().await?
+    );
+
+    // 4c. Cannot change a user's login type.
+    let resp = authed(
+        client().post(format!("{base}/set_login_type/test2@windmill.dev")),
+        &sa_wm,
+    )
+    .json(&json!({ "login_type": "password" }))
+    .send()
+    .await?;
+    assert_eq!(
+        resp.status(),
+        401,
+        "superadmin WM_TOKEN must not change login type: {}",
+        resp.text().await?
+    );
+
+    // 4d. Cannot offboard a global user (deletes user, tokens, password, invites,
+    //     instance-group membership and reassigns their assets).
+    let resp = authed(
+        client().post(format!("{base}/offboard/test2@windmill.dev")),
+        &sa_wm,
+    )
+    .json(&json!({}))
+    .send()
+    .await?;
+    assert_eq!(
+        resp.status(),
+        401,
+        "superadmin WM_TOKEN must not offboard users: {}",
+        resp.text().await?
+    );
+
+    // 4e. Cannot export the global user table (leaks every user's password_hash).
+    let resp = authed(client().get(format!("{base}/export")), &sa_wm)
+        .send()
+        .await?;
+    assert_eq!(
+        resp.status(),
+        401,
+        "superadmin WM_TOKEN must not export global users: {}",
+        resp.text().await?
+    );
+
+    // 5. Escape hatch / no false positive: a real superadmin API token
+    //    (SECRET_TOKEN, no job_id) can still create tokens.
+    let resp = authed(
+        client().post(format!("{base}/tokens/create")),
+        "SECRET_TOKEN",
+    )
+    .json(&json!({ "label": "ci" }))
+    .send()
+    .await?;
+    assert_eq!(
+        resp.status(),
+        201,
+        "a real superadmin token must still create tokens: {}",
+        resp.text().await?
+    );
+
+    // 6. No collateral: a non-superadmin WM_TOKEN can still create its own
+    //    token — the guard only fires for superadmin-capable job tokens.
+    let user_wm = wm_token("test2@windmill.dev", false).await;
+    let resp = authed(client().post(format!("{base}/tokens/create")), &user_wm)
+        .json(&json!({ "label": "from-script" }))
+        .send()
+        .await?;
+    assert_eq!(
+        resp.status(),
+        201,
+        "non-superadmin WM_TOKEN must still create its own token: {}",
+        resp.text().await?
+    );
+
+    Ok(())
+}
@@ -450,6 +450,33 @@ struct GraphRunnableNode {
    // pipeline-member visual state on the frontend.
    #[serde(skip_serializing_if = "std::ops::Not::not", default)]
    in_pipeline: bool,
+    // Annotation badges parsed from the deployed script body, so the canvas
+    // shows partition/freshness/tag/retry/data-test chips on *deployed* nodes
+    // (not only on live-edited drafts, which the frontend parses itself). Kept
+    // in lockstep with the TS `AssetGraphRunnableNode` fields the node renders.
+    #[serde(skip_serializing_if = "Option::is_none", default)]
+    partition_kind: Option<String>,
+    #[serde(skip_serializing_if = "Option::is_none", default)]
+    freshness: Option<String>,
+    #[serde(skip_serializing_if = "Option::is_none", default)]
+    tag: Option<String>,
+    #[serde(skip_serializing_if = "Option::is_none", default)]
+    retry: Option<windmill_common::assets::RetrySpec>,
+    #[serde(skip_serializing_if = "Vec::is_empty", default)]
+    data_tests: Vec<windmill_common::assets::DataTest>,
+}
+
+// The partition's kind word for the node badge (the full PartitionSpec carries
+// tz/format/start, which the badge doesn't need).
+fn partition_kind_word(kind: &windmill_common::assets::PartitionKind) -> &'static str {
+    use windmill_common::assets::PartitionKind::*;
+    match kind {
+        Daily => "daily",
+        Hourly => "hourly",
+        Weekly => "weekly",
+        Monthly => "monthly",
+        Dynamic { .. } => "dynamic",
+    }
 }

 // Lineage edge from parsed r/w usages. One per (runnable, asset, access_type)
@@ -642,15 +669,21 @@ async fn asset_graph(
    .await?;

    // Which scripts in scope are pipeline members (have `// pipeline`).
+    // Pipeline members + their latest deployed body, so the graph can surface
+    // annotation badges (partition/freshness/tag/retry/data_test) on deployed
+    // nodes. `DISTINCT ON (path) … ORDER BY created_at DESC` picks the newest
+    // non-archived version per path (a redeploy archives the prior one, but be
+    // defensive against transient overlaps).
    let pipeline_member_paths = sqlx::query!(
        r#"
-        SELECT path AS "path!"
+        SELECT DISTINCT ON (path) path AS "path!", content AS "content!"
        FROM script
        WHERE workspace_id = $1
          AND auto_kind = 'pipeline'
          AND archived = false
          AND deleted = false
          AND ($2::text IS NULL OR path LIKE $2)
+        ORDER BY path, created_at DESC
        "#,
        &w_id,
        folder_filter.as_deref(),
@@ -681,6 +714,20 @@ async fn asset_graph(

    tx.commit().await?;

+    // Parse each pipeline member's body once into its badge annotations, keyed
+    // by path, for the runnable-node construction below.
+    let annotations_by_path: std::collections::HashMap<
+        String,
+        windmill_common::assets::PipelineAnnotations,
+    > = pipeline_member_paths
+        .iter()
+        .map(|r| {
+            (
+                r.path.clone(),
+                windmill_common::assets::parse_pipeline_annotations(&r.content),
+            )
+        })
+        .collect();
    let pipeline_member_script_paths: std::collections::HashSet<String> =
        pipeline_member_paths.into_iter().map(|r| r.path).collect();
    let existing_script_paths: std::collections::HashSet<String> =
@@ -804,7 +851,26 @@ async fn asset_graph(
        .map(|(usage_kind, path)| {
            let in_pipeline = usage_kind == AssetUsageKind::Script
                && pipeline_member_script_paths.contains(&path);
-            GraphRunnableNode { path, usage_kind, in_pipeline }
+            // Annotation badges, only for pipeline-member scripts (the only
+            // bodies we parsed). Gate on the runnable kind too: a flow sharing a
+            // path with a pipeline script must not inherit its badges.
+            let ann = (usage_kind == AssetUsageKind::Script)
+                .then(|| annotations_by_path.get(&path))
+                .flatten();
+            GraphRunnableNode {
+                in_pipeline,
+                partition_kind: ann
+                    .and_then(|a| a.partition.as_ref())
+                    .map(|p| partition_kind_word(&p.kind).to_string()),
+                freshness: ann
+                    .and_then(|a| a.freshness.as_ref())
+                    .map(|f| f.duration.clone()),
+                tag: ann.and_then(|a| a.tag.clone()),
+                retry: ann.and_then(|a| a.retry.clone()),
+                data_tests: ann.map(|a| a.data_tests.clone()).unwrap_or_default(),
+                path,
+                usage_kind,
+            }
        })
        .collect();
    runnables.sort_by(|a, b| a.path.cmp(&b.path));
@@ -205,6 +205,33 @@ pub async fn require_super_admin(db: &DB, email: &str) -> error::Result<()> {
    }
 }

+/// Forbid sensitive global user/token management when authenticated as a
+/// superadmin *via a job token* (`WM_TOKEN`).
+///
+/// A `WM_TOKEN`'s identity is derived from an app/flow `on_behalf_of`, which a
+/// non-admin `wm_deployers` member can point at a superadmin. Trusting it for
+/// these operations would let them establish *persistent* superadmin (promote a
+/// user, reset a superadmin's password, mint a superadmin token, ...). `job_id`
+/// is set only for `WM_TOKEN`s; regular session/API tokens have it `None`, so a
+/// real superadmin who needs this from a script uses a dedicated superadmin API
+/// token (which only a real superadmin can create) instead of `$WM_TOKEN`.
+pub async fn forbid_superadmin_job_token(
+    db: &DB,
+    email: &str,
+    job_id: Option<uuid::Uuid>,
+) -> error::Result<()> {
+    if job_id.is_some() && is_super_admin_email(db, email).await? {
+        return Err(Error::NotAuthorized(
+            "This operation cannot be performed with a job token ($WM_TOKEN) that runs as a \
+             superadmin. If a script genuinely needs to do this, create a dedicated superadmin \
+             token from the User settings drawer (the 'Tokens' section), store it as a secret, \
+             and use that token explicitly instead of $WM_TOKEN."
+                .to_owned(),
+        ));
+    }
+    Ok(())
+}
+
 pub fn check_scopes<F>(authed: &ApiAuthed, required: F) -> error::Result<()>
 where
    F: FnOnce() -> String,
@@ -996,6 +1023,18 @@ pub fn require_path_read_access_for_preview(
        return Ok(());
    };

+    // Reject path traversal before any privilege-based short-circuit. A Preview's
+    // path is request-supplied and bypasses the DB `proper_id` CHECK that deployed
+    // runnables get; it then flows to the worker where it builds on-disk module
+    // directories. A `..` segment or an absolute path could let a write escape the
+    // per-job dir.
+    if path.starts_with('/') || path.split('/').any(|seg| seg == "..") || path.contains('\0') {
+        return Err(Error::BadRequest(format!(
+            "Invalid path for preview job: {}",
+            path
+        )));
+    }
+
    if authed.is_admin {
        return Ok(());
    }
@@ -1053,6 +1092,45 @@ mod tests {
        }
    }

+    // Regression tests for the Preview path traversal: a Preview's path skips the
+    // DB `proper_id` CHECK and reaches the worker, where it builds on-disk module
+    // dirs. Traversal must be rejected even for admins, who otherwise bypass the
+    // namespace/folder access check.
+    #[test]
+    fn preview_path_rejects_traversal() {
+        let admin = ApiAuthed { is_admin: true, username: "admin".into(), ..Default::default() };
+        for path in [
+            "u/admin/../../../../../../tmp/evil/payload",
+            "../../tmp/evil",
+            "/tmp/evil",
+            "u/admin/ok/../../../../etc/cron.d/x",
+        ] {
+            assert!(
+                require_path_read_access_for_preview(&admin, &Some(path.to_string())).is_err(),
+                "expected traversal path to be rejected: {path}"
+            );
+        }
+    }
+
+    #[test]
+    fn preview_path_allows_legitimate_paths() {
+        let alice = ApiAuthed { username: "alice".into(), ..Default::default() };
+        assert!(require_path_read_access_for_preview(&alice, &None).is_ok());
+        assert!(require_path_read_access_for_preview(&alice, &Some(String::new())).is_ok());
+        assert!(
+            require_path_read_access_for_preview(&alice, &Some("u/alice/my_script".into())).is_ok()
+        );
+
+        let admin = ApiAuthed { is_admin: true, username: "admin".into(), ..Default::default() };
+        assert!(
+            require_path_read_access_for_preview(&admin, &Some("hub/foo/bar/baz".into())).is_ok()
+        );
+        // `..` only as a substring of a segment is a valid name, not traversal.
+        assert!(
+            require_path_read_access_for_preview(&admin, &Some("f/team/my..script".into())).is_ok()
+        );
+    }
+
    #[test]
    fn predicate_no_scopes_allows_all() {
        let authed = authed_with_scopes(None);
@@ -646,6 +646,20 @@ async fn test_workspace_endpoints(db: Pool<Postgres>) -> anyhow::Result<()> {
            .unwrap();
        assert_eq!(resp.json::<bool>().await?, true);

+        // Regression: changing a fork's workspace id must preserve its parent
+        // linkage. Dropping it leaves a wm-fork- workspace with no parent — a
+        // "fork of nothing" that can no longer be compared or merged.
+        let parent: Option<String> =
+            sqlx::query_scalar("SELECT parent_workspace_id FROM workspace WHERE id = $1")
+                .bind("wm-fork-renamed")
+                .fetch_one(&db)
+                .await?;
+        assert_eq!(
+            parent.as_deref(),
+            Some("new-test-ws"),
+            "renamed fork must keep its parent_workspace_id"
+        );
+
        // --- create_fork over an existing (active) workspace id: clear 400, not a raw SQL 500 ---
        let resp = authed(client().post(format!("{new_ws_base}/create_fork")))
            .json(&json!({
@@ -256,52 +256,300 @@ pub async fn test_s3_bucket(
    use bytes::Bytes;
    use futures::StreamExt;

-    require_super_admin(&db, &authed.email).await?;
+    // The probe executes on the API server itself. On multi-tenant Cloud that is a shared control
+    // plane, so we constrain untrusted callers to remove the SSRF / credential-exfiltration /
+    // local-filesystem surface (see validate_object_storage_test). On self-hosted instances the
+    // object store usually lives on the local/private network and all authenticated users are
+    // trusted, so testing there stays unrestricted. Super admins keep the unrestricted path too.
+    let is_super_admin = is_super_admin_email(&db, &authed.email).await?;
+    let restrict = !is_super_admin && *CLOUD_HOSTED;
+    if restrict {
+        validate_object_storage_test(&test_s3_bucket).await?;
+    }

    let client = build_object_store_from_settings(test_s3_bucket, Some(&db))
        .await?
        .store;

-    let mut list = client.list(Some(
-        &windmill_object_store::object_store_reexports::Path::from("".to_string()),
-    ));
-    let first_file = list.next().await;
-    if first_file.is_some() {
-        if let Err(e) = first_file.as_ref().unwrap() {
-            tracing::error!("error listing bucket: {e:#}");
-            error::Error::internal_err(format!("Failed to list files in blob storage: {e:#}"));
+    let run = async {
+        let mut list = client.list(Some(
+            &windmill_object_store::object_store_reexports::Path::from("".to_string()),
+        ));
+        let first_file = list.next().await;
+        if first_file.is_some() {
+            if let Err(e) = first_file.as_ref().unwrap() {
+                tracing::error!("error listing bucket: {e:#}");
+                error::Error::internal_err(format!("Failed to list files in blob storage: {e:#}"));
+            }
+            tracing::info!("Listed files: {:?}", first_file.unwrap());
+        } else {
+            tracing::info!("No files in blob storage");
        }
-        tracing::info!("Listed files: {:?}", first_file.unwrap());
+
+        let path = windmill_object_store::object_store_reexports::Path::from(format!(
+            "/test-s3-bucket-{uuid}",
+            uuid = uuid::Uuid::new_v4()
+        ));
+        tracing::info!("Testing blob storage at path: {path}");
+        client
+            .put(
+                &path,
+                windmill_object_store::object_store_reexports::PutPayload::from_static(b"hello"),
+            )
+            .await
+            .map_err(|e| anyhow::anyhow!("error writing file to {path}: {e:#}"))?;
+        let content = client
+            .get(&path)
+            .await
+            .map_err(to_anyhow)?
+            .bytes()
+            .await
+            .map_err(to_anyhow)?;
+        if content != Bytes::from_static(b"hello") {
+            return Err(error::Error::internal_err(
+                "Failed to read back from blob storage".to_string(),
+            ));
+        }
+        client.delete(&path).await.map_err(to_anyhow)?;
+        Ok::<String, error::Error>("Tested blob storage successfully".to_string())
+    };
+
+    if restrict {
+        // The object-store client is built with timeouts disabled, so a malicious endpoint could
+        // otherwise hold the API server connection open indefinitely.
+        tokio::time::timeout(Duration::from_secs(15), run)
+            .await
+            .map_err(|_| {
+                error::Error::internal_err("Object storage connectivity test timed out".to_string())
+            })?
    } else {
-        tracing::info!("No files in blob storage");
+        run.await
+    }
+}
+
+// Hardening for the object-storage connectivity test by an untrusted (non-super-admin) caller on
+// Cloud. The probe runs on the shared API server, so without these constraints an authenticated
+// user could coerce the server into connecting to arbitrary internal endpoints (SSRF), signing
+// requests with the instance role (credential exfiltration), or reading/writing the server's local
+// disk (filesystem object store).
+#[cfg(feature = "parquet")]
+async fn validate_object_storage_test(settings: &ObjectSettings) -> error::Result<()> {
+    fn non_empty(opt: &Option<String>) -> bool {
+        opt.as_ref().is_some_and(|s| !s.is_empty())
    }

-    let path = windmill_object_store::object_store_reexports::Path::from(format!(
-        "/test-s3-bucket-{uuid}",
-        uuid = uuid::Uuid::new_v4()
-    ));
-    tracing::info!("Testing blob storage at path: {path}");
-    client
-        .put(
-            &path,
-            windmill_object_store::object_store_reexports::PutPayload::from_static(b"hello"),
-        )
+    // Reject backends that rely on the server's identity or local filesystem, require explicit
+    // credentials for the rest (so the server never falls back to its own ambient credentials), and
+    // resolve the host the client will actually connect to. We derive the *effective* endpoint here
+    // — mirroring build_*_from_settings: the region/account-derived default and the virtual-hosted
+    // bucket prefix — rather than only validating a caller-supplied `endpoint`, so caller-controlled
+    // `region`/`account_name`/`bucket` cannot smuggle an internal host past the check (e.g. an empty
+    // endpoint with region = "@169.254.169.254/" otherwise resolves to the cloud metadata service).
+    let effective_endpoint: Option<String> = match settings {
+        ObjectSettings::Filesystem(_) => {
+            return Err(error::Error::NotAuthorized(
+                "Testing a local filesystem object store requires a super admin".to_string(),
+            ));
+        }
+        ObjectSettings::AwsOidc(_) => {
+            return Err(error::Error::NotAuthorized(
+                "Testing OIDC-based object storage requires a super admin".to_string(),
+            ));
+        }
+        ObjectSettings::S3(s3) => {
+            if !(non_empty(&s3.access_key) && non_empty(&s3.secret_key)) {
+                return Err(error::Error::NotAuthorized(
+                    "Testing S3 storage without explicit credentials requires a super admin"
+                        .to_string(),
+                ));
+            }
+            let region = s3
+                .region
+                .clone()
+                .filter(|r| !r.is_empty())
+                .or_else(|| std::env::var("AWS_REGION").ok().filter(|r| !r.is_empty()))
+                .unwrap_or_else(|| "us-east-1".to_string());
+            let raw_endpoint = s3
+                .endpoint
+                .clone()
+                .filter(|e| !e.is_empty())
+                .or_else(|| std::env::var("S3_ENDPOINT").ok().filter(|e| !e.is_empty()))
+                .unwrap_or_else(|| format!("s3.{region}.amazonaws.com"));
+            Some(windmill_object_store::render_endpoint(
+                raw_endpoint,
+                !s3.allow_http.unwrap_or(true),
+                s3.port,
+                s3.path_style,
+                s3.bucket.clone().unwrap_or_default(),
+            ))
+        }
+        ObjectSettings::Azure(azure) => {
+            if !non_empty(&azure.access_key) {
+                return Err(error::Error::NotAuthorized(
+                    "Testing Azure storage without an explicit access key requires a super admin"
+                        .to_string(),
+                ));
+            }
+            Some(
+                azure
+                    .endpoint
+                    .clone()
+                    .filter(|e| !e.is_empty())
+                    .unwrap_or_else(|| format!("{}.blob.core.windows.net", azure.account_name)),
+            )
+        }
+        ObjectSettings::Gcs(gcs) => {
+            if gcs.service_account_key.is_empty() {
+                return Err(error::Error::NotAuthorized(
+                    "Testing GCS storage without a service account key requires a super admin"
+                        .to_string(),
+                ));
+            }
+            // The service-account-key JSON can override the data-plane URL (`gcs_base_url`) and the
+            // OAuth token endpoint (`token_uri`); the GCS client connects to whatever they point at.
+            // Validate every http(s) URL embedded in the key. When none override it, the host stays
+            // the public storage.googleapis.com, so no further check is needed.
+            if let Ok(serde_json::Value::Object(map)) =
+                serde_json::from_str::<serde_json::Value>(&gcs.service_account_key)
+            {
+                for value in map.values() {
+                    if let Some(url) = value.as_str() {
+                        // Match how the URL parser reads the value: leading whitespace/control is
+                        // ignored and the scheme is case-insensitive.
+                        let url =
+                            url.trim_start_matches(|c: char| c.is_whitespace() || c.is_control());
+                        if strip_http_scheme(url).is_some() {
+                            validate_public_endpoint(url).await?;
+                        }
+                    }
+                }
+            }
+            None
+        }
+    };
+
+    // Block non-public network targets (internal services, cloud metadata, loopback, ...).
+    if let Some(endpoint) = effective_endpoint {
+        validate_public_endpoint(&endpoint).await?;
+    }
+    Ok(())
+}
+
+#[cfg(feature = "parquet")]
+async fn validate_public_endpoint(endpoint: &str) -> error::Result<()> {
+    let host = extract_host(endpoint).ok_or_else(|| {
+        error::Error::BadRequest(format!("Invalid object storage endpoint: {endpoint}"))
+    })?;
+
+    let addrs: Vec<std::net::SocketAddr> = tokio::net::lookup_host((host.as_str(), 443u16))
        .await
-        .map_err(|e| anyhow::anyhow!("error writing file to {path}: {e:#}"))?;
-    let content = client
-        .get(&path)
-        .await
-        .map_err(to_anyhow)?
-        .bytes()
-        .await
-        .map_err(to_anyhow)?;
-    if content != Bytes::from_static(b"hello") {
-        return Err(error::Error::internal_err(
-            "Failed to read back from blob storage".to_string(),
-        ));
+        .map_err(|e| {
+            error::Error::BadRequest(format!(
+                "Could not resolve object storage endpoint '{host}': {e}"
+            ))
+        })?
+        .collect();
+
+    if addrs.is_empty() {
+        return Err(error::Error::BadRequest(format!(
+            "Could not resolve object storage endpoint '{host}'"
+        )));
+    }
+
+    // Reject if any resolved address is non-public, which also defeats the simplest DNS-rebinding
+    // attempts (a name resolving to both a public and a private address).
+    for addr in addrs {
+        if is_forbidden_ip(addr.ip()) {
+            return Err(error::Error::NotAuthorized(
+                "Testing object storage at a private, loopback, or link-local endpoint requires a super admin"
+                    .to_string(),
+            ));
+        }
+    }
+    Ok(())
+}
+
+// Strip a leading `http://`/`https://` scheme case-insensitively (URL schemes are
+// case-insensitive), returning the remainder when one was present.
+#[cfg(feature = "parquet")]
+fn strip_http_scheme(s: &str) -> Option<&str> {
+    for scheme in ["https://", "http://"] {
+        let b = scheme.as_bytes();
+        if s.len() >= b.len() && s.as_bytes()[..b.len()].eq_ignore_ascii_case(b) {
+            return Some(&s[b.len()..]);
+        }
+    }
+    None
+}
+
+#[cfg(feature = "parquet")]
+fn extract_host(endpoint: &str) -> Option<String> {
+    let mut s = endpoint.trim();
+    if let Some(rest) = strip_http_scheme(s) {
+        s = rest;
+    }
+    s = s.split(['/', '?', '#', '\\']).next().unwrap_or(s);
+    if let Some((_, rest)) = s.rsplit_once('@') {
+        s = rest;
+    }
+    let host = if let Some(rest) = s.strip_prefix('[') {
+        // IPv6 literal, e.g. [::1]:9000
+        rest.split(']').next().unwrap_or(rest)
+    } else {
+        // host or host:port
+        s.split(':').next().unwrap_or(s)
+    }
+    .trim();
+    if host.is_empty() {
+        None
+    } else {
+        Some(host.to_string())
+    }
+}
+
+#[cfg(feature = "parquet")]
+fn is_forbidden_ip(ip: std::net::IpAddr) -> bool {
+    use std::net::{IpAddr, Ipv4Addr};
+    match ip {
+        IpAddr::V4(v4) => {
+            v4.is_loopback()
+                || v4.is_private()
+                || v4.is_link_local() // 169.254.0.0/16, incl. the cloud metadata endpoint
+                || v4.is_unspecified()
+                || v4.is_broadcast()
+                || v4.is_documentation()
+                || v4.is_multicast()
+                || v4.octets()[0] == 0 // 0.0.0.0/8
+                || (v4.octets()[0] == 100 && (v4.octets()[1] & 0xc0) == 64) // 100.64.0.0/10 CGNAT
+        }
+        IpAddr::V6(v6) => {
+            // Any IPv4 embedded in an IPv6 address (IPv4-mapped ::ffff:0:0/96, IPv4-compatible
+            // ::/96, or NAT64 64:ff9b::/96) is re-checked against the IPv4 rules, so e.g.
+            // 64:ff9b::169.254.169.254 cannot route to the metadata endpoint in a NAT64 network.
+            let seg = v6.segments();
+            let is_v4_compatible = seg[0..6] == [0, 0, 0, 0, 0, 0];
+            let is_nat64 = seg[0] == 0x0064 && seg[1] == 0xff9b && seg[2..6] == [0, 0, 0, 0];
+            if let Some(v4) = v6.to_ipv4_mapped() {
+                return is_forbidden_ip(IpAddr::V4(v4));
+            }
+            if is_v4_compatible || is_nat64 {
+                let embedded = Ipv4Addr::new(
+                    (seg[6] >> 8) as u8,
+                    (seg[6] & 0xff) as u8,
+                    (seg[7] >> 8) as u8,
+                    (seg[7] & 0xff) as u8,
+                );
+                if is_forbidden_ip(IpAddr::V4(embedded)) {
+                    return true;
+                }
+            }
+            v6.is_loopback()
+                || v6.is_unspecified()
+                || v6.is_multicast()
+                || (seg[0] & 0xfe00) == 0xfc00 // fc00::/7 unique local
+                || (seg[0] & 0xffc0) == 0xfe80 // fe80::/10 link-local
+        }
    }
-    client.delete(&path).await.map_err(to_anyhow)?;
-    Ok("Tested blob storage successfully".to_string())
 }

 #[cfg(feature = "parquet")]
@@ -1275,8 +1523,8 @@ async fn setup_custom_instance_pg_database_inner(
    // Validate name to ensure it only contains alphanumeric characters
    // Prevents SQL injection on the instance database
    lazy_static::lazy_static! {
-        // Must start with a letter, then alphanumeric/underscore
-        static ref VALID_NAME: regex::Regex = regex::Regex::new(r"^[a-zA-Z][a-zA-Z0-9_]*$").unwrap();
+        // Must start with a letter, then alphanumeric/underscore/hyphen
+        static ref VALID_NAME: regex::Regex = regex::Regex::new(r"^[a-zA-Z][a-zA-Z0-9_-]*$").unwrap();
    }
    let dbname = dbname.trim();
    if dbname.is_empty() {
@@ -1292,7 +1540,7 @@ async fn setup_custom_instance_pg_database_inner(
    }
    if !VALID_NAME.is_match(dbname) {
        return Err(error::Error::BadRequest(
-            "Database name must start with a letter and contain only alphanumeric characters or underscores".to_string(),
+            "Database name must start with a letter and contain only alphanumeric characters, underscores, or hyphens".to_string(),
        ));
    }
    // Additional check: block PostgreSQL reserved/special names
@@ -1861,3 +2109,111 @@ mod tests {
        );
    }
 }
+
+#[cfg(all(test, feature = "parquet"))]
+mod object_storage_test_hardening {
+    use super::{extract_host, is_forbidden_ip, validate_object_storage_test};
+    use std::net::IpAddr;
+    use windmill_object_store::ObjectSettings;
+
+    // IP literals (not hostnames) keep validate_public_endpoint deterministic — `lookup_host`
+    // parses them without any network round-trip.
+    fn gcs_settings(gcs_base_url: &str) -> ObjectSettings {
+        serde_json::from_value(serde_json::json!({
+            "type": "Gcs",
+            "bucket": "b",
+            "serviceAccountKey": { "gcs_base_url": gcs_base_url, "client_email": "x@y.z" }
+        }))
+        .unwrap()
+    }
+
+    #[tokio::test]
+    async fn rejects_gcs_internal_base_url() {
+        // gcs_base_url in the service-account key must not smuggle an internal host past the check,
+        // including via a mixed-case scheme (URL schemes are case-insensitive).
+        for url in [
+            "http://169.254.169.254",
+            "HTTP://169.254.169.254",
+            "Https://10.0.0.5",
+        ] {
+            assert!(
+                validate_object_storage_test(&gcs_settings(url))
+                    .await
+                    .is_err(),
+                "{url} should be rejected"
+            );
+        }
+    }
+
+    #[tokio::test]
+    async fn allows_gcs_public_base_url() {
+        assert!(
+            validate_object_storage_test(&gcs_settings("https://8.8.8.8"))
+                .await
+                .is_ok()
+        );
+    }
+
+    fn ip(s: &str) -> IpAddr {
+        s.parse().unwrap()
+    }
+
+    #[test]
+    fn forbids_internal_ips() {
+        for s in [
+            "127.0.0.1",                // loopback
+            "169.254.169.254",          // cloud metadata (link-local)
+            "10.0.0.5",                 // private
+            "172.16.3.4",               // private
+            "192.168.1.10",             // private
+            "0.0.0.0",                  // unspecified
+            "100.64.0.1",               // CGNAT
+            "::1",                      // IPv6 loopback
+            "fe80::1",                  // IPv6 link-local
+            "fc00::1",                  // IPv6 unique local
+            "::ffff:127.0.0.1",         // IPv4-mapped loopback
+            "::ffff:169.254.169.254",   // IPv4-mapped metadata
+            "::169.254.169.254",        // IPv4-compatible metadata
+            "64:ff9b::169.254.169.254", // NAT64-embedded metadata
+            "64:ff9b::a9fe:a9fe",       // NAT64-embedded metadata (hex form)
+        ] {
+            assert!(is_forbidden_ip(ip(s)), "{s} should be forbidden");
+        }
+    }
+
+    #[test]
+    fn allows_public_ips() {
+        for s in ["8.8.8.8", "1.1.1.1", "52.95.110.1", "2606:4700:4700::1111"] {
+            assert!(!is_forbidden_ip(ip(s)), "{s} should be allowed");
+        }
+    }
+
+    #[test]
+    fn extracts_host_from_endpoint() {
+        let cases = [
+            ("s3.amazonaws.com", Some("s3.amazonaws.com")),
+            ("https://minio.internal:9000", Some("minio.internal")),
+            ("http://10.0.0.5:9000/bucket", Some("10.0.0.5")),
+            ("user:pass@host.example:443", Some("host.example")),
+            ("[::1]:9000", Some("::1")),
+            ("https://[fe80::1]/x", Some("fe80::1")),
+            ("", None),
+            // Injection via region/bucket interpolation into the default endpoint string: the
+            // userinfo `@` and the path `/` must not hide the real authority from the host check.
+            (
+                "https://s3.@169.254.169.254/.amazonaws.com",
+                Some("169.254.169.254"),
+            ),
+            (
+                "https://@169.254.169.254/mybucket.s3.amazonaws.com",
+                Some("169.254.169.254"),
+            ),
+            ("s3.#@169.254.169.254/x.amazonaws.com", Some("s3.")),
+            // Scheme is case-insensitive.
+            ("HTTP://169.254.169.254", Some("169.254.169.254")),
+        ];
+        for (input, expected) in cases {
+            assert_eq!(extract_host(input).as_deref(), expected, "input: {input}");
+        }
+    }
+}
@@ -339,13 +339,15 @@ async fn cleanup_job_logs(
        return Ok(());
    }

+    let mut completed_at_floor: Option<DateTime<Utc>> = None;
    loop {
-        let (deleted_count, rel_paths) =
-            delete_expired_jobs_batch(db, retention_secs, JOB_BATCH).await?;
+        let (deleted_count, rel_paths, max_completed_at) =
+            delete_expired_jobs_batch(db, retention_secs, JOB_BATCH, completed_at_floor).await?;

        if deleted_count == 0 {
            break;
        }
+        completed_at_floor = max_completed_at.or(completed_at_floor);

        let s3_paths: Vec<ObjectPath> = rel_paths
            .iter()
@@ -382,7 +384,8 @@ async fn delete_expired_jobs_batch(
    db: &DB,
    job_retention_secs: i64,
    batch_size: i64,
-) -> error::Result<(usize, Vec<String>)> {
+    completed_at_floor: Option<DateTime<Utc>>,
+) -> error::Result<(usize, Vec<String>, Option<DateTime<Utc>>)> {
    let mut tx = db.begin().await?;

    let active_root_job_ids: Vec<Uuid> = sqlx::query_scalar!(
@@ -395,33 +398,61 @@ async fn delete_expired_jobs_batch(
    .fetch_all(&mut *tx)
    .await?;

-    // Active-root exclusion via NOT IN (hashed SubPlan) instead of `!= ALL($3)`;
-    // see backend/src/monitor.rs::delete_expired_jobs_batch for the rationale.
-    let deleted_jobs: Vec<Uuid> = sqlx::query_scalar!(
-        "DELETE FROM v2_job_completed
-         WHERE id IN (
-             SELECT jc.id FROM v2_job_completed jc
-             LEFT JOIN v2_job j ON j.id = jc.id
-             WHERE jc.completed_at <= now() - ($1::bigint::text || ' s')::interval
-               AND COALESCE(j.root_job, j.flow_innermost_root_job, jc.id) NOT IN (
-                   SELECT u FROM unnest($3::uuid[]) AS u WHERE u IS NOT NULL
-               )
-             ORDER BY jc.completed_at ASC
-             LIMIT $2
-             FOR UPDATE OF jc SKIP LOCKED
-         )
-         RETURNING id",
-        job_retention_secs,
-        batch_size,
-        &active_root_job_ids
-    )
-    .fetch_all(&mut *tx)
-    .await?;
+    // `completed_at_floor` carries a watermark across batches so each one resumes after the rows
+    // the previous batch processed instead of re-scanning the (potentially undeletable) oldest
+    // prefix; the empty-active-roots branch skips the v2_job join entirely. See
+    // backend/src/monitor.rs::delete_expired_jobs_batch for the full rationale.
+    let (deleted_jobs, max_completed_at) = if active_root_job_ids.is_empty() {
+        let rows = sqlx::query!(
+            "DELETE FROM v2_job_completed
+             WHERE id IN (
+                 SELECT id FROM v2_job_completed
+                 WHERE completed_at <= now() - ($1::bigint::text || ' s')::interval
+                   AND ($3::timestamptz IS NULL OR completed_at >= $3)
+                 ORDER BY completed_at ASC
+                 LIMIT $2
+                 FOR UPDATE SKIP LOCKED
+             )
+             RETURNING id, completed_at",
+            job_retention_secs,
+            batch_size,
+            completed_at_floor,
+        )
+        .fetch_all(&mut *tx)
+        .await?;
+        let max = rows.iter().map(|r| r.completed_at).max();
+        (rows.into_iter().map(|r| r.id).collect::<Vec<Uuid>>(), max)
+    } else {
+        let rows = sqlx::query!(
+            "DELETE FROM v2_job_completed
+             WHERE id IN (
+                 SELECT jc.id FROM v2_job_completed jc
+                 LEFT JOIN v2_job j ON j.id = jc.id
+                 WHERE jc.completed_at <= now() - ($1::bigint::text || ' s')::interval
+                   AND ($4::timestamptz IS NULL OR jc.completed_at >= $4)
+                   AND COALESCE(j.root_job, j.flow_innermost_root_job, jc.id) NOT IN (
+                       SELECT u FROM unnest($3::uuid[]) AS u WHERE u IS NOT NULL
+                   )
+                 ORDER BY jc.completed_at ASC
+                 LIMIT $2
+                 FOR UPDATE OF jc SKIP LOCKED
+             )
+             RETURNING id, completed_at",
+            job_retention_secs,
+            batch_size,
+            &active_root_job_ids,
+            completed_at_floor,
+        )
+        .fetch_all(&mut *tx)
+        .await?;
+        let max = rows.iter().map(|r| r.completed_at).max();
+        (rows.into_iter().map(|r| r.id).collect::<Vec<Uuid>>(), max)
+    };

    let deleted_count = deleted_jobs.len();
    if deleted_count == 0 {
        tx.commit().await?;
-        return Ok((0, Vec::new()));
+        return Ok((0, Vec::new(), max_completed_at));
    }

    if let Err(e) = sqlx::query!(
@@ -471,7 +502,7 @@ async fn delete_expired_jobs_batch(

    tx.commit().await?;

-    Ok((deleted_count, log_paths))
+    Ok((deleted_count, log_paths, max_completed_at))
 }

 /// Scan S3 under the `logs/` prefix for orphan log files and delete them.
@@ -27,7 +27,7 @@ use axum::{
    Json, Router,
 };
 use hyper::{header::LOCATION, StatusCode};
-use windmill_api_auth::require_super_admin;
+use windmill_api_auth::{forbid_superadmin_job_token, require_super_admin, OptJobAuthed};
 use windmill_common::usernames::{
    generate_instance_wide_unique_username, get_instance_username_or_create_pending,
 };
@@ -1415,11 +1415,13 @@ async fn convert_user_to_group(

 async fn update_user(
    authed: ApiAuthed,
+    OptJobAuthed { job_id, .. }: OptJobAuthed,
    Path(email_to_update): Path<String>,
    Extension(db): Extension<DB>,
    Json(eu): Json<EditUser>,
 ) -> Result<String> {
    require_super_admin(&db, &authed.email).await?;
+    forbid_superadmin_job_token(&db, &authed.email, job_id).await?;
    let mut tx = db.begin().await?;

    let mut new_super_admin: Option<bool> = None;
@@ -1581,10 +1583,12 @@ async fn update_user(

 async fn delete_user(
    authed: ApiAuthed,
+    OptJobAuthed { job_id, .. }: OptJobAuthed,
    Path(email_to_delete): Path<String>,
    Extension(db): Extension<DB>,
 ) -> Result<String> {
    require_super_admin(&db, &authed.email).await?;
+    forbid_superadmin_job_token(&db, &authed.email, job_id).await?;
    let mut tx = db.begin().await?;

    sqlx::query!("DELETE FROM token WHERE email = $1", &email_to_delete)
@@ -1877,9 +1881,11 @@ async fn set_login_type(
    Extension(db): Extension<DB>,
    Path(email): Path<String>,
    authed: ApiAuthed,
+    OptJobAuthed { job_id, .. }: OptJobAuthed,
    Json(et): Json<EditLoginType>,
 ) -> Result<String> {
    require_super_admin(&db, &authed.email).await?;
+    forbid_superadmin_job_token(&db, &authed.email, job_id).await?;
    let mut tx = db.begin().await?;

    sqlx::query!(
@@ -2159,8 +2165,10 @@ pub async fn create_session_token<'c>(
 async fn create_token(
    Extension(db): Extension<DB>,
    authed: ApiAuthed,
+    OptJobAuthed { job_id, .. }: OptJobAuthed,
    Json(token_config): Json<NewToken>,
 ) -> Result<(StatusCode, String)> {
+    forbid_superadmin_job_token(&db, &authed.email, job_id).await?;
    check_token_create_rate_limit(&authed.username)?;

    windmill_api_auth::ensure_scopes_within_caller(&authed, token_config.scopes.as_deref())?;
@@ -2176,6 +2184,7 @@ async fn create_token(
 async fn impersonate(
    Extension(db): Extension<DB>,
    authed: ApiAuthed,
+    OptJobAuthed { job_id, .. }: OptJobAuthed,
    Json(new_token): Json<NewToken>,
 ) -> Result<(StatusCode, String)> {
    use windmill_common::min_version::MIN_VERSION_SUPPORTS_TOKEN_HASH;
@@ -2189,6 +2198,7 @@ async fn impersonate(
        Some(&token)
    };
    require_super_admin(&db, &authed.email).await?;
+    forbid_superadmin_job_token(&db, &authed.email, job_id).await?;

    if new_token.impersonate_email.is_none() {
        return Err(Error::BadRequest(
@@ -2707,8 +2717,10 @@ struct ExportedGlobalUser {
 async fn export_global_users(
    Extension(db): Extension<DB>,
    authed: ApiAuthed,
+    OptJobAuthed { job_id, .. }: OptJobAuthed,
 ) -> JsonResult<Vec<ExportedGlobalUser>> {
    require_super_admin(&db, &authed.email).await?;
+    forbid_superadmin_job_token(&db, &authed.email, job_id).await?;
    let mut tx = db.begin().await?;
    let users = sqlx::query_as!(
        ExportedGlobalUser,
@@ -2744,9 +2756,11 @@ async fn export_global_users() -> JsonResult<String> {
 async fn overwrite_global_users(
    Extension(db): Extension<DB>,
    authed: ApiAuthed,
+    OptJobAuthed { job_id, .. }: OptJobAuthed,
    Json(users): Json<Vec<ExportedGlobalUser>>,
 ) -> Result<String> {
    require_super_admin(&db, &authed.email).await?;
+    forbid_superadmin_job_token(&db, &authed.email, job_id).await?;
    let mut tx = db.begin().await?;
    sqlx::query!("DELETE FROM password")
        .execute(&mut *tx)
@@ -717,16 +717,27 @@ async fn create_deployment_request_comment(
 // ---- helpers ------------------------------------------------------------

 async fn parent_of_fork(db: &DB, w_id: &str) -> Result<String> {
-    sqlx::query_scalar!(
-        "SELECT parent_workspace_id FROM workspace WHERE id = $1",
+    // Resolve the fork's parent and require it to still exist and be active. A
+    // parent that is archived (soft-deleted) can no longer be accessed, so a
+    // diff or deployment request against it targets an unreachable workspace.
+    let parent = sqlx::query!(
+        "SELECT p.id AS \"id!\", p.deleted AS \"deleted!\"
+         FROM workspace f
+         JOIN workspace p ON p.id = f.parent_workspace_id
+         WHERE f.id = $1",
        w_id,
    )
    .fetch_optional(db)
-    .await?
-    .flatten()
-    .ok_or_else(|| {
-        Error::BadRequest(format!(
+    .await?;
+
+    match parent {
+        None => Err(Error::BadRequest(format!(
            "workspace {w_id} is not a fork (no parent_workspace_id)"
-        ))
-    })
+        ))),
+        Some(p) if p.deleted => Err(Error::BadRequest(format!(
+            "parent workspace {} of fork {w_id} is archived",
+            p.id
+        ))),
+        Some(p) => Ok(p.id),
+    }
 }
@@ -197,6 +197,7 @@ pub fn global_service() -> Router {
        .route("/list_as_superadmin", get(list_workspaces_as_super_admin))
        .route("/list", get(list_workspaces))
        .route("/users", get(user_workspaces))
+        .route("/session_workspace_status", post(session_workspace_status))
        .route("/create", post(create_workspace))
        .route("/create_fork", post(deprecated_create_workspace_fork))
        .route("/exists", post(exists_workspace))
@@ -3623,6 +3624,47 @@ async fn user_workspaces(
    Ok(Json(WorkspaceList { email, workspaces }))
 }

+#[derive(Deserialize)]
+struct SessionWorkspaceStatusRequest {
+    workspace_ids: Vec<String>,
+}
+
+/// Reconciliation support for client-side AI sessions, which the backend cannot touch
+/// directly. The client posts the workspace ids its sessions reference and uses the
+/// per-id status to keep sessions in sync with workspace lifecycle: `deleted` (no row /
+/// no access → unresolvable) drops the sessions, `archived` (soft-deleted, still a
+/// member) archives them, `active` restores ones previously archived-by-workspace.
+/// Archived and hard-deleted workspaces are absent from `user_workspaces`, so this is the
+/// only way the client learns about a change made while it was away or on another device.
+async fn session_workspace_status(
+    Extension(db): Extension<DB>,
+    ApiAuthed { email, .. }: ApiAuthed,
+    Json(req): Json<SessionWorkspaceStatusRequest>,
+) -> JsonResult<HashMap<String, String>> {
+    if req.workspace_ids.len() > 1000 {
+        return Err(Error::BadRequest(
+            "Too many workspace ids (max 1000)".to_string(),
+        ));
+    }
+    let rows = sqlx::query!(
+        "SELECT req.id AS \"id!\",
+                (CASE
+                    WHEN usr.email IS NULL THEN 'deleted'
+                    WHEN workspace.deleted THEN 'archived'
+                    ELSE 'active'
+                END) AS \"status!\"
+         FROM unnest($1::text[]) AS req(id)
+         LEFT JOIN workspace ON workspace.id = req.id
+         LEFT JOIN usr ON usr.workspace_id = workspace.id AND usr.email = $2",
+        &req.workspace_ids[..],
+        email,
+    )
+    .fetch_all(&db)
+    .await?;
+    let statuses = rows.into_iter().map(|r| (r.id, r.status)).collect();
+    Ok(Json(statuses))
+}
+
 pub async fn check_w_id_conflict<'c>(tx: &mut Transaction<'c, Postgres>, w_id: &str) -> Result<()> {
    if w_id == "global" {
        return Err(windmill_common::error::Error::BadRequest(
@@ -65,13 +65,22 @@ pub(crate) async fn change_workspace_id(
        old_id, rw.new_id
    );

-    // Create new workspace with new id and name
+    // Create new workspace with new id and name. A fork that keeps a wm-fork-
+    // id must carry its parent_workspace_id over, otherwise it becomes a
+    // parentless "fork of nothing" with no source to compare or merge against.
+    // A non-fork target id means the workspace is being promoted out of a fork,
+    // so the parent pointer is intentionally cleared.
    info!("Creating new workspace row");
+    let new_is_fork = rw.new_id.starts_with(WM_FORK_PREFIX);
    sqlx::query!(
-        "INSERT INTO workspace SELECT $1, $2, owner, false, premium FROM workspace WHERE id = $3",
+        "INSERT INTO workspace (id, name, owner, deleted, premium, parent_workspace_id)
+         SELECT $1, $2, owner, false, premium,
+                CASE WHEN $4 THEN parent_workspace_id ELSE NULL END
+         FROM workspace WHERE id = $3",
        &rw.new_id,
        &rw.new_name,
-        &old_id
+        &old_id,
+        new_is_fork
    )
    .execute(&mut *tx)
    .await?;
@@ -347,6 +356,18 @@ pub(crate) async fn change_workspace_id(
    .execute(&mut *tx)
    .await?;

+    // Re-parent child forks: any fork whose parent_workspace_id was the old id
+    // must follow the renamed parent to the new id, otherwise it is left
+    // pointing at the soft-deleted old shell (whose data has moved here).
+    info!("Re-parenting child forks to the new workspace id");
+    sqlx::query!(
+        "UPDATE workspace SET parent_workspace_id = $1 WHERE parent_workspace_id = $2",
+        &rw.new_id,
+        &old_id
+    )
+    .execute(&mut *tx)
+    .await?;
+
    info!("Updating workspace_protection_rule table");
    sqlx::query!(
        "UPDATE workspace_protection_rule SET workspace_id = $1 WHERE workspace_id = $2",
@@ -1,7 +1,7 @@
 openapi: "3.0.3"

 info:
-  version: 1.737.0
+  version: 1.739.0
  title: Windmill API

  contact:
@@ -997,6 +997,39 @@ paths:
              schema:
                $ref: "#/components/schemas/UserWorkspaceList"

+  /workspaces/session_workspace_status:
+    post:
+      summary: get the lifecycle status of workspaces referenced by client-side sessions
+      operationId: getSessionWorkspaceStatus
+      tags:
+        - workspace
+      requestBody:
+        required: true
+        content:
+          application/json:
+            schema:
+              type: object
+              properties:
+                workspace_ids:
+                  type: array
+                  items:
+                    type: string
+              required:
+                - workspace_ids
+      responses:
+        "200":
+          description: map of workspace id to status (active, archived, or deleted)
+          content:
+            application/json:
+              schema:
+                type: object
+                additionalProperties:
+                  type: string
+                  enum:
+                    - active
+                    - archived
+                    - deleted
+
  /w/{workspace}/workspaces/get_as_superadmin:
    get:
      summary: get workspace as super admin (require to be super admin)
@@ -84,6 +84,12 @@ lazy_static::lazy_static! {
                    (20260228000000, include_str!(
                        "../../migrations/20260228000000_v2_job_completed_failure_index.up.sql"
                    ).replace("CREATE INDEX", "CREATE INDEX CONCURRENTLY")),
+                    (20260610151334, include_str!(
+                        "../../migrations/20260610151334_folder_labels.up.sql"
+                    ).replace("SET search_path = public", "SET search_path FROM CURRENT").to_string()),
+                    (20260614075900, include_str!(
+                        "../../migrations/20260614075900_dedup_folder_labels.up.sql"
+                    ).replace("SET search_path = public", "SET search_path FROM CURRENT").to_string()),
                    ].into_iter().collect();
 }

@@ -1,13 +1,13 @@
 use std::collections::HashMap;

-use crate::db::ApiAuthed;
+use crate::db::{ApiAuthed, OptJobAuthed};
 use crate::secret_backend_ext::rename_vault_secrets_with_prefix;
 use axum::{
    extract::{Extension, Path},
    Json,
 };
 use serde::{Deserialize, Serialize};
-use windmill_api_auth::require_super_admin;
+use windmill_api_auth::{forbid_superadmin_job_token, require_super_admin};
 use windmill_api_users::users::delete_workspace_user_internal;
 use windmill_audit::audit_oss::audit_log;
 use windmill_audit::ActionKind;
@@ -483,11 +483,13 @@ pub(crate) async fn global_offboard_preview(

 pub(crate) async fn offboard_global_user(
    authed: ApiAuthed,
+    OptJobAuthed { job_id, .. }: OptJobAuthed,
    Extension(db): Extension<DB>,
    Path(email): Path<String>,
    Json(req): Json<GlobalOffboardRequest>,
 ) -> Result<Json<OffboardResponse>> {
    require_super_admin(&db, &authed.email).await?;
+    forbid_superadmin_job_token(&db, &authed.email, job_id).await?;

    let workspaces = sqlx::query!(
        "SELECT workspace_id, username FROM usr WHERE email = $1",
@@ -8,245 +8,24 @@

 //! Secret backend extension for the API layer
 //!
-//! This module provides helper functions for integrating the SecretBackend
-//! trait with variable operations in the API.
+//! Backend resolution and read helpers live in
+//! `windmill_common::secret_backend`; this module keeps the API-specific bulk
+//! rename helper used when renaming users.
 //!
 //! Note: HashiCorp Vault integration requires Enterprise Edition.
 //! The OSS version only supports the database backend.

-#[cfg(all(feature = "private", feature = "enterprise"))]
-use std::sync::Arc;
-
 use windmill_common::{db::DB, error::Result};

-#[cfg(all(feature = "private", feature = "enterprise"))]
-use windmill_common::error::Error;
-
-#[cfg(all(feature = "private", feature = "enterprise"))]
-use windmill_common::secret_backend::{database::DatabaseBackend, SecretBackend};
-
 #[cfg(all(feature = "private", feature = "enterprise"))]
 use windmill_common::{
-    global_settings::{load_value_from_global_settings, SECRET_BACKEND_SETTING},
+    error::Error,
    secret_backend::{
-        AwsSecretsManagerBackend, AwsSecretsManagerSettings, AzureKeyVaultBackend,
-        AzureKeyVaultSettings, SecretBackendConfig, VaultBackend, VaultSettings,
+        get_secret_backend, is_aws_sm_stored_value, is_azure_kv_stored_value,
+        is_external_stored_value, is_vault_backend_configured,
    },
 };

-#[cfg(all(feature = "private", feature = "enterprise"))]
-use tokio::sync::RwLock;
-
-// Cached Vault backend to avoid recreating it for every request
-// This enables connection pooling and avoids repeated setup overhead
-#[cfg(all(feature = "private", feature = "enterprise"))]
-struct CachedVaultBackend {
-    backend: Arc<dyn SecretBackend>,
-    settings: VaultSettings,
-}
-
-#[cfg(all(feature = "private", feature = "enterprise"))]
-lazy_static::lazy_static! {
-    static ref VAULT_BACKEND_CACHE: RwLock<Option<CachedVaultBackend>> = RwLock::new(None);
-}
-
-// Cached Azure Key Vault backend
-#[cfg(all(feature = "private", feature = "enterprise"))]
-struct CachedAzureKvBackend {
-    backend: Arc<dyn SecretBackend>,
-    settings: AzureKeyVaultSettings,
-}
-
-#[cfg(all(feature = "private", feature = "enterprise"))]
-lazy_static::lazy_static! {
-    static ref AZURE_KV_BACKEND_CACHE: RwLock<Option<CachedAzureKvBackend>> = RwLock::new(None);
-}
-
-// Cached AWS Secrets Manager backend
-#[cfg(all(feature = "private", feature = "enterprise"))]
-struct CachedAwsSmBackend {
-    backend: Arc<dyn SecretBackend>,
-    settings: AwsSecretsManagerSettings,
-}
-
-#[cfg(all(feature = "private", feature = "enterprise"))]
-lazy_static::lazy_static! {
-    static ref AWS_SM_BACKEND_CACHE: RwLock<Option<CachedAwsSmBackend>> = RwLock::new(None);
-}
-
-/// Get the current secret backend based on global settings (EE only)
-#[cfg(all(feature = "private", feature = "enterprise"))]
-async fn get_secret_backend(db: &DB) -> Result<Arc<dyn SecretBackend>> {
-    let config = match load_value_from_global_settings(db, SECRET_BACKEND_SETTING).await? {
-        Some(value) => serde_json::from_value::<SecretBackendConfig>(value).unwrap_or_default(),
-        None => SecretBackendConfig::default(),
-    };
-
-    match config {
-        SecretBackendConfig::Database => Ok(Arc::new(DatabaseBackend::new(db.clone()))),
-        SecretBackendConfig::HashiCorpVault(settings) => {
-            get_or_create_vault_backend(db, settings).await
-        }
-        SecretBackendConfig::AzureKeyVault(settings) => {
-            get_or_create_azure_kv_backend(db, settings).await
-        }
-        SecretBackendConfig::AwsSecretsManager(settings) => {
-            get_or_create_aws_sm_backend(db, settings).await
-        }
-    }
-}
-
-/// Get a cached Vault backend or create a new one if settings changed
-#[cfg(all(feature = "private", feature = "enterprise"))]
-async fn get_or_create_vault_backend(
-    _db: &DB,
-    settings: VaultSettings,
-) -> Result<Arc<dyn SecretBackend>> {
-    // Check if we have a cached backend with matching settings (read lock)
-    {
-        let cache = VAULT_BACKEND_CACHE.read().await;
-        if let Some(ref cached) = *cache {
-            if cached.settings == settings {
-                return Ok(cached.backend.clone());
-            }
-        }
-    }
-
-    // Need to create a new backend - acquire write lock
-    let mut cache = VAULT_BACKEND_CACHE.write().await;
-
-    // Double-check (another task may have created it while we waited)
-    if let Some(ref cached) = *cache {
-        if cached.settings == settings {
-            return Ok(cached.backend.clone());
-        }
-    }
-
-    // Create new backend
-    let backend: Arc<dyn SecretBackend> = {
-        #[cfg(feature = "openidconnect")]
-        if settings.token.is_none() {
-            Arc::new(VaultBackend::new_with_db(settings.clone(), _db.clone()))
-        } else {
-            Arc::new(VaultBackend::new(settings.clone()))
-        }
-
-        #[cfg(not(feature = "openidconnect"))]
-        Arc::new(VaultBackend::new(settings.clone()))
-    };
-
-    // Cache it
-    *cache = Some(CachedVaultBackend { backend: backend.clone(), settings });
-
-    Ok(backend)
-}
-
-/// Get a cached Azure Key Vault backend or create a new one if settings changed
-#[cfg(all(feature = "private", feature = "enterprise"))]
-async fn get_or_create_azure_kv_backend(
-    _db: &DB,
-    settings: AzureKeyVaultSettings,
-) -> Result<Arc<dyn SecretBackend>> {
-    // Check if we have a cached backend with matching settings (read lock)
-    {
-        let cache = AZURE_KV_BACKEND_CACHE.read().await;
-        if let Some(ref cached) = *cache {
-            if cached.settings == settings {
-                return Ok(cached.backend.clone());
-            }
-        }
-    }
-
-    // Need to create a new backend - acquire write lock
-    let mut cache = AZURE_KV_BACKEND_CACHE.write().await;
-
-    // Double-check (another task may have created it while we waited)
-    if let Some(ref cached) = *cache {
-        if cached.settings == settings {
-            return Ok(cached.backend.clone());
-        }
-    }
-
-    // Create new backend
-    let backend: Arc<dyn SecretBackend> = Arc::new(AzureKeyVaultBackend::new(settings.clone()));
-
-    // Cache it
-    *cache = Some(CachedAzureKvBackend { backend: backend.clone(), settings });
-
-    Ok(backend)
-}
-
-/// Get a cached AWS SM backend or create a new one if settings changed
-#[cfg(all(feature = "private", feature = "enterprise"))]
-async fn get_or_create_aws_sm_backend(
-    _db: &DB,
-    settings: AwsSecretsManagerSettings,
-) -> Result<Arc<dyn SecretBackend>> {
-    {
-        let cache = AWS_SM_BACKEND_CACHE.read().await;
-        if let Some(ref cached) = *cache {
-            if cached.settings == settings {
-                return Ok(cached.backend.clone());
-            }
-        }
-    }
-
-    let mut cache = AWS_SM_BACKEND_CACHE.write().await;
-
-    if let Some(ref cached) = *cache {
-        if cached.settings == settings {
-            return Ok(cached.backend.clone());
-        }
-    }
-
-    let backend: Arc<dyn SecretBackend> =
-        Arc::new(AwsSecretsManagerBackend::new_with_client(settings.clone()).await?);
-
-    *cache = Some(CachedAwsSmBackend { backend: backend.clone(), settings });
-
-    Ok(backend)
-}
-
-/// Check if an external secret backend is currently configured (EE only)
-#[cfg(all(feature = "private", feature = "enterprise"))]
-async fn is_vault_backend_configured(db: &DB) -> Result<bool> {
-    let config = match load_value_from_global_settings(db, SECRET_BACKEND_SETTING).await? {
-        Some(value) => serde_json::from_value::<SecretBackendConfig>(value).unwrap_or_default(),
-        None => SecretBackendConfig::default(),
-    };
-
-    Ok(matches!(
-        config,
-        SecretBackendConfig::HashiCorpVault(_)
-            | SecretBackendConfig::AzureKeyVault(_)
-            | SecretBackendConfig::AwsSecretsManager(_)
-    ))
-}
-
-/// Check if a value is stored in Vault (indicated by the $vault: prefix)
-#[cfg(all(feature = "private", feature = "enterprise"))]
-fn is_vault_stored_value(value: &str) -> bool {
-    value.starts_with("$vault:")
-}
-
-/// Check if a value is stored in Azure Key Vault (indicated by the $azure_kv: prefix)
-#[cfg(all(feature = "private", feature = "enterprise"))]
-fn is_azure_kv_stored_value(value: &str) -> bool {
-    value.starts_with("$azure_kv:")
-}
-
-/// Check if a value is stored in AWS Secrets Manager
-#[cfg(all(feature = "private", feature = "enterprise"))]
-fn is_aws_sm_stored_value(value: &str) -> bool {
-    value.starts_with("$aws_sm:")
-}
-
-/// Check if a value is stored in any external secret backend
-#[cfg(all(feature = "private", feature = "enterprise"))]
-fn is_external_stored_value(value: &str) -> bool {
-    is_vault_stored_value(value) || is_azure_kv_stored_value(value) || is_aws_sm_stored_value(value)
-}
-
 /// Bulk rename secrets in Vault when a path prefix changes (e.g., user rename)
 /// EE only feature.
 ///
@@ -11,7 +11,7 @@ pub use windmill_api_users::users::*;

 use std::sync::Arc;

-use crate::db::ApiAuthed;
+use crate::db::{ApiAuthed, OptJobAuthed};
 use crate::secret_backend_ext::rename_vault_secrets_with_prefix;
 use argon2::Argon2;
 use axum::{
@@ -21,7 +21,7 @@ use axum::{
 };
 use hyper::StatusCode;
 use serde::Deserialize;
-use windmill_api_auth::require_super_admin;
+use windmill_api_auth::{forbid_superadmin_job_token, require_super_admin};
 use windmill_audit::audit_oss::audit_log;
 use windmill_audit::ActionKind;
 use windmill_common::audit::AuditAuthor;
@@ -71,11 +71,13 @@ pub fn make_unauthed_service() -> Router {

 async fn create_user(
    authed: ApiAuthed,
+    OptJobAuthed { job_id, .. }: OptJobAuthed,
    Extension(db): Extension<DB>,
    Extension(webhook): Extension<windmill_common::webhook::WebhookShared>,
    Extension(argon2): Extension<Arc<Argon2<'_>>>,
    Json(nu): Json<NewUser>,
 ) -> Result<(StatusCode, String)> {
+    forbid_superadmin_job_token(&db, &authed.email, job_id).await?;
    crate::users_oss::create_user(authed, db, webhook, argon2, nu).await
 }

@@ -141,8 +143,10 @@ async fn set_password(
    Extension(db): Extension<DB>,
    Extension(argon2): Extension<Arc<Argon2<'_>>>,
    authed: ApiAuthed,
+    OptJobAuthed { job_id, .. }: OptJobAuthed,
    Json(ep): Json<EditPassword>,
 ) -> Result<String> {
+    forbid_superadmin_job_token(&db, &authed.email, job_id).await?;
    let email = authed.email.clone();
    crate::users_oss::set_password(db, argon2, authed, &email, ep).await
 }
@@ -152,9 +156,11 @@ async fn set_password_of_user(
    Extension(argon2): Extension<Arc<Argon2<'_>>>,
    Path(email): Path<String>,
    authed: ApiAuthed,
+    OptJobAuthed { job_id, .. }: OptJobAuthed,
    Json(ep): Json<EditPassword>,
 ) -> Result<String> {
    require_super_admin(&db, &authed.email).await?;
+    forbid_superadmin_job_token(&db, &authed.email, job_id).await?;
    crate::users_oss::set_password(db, argon2, authed, &email, ep).await
 }

@@ -165,11 +171,13 @@ struct RenameUser {

 async fn rename_user(
    authed: ApiAuthed,
+    OptJobAuthed { job_id, .. }: OptJobAuthed,
    Path(user_email): Path<String>,
    Extension(db): Extension<DB>,
    Json(ru): Json<RenameUser>,
 ) -> Result<String> {
    require_super_admin(&db, &authed.email).await?;
+    forbid_superadmin_job_token(&db, &authed.email, job_id).await?;

    let mut tx = db.begin().await?;

@@ -4,7 +4,10 @@ use sqlx::{PgExecutor, Postgres, Transaction};

 use crate::{error, scripts::ScriptHash};

-pub use windmill_parser::asset_parser::{parse_pipeline_annotations, TriggerSpec, PARTITION_TOKEN};
+pub use windmill_parser::asset_parser::{
+    parse_pipeline_annotations, DataTest, PartitionKind, PipelineAnnotations, RetrySpec,
+    TriggerSpec, PARTITION_TOKEN,
+};
 pub use windmill_types::assets::*;

 #[derive(sqlx::Type, Debug, Clone, Copy, PartialEq)]
@@ -529,6 +529,31 @@ mod classify_python_logging_line_tests {
    }
 }

+#[cfg(test)]
+mod validate_dbname_tests {
+    use super::validate_dbname;
+
+    #[test]
+    fn accepts_letters_digits_underscores_and_hyphens() {
+        assert!(validate_dbname("mydb").is_ok());
+        assert!(validate_dbname("my_db").is_ok());
+        assert!(validate_dbname("my-database").is_ok());
+        assert!(validate_dbname("My-Db_1").is_ok());
+    }
+
+    #[test]
+    fn rejects_invalid_names() {
+        // Must start with a letter (hyphen/digit/underscore leads are rejected).
+        assert!(validate_dbname("-db").is_err());
+        assert!(validate_dbname("1db").is_err());
+        assert!(validate_dbname("_db").is_err());
+        // No other special characters or whitespace.
+        assert!(validate_dbname("my db").is_err());
+        assert!(validate_dbname("my;db").is_err());
+        assert!(validate_dbname("").is_err());
+    }
+}
+
 #[derive(Serialize, Debug)]
 pub struct PrepareQueryColumnInfo {
    pub name: String,
@@ -812,7 +837,7 @@ impl PgDatabase {
 }

 /// Validate a database name to prevent SQL injection.
-/// Must start with a letter, contain only alphanumeric characters or underscores, and be <= 63 chars.
+/// Must start with a letter, contain only alphanumeric characters, underscores, or hyphens, and be <= 63 chars.
 pub fn validate_dbname(dbname: &str) -> error::Result<()> {
    let dbname = dbname.trim();
    if dbname.is_empty() {
@@ -836,10 +861,11 @@ pub fn validate_dbname(dbname: &str) -> error::Result<()> {
    }
    if !dbname
        .chars()
-        .all(|c| c.is_ascii_alphanumeric() || c == '_')
+        .all(|c| c.is_ascii_alphanumeric() || c == '_' || c == '-')
    {
        return Err(error::Error::BadRequest(
-            "Database name must contain only alphanumeric characters or underscores".to_string(),
+            "Database name must contain only alphanumeric characters, underscores, or hyphens"
+                .to_string(),
        ));
    }
    Ok(())
@@ -172,6 +172,19 @@ pub struct SimpleColumn {
 pub struct SelectOptions {
    pub limit: Option<i64>,
    pub offset: Option<i64>,
+    /// DuckLake time-travel: when set (DuckDB only), the read is pinned to this
+    /// catalog snapshot via `AT (VERSION => n)`. Ignored for other db types.
+    pub version: Option<i64>,
+}
+
+/// DuckLake time-travel suffix appended after a table name in a FROM clause.
+/// `n` is a server-controlled `i64` (a snapshot id), so inlining it is
+/// injection-safe. Empty string when unpinned (reads the latest snapshot).
+fn duckdb_version_suffix(version: Option<i64>) -> String {
+    match version {
+        Some(v) => format!(" AT (VERSION => {})", v),
+        None => String::new(),
+    }
 }

 // ---------------------------------------------------------------------------
@@ -190,6 +203,8 @@ struct SelectPayload {
    #[serde(rename = "fixPgIntTypes")]
    fix_pg_int_types: Option<bool>,
    ducklake: Option<String>,
+    /// DuckLake snapshot to time-travel the read to (DuckDB only).
+    version: Option<i64>,
 }

 #[derive(Deserialize)]
@@ -200,6 +215,21 @@ struct CountPayload {
    #[serde(rename = "whereClause")]
    where_clause: Option<String>,
    ducklake: Option<String>,
+    /// DuckLake snapshot to time-travel the count to (DuckDB only).
+    version: Option<i64>,
+}
+
+/// `WM_INTERNAL_DB_DUCKLAKE_SNAPSHOTS` payload — lists the time-travel history
+/// of a ducklake table. DuckLake snapshots are catalog-wide commits, so without
+/// a `table` this lists every commit; with one it is scoped to snapshots where
+/// that table exists (see `expand_ducklake_snapshots`).
+#[derive(Deserialize)]
+struct DucklakeSnapshotsPayload {
+    ducklake: String,
+    /// Schema-qualified table name (e.g. `main.events_daily`) to scope the
+    /// history to. Snapshots predating the table's creation are excluded — a
+    /// time-travel read can't target a version where the table didn't exist.
+    table: Option<String>,
 }

 #[derive(Deserialize)]
@@ -304,6 +334,10 @@ pub fn try_expand_internal_db_query(
            expand_primary_key_constraint(json_str, db_type).map(ExpandedQuery::sql)
        }
        "SNOWFLAKE_PRIMARY_KEYS" => expand_snowflake_primary_keys(json_str).map(ExpandedQuery::sql),
+        // DuckLake time-travel: list a ducklake's snapshot history
+        "DUCKLAKE_SNAPSHOTS" => {
+            expand_ducklake_snapshots(json_str, db_type).map(ExpandedQuery::sql)
+        }
        _ => Err(format!("Unknown WM_INTERNAL_DB operation: {}", op)),
    };

@@ -324,7 +358,8 @@ fn expand_select(json_str: &str, db_type: DbType) -> Result<String, String> {
    let payload: SelectPayload =
        serde_json::from_str(json_str).map_err(|e| format!("Invalid SELECT payload: {}", e))?;

-    let options = SelectOptions { limit: payload.limit, offset: payload.offset };
+    let options =
+        SelectOptions { limit: payload.limit, offset: payload.offset, version: payload.version };
    let breaking = payload
        .fix_pg_int_types
        .map(|v| BreakingFeatures { fix_pg_int_types: v });
@@ -350,11 +385,47 @@ fn expand_count(json_str: &str, db_type: DbType) -> Result<String, String> {
        &payload.table,
        payload.where_clause.as_deref(),
        &payload.column_defs,
+        payload.version,
    )?;

    Ok(maybe_wrap_ducklake(query, payload.ducklake.as_deref()))
 }

+/// Expand `DUCKLAKE_SNAPSHOTS` into the catalog's time-travel history. DuckLake
+/// snapshots are catalog-wide commits, so `ducklake_snapshots('dl')` (the alias
+/// `maybe_wrap_ducklake` attaches) lists every version any `AT (VERSION => n)`
+/// read can target, newest first.
+fn expand_ducklake_snapshots(json_str: &str, db_type: DbType) -> Result<String, String> {
+    if db_type != DbType::Duckdb {
+        return Err("DUCKLAKE_SNAPSHOTS is only supported for DuckDB".to_string());
+    }
+    let payload: DucklakeSnapshotsPayload = serde_json::from_str(json_str)
+        .map_err(|e| format!("Invalid DUCKLAKE_SNAPSHOTS payload: {}", e))?;
+    // `dl` is the alias `wrap_ducklake_query` attaches and `USE`s below.
+    let query = match &payload.table {
+        // Scope to snapshots from the table's first creation onward. A DuckLake
+        // table created at snapshot N can't be read before N (the catalog-wide
+        // list would otherwise offer impossible versions). The creation snapshot
+        // is the earliest whose `changes.tables_created` names the table;
+        // COALESCE to 0 (show all) if it is never found.
+        Some(table) => {
+            let table = escape_sql_literal(table);
+            format!(
+                "SELECT snapshot_id, snapshot_time FROM ducklake_snapshots('dl') \
+                 WHERE snapshot_id >= COALESCE((\
+                   SELECT min(snapshot_id) FROM ducklake_snapshots('dl') \
+                   WHERE list_contains(changes.tables_created, '{table}')), 0) \
+                 ORDER BY snapshot_id DESC"
+            )
+        }
+        None => {
+            "SELECT snapshot_id, snapshot_time FROM ducklake_snapshots('dl') ORDER BY snapshot_id DESC"
+                .to_string()
+        }
+    };
+    Ok(maybe_wrap_ducklake(query, Some(&payload.ducklake)))
+}
+
 /// Filter columns to primary keys only; fall back to all columns if none are marked.
 fn pk_columns_or_all(columns: &[ColumnDef]) -> Vec<ColumnDef> {
    let pks: Vec<ColumnDef> = columns.iter().filter(|c| c.isprimarykey).cloned().collect();
@@ -950,9 +1021,10 @@ pub fn make_select_query(
            );

            query.push_str(&format!(
-                "SELECT {} FROM {}\n",
+                "SELECT {} FROM {}{}\n",
                filtered_columns.join(", "),
-                quote_table_name(table, db_type)
+                quote_table_name(table, db_type),
+                duckdb_version_suffix(options.and_then(|o| o.version))
            ));
            query.push_str(&format!(
                " WHERE {} {}\n",
@@ -977,6 +1049,8 @@ pub fn make_count_query(
    table: &str,
    where_clause: Option<&str>,
    column_defs: &[ColumnDef],
+    // DuckLake time-travel snapshot (DuckDB only); `None` counts the latest.
+    version: Option<i64>,
 ) -> Result<String, String> {
    let where_prefix = " WHERE ";
    let and_condition = " AND ";
@@ -1118,8 +1192,9 @@ pub fn make_count_query(
                quicksearch_condition.push_str(" ($quicksearch = '' OR 1 = 1)");
            }
            query.push_str(&format!(
-                "SELECT COUNT(*) as count FROM {}",
-                quote_table_name(table, db_type)
+                "SELECT COUNT(*) as count FROM {}{}",
+                quote_table_name(table, db_type),
+                duckdb_version_suffix(version)
            ));
        }
    }
@@ -2998,7 +3073,7 @@ mod tests {
    #[test]
    fn test_select_snowflake_custom_limit() {
        let cols = vec![col("id", "int")];
-        let opts = SelectOptions { limit: Some(50), offset: Some(10) };
+        let opts = SelectOptions { limit: Some(50), offset: Some(10), version: None };
        let result = make_select_query(
            "my_table",
            &cols,
@@ -3072,7 +3147,7 @@ mod tests {
    #[test]
    fn test_count_postgresql_basic() {
        let cols = vec![col("id", "int4"), col("name", "text")];
-        let result = make_count_query(DbType::Postgresql, "my_table", None, &cols).unwrap();
+        let result = make_count_query(DbType::Postgresql, "my_table", None, &cols, None).unwrap();

        assert!(result.contains("-- $1 quicksearch (text)"));
        assert!(result.contains("SELECT COUNT(*) as count FROM \"my_table\""));
@@ -3090,6 +3165,7 @@ mod tests {
            "my_table",
            Some("status = 'active'"),
            &cols,
+            None,
        )
        .unwrap();

@@ -3105,7 +3181,7 @@ mod tests {
            c.ignored = Some(true);
            c
        }];
-        let result = make_count_query(DbType::Postgresql, "my_table", None, &cols).unwrap();
+        let result = make_count_query(DbType::Postgresql, "my_table", None, &cols, None).unwrap();
        assert!(result.contains("($1 = '' OR 1 = 1)"));
    }

@@ -3116,7 +3192,7 @@ mod tests {
    #[test]
    fn test_count_mysql_basic() {
        let cols = vec![col("id", "int"), col("name", "varchar")];
-        let result = make_count_query(DbType::Mysql, "my_table", None, &cols).unwrap();
+        let result = make_count_query(DbType::Mysql, "my_table", None, &cols, None).unwrap();

        assert!(result.contains("-- :quicksearch (text)"));
        assert!(result.contains("SELECT COUNT(*) as count FROM `my_table`"));
@@ -3130,7 +3206,7 @@ mod tests {
    #[test]
    fn test_count_mssql_basic() {
        let cols = vec![col("id", "int"), col("name", "nvarchar")];
-        let result = make_count_query(DbType::MsSqlServer, "my_table", None, &cols).unwrap();
+        let result = make_count_query(DbType::MsSqlServer, "my_table", None, &cols, None).unwrap();

        assert!(result.contains("SELECT COUNT(*) as count FROM [my_table]"));
        assert!(result.contains("(@p1 = '' OR CONCAT([id], [name]) LIKE '%' + @p1 + '%')"));
@@ -3143,7 +3219,7 @@ mod tests {
    #[test]
    fn test_count_snowflake_basic() {
        let cols = vec![col("id", "int"), col("name", "text")];
-        let result = make_count_query(DbType::Snowflake, "my_table", None, &cols).unwrap();
+        let result = make_count_query(DbType::Snowflake, "my_table", None, &cols, None).unwrap();

        // Two quicksearch params for snowflake with visible columns
        assert!(result.contains("-- ? quicksearch (text)\n-- ? quicksearch (text)"));
@@ -3158,7 +3234,7 @@ mod tests {
            c.ignored = Some(true);
            c
        }];
-        let result = make_count_query(DbType::Snowflake, "my_table", None, &cols).unwrap();
+        let result = make_count_query(DbType::Snowflake, "my_table", None, &cols, None).unwrap();
        // One quicksearch param
        let param_lines: Vec<&str> = result.lines().filter(|l| l.starts_with("-- ?")).collect();
        assert_eq!(param_lines.len(), 1);
@@ -3172,7 +3248,7 @@ mod tests {
    #[test]
    fn test_count_bigquery_basic() {
        let cols = vec![col("id", "INTEGER"), col("name", "STRING")];
-        let result = make_count_query(DbType::Bigquery, "my_table", None, &cols).unwrap();
+        let result = make_count_query(DbType::Bigquery, "my_table", None, &cols, None).unwrap();

        assert!(result.contains("-- @quicksearch (string)"));
        assert!(result.contains("SELECT COUNT(*) as count FROM `my_table`"));
@@ -3182,7 +3258,7 @@ mod tests {
    #[test]
    fn test_count_bigquery_json_type() {
        let cols = vec![col("id", "INTEGER"), col("data", "JSON")];
-        let result = make_count_query(DbType::Bigquery, "my_table", None, &cols).unwrap();
+        let result = make_count_query(DbType::Bigquery, "my_table", None, &cols, None).unwrap();
        assert!(result.contains("TO_JSON_STRING(`data`)"));
    }

@@ -3193,7 +3269,7 @@ mod tests {
    #[test]
    fn test_count_duckdb_basic() {
        let cols = vec![col("id", "int"), col("name", "text")];
-        let result = make_count_query(DbType::Duckdb, "my_table", None, &cols).unwrap();
+        let result = make_count_query(DbType::Duckdb, "my_table", None, &cols, None).unwrap();

        assert!(result.contains("-- $quicksearch (text)"));
        assert!(result.contains("SELECT COUNT(*) as count FROM \"my_table\""));
@@ -3202,6 +3278,74 @@ mod tests {
        );
    }

+    // -----------------------------------------------------------------------
+    // DuckLake time-travel (AT VERSION) + snapshot history
+    // -----------------------------------------------------------------------
+
+    #[test]
+    fn test_select_duckdb_time_travel() {
+        let cols = vec![col("id", "int"), col("name", "text")];
+        let opts = SelectOptions { limit: None, offset: None, version: Some(42) };
+        let result =
+            make_select_query("orders", &cols, None, DbType::Duckdb, Some(&opts), None).unwrap();
+        // Read is pinned to the catalog snapshot via AT (VERSION => n).
+        assert!(result.contains("FROM \"orders\" AT (VERSION => 42)\n"));
+    }
+
+    #[test]
+    fn test_select_duckdb_no_version_unpinned() {
+        let cols = vec![col("id", "int")];
+        let result = make_select_query("orders", &cols, None, DbType::Duckdb, None, None).unwrap();
+        // Without a version the read targets the latest snapshot — no AT clause.
+        assert!(result.contains("FROM \"orders\"\n"));
+        assert!(!result.contains("AT (VERSION"));
+    }
+
+    #[test]
+    fn test_count_duckdb_time_travel() {
+        let cols = vec![col("id", "int")];
+        let result = make_count_query(DbType::Duckdb, "orders", None, &cols, Some(7)).unwrap();
+        assert!(result.contains("FROM \"orders\" AT (VERSION => 7)"));
+    }
+
+    #[test]
+    fn test_version_ignored_for_non_duckdb() {
+        // AT (VERSION) is DuckLake-only; other dialects must never emit it even
+        // if a version is somehow passed through.
+        let cols = vec![col("id", "int4")];
+        let opts = SelectOptions { limit: None, offset: None, version: Some(5) };
+        let result =
+            make_select_query("orders", &cols, None, DbType::Postgresql, Some(&opts), None)
+                .unwrap();
+        assert!(!result.contains("AT (VERSION"));
+    }
+
+    #[test]
+    fn test_expand_ducklake_snapshots() {
+        let json = r#"{"ducklake": "analytics"}"#;
+        let result = expand_ducklake_snapshots(json, DbType::Duckdb).unwrap();
+        assert!(result.contains("ATTACH 'ducklake://analytics' AS dl;USE dl;"));
+        assert!(result.contains("ducklake_snapshots('dl')"));
+        assert!(result.contains("ORDER BY snapshot_id DESC"));
+        // Unscoped: no per-table existence filter.
+        assert!(!result.contains("tables_created"));
+    }
+
+    #[test]
+    fn test_expand_ducklake_snapshots_scoped_to_table() {
+        let json = r#"{"ducklake": "analytics", "table": "main.events_daily"}"#;
+        let result = expand_ducklake_snapshots(json, DbType::Duckdb).unwrap();
+        // Scoped to snapshots from the table's first creation onward.
+        assert!(result.contains("list_contains(changes.tables_created, 'main.events_daily')"));
+        assert!(result.contains("snapshot_id >= COALESCE"));
+    }
+
+    #[test]
+    fn test_expand_ducklake_snapshots_non_duckdb_errors() {
+        let json = r#"{"ducklake": "analytics"}"#;
+        assert!(expand_ducklake_snapshots(json, DbType::Postgresql).is_err());
+    }
+
    // -----------------------------------------------------------------------
    // DELETE - all DB types
    // -----------------------------------------------------------------------
@@ -3500,7 +3644,7 @@ mod tests {
    #[test]
    fn test_count_mssql_no_where() {
        let cols = vec![col("id", "int")];
-        let result = make_count_query(DbType::MsSqlServer, "my_table", None, &cols).unwrap();
+        let result = make_count_query(DbType::MsSqlServer, "my_table", None, &cols, None).unwrap();
        // MSSQL uses WHERE directly (no AND replacement)
        assert!(result.contains("SELECT COUNT(*) as count FROM [my_table] WHERE "));
    }
@@ -3508,7 +3652,7 @@ mod tests {
    #[test]
    fn test_count_mysql_no_where_uses_where_keyword() {
        let cols = vec![col("id", "int")];
-        let result = make_count_query(DbType::Mysql, "my_table", None, &cols).unwrap();
+        let result = make_count_query(DbType::Mysql, "my_table", None, &cols, None).unwrap();
        // The AND should be replaced with WHERE
        assert!(result.contains("FROM `my_table` WHERE "));
        assert!(!result.contains("FROM `my_table` AND "));
@@ -13,6 +13,9 @@
 //! vaults like HashiCorp Vault (Enterprise Edition).

 pub mod database;
+pub mod resolver;
+
+pub use resolver::*;

 #[cfg(feature = "private")]
 pub mod vault_ee;
@@ -0,0 +1,324 @@
+/*
+ * Author: Ruben Fiszel
+ * Copyright: Windmill Labs, Inc 2024
+ * This file and its contents are licensed under the AGPLv3 License.
+ * Please see the included NOTICE for copyright information and
+ * LICENSE-AGPL for a copy of the license.
+ */
+
+//! Resolution of the configured secret backend.
+//!
+//! Lives in `windmill-common` (rather than the API/store crates) so that
+//! lower-level helpers such as [`crate::variables::get_variable_or_self`] can
+//! route secret reads through the configured backend. With an external backend
+//! (Vault / Azure Key Vault / AWS Secrets Manager), the `variable.value` column
+//! holds a `$vault:`/`$azure_kv:`/`$aws_sm:` marker rather than base64
+//! ciphertext, so decrypting it directly fails — reads must go through the
+//! backend instead.
+//!
+//! Note: external backends require Enterprise Edition. The OSS version only
+//! supports the database backend.
+
+use std::sync::Arc;
+
+use crate::{
+    db::DB,
+    error::{Error, Result},
+    secret_backend::{database::DatabaseBackend, SecretBackend},
+    variables::{build_crypt, decrypt},
+};
+
+#[cfg(all(feature = "private", feature = "enterprise"))]
+use crate::{
+    global_settings::{load_value_from_global_settings, SECRET_BACKEND_SETTING},
+    secret_backend::{
+        AwsSecretsManagerBackend, AwsSecretsManagerSettings, AzureKeyVaultBackend,
+        AzureKeyVaultSettings, SecretBackendConfig, VaultBackend, VaultSettings,
+    },
+};
+
+#[cfg(all(feature = "private", feature = "enterprise"))]
+use tokio::sync::RwLock;
+
+// Cached Vault backend to avoid recreating it for every request
+// This enables connection pooling and avoids repeated setup overhead
+#[cfg(all(feature = "private", feature = "enterprise"))]
+struct CachedVaultBackend {
+    backend: Arc<dyn SecretBackend>,
+    settings: VaultSettings,
+}
+
+#[cfg(all(feature = "private", feature = "enterprise"))]
+lazy_static::lazy_static! {
+    static ref VAULT_BACKEND_CACHE: RwLock<Option<CachedVaultBackend>> = RwLock::new(None);
+}
+
+#[cfg(all(feature = "private", feature = "enterprise"))]
+struct CachedAzureKvBackend {
+    backend: Arc<dyn SecretBackend>,
+    settings: AzureKeyVaultSettings,
+}
+
+#[cfg(all(feature = "private", feature = "enterprise"))]
+lazy_static::lazy_static! {
+    static ref AZURE_KV_BACKEND_CACHE: RwLock<Option<CachedAzureKvBackend>> = RwLock::new(None);
+}
+
+// Cached AWS Secrets Manager backend
+#[cfg(all(feature = "private", feature = "enterprise"))]
+struct CachedAwsSmBackend {
+    backend: Arc<dyn SecretBackend>,
+    settings: AwsSecretsManagerSettings,
+}
+
+#[cfg(all(feature = "private", feature = "enterprise"))]
+lazy_static::lazy_static! {
+    static ref AWS_SM_BACKEND_CACHE: RwLock<Option<CachedAwsSmBackend>> = RwLock::new(None);
+}
+
+/// Get the current secret backend based on global settings
+///
+/// OSS: Always returns DatabaseBackend
+/// EE: Returns configured backend (Database or Vault)
+#[cfg(not(all(feature = "private", feature = "enterprise")))]
+pub async fn get_secret_backend(db: &DB) -> Result<Arc<dyn SecretBackend>> {
+    Ok(Arc::new(DatabaseBackend::new(db.clone())))
+}
+
+#[cfg(all(feature = "private", feature = "enterprise"))]
+pub async fn get_secret_backend(db: &DB) -> Result<Arc<dyn SecretBackend>> {
+    let config = match load_value_from_global_settings(db, SECRET_BACKEND_SETTING).await? {
+        Some(value) => serde_json::from_value::<SecretBackendConfig>(value).unwrap_or_default(),
+        None => SecretBackendConfig::default(),
+    };
+
+    match config {
+        SecretBackendConfig::Database => Ok(Arc::new(DatabaseBackend::new(db.clone()))),
+        SecretBackendConfig::HashiCorpVault(settings) => {
+            get_or_create_vault_backend(db, settings).await
+        }
+        SecretBackendConfig::AzureKeyVault(settings) => {
+            get_or_create_azure_kv_backend(db, settings).await
+        }
+        SecretBackendConfig::AwsSecretsManager(settings) => {
+            get_or_create_aws_sm_backend(db, settings).await
+        }
+    }
+}
+
+/// Get a cached Vault backend or create a new one if settings changed
+#[cfg(all(feature = "private", feature = "enterprise"))]
+async fn get_or_create_vault_backend(
+    _db: &DB,
+    settings: VaultSettings,
+) -> Result<Arc<dyn SecretBackend>> {
+    // Check if we have a cached backend with matching settings (read lock)
+    {
+        let cache = VAULT_BACKEND_CACHE.read().await;
+        if let Some(ref cached) = *cache {
+            if cached.settings == settings {
+                return Ok(cached.backend.clone());
+            }
+        }
+    }
+
+    // Need to create a new backend - acquire write lock
+    let mut cache = VAULT_BACKEND_CACHE.write().await;
+
+    // Double-check (another task may have created it while we waited)
+    if let Some(ref cached) = *cache {
+        if cached.settings == settings {
+            return Ok(cached.backend.clone());
+        }
+    }
+
+    // Create new backend
+    let backend: Arc<dyn SecretBackend> = {
+        #[cfg(feature = "openidconnect")]
+        if settings.token.is_none() {
+            Arc::new(VaultBackend::new_with_db(settings.clone(), _db.clone()))
+        } else {
+            Arc::new(VaultBackend::new(settings.clone()))
+        }
+
+        #[cfg(not(feature = "openidconnect"))]
+        Arc::new(VaultBackend::new(settings.clone()))
+    };
+
+    // Cache it
+    *cache = Some(CachedVaultBackend { backend: backend.clone(), settings });
+
+    Ok(backend)
+}
+
+/// Get a cached Azure Key Vault backend or create a new one if settings changed
+#[cfg(all(feature = "private", feature = "enterprise"))]
+async fn get_or_create_azure_kv_backend(
+    _db: &DB,
+    settings: AzureKeyVaultSettings,
+) -> Result<Arc<dyn SecretBackend>> {
+    // Check if we have a cached backend with matching settings (read lock)
+    {
+        let cache = AZURE_KV_BACKEND_CACHE.read().await;
+        if let Some(ref cached) = *cache {
+            if cached.settings == settings {
+                return Ok(cached.backend.clone());
+            }
+        }
+    }
+
+    // Need to create a new backend - acquire write lock
+    let mut cache = AZURE_KV_BACKEND_CACHE.write().await;
+
+    // Double-check (another task may have created it while we waited)
+    if let Some(ref cached) = *cache {
+        if cached.settings == settings {
+            return Ok(cached.backend.clone());
+        }
+    }
+
+    // Create new backend
+    let backend: Arc<dyn SecretBackend> = Arc::new(AzureKeyVaultBackend::new(settings.clone()));
+
+    // Cache it
+    *cache = Some(CachedAzureKvBackend { backend: backend.clone(), settings });
+
+    Ok(backend)
+}
+
+/// Get a cached AWS SM backend or create a new one if settings changed
+#[cfg(all(feature = "private", feature = "enterprise"))]
+async fn get_or_create_aws_sm_backend(
+    _db: &DB,
+    settings: AwsSecretsManagerSettings,
+) -> Result<Arc<dyn SecretBackend>> {
+    {
+        let cache = AWS_SM_BACKEND_CACHE.read().await;
+        if let Some(ref cached) = *cache {
+            if cached.settings == settings {
+                return Ok(cached.backend.clone());
+            }
+        }
+    }
+
+    let mut cache = AWS_SM_BACKEND_CACHE.write().await;
+
+    if let Some(ref cached) = *cache {
+        if cached.settings == settings {
+            return Ok(cached.backend.clone());
+        }
+    }
+
+    let backend: Arc<dyn SecretBackend> =
+        Arc::new(AwsSecretsManagerBackend::new_with_client(settings.clone()).await?);
+
+    *cache = Some(CachedAwsSmBackend { backend: backend.clone(), settings });
+
+    Ok(backend)
+}
+
+/// Check if a Vault backend is currently configured
+///
+/// OSS: Always returns false
+/// EE: Checks global settings
+#[cfg(not(all(feature = "private", feature = "enterprise")))]
+pub async fn is_vault_backend_configured(_db: &DB) -> Result<bool> {
+    Ok(false)
+}
+
+#[cfg(all(feature = "private", feature = "enterprise"))]
+pub async fn is_vault_backend_configured(db: &DB) -> Result<bool> {
+    let config = match load_value_from_global_settings(db, SECRET_BACKEND_SETTING).await? {
+        Some(value) => serde_json::from_value::<SecretBackendConfig>(value).unwrap_or_default(),
+        None => SecretBackendConfig::default(),
+    };
+
+    Ok(matches!(
+        config,
+        SecretBackendConfig::HashiCorpVault(_)
+            | SecretBackendConfig::AzureKeyVault(_)
+            | SecretBackendConfig::AwsSecretsManager(_)
+    ))
+}
+
+/// Get a secret value using the configured backend
+///
+/// For database backend: decrypts `encrypted_value` using the workspace key
+/// For external backends (EE only): fetches from the backend at `path`,
+/// ignoring `encrypted_value` (which holds only a `$...:` marker)
+pub async fn get_secret_value(
+    db: &DB,
+    workspace_id: &str,
+    path: &str,
+    encrypted_value: &str,
+) -> Result<String> {
+    let backend = get_secret_backend(db).await?;
+
+    match backend.backend_name() {
+        "database" => {
+            // Use existing database decryption
+            let mc = build_crypt(db, workspace_id).await?;
+            decrypt(&mc, encrypted_value.to_string()).map_err(|e| {
+                Error::internal_err(format!("Error decrypting variable {}: {}", path, e))
+            })
+        }
+        "hashicorp_vault" => {
+            // Fetch from Vault directly
+            backend.get_secret(workspace_id, path).await
+        }
+        "azure_key_vault" => backend.get_secret(workspace_id, path).await,
+        "aws_secrets_manager" => backend.get_secret(workspace_id, path).await,
+        _ => Err(Error::internal_err(format!(
+            "Unknown backend: {}",
+            backend.backend_name()
+        ))),
+    }
+}
+
+/// Check if a value is stored in Vault (indicated by the $vault: prefix)
+pub fn is_vault_stored_value(value: &str) -> bool {
+    value.starts_with("$vault:")
+}
+
+/// Check if a value is stored in Azure Key Vault (indicated by the $azure_kv: prefix)
+pub fn is_azure_kv_stored_value(value: &str) -> bool {
+    value.starts_with("$azure_kv:")
+}
+
+/// Check if a value is stored in AWS Secrets Manager (indicated by the $aws_sm: prefix)
+pub fn is_aws_sm_stored_value(value: &str) -> bool {
+    value.starts_with("$aws_sm:")
+}
+
+/// Check if a value is stored in any external secret backend
+pub fn is_external_stored_value(value: &str) -> bool {
+    is_vault_stored_value(value) || is_azure_kv_stored_value(value) || is_aws_sm_stored_value(value)
+}
+
+#[cfg(test)]
+mod tests {
+    use super::*;
+
+    #[test]
+    fn external_markers_are_detected() {
+        assert!(is_external_stored_value("$vault:u/admin/secret"));
+        assert!(is_external_stored_value("$azure_kv:u/admin/secret"));
+        assert!(is_external_stored_value("$aws_sm:u/admin/secret"));
+    }
+
+    #[test]
+    fn base64_ciphertext_is_not_treated_as_external() {
+        // A base64 magic_crypt blob must route through `decrypt`, never the
+        // external backend. The leading `$` is what distinguishes a marker from
+        // ciphertext; decrypting a marker fails with "Invalid byte 36" (`$`),
+        // which is the bug this gate prevents.
+        for v in [
+            "bm90LWEtbWFya2Vy",
+            "AAAA1234+/abcd==",
+            "",
+            "$something_else",
+        ] {
+            assert!(!is_external_stored_value(v), "unexpected external: {v:?}");
+        }
+    }
+}
@@ -9,6 +9,7 @@
 use crate::db::{Authable, UserDB};
 use crate::error::{self, Error};
 use crate::scripts::ScriptHash;
+use crate::secret_backend::{get_secret_value, is_external_stored_value};
 use crate::utils::WarnAfterExt;
 use crate::worker::Connection;
 use crate::{worker::WORKER_GROUP, BASE_URL, DB};
@@ -234,13 +235,17 @@ pub async fn get_secret_value_as_admin(
    let r = if variable.is_secret {
        let value = variable.value;
        if !value.is_empty() {
-            let mc = build_crypt(db, w_id).await?;
-            decrypt(&mc, value).map_err(|e| {
-                crate::error::Error::internal_err(format!(
-                    "Error decrypting variable {}: {}",
-                    variable.path, e
-                ))
-            })?
+            if is_external_stored_value(&value) {
+                get_secret_value(db, w_id, &variable.path, &value).await?
+            } else {
+                let mc = build_crypt(db, w_id).await?;
+                decrypt(&mc, value).map_err(|e| {
+                    crate::error::Error::internal_err(format!(
+                        "Error decrypting variable {}: {}",
+                        variable.path, e
+                    ))
+                })?
+            }
        } else {
            "".to_string()
        }
@@ -546,10 +551,14 @@ pub async fn get_variable_or_self(
    if let Some(record) = record {
        let mut value = record.value;
        if record.is_secret {
-            let mc = build_crypt(db, w_id).await?;
-            value = decrypt(&mc, value).map_err(|e| {
-                Error::internal_err(format!("Error decrypting variable {}: {}", path, e))
-            })?;
+            if is_external_stored_value(&value) {
+                value = get_secret_value(db, w_id, &path, &value).await?;
+            } else {
+                let mc = build_crypt(db, w_id).await?;
+                value = decrypt(&mc, value).map_err(|e| {
+                    Error::internal_err(format!("Error decrypting variable {}: {}", path, e))
+                })?;
+            }
        }

        Ok(value)
@@ -591,10 +600,14 @@ pub async fn get_variable_or_self_as<T: Authable + Sync>(
    if let Some(record) = record {
        let mut value = record.value;
        if record.is_secret {
-            let mc = build_crypt(db, w_id).await?;
-            value = decrypt(&mc, value).map_err(|e| {
-                Error::internal_err(format!("Error decrypting variable {}: {}", var_path, e))
-            })?;
+            if is_external_stored_value(&value) {
+                value = get_secret_value(db, w_id, &var_path, &value).await?;
+            } else {
+                let mc = build_crypt(db, w_id).await?;
+                value = decrypt(&mc, value).map_err(|e| {
+                    Error::internal_err(format!("Error decrypting variable {}: {}", var_path, e))
+                })?;
+            }
        }

        Ok(value)
@@ -693,8 +693,6 @@ pub fn is_allowed_file_location(job_dir: &str, user_defined_path: &str) -> error

    let full_path = job_dir.join(&user_path);

-    // let normalized_job_dir = std::fs::canonicalize(job_dir)?;
-    // let normalized_full_path = std::fs::canonicalize(&full_path)?;
    let normalized_job_dir = normalize_path(job_dir);
    let normalized_full_path = normalize_path(&full_path);

@@ -706,6 +704,36 @@ pub fn is_allowed_file_location(job_dir: &str, user_defined_path: &str) -> error
        .into());
    }

+    // The lexical check above cannot see symlinks: a symlink planted inside the
+    // job dir - e.g. by an earlier Ansible `git_repos` clone whose tracked
+    // content includes one - would let a later `git clone` or file write follow
+    // it out of the job dir while still passing the textual `starts_with` check.
+    // Walk the *normalized* relative path (`..`/`.` already collapsed) so each
+    // step matches the real on-disk resolution, and reject any existing component
+    // that is a symlink. Walking the raw user path would drift on an in-bounds
+    // `..` (e.g. `foo/../link`, which normalizes back inside the job dir) and miss
+    // the real symlinked component. Not-yet-existing components are safe: a path
+    // that does not exist cannot itself be a symlink.
+    let relative = normalized_full_path
+        .strip_prefix(&normalized_job_dir)
+        .unwrap_or(&normalized_full_path);
+    let mut current = normalized_job_dir.clone();
+    for component in relative.components() {
+        if let Component::Normal(c) = component {
+            current.push(c);
+            if std::fs::symlink_metadata(&current)
+                .map(|m| m.file_type().is_symlink())
+                .unwrap_or(false)
+            {
+                return Err(std::io::Error::new(
+                    std::io::ErrorKind::PermissionDenied,
+                    "Path traverses a symlink, which is not allowed.",
+                )
+                .into());
+            }
+        }
+    }
+
    Ok(normalized_full_path)
 }

@@ -2828,4 +2856,71 @@ mod tests {

        let _ = std::fs::remove_dir_all(&base);
    }
+
+    #[test]
+    fn test_is_allowed_file_location_allows_plain_relative() {
+        let base = std::env::temp_dir().join(format!("wm_allowed_loc_ok_{}", uuid::Uuid::new_v4()));
+        let job_dir = base.join("job");
+        std::fs::create_dir_all(&job_dir).unwrap();
+        let job_dir_str = job_dir.to_str().unwrap();
+
+        let out = is_allowed_file_location(job_dir_str, "repo/sub/playbook.yml").unwrap();
+        assert_eq!(out, normalize_path(&job_dir.join("repo/sub/playbook.yml")));
+
+        let _ = std::fs::remove_dir_all(&base);
+    }
+
+    #[test]
+    fn test_is_allowed_file_location_rejects_parent_and_absolute() {
+        let base =
+            std::env::temp_dir().join(format!("wm_allowed_loc_esc_{}", uuid::Uuid::new_v4()));
+        let job_dir = base.join("job");
+        std::fs::create_dir_all(&job_dir).unwrap();
+        let job_dir_str = job_dir.to_str().unwrap();
+
+        assert!(is_allowed_file_location(job_dir_str, "../escape").is_err());
+        assert!(is_allowed_file_location(job_dir_str, "a/../../escape").is_err());
+        assert!(is_allowed_file_location(job_dir_str, "/etc/passwd").is_err());
+
+        let _ = std::fs::remove_dir_all(&base);
+    }
+
+    // Regression for GHSA-v934-cvpf-6fjw: a symlink planted inside the job dir
+    // (e.g. by an earlier `git_repos` clone) must not let a later target traverse
+    // it out of the job dir, even though the lexical path stays "inside".
+    #[cfg(unix)]
+    #[test]
+    fn test_is_allowed_file_location_rejects_symlink_traversal() {
+        let base =
+            std::env::temp_dir().join(format!("wm_allowed_loc_symlink_{}", uuid::Uuid::new_v4()));
+        let job_dir = base.join("job");
+        std::fs::create_dir_all(&job_dir).unwrap();
+        // Stand-in for the shared cache dir living outside the job dir.
+        let outside = base.join("outside");
+        std::fs::create_dir_all(&outside).unwrap();
+        let job_dir_str = job_dir.to_str().unwrap();
+
+        // Plant `job/repo` -> `../outside`, as a malicious first clone would.
+        let planted = job_dir.join("repo");
+        std::os::unix::fs::symlink(&outside, &planted).unwrap();
+
+        // Both the symlink itself and any path traversing it are rejected.
+        assert!(is_allowed_file_location(job_dir_str, "repo").is_err());
+        assert!(is_allowed_file_location(job_dir_str, "repo/payload").is_err());
+        assert!(is_allowed_file_location(job_dir_str, "repo/sub/payload").is_err());
+
+        // An in-bounds `..` must not bypass the check: `foo/../repo/payload`
+        // normalizes back to `repo/payload` and still traverses the symlink.
+        assert!(is_allowed_file_location(job_dir_str, "foo/../repo/payload").is_err());
+        std::fs::create_dir(job_dir.join("real")).unwrap();
+        assert!(is_allowed_file_location(job_dir_str, "real/../repo/payload").is_err());
+
+        // A dangling symlink (target does not exist yet) is still caught:
+        // `symlink_metadata` does not follow the link.
+        let dangling = job_dir.join("dangling");
+        std::os::unix::fs::symlink(base.join("nonexistent"), &dangling).unwrap();
+        assert!(is_allowed_file_location(job_dir_str, "dangling/payload").is_err());
+
+        let _ = std::fs::remove_dir_all(&base);
+    }
 }
@@ -8,6 +8,7 @@ use strum::AsRefStr;
 use crate::{
    error::{self, to_anyhow, Error, Result},
    get_database_url,
+    secret_backend::{get_secret_value, is_external_stored_value},
    utils::get_custom_pg_instance_password,
    variables::{build_crypt, decrypt},
    PgDatabase, DB,
@@ -725,10 +726,14 @@ async fn transform_json_unchecked(
            .await
            .map_err(to_anyhow)?;
            let value = if is_secret {
-                let mc = build_crypt(&db, &w_id).await?;
-                decrypt(&mc, value).map_err(|e| {
-                    Error::internal_err(format!("Error decrypting variable {}: {}", &s, e))
-                })?
+                if is_external_stored_value(&value) {
+                    get_secret_value(db, w_id, &s[5..], &value).await?
+                } else {
+                    let mc = build_crypt(&db, &w_id).await?;
+                    decrypt(&mc, value).map_err(|e| {
+                        Error::internal_err(format!("Error decrypting variable {}: {}", &s, e))
+                    })?
+                }
            } else {
                value
            };
@@ -6,269 +6,27 @@
 * LICENSE-AGPL for a copy of the license.
 */

-//! Secret backend extension for the API layer
+//! Secret backend extension for the store layer
 //!
-//! This module provides helper functions for integrating the SecretBackend
-//! trait with variable operations in the API.
+//! Write-side helpers for integrating the SecretBackend trait with variable
+//! operations. Backend resolution and read helpers live in
+//! `windmill_common::secret_backend` (so lower-level crates can resolve secrets
+//! too) and are re-exported here for existing callers.
 //!
 //! Note: HashiCorp Vault integration requires Enterprise Edition.
 //! The OSS version only supports the database backend.

-use std::sync::Arc;
-
 use windmill_common::{
    db::DB,
    error::{Error, Result},
-    secret_backend::{database::DatabaseBackend, SecretBackend},
-    variables::{build_crypt, decrypt, encrypt},
+    variables::{build_crypt, encrypt},
 };

-#[cfg(all(feature = "private", feature = "enterprise"))]
-use windmill_common::{
-    global_settings::{load_value_from_global_settings, SECRET_BACKEND_SETTING},
-    secret_backend::{
-        AwsSecretsManagerBackend, AwsSecretsManagerSettings, AzureKeyVaultBackend,
-        AzureKeyVaultSettings, SecretBackendConfig, VaultBackend, VaultSettings,
-    },
+pub use windmill_common::secret_backend::{
+    get_secret_backend, get_secret_value, is_aws_sm_stored_value, is_azure_kv_stored_value,
+    is_external_stored_value, is_vault_backend_configured, is_vault_stored_value,
 };

-#[cfg(all(feature = "private", feature = "enterprise"))]
-use tokio::sync::RwLock;
-
-// Cached Vault backend to avoid recreating it for every request
-// This enables connection pooling and avoids repeated setup overhead
-#[cfg(all(feature = "private", feature = "enterprise"))]
-struct CachedVaultBackend {
-    backend: Arc<dyn SecretBackend>,
-    settings: VaultSettings,
-}
-
-#[cfg(all(feature = "private", feature = "enterprise"))]
-lazy_static::lazy_static! {
-    static ref VAULT_BACKEND_CACHE: RwLock<Option<CachedVaultBackend>> = RwLock::new(None);
-}
-
-#[cfg(all(feature = "private", feature = "enterprise"))]
-struct CachedAzureKvBackend {
-    backend: Arc<dyn SecretBackend>,
-    settings: AzureKeyVaultSettings,
-}
-
-#[cfg(all(feature = "private", feature = "enterprise"))]
-lazy_static::lazy_static! {
-    static ref AZURE_KV_BACKEND_CACHE: RwLock<Option<CachedAzureKvBackend>> = RwLock::new(None);
-}
-
-// Cached AWS Secrets Manager backend
-#[cfg(all(feature = "private", feature = "enterprise"))]
-struct CachedAwsSmBackend {
-    backend: Arc<dyn SecretBackend>,
-    settings: AwsSecretsManagerSettings,
-}
-
-#[cfg(all(feature = "private", feature = "enterprise"))]
-lazy_static::lazy_static! {
-    static ref AWS_SM_BACKEND_CACHE: RwLock<Option<CachedAwsSmBackend>> = RwLock::new(None);
-}
-
-/// Get the current secret backend based on global settings
-///
-/// OSS: Always returns DatabaseBackend
-/// EE: Returns configured backend (Database or Vault)
-#[cfg(not(all(feature = "private", feature = "enterprise")))]
-pub async fn get_secret_backend(db: &DB) -> Result<Arc<dyn SecretBackend>> {
-    Ok(Arc::new(DatabaseBackend::new(db.clone())))
-}
-
-#[cfg(all(feature = "private", feature = "enterprise"))]
-pub async fn get_secret_backend(db: &DB) -> Result<Arc<dyn SecretBackend>> {
-    let config = match load_value_from_global_settings(db, SECRET_BACKEND_SETTING).await? {
-        Some(value) => serde_json::from_value::<SecretBackendConfig>(value).unwrap_or_default(),
-        None => SecretBackendConfig::default(),
-    };
-
-    match config {
-        SecretBackendConfig::Database => Ok(Arc::new(DatabaseBackend::new(db.clone()))),
-        SecretBackendConfig::HashiCorpVault(settings) => {
-            get_or_create_vault_backend(db, settings).await
-        }
-        SecretBackendConfig::AzureKeyVault(settings) => {
-            get_or_create_azure_kv_backend(db, settings).await
-        }
-        SecretBackendConfig::AwsSecretsManager(settings) => {
-            get_or_create_aws_sm_backend(db, settings).await
-        }
-    }
-}
-
-/// Get a cached Vault backend or create a new one if settings changed
-#[cfg(all(feature = "private", feature = "enterprise"))]
-async fn get_or_create_vault_backend(
-    _db: &DB,
-    settings: VaultSettings,
-) -> Result<Arc<dyn SecretBackend>> {
-    // Check if we have a cached backend with matching settings (read lock)
-    {
-        let cache = VAULT_BACKEND_CACHE.read().await;
-        if let Some(ref cached) = *cache {
-            if cached.settings == settings {
-                return Ok(cached.backend.clone());
-            }
-        }
-    }
-
-    // Need to create a new backend - acquire write lock
-    let mut cache = VAULT_BACKEND_CACHE.write().await;
-
-    // Double-check (another task may have created it while we waited)
-    if let Some(ref cached) = *cache {
-        if cached.settings == settings {
-            return Ok(cached.backend.clone());
-        }
-    }
-
-    // Create new backend
-    let backend: Arc<dyn SecretBackend> = {
-        #[cfg(feature = "openidconnect")]
-        if settings.token.is_none() {
-            Arc::new(VaultBackend::new_with_db(settings.clone(), _db.clone()))
-        } else {
-            Arc::new(VaultBackend::new(settings.clone()))
-        }
-
-        #[cfg(not(feature = "openidconnect"))]
-        Arc::new(VaultBackend::new(settings.clone()))
-    };
-
-    // Cache it
-    *cache = Some(CachedVaultBackend { backend: backend.clone(), settings });
-
-    Ok(backend)
-}
-
-/// Get a cached Azure Key Vault backend or create a new one if settings changed
-#[cfg(all(feature = "private", feature = "enterprise"))]
-async fn get_or_create_azure_kv_backend(
-    _db: &DB,
-    settings: AzureKeyVaultSettings,
-) -> Result<Arc<dyn SecretBackend>> {
-    // Check if we have a cached backend with matching settings (read lock)
-    {
-        let cache = AZURE_KV_BACKEND_CACHE.read().await;
-        if let Some(ref cached) = *cache {
-            if cached.settings == settings {
-                return Ok(cached.backend.clone());
-            }
-        }
-    }
-
-    // Need to create a new backend - acquire write lock
-    let mut cache = AZURE_KV_BACKEND_CACHE.write().await;
-
-    // Double-check (another task may have created it while we waited)
-    if let Some(ref cached) = *cache {
-        if cached.settings == settings {
-            return Ok(cached.backend.clone());
-        }
-    }
-
-    // Create new backend
-    let backend: Arc<dyn SecretBackend> = Arc::new(AzureKeyVaultBackend::new(settings.clone()));
-
-    // Cache it
-    *cache = Some(CachedAzureKvBackend { backend: backend.clone(), settings });
-
-    Ok(backend)
-}
-
-/// Get a cached AWS SM backend or create a new one if settings changed
-#[cfg(all(feature = "private", feature = "enterprise"))]
-async fn get_or_create_aws_sm_backend(
-    _db: &DB,
-    settings: AwsSecretsManagerSettings,
-) -> Result<Arc<dyn SecretBackend>> {
-    {
-        let cache = AWS_SM_BACKEND_CACHE.read().await;
-        if let Some(ref cached) = *cache {
-            if cached.settings == settings {
-                return Ok(cached.backend.clone());
-            }
-        }
-    }
-
-    let mut cache = AWS_SM_BACKEND_CACHE.write().await;
-
-    if let Some(ref cached) = *cache {
-        if cached.settings == settings {
-            return Ok(cached.backend.clone());
-        }
-    }
-
-    let backend: Arc<dyn SecretBackend> =
-        Arc::new(AwsSecretsManagerBackend::new_with_client(settings.clone()).await?);
-
-    *cache = Some(CachedAwsSmBackend { backend: backend.clone(), settings });
-
-    Ok(backend)
-}
-
-/// Check if a Vault backend is currently configured
-///
-/// OSS: Always returns false
-/// EE: Checks global settings
-#[cfg(not(all(feature = "private", feature = "enterprise")))]
-pub async fn is_vault_backend_configured(_db: &DB) -> Result<bool> {
-    Ok(false)
-}
-
-#[cfg(all(feature = "private", feature = "enterprise"))]
-pub async fn is_vault_backend_configured(db: &DB) -> Result<bool> {
-    let config = match load_value_from_global_settings(db, SECRET_BACKEND_SETTING).await? {
-        Some(value) => serde_json::from_value::<SecretBackendConfig>(value).unwrap_or_default(),
-        None => SecretBackendConfig::default(),
-    };
-
-    Ok(matches!(
-        config,
-        SecretBackendConfig::HashiCorpVault(_)
-            | SecretBackendConfig::AzureKeyVault(_)
-            | SecretBackendConfig::AwsSecretsManager(_)
-    ))
-}
-
-/// Get a secret value using the configured backend
-///
-/// For database backend: decrypts using workspace key
-/// For vault backend (EE only): fetches from Vault directly
-pub async fn get_secret_value(
-    db: &DB,
-    workspace_id: &str,
-    path: &str,
-    encrypted_value: &str,
-) -> Result<String> {
-    let backend = get_secret_backend(db).await?;
-
-    match backend.backend_name() {
-        "database" => {
-            // Use existing database decryption
-            let mc = build_crypt(db, workspace_id).await?;
-            decrypt(&mc, encrypted_value.to_string()).map_err(|e| {
-                Error::internal_err(format!("Error decrypting variable {}: {}", path, e))
-            })
-        }
-        "hashicorp_vault" => {
-            // Fetch from Vault directly
-            backend.get_secret(workspace_id, path).await
-        }
-        "azure_key_vault" => backend.get_secret(workspace_id, path).await,
-        "aws_secrets_manager" => backend.get_secret(workspace_id, path).await,
-        _ => Err(Error::internal_err(format!(
-            "Unknown backend: {}",
-            backend.backend_name()
-        ))),
-    }
-}
-
 /// Store a secret value using the configured backend
 ///
 /// For database backend: encrypts using workspace key and returns encrypted value
@@ -412,26 +170,6 @@ pub async fn delete_secret_from_backend(db: &DB, workspace_id: &str, path: &str)
    }
 }

-/// Check if a value is stored in Vault (indicated by the $vault: prefix)
-pub fn is_vault_stored_value(value: &str) -> bool {
-    value.starts_with("$vault:")
-}
-
-/// Check if a value is stored in Azure Key Vault (indicated by the $azure_kv: prefix)
-pub fn is_azure_kv_stored_value(value: &str) -> bool {
-    value.starts_with("$azure_kv:")
-}
-
-/// Check if a value is stored in AWS Secrets Manager (indicated by the $aws_sm: prefix)
-pub fn is_aws_sm_stored_value(value: &str) -> bool {
-    value.starts_with("$aws_sm:")
-}
-
-/// Check if a value is stored in any external secret backend
-pub fn is_external_stored_value(value: &str) -> bool {
-    is_vault_stored_value(value) || is_azure_kv_stored_value(value) || is_aws_sm_stored_value(value)
-}
-
 /// Rename a secret in Vault when a variable path changes (EE only)
 #[cfg(not(all(feature = "private", feature = "enterprise")))]
 pub async fn rename_vault_secret(
@@ -39,6 +39,47 @@ struct MaterializeExec {
    asset_kind: windmill_common::assets::AssetKind,
    asset_path: String,
    partition: String,
+    // Number of `// data_test` checks the codegen embedded. Enforcement recovers
+    // the per-test outcomes from the summary row; if it recovers fewer than this
+    // (e.g. an FFI serialization change drops the column), we fail loud rather
+    // than silently pass declared-but-unverified tests.
+    n_data_tests: usize,
+}
+
+// Fetch and validate a custom data-test script's body. v1 custom tests are
+// DuckDB scripts holding a single SELECT/CTE that returns the violating rows
+// (dbt's singular-test convention); the worker embeds that query as a subquery
+// check in the materialize connection (the single-statement constraint is
+// enforced in sql_materialize.rs). Server workers only — agent (Http) workers
+// have no script cache to read deployed content from.
+async fn fetch_custom_test_body(conn: &Connection, w_id: &str, path: &str) -> Result<String> {
+    let Connection::Sql(db) = conn else {
+        return Err(Error::ExecutionErr(format!(
+            "data_test custom `{path}`: custom tests require a server worker (not supported on \
+             agent workers in v1)"
+        )));
+    };
+    let hash = windmill_common::get_latest_script_hash(db, path, w_id)
+        .await?
+        .ok_or_else(|| {
+            Error::ExecutionErr(format!(
+                "data_test custom `{path}`: no deployed script found at this path"
+            ))
+        })?;
+    let content =
+        crate::get_script_content_by_hash(&windmill_common::scripts::ScriptHash(hash), w_id, conn)
+            .await?;
+    if !matches!(
+        content.language,
+        Some(windmill_common::scripts::ScriptLang::DuckDb)
+    ) {
+        return Err(Error::ExecutionErr(format!(
+            "data_test custom `{path}`: must be a DuckDB script returning the violating rows \
+             (got language {:?})",
+            content.language
+        )));
+    }
+    Ok(content.content)
 }

 // If `query` declares `// materialize <ducklake>`, return what to record plus,
@@ -46,22 +87,47 @@ struct MaterializeExec {
 // mode the script writes its own DDL, so the rewrite is `None`). The rewritten
 // SQL contains a synthetic `ATTACH 'ducklake://<name>' AS _wm_target` that the
 // normal ATTACH-transform pass resolves to real credentials — the same path as
-// the user's own ATTACH. Returns `None` when there is no materialize annotation
-// or the target isn't a ducklake (only ducklake is materialized in v1).
+// the user's own ATTACH. `// data_test` lines append verifier probes that run
+// against the freshly-materialized target and raise (failing the run) on
+// violation. Returns `None` when there is no materialize annotation or the
+// target isn't a ducklake (only ducklake is materialized in v1).
 fn build_materialized_query(
    query: &str,
    partition_value: Option<&str>,
+    // Custom (`// data_test <path>`) test bodies, pre-fetched by the caller
+    // (`fetch_custom_test_bodies`) so this stays pure/sync and unit-testable —
+    // the DB read is the only thing that needs a connection. Keyed by script path.
+    custom_test_bodies: &std::collections::HashMap<String, String>,
 ) -> Result<Option<(Option<String>, MaterializeExec)>> {
-    use windmill_parser::asset_parser::{parse_pipeline_annotations, AssetKind as PAssetKind};
+    use windmill_parser::asset_parser::{
+        parse_pipeline_annotations, AssetKind as PAssetKind, DataTest,
+    };
    use windmill_parser::sql_materialize::{
-        build_wrap_blocks, classify_wrap, MaterializeStrategy, TARGET_ALIAS,
+        build_wrap_blocks, DataTestResolved, MaterializeStrategy, TARGET_ALIAS,
    };

    let ann = parse_pipeline_annotations(query);
+    let has_tests = !ann.data_tests.is_empty();
    let Some(m) = ann.materialize else {
+        // Data tests run *against the materialized asset*; without a
+        // `// materialize` target there is nothing to test. Fail loudly rather
+        // than silently skip the declared checks.
+        if has_tests {
+            return Err(Error::ExecutionErr(
+                "data_test: requires a `// materialize` target — data tests run against the \
+                 materialized asset"
+                    .to_string(),
+            ));
+        }
        return Ok(None);
    };
    if m.target_kind != PAssetKind::Ducklake {
+        if has_tests {
+            return Err(Error::ExecutionErr(
+                "data_test: only `ducklake://` materialization targets support data tests in v1"
+                    .to_string(),
+            ));
+        }
        return Ok(None);
    }
    let partitioned = ann.partition.is_some();
@@ -86,10 +152,36 @@ fn build_materialized_query(
        asset_kind: windmill_common::assets::AssetKind::Ducklake,
        asset_path: m.target_path.clone(),
        partition: partition.clone(),
+        n_data_tests: ann.data_tests.len(),
+    };
+
+    // `{partition}` → escaped SQL literal substitution, applied to the managed
+    // SELECT, its setup, and any custom-test body so a partitioned test can
+    // filter by the active slice. Always a complete `'…'` literal (with `'`
+    // doubled) whether or not the author quoted it, so a run caller can't break
+    // out and alter statement boundaries. The pre-quoted `'{partition}'` form is
+    // matched first so it doesn't become `''…''`. No-op when unpartitioned.
+    let lit = format!("'{}'", partition.replace('\'', "''"));
+    let substitute = |s: &str| -> String {
+        if !partitioned {
+            return s.to_string();
+        }
+        let tok = windmill_common::assets::PARTITION_TOKEN;
+        let quoted_tok = format!("'{tok}'");
+        s.replace(&quoted_tok, &lit).replace(tok, &lit)
    };

    if m.manual {
-        // Escape hatch: the script owns its DDL; we only record state.
+        // Escape hatch: the script owns its DDL. We can't reliably attach the
+        // managed target or know the partition column it wrote, so data tests
+        // are not generated for manual mode in v1.
+        if has_tests {
+            return Err(Error::ExecutionErr(
+                "data_test: not supported with `// materialize manual` in v1 — use managed \
+                 `// materialize`"
+                    .to_string(),
+            ));
+        }
        return Ok(Some((None, meta)));
    }
    if table.is_empty() {
@@ -98,24 +190,10 @@ fn build_materialized_query(
            m.target_path
        )));
    }
-    let mut plan = classify_wrap(query).map_err(|e| Error::ExecutionErr(e.message()))?;
-    // Resolve the `{partition}` token (same token `// on` asset URIs use) to the
-    // current partition value everywhere in the managed script, so a partitioned
-    // materialize can filter its source by the active slice, e.g.
-    // `WHERE day = {partition}`. The token is always replaced by a *complete*
-    // escaped SQL literal (`'…'` with `'` doubled) whether or not the author
-    // quoted it — so a run caller can't pass metacharacters that break out of
-    // the literal and alter statement boundaries. The pre-quoted form
-    // `'{partition}'` is matched first so it doesn't become `''…''`. Only
-    // meaningful when partitioned.
-    if partitioned {
-        let lit = format!("'{}'", partition.replace('\'', "''"));
-        let tok = windmill_common::assets::PARTITION_TOKEN;
-        let quoted_tok = format!("'{tok}'");
-        plan.output = plan.output.replace(&quoted_tok, &lit).replace(tok, &lit);
-        for s in plan.setup.iter_mut() {
-            *s = s.replace(&quoted_tok, &lit).replace(tok, &lit);
-        }
+    let mut plan = classify_wrap_or_err(query)?;
+    plan.output = substitute(&plan.output);
+    for s in plan.setup.iter_mut() {
+        *s = substitute(s);
    }
    let strategy = if m.append {
        MaterializeStrategy::Append
@@ -126,8 +204,30 @@ fn build_materialized_query(
    };
    // Inline the partition as an escaped SQL literal (DuckLake has no bind for
    // the partition column in our generated DDL).
-    let pval = format!("'{}'", partition.replace('\'', "''"));
+    let pval = lit.clone();
    let synthetic_attach = format!("ATTACH 'ducklake://{ducklake_name}' AS {TARGET_ALIAS};");
+
+    // Resolve data tests (fetch + partition-substitute custom bodies) so codegen
+    // can embed every check's violating-row count in the materialize summary.
+    // The summary then carries the full per-test breakdown back to the worker,
+    // which runs them all and decides pass/fail (no abort-on-first). Empty when
+    // there are no `// data_test` lines.
+    let mut resolved = Vec::with_capacity(ann.data_tests.len());
+    for test in &ann.data_tests {
+        match test {
+            DataTest::Custom { path } => {
+                let raw = custom_test_bodies.get(path).ok_or_else(|| {
+                    Error::ExecutionErr(format!(
+                        "data_test custom `{path}`: body not fetched before codegen (internal)"
+                    ))
+                })?;
+                resolved
+                    .push(DataTestResolved::Custom { path: path.clone(), body: substitute(raw) });
+            }
+            other => resolved.push(DataTestResolved::BuiltIn(other.clone())),
+        }
+    }
+
    let blocks = build_wrap_blocks(
        &plan,
        &synthetic_attach,
@@ -137,10 +237,44 @@ fn build_materialized_query(
        &pval,
        partitioned,
        strategy,
-    );
+        &resolved,
+    )
+    .map_err(Error::ExecutionErr)?;
+
    Ok(Some((Some(blocks.join("\n")), meta)))
 }

+// Fetch the deployed body of every `// data_test <path>` custom test declared in
+// `query`, keyed by path, so the sync `build_materialized_query` can splice them
+// in. The DB read is the only part of materialize codegen that needs a
+// connection; isolating it here keeps the codegen pure and unit-testable.
+// Server workers only (`fetch_custom_test_body` errors on agent workers). Empty
+// when there are no custom tests.
+async fn fetch_custom_test_bodies(
+    query: &str,
+    conn: &Connection,
+    w_id: &str,
+) -> Result<std::collections::HashMap<String, String>> {
+    use windmill_parser::asset_parser::{parse_pipeline_annotations, DataTest};
+    let ann = parse_pipeline_annotations(query);
+    let mut bodies = std::collections::HashMap::new();
+    for test in &ann.data_tests {
+        if let DataTest::Custom { path } = test {
+            if !bodies.contains_key(path) {
+                let body = fetch_custom_test_body(conn, w_id, path).await?;
+                bodies.insert(path.clone(), body);
+            }
+        }
+    }
+    Ok(bodies)
+}
+
+// classify_wrap with the spec's actionable message turned into an executor error.
+fn classify_wrap_or_err(query: &str) -> Result<windmill_parser::sql_materialize::WrapPlan> {
+    windmill_parser::sql_materialize::classify_wrap(query)
+        .map_err(|e| Error::ExecutionErr(e.message()))
+}
+
 // Pull a named i64 field (`snapshot_id` / `rows`) out of the trailing summary
 // read — which in wrap mode is the job result. Shape-tolerant (object / array /
 // nested), returns None if absent (literal mode, or capture failed).
@@ -156,6 +290,75 @@ fn extract_i64(result: &RawValue, field: &str) -> Option<i64> {
    find(&serde_json::from_str::<Value>(result.get()).ok()?, field)
 }

+// One data test's outcome as carried by the materialize summary's `data_tests`
+// column: its display name and how many rows violated it (0 = pass).
+struct DataTestOutcome {
+    name: String,
+    violating: i64,
+}
+
+// Pull the per-test breakdown out of the materialize summary result. The
+// `data_tests` column is a DuckLake list-of-struct `[{test, violating}, …]`;
+// the FFI may surface it as a nested JSON array or as a JSON string, so accept
+// both. Returns empty when there are no tests (the column is absent).
+fn extract_data_tests(result: &RawValue) -> Vec<DataTestOutcome> {
+    fn collect(v: &Value, out: &mut Vec<DataTestOutcome>) {
+        if let Value::Array(arr) = v {
+            for item in arr {
+                if let Value::Object(o) = item {
+                    if let Some(Value::String(name)) = o.get("test") {
+                        let violating = o
+                            .get("violating")
+                            .and_then(|x| x.as_i64().or_else(|| x.as_f64().map(|f| f as i64)))
+                            .unwrap_or(0);
+                        out.push(DataTestOutcome { name: name.clone(), violating });
+                    }
+                }
+            }
+        }
+    }
+    fn find_field(v: &Value) -> Option<&Value> {
+        match v {
+            Value::Object(o) => o.get("data_tests"),
+            Value::Array(a) => a.iter().find_map(find_field),
+            _ => None,
+        }
+    }
+    let mut out = Vec::new();
+    let Ok(root) = serde_json::from_str::<Value>(result.get()) else {
+        return out;
+    };
+    match find_field(&root) {
+        Some(arr @ Value::Array(_)) => collect(arr, &mut out),
+        // FFI serialized the list-of-struct as a JSON string — parse it.
+        Some(Value::String(s)) => {
+            if let Ok(parsed) = serde_json::from_str::<Value>(s) {
+                collect(&parsed, &mut out);
+            }
+        }
+        _ => {}
+    }
+    out
+}
+
+// Render the full pass/fail breakdown for a failed data-test run — every test,
+// not just the first failure, so the user sees the whole picture in one place.
+fn format_data_test_breakdown(asset_path: &str, tests: &[DataTestOutcome]) -> String {
+    let failed = tests.iter().filter(|t| t.violating > 0).count();
+    let mut lines = vec![format!(
+        "data tests failed on {asset_path} ({failed}/{} failed):",
+        tests.len()
+    )];
+    for t in tests {
+        if t.violating > 0 {
+            lines.push(format!("  ✗ {} — {} violating row(s)", t.name, t.violating));
+        } else {
+            lines.push(format!("  ✓ {}", t.name));
+        }
+    }
+    lines.join("\n")
+}
+
 // Best-effort record of a materialization outcome. On a Sql connection it writes
 // the row directly; on an agent worker (Http, no direct DB) it posts to the API
 // so state lands the same way. Never fails the job — a lost row degrades the
@@ -250,8 +453,12 @@ pub async fn do_duckdb(
            .and_then(|a| a.0.get(windmill_common::partition::PARTITION_ARG))
            .and_then(|rv| serde_json::from_str::<String>(rv.get()).ok())
            .filter(|s| !s.is_empty());
-        let materialize = if query.contains("materialize") {
-            build_materialized_query(query, partition_value.as_deref())?
+        let materialize = if query.contains("materialize") || query.contains("data_test") {
+            // Custom-test bodies need a DB read; fetch them first so the codegen
+            // itself stays pure/sync.
+            let custom_test_bodies =
+                fetch_custom_test_bodies(query, conn, &job.workspace_id).await?;
+            build_materialized_query(query, partition_value.as_deref(), &custom_test_bodies)?
        } else {
            None
        };
@@ -271,6 +478,19 @@ pub async fn do_duckdb(
            }
            _ => query,
        };
+
+        // Managed materialize generates its own trailing summary row (asset /
+        // rows / snapshot_id / data_tests), and data-test enforcement below reads
+        // the `data_tests` column off that row. The row shape is ours, not the
+        // user's — so force the full-last-row strategy regardless of any
+        // `// result_collection` annotation, which would otherwise reshape it
+        // (e.g. a scalar mode drops every column but the first) and silently
+        // bypass test enforcement.
+        let collection_strategy = if matches!(&materialize, Some((Some(_), _))) {
+            SqlResultCollectionStrategy::LastStatementAllRows
+        } else {
+            collection_strategy
+        };
        let mut job_args = build_args_values(job, client, conn).await?;

        let reserved_variables =
@@ -427,9 +647,55 @@ pub async fn do_duckdb(

        if let Some((_, meta)) = &materialize {
            // In wrap mode the job result is the summary read (snapshot_id +
-            // rows); in literal mode there is none, so both stay None.
+            // rows + the per-test breakdown); in literal mode there is none.
            let snapshot_id = extract_i64(&result, "snapshot_id");
            let row_count = extract_i64(&result, "rows");
+            // Data tests all ran (every check counted in one query); decide
+            // pass/fail here. Any violation fails the run — the write is already
+            // committed (like dbt), so the slice is recorded `Failed` and the
+            // cascade stops. The error lists *every* test so the user sees the
+            // whole picture, not just the first failure.
+            let tests = extract_data_tests(&result);
+            // Defense-in-depth: codegen embedded `n_data_tests` checks, so the
+            // summary row must carry that many outcomes. Recovering fewer means
+            // the `data_tests` column was dropped/reshaped before we read it —
+            // fail loud rather than silently pass unverified tests.
+            if tests.len() < meta.n_data_tests {
+                let msg = format!(
+                    "data tests on {}: expected {} test outcome(s) but recovered {} from the \
+                     result — aborting to avoid a silent pass",
+                    meta.asset_path,
+                    meta.n_data_tests,
+                    tests.len()
+                );
+                record_mat(
+                    conn,
+                    &job.workspace_id,
+                    job.id,
+                    meta,
+                    windmill_common::materialization::MaterializationStatus::Failed,
+                    snapshot_id,
+                    row_count,
+                    Some(&msg),
+                )
+                .await;
+                return Err(Error::ExecutionErr(msg));
+            }
+            if tests.iter().any(|t| t.violating > 0) {
+                let breakdown = format_data_test_breakdown(&meta.asset_path, &tests);
+                record_mat(
+                    conn,
+                    &job.workspace_id,
+                    job.id,
+                    meta,
+                    windmill_common::materialization::MaterializationStatus::Failed,
+                    snapshot_id,
+                    row_count,
+                    Some(&breakdown),
+                )
+                .await;
+                return Err(Error::ExecutionErr(breakdown));
+            }
            record_mat(
                conn,
                &job.workspace_id,
@@ -1133,9 +1399,11 @@ mod tests {
        assert_eq!(file_arg.otyp.as_deref(), Some("s3object"));

        // The wrapped query still references `$file`, so the parsed sig binds it.
-        let (rewritten, _) = build_materialized_query(script, None)
-            .expect("materialize builds")
-            .expect("materialize present");
+        // No custom data tests here, so no fetched bodies are needed.
+        let (rewritten, _) =
+            build_materialized_query(script, None, &std::collections::HashMap::new())
+                .expect("materialize builds")
+                .expect("materialize present");
        let rewritten = rewritten.expect("managed mode rewrites the query");
        assert!(
            rewritten.contains("$file"),
@@ -1504,4 +1772,54 @@ mod tests {
        let serialized = serde_json::to_string(&arg).unwrap();
        assert!(serialized.contains("\"json_value\":{\"key\":\"value\"}"));
    }
+
+    fn raw(s: &str) -> Box<RawValue> {
+        serde_json::from_str(s).unwrap()
+    }
+
+    #[test]
+    fn extract_data_tests_parses_nested_array() {
+        // The real result shape: an array of one summary row carrying a nested
+        // `data_tests` array (how the FFI serialises the list-of-struct).
+        let r = raw(
+            r#"[{"rows":3,"snapshot_id":17,"materialized":"ducklake://a/b",
+                "data_tests":[{"test":"unique(order_id)","violating":0},
+                              {"test":"accepted_values(status)","violating":2}]}]"#,
+        );
+        let out = extract_data_tests(&r);
+        assert_eq!(out.len(), 2);
+        assert_eq!(out[0].name, "unique(order_id)");
+        assert_eq!(out[0].violating, 0);
+        assert_eq!(out[1].name, "accepted_values(status)");
+        assert_eq!(out[1].violating, 2);
+    }
+
+    #[test]
+    fn extract_data_tests_handles_string_encoded_and_absent() {
+        // Fallback: some serialisations surface the list-of-struct as a JSON string.
+        let s = raw(r#"{"data_tests":"[{\"test\":\"not_null(x)\",\"violating\":1}]"}"#);
+        let out = extract_data_tests(&s);
+        assert_eq!(out.len(), 1);
+        assert_eq!(out[0].name, "not_null(x)");
+        assert_eq!(out[0].violating, 1);
+        // Absent column (no tests) -> empty, no panic.
+        assert!(extract_data_tests(&raw(r#"[{"rows":3}]"#)).is_empty());
+    }
+
+    #[test]
+    fn format_data_test_breakdown_lists_all_with_marks() {
+        let tests = vec![
+            DataTestOutcome { name: "unique(order_id)".into(), violating: 1 },
+            DataTestOutcome { name: "not_null(user_id)".into(), violating: 0 },
+            DataTestOutcome { name: "accepted_values(status)".into(), violating: 2 },
+        ];
+        let msg = format_data_test_breakdown("analytics/orders", &tests);
+        assert_eq!(
+            msg,
+            "data tests failed on analytics/orders (2/3 failed):\n  \
+             ✗ unique(order_id) — 1 violating row(s)\n  \
+             ✓ not_null(user_id)\n  \
+             ✗ accepted_values(status) — 2 violating row(s)"
+        );
+    }
 }
@@ -37,8 +37,8 @@ use windmill_common::{
    scripts::ScriptLang,
    utils::calculate_hash,
    worker::{
-        copy_dir_recursively, pad_string, split_python_requirements, write_file, Connection,
-        PyVAlias, PythonAnnotations, WORKER_CONFIG,
+        copy_dir_recursively, is_allowed_file_location, pad_string, split_python_requirements,
+        write_file, Connection, PyVAlias, PythonAnnotations, WORKER_CONFIG,
    },
 };

@@ -664,10 +664,16 @@ pub fn compute_python_module_dir(script_path: &str) -> String {
        .replace("-", "_")
        .replace("@", ".");
    if dirs_full.len() > 0 {
-        dirs_full
-            .strip_prefix("/")
-            .unwrap_or(&dirs_full)
-            .to_string()
+        let dirs = dirs_full.strip_prefix("/").unwrap_or(&dirs_full);
+        // This directory is appended to job_dir and written to. Neutralize any
+        // `.`/`..` segment so the result stays a relative path inside job_dir: a
+        // Preview path is request-supplied and skips the DB `proper_id` CHECK that
+        // deployed runnables get, and the `@`->`.` rewrite above can also turn a
+        // segment like `@.` into `..`.
+        dirs.split('/')
+            .map(|seg| if seg == "." || seg == ".." { "_" } else { seg })
+            .collect::<Vec<_>>()
+            .join("/")
    } else {
        "tmp".to_string()
    }
@@ -1668,6 +1674,10 @@ async fn prepare_wrapper(
        last
    };
    let module_dir = format!("{}/{}", job_dir, dirs);
+    // Defense-in-depth: `dirs`/`last` derive from the (request-supplied for
+    // previews) script path. compute_python_module_dir already neutralizes `..`,
+    // but assert containment here too so the write can never escape job_dir.
+    is_allowed_file_location(job_dir, &format!("{dirs}/{last}.py"))?;
    tokio::fs::create_dir_all(format!("{module_dir}/")).await?;

    let _ = write_file(&module_dir, &format!("{last}.py"), inner_content)?;
@@ -2357,6 +2367,35 @@ async fn verify_wheel_record(venv_p: &str) -> Result<(), String> {
    }
 }

+lazy_static::lazy_static! {
+    /// Per-target-directory install locks. The wheel cache
+    /// (`ROOT_CACHE_DIR/python_<v>/<pkg>==<ver>`) is shared by every job on a
+    /// worker, and `uv pip install --reinstall --target <dir>` transiently
+    /// empties that directory while it runs. Two jobs installing the same
+    /// package concurrently would clobber each other's files — and a job that
+    /// imports from the dir mid-reinstall hits a flaky `ModuleNotFoundError`
+    /// (e.g. wmill's `httpx` vanishing during a WAC run). We serialize installs
+    /// per target dir and re-check the `.valid.windmill` marker once the lock is
+    /// held, so a waiter reuses the freshly populated cache instead of
+    /// reinstalling over it.
+    static ref PY_INSTALL_LOCKS: std::sync::Mutex<HashMap<String, Arc<tokio::sync::Mutex<()>>>> =
+        std::sync::Mutex::new(HashMap::new());
+}
+
+/// Returns the install lock guarding `venv_p`, creating it on first use.
+fn py_install_lock_for(venv_p: &str) -> Arc<tokio::sync::Mutex<()>> {
+    let mut locks = PY_INSTALL_LOCKS.lock().unwrap();
+    locks
+        .entry(venv_p.to_string())
+        .or_insert_with(|| Arc::new(tokio::sync::Mutex::new(())))
+        .clone()
+}
+
+/// A venv dir is usable iff its install wrote the `.valid.windmill` marker.
+async fn is_venv_valid(venv_p: &str) -> bool {
+    metadata(format!("{venv_p}/.valid.windmill")).await.is_ok()
+}
+
 /// uv pip install, include cached or pull from S3
 pub async fn handle_python_reqs(
    requirements: Vec<String>,
@@ -2716,6 +2755,34 @@ pub async fn handle_python_reqs(
            );

            let start = std::time::Instant::now();
+
+            // Serialize population of this shared cache dir against other jobs.
+            // `--reinstall` (and the S3 tar extraction below) rewrite `venv_p` in
+            // place, so a concurrent installer must not run while another job is
+            // still populating it.
+            let install_lock = py_install_lock_for(&venv_p);
+            let _install_guard = install_lock.lock().await;
+            // Another job may have finished the install while we waited on the
+            // lock — reuse its result instead of reinstalling over a dir other
+            // jobs may now be importing from.
+            if is_venv_valid(&venv_p).await {
+                print_success(
+                    false,
+                    false,
+                    &job_id,
+                    &w_id,
+                    &req,
+                    req_tl,
+                    counter_arc,
+                    total_to_install,
+                    start,
+                    &conn,
+                )
+                .await;
+                pids.lock().await.get_mut(i).and_then(|e| e.take());
+                return Ok(());
+            }
+
            #[cfg(all(feature = "enterprise", feature = "parquet"))]
            if is_not_pro {
                if let Some(os) = windmill_object_store::get_object_store().await {
@@ -3302,6 +3369,36 @@ pub async fn start_worker(
 mod tests {
    use super::*;

+    #[tokio::test]
+    async fn py_install_lock_serializes_same_target_only() {
+        // The shared wheel cache race fix relies on: same target dir -> same lock
+        // (serialized installs), different target dir -> independent locks. If this
+        // keying ever regresses, concurrent `--reinstall` could clobber a dir a job
+        // is importing from, reintroducing the flaky ModuleNotFoundError.
+        let a1 = py_install_lock_for("/cache/python_3_12/wmill==1.0.0");
+        let a2 = py_install_lock_for("/cache/python_3_12/wmill==1.0.0");
+        let b = py_install_lock_for("/cache/python_3_12/httpx==0.27.0");
+
+        assert!(
+            Arc::ptr_eq(&a1, &a2),
+            "same target dir must share one install lock"
+        );
+        assert!(
+            !Arc::ptr_eq(&a1, &b),
+            "different target dirs must use independent install locks"
+        );
+
+        let _held = a1.lock().await;
+        assert!(
+            a2.try_lock().is_err(),
+            "installs into the same target dir must be mutually exclusive"
+        );
+        assert!(
+            b.try_lock().is_ok(),
+            "installs into different target dirs must not block each other"
+        );
+    }
+
    #[test]
    fn test_compute_python_module_dir_nested_path() {
        assert_eq!(
@@ -3357,6 +3454,17 @@ mod tests {
        assert_eq!(compute_python_module_dir("f/in/script"), "f/_in");
    }

+    #[test]
+    fn test_compute_python_module_dir_neutralizes_traversal() {
+        // A Preview path skips the DB `proper_id` CHECK, so it can carry `..`.
+        // `..`/`.` segments must be neutralized so the dir stays inside job_dir.
+        let dirs = compute_python_module_dir("u/x/../../../../tmp/evil/payload");
+        assert!(!dirs.split('/').any(|s| s == ".." || s == "."));
+        assert_eq!(dirs, "u/x/_/_/_/_/tmp/evil");
+        // The `@`->`.` rewrite must not be able to synthesize a `..` segment.
+        assert_eq!(compute_python_module_dir("u/@./script"), "u/_");
+    }
+
    #[test]
    fn test_compute_py_codegen_basic_args() {
        let code = "def main(x: str, y: int):\n    return x\n";
@@ -2,7 +2,7 @@ import { sleep } from "https://deno.land/x/sleep@v1.2.1/mod.ts";
 import * as windmill from "https://deno.land/x/windmill@v1.174.0/mod.ts";
 import * as api from "https://deno.land/x/windmill@v1.174.0/windmill-api/index.ts";

-export const VERSION = "v1.737.0";
+export const VERSION = "v1.739.0";

 export async function login(email: string, password: string): Promise<string> {
  return await windmill.UserService.login({
@@ -0,0 +1,241 @@
+// Bounded-cascade graph engine for `wmill pipeline run --to`.
+//
+// MIRROR of frontend/src/lib/components/assets/AssetGraph/boundedCascade.ts —
+// the two have no shared import path (frontend is a separate package), so keep
+// them in sync. The grammar is intentionally tiny: there is no dbt-style
+// `--select` string. The user names a start (a schedule / manual root) and one
+// or more end nodes; the run is the "path between" them:
+//
+//     descendants(start) ∩ (ancestors(ends) ∪ ends) ∪ {start}
+//
+// Node ids: assets `${kind}:${path}` (e.g. `datatable:main/raw`); runnables
+// `script:${path}`. Operates on the asset-graph payload shape used by
+// pipeline.ts.
+
+export type BCGraph = {
+  runnables: { path: string; usage_kind: "script" | "flow" | "job" }[];
+  assets: { kind: string; path: string }[];
+  edges: {
+    runnable_kind: string;
+    runnable_path: string;
+    asset_kind: string;
+    asset_path: string;
+    access_type?: "r" | "w" | "rw";
+  }[];
+  triggers: (
+    | {
+        trigger_kind: "asset";
+        asset_kind: string;
+        asset_path: string;
+        runnable_kind: string;
+        runnable_path: string;
+      }
+    | { trigger_kind: string; runnable_kind: string; runnable_path: string }
+  )[];
+};
+
+export const SCRIPT_PREFIX = "script:";
+export const scriptNodeId = (path: string): string => `${SCRIPT_PREFIX}${path}`;
+export const isScriptNode = (id: string): boolean => id.startsWith(SCRIPT_PREFIX);
+export const scriptPathOf = (id: string): string => id.slice(SCRIPT_PREFIX.length);
+const assetNodeId = (kind: string, path: string): string => `${kind}:${path}`;
+
+// Native trigger kinds that fan out per-event — never bounded-run starts.
+// `webhook` / `data_upload` have no trigger row in the graph payload, so a root
+// whose only entry is one of those reads as a manual root.
+const EVENT_TRIGGER_KINDS = new Set([
+  "kafka",
+  "mqtt",
+  "nats",
+  "postgres",
+  "sqs",
+  "gcp",
+  "email",
+]);
+
+/** Resolve an asset URI (`datatable://x`, `s3://b/k`, …) to its node id. */
+export function assetUriToNodeId(uri: string): string | undefined {
+  const m = uri.match(/^([a-z0-9_]+):\/\/(.+)$/i);
+  if (!m) return undefined;
+  const prefix = m[1].toLowerCase();
+  const kind = prefix === "s3" ? "s3object" : prefix;
+  return `${kind}:${m[2]}`;
+}
+
+export type LineageDag = {
+  down: Map<string, Set<string>>;
+  up: Map<string, Set<string>>;
+  nodes: Set<string>;
+};
+
+function addEdge(dag: LineageDag, a: string, b: string) {
+  if (a === b) return;
+  dag.nodes.add(a);
+  dag.nodes.add(b);
+  (dag.down.get(a) ?? dag.down.set(a, new Set()).get(a)!).add(b);
+  (dag.up.get(b) ?? dag.up.set(b, new Set()).get(b)!).add(a);
+}
+
+/** Unified upstream→downstream lineage DAG over scripts ∪ assets. */
+export function buildLineageDag(g: BCGraph): LineageDag {
+  const dag: LineageDag = { down: new Map(), up: new Map(), nodes: new Set() };
+  for (const r of g.runnables ?? []) {
+    if (r.usage_kind === "script") dag.nodes.add(scriptNodeId(r.path));
+  }
+  for (const a of g.assets ?? []) dag.nodes.add(assetNodeId(a.kind, a.path));
+  for (const e of g.edges ?? []) {
+    if (e.runnable_kind !== "script") continue;
+    const aid = assetNodeId(e.asset_kind, e.asset_path);
+    const access = e.access_type ?? "r";
+    if (access === "w" || access === "rw") {
+      addEdge(dag, scriptNodeId(e.runnable_path), aid); // producer
+    } else if (access === "r") {
+      addEdge(dag, aid, scriptNodeId(e.runnable_path)); // pure reader
+    }
+  }
+  for (const t of g.triggers ?? []) {
+    if (t.trigger_kind !== "asset" || t.runnable_kind !== "script") continue;
+    const at = t as Extract<BCGraph["triggers"][number], { trigger_kind: "asset" }>;
+    addEdge(dag, assetNodeId(at.asset_kind, at.asset_path), scriptNodeId(at.runnable_path));
+  }
+  return dag;
+}
+
+function closure(adj: Map<string, Set<string>>, start: string): Set<string> {
+  const seen = new Set<string>();
+  const queue = [start];
+  while (queue.length > 0) {
+    const cur = queue.shift()!;
+    for (const n of adj.get(cur) ?? []) {
+      if (seen.has(n)) continue;
+      seen.add(n);
+      queue.push(n);
+    }
+  }
+  // A cycle back to `start` would have re-added it; the contract excludes
+  // the node itself.
+  seen.delete(start);
+  return seen;
+}
+
+export const descendants = (dag: LineageDag, n: string): Set<string> => closure(dag.down, n);
+export const ancestors = (dag: LineageDag, n: string): Set<string> => closure(dag.up, n);
+
+export type BoundedResult = {
+  nodes: Set<string>;
+  reachableEnds: string[];
+  droppedEnds: string[];
+};
+
+/** Path-between node set for `start` and `ends` (inclusive). */
+export function boundedSet(dag: LineageDag, start: string, ends: string[]): BoundedResult {
+  const downSet = new Set(descendants(dag, start));
+  downSet.add(start);
+  const reachableEnds = ends.filter((e) => downSet.has(e));
+  const droppedEnds = ends.filter((e) => !downSet.has(e));
+  if (reachableEnds.length === 0) {
+    return { nodes: new Set([start]), reachableEnds, droppedEnds };
+  }
+  const upClosure = new Set<string>();
+  for (const e of reachableEnds) {
+    upClosure.add(e);
+    for (const a of ancestors(dag, e)) upClosure.add(a);
+  }
+  const nodes = new Set<string>();
+  for (const n of downSet) if (upClosure.has(n)) nodes.add(n);
+  nodes.add(start);
+  return { nodes, reachableEnds, droppedEnds };
+}
+
+/** Script node ids eligible to start a bounded run. */
+export function validStarts(g: BCGraph): Set<string> {
+  const subscribers = new Set<string>();
+  const scheduleScripts = new Set<string>();
+  const eventScripts = new Set<string>();
+  for (const t of g.triggers ?? []) {
+    if (t.runnable_kind !== "script") continue;
+    if (t.trigger_kind === "asset") subscribers.add(t.runnable_path);
+    else if (t.trigger_kind === "schedule") scheduleScripts.add(t.runnable_path);
+    else if (EVENT_TRIGGER_KINDS.has(t.trigger_kind)) eventScripts.add(t.runnable_path);
+  }
+  const out = new Set<string>();
+  for (const r of g.runnables ?? []) {
+    if (r.usage_kind !== "script") continue;
+    const p = r.path;
+    if (scheduleScripts.has(p)) out.add(scriptNodeId(p));
+    else if (!subscribers.has(p) && !eventScripts.has(p)) out.add(scriptNodeId(p));
+  }
+  return out;
+}
+
+/** Project a node-id set to the script paths it contains. */
+export function scriptsOf(nodes: Iterable<string>): string[] {
+  const out: string[] = [];
+  for (const id of nodes) if (isScriptNode(id)) out.push(scriptPathOf(id));
+  return out;
+}
+
+/**
+ * Resolve a CLI `--to` / `--from` token to a node id, or undefined if it
+ * matches nothing. Asset URIs (`kind://path`) resolve to the asset node; a bare
+ * token matches a runnable by exact path or by short (last-segment) name.
+ */
+export function resolveToken(g: BCGraph, token: string): string | undefined {
+  if (token.includes("://")) {
+    const id = assetUriToNodeId(token);
+    return id && g.assets.some((a) => `${a.kind}:${a.path}` === id) ? id : undefined;
+  }
+  const scripts = (g.runnables ?? []).filter((r) => r.usage_kind === "script");
+  const exact = scripts.find((r) => r.path === token);
+  if (exact) return scriptNodeId(exact.path);
+  const byShort = scripts.filter((r) => (r.path.split("/").pop() ?? r.path) === token);
+  return byShort.length === 1 ? scriptNodeId(byShort[0].path) : undefined;
+}
+
+/**
+ * Topological order of `scripts` over the in-set producer→subscriber edges
+ * (assets collapsed). Scripts on a cycle are returned in `cyclic` and excluded
+ * from `order`. Serial-run friendly: every script comes after its in-set
+ * upstreams.
+ */
+export function topoOrder(
+  g: BCGraph,
+  scripts: Set<string>,
+): { order: string[]; cyclic: string[] } {
+  const dag = buildLineageDag(g);
+  const down = new Map<string, Set<string>>();
+  const indegree = new Map<string, number>();
+  for (const s of scripts) indegree.set(s, 0);
+  // One-hop (through a single asset) script→script edges, restricted to the set.
+  for (const s of scripts) {
+    const sid = scriptNodeId(s);
+    const oneHop = new Set<string>();
+    for (const asset of dag.down.get(sid) ?? []) {
+      for (const sub of dag.down.get(asset) ?? []) {
+        if (isScriptNode(sub)) {
+          const p = scriptPathOf(sub);
+          if (p !== s && scripts.has(p)) oneHop.add(p);
+        }
+      }
+    }
+    if (oneHop.size > 0) {
+      down.set(s, oneHop);
+      for (const p of oneHop) indegree.set(p, (indegree.get(p) ?? 0) + 1);
+    }
+  }
+  const ready = [...scripts].filter((s) => (indegree.get(s) ?? 0) === 0);
+  const remaining = new Map(indegree);
+  const order: string[] = [];
+  while (ready.length > 0) {
+    const n = ready.shift()!;
+    order.push(n);
+    for (const p of down.get(n) ?? []) {
+      const d = (remaining.get(p) ?? 0) - 1;
+      remaining.set(p, d);
+      if (d === 0) ready.push(p);
+    }
+  }
+  const orderedSet = new Set(order);
+  const cyclic = [...scripts].filter((s) => !orderedSet.has(s));
+  return { order, cyclic };
+}
@@ -8,6 +8,18 @@ import { requireLogin } from "../../core/auth.ts";
 import { resolveWorkspace } from "../../core/context.ts";
 import * as log from "../../core/log.ts";
 import { GlobalOptions } from "../../types.ts";
+import {
+  type BCGraph,
+  boundedSet,
+  buildLineageDag,
+  descendants,
+  resolveToken,
+  scriptNodeId,
+  scriptPathOf,
+  scriptsOf,
+  topoOrder,
+  validStarts,
+} from "./boundedCascade.ts";

 // Mirrors the asset-graph endpoint payload (backend/windmill-api-assets).
 // TODO: the checked-in generated client (cli/gen, last regenerated 2025-04)
@@ -269,6 +281,196 @@ async function show(
  console.log(lines.join("\n"));
 }

+// Poll a launched job to a terminal state. Modest fixed cadence; capped so a
+// wedged job can't hang the CLI forever.
+async function waitJob(workspace: string, id: string): Promise<boolean> {
+  const MAX_RETRIES = 6000; // ~10min at 100ms
+  for (let i = 0; i < MAX_RETRIES; i++) {
+    try {
+      const r = await wmill.getCompletedJobResultMaybe({
+        workspace,
+        id,
+        getStarted: false,
+      });
+      // A completed job without an explicit `success: true` is a failure
+      // (mirrors the frontend `waitJobTerminal`): the cascade only advances on
+      // a confirmed success.
+      if (r.completed) return r.success === true;
+    } catch {
+      // transient — retry
+    }
+    await new Promise((res) => setTimeout(res, 100));
+  }
+  throw new Error(`Timed out waiting for job ${id}`);
+}
+
+// Bounded-cascade run: start at a schedule / manual root, fan downstream, but
+// stop at the `--to` end node(s). Scripts run in topological order; each is
+// launched with `_wmill_skip_asset_dispatch` so the CLI owns the whole closure
+// (the backend dispatcher never double-fires the deployed part). With no
+// `--to`, runs the full read-aware downstream of `--from` (every descendant in
+// the lineage DAG, pure readers included — broader than the canvas cascade,
+// which dispatches subscribers only).
+async function run(
+  opts: GlobalOptions & {
+    from?: string;
+    to?: string[];
+    dryRun?: boolean;
+    json?: boolean;
+  },
+  folder: string,
+) {
+  if (opts.json) log.setSilent(true);
+  const workspace = await resolveWorkspace(opts);
+  await requireLogin(opts);
+
+  const f = folder.replace(/^f\//, "").replace(/\/$/, "");
+  const graph = await apiGet<BCGraph>(
+    `/w/${workspace.workspaceId}/assets/graph?folder=${encodeURIComponent(f)}&asset_kinds=${ASSET_KINDS}`,
+  );
+
+  // Resolve the start: explicit --from (must be a valid start) or the folder's
+  // sole valid start.
+  const starts = validStarts(graph);
+  let start: string;
+  if (opts.from) {
+    const resolved = resolveToken(graph, opts.from);
+    if (!resolved) {
+      // Distinguish "no match" from "ambiguous short name" (resolveToken
+      // returns undefined for both) so the hint is actionable.
+      const matches = graph.runnables.filter(
+        (r) => r.usage_kind === "script" && (r.path.split("/").pop() ?? r.path) === opts.from,
+      );
+      if (matches.length > 1) {
+        throw new Error(
+          `--from '${opts.from}' matches multiple scripts (${matches.map((r) => r.path).sort().join(", ")}) — pass the full path.`,
+        );
+      }
+      throw new Error(`--from '${opts.from}' matched no script in f/${f}.`);
+    }
+    if (!starts.has(resolved)) {
+      throw new Error(
+        `--from '${opts.from}' is not a valid bounded-run start. Starts must be schedule-triggered or manual roots; row-backed event triggers (kafka/mqtt/nats/postgres/sqs/gcp/email) fan out per-event and can't be bounded.`,
+      );
+    }
+    start = resolved;
+  } else if (starts.size === 1) {
+    start = [...starts][0];
+  } else if (starts.size === 0) {
+    throw new Error(
+      `No schedule or manual root in f/${f} to start a bounded run from.`,
+    );
+  } else {
+    throw new Error(
+      `f/${f} has ${starts.size} possible starts — pass --from <script>. Candidates: ${scriptsOf(starts).sort().join(", ")}`,
+    );
+  }
+
+  // Resolve --to end node(s): split on comma, resolve each. An unresolved or
+  // ambiguous token is a hard error — silently ignoring it would run a
+  // different subset than the user asked for.
+  const toTokens = (opts.to ?? []).flatMap((t) => t.split(",")).map((t) => t.trim()).filter(
+    Boolean,
+  );
+  const ends: string[] = [];
+  const unresolved: string[] = [];
+  for (const tok of toTokens) {
+    const id = resolveToken(graph, tok);
+    if (!id) unresolved.push(tok);
+    else ends.push(id);
+  }
+  if (unresolved.length > 0) {
+    const details = unresolved.map((tok) => {
+      const matches = graph.runnables.filter(
+        (r) => r.usage_kind === "script" && (r.path.split("/").pop() ?? r.path) === tok,
+      );
+      return matches.length > 1
+        ? `'${tok}' (ambiguous: ${matches.map((r) => r.path).sort().join(", ")} — use the full path)`
+        : `'${tok}' (no match in f/${f})`;
+    });
+    throw new Error(`--to could not be resolved: ${details.join("; ")}.`);
+  }
+
+  // Readable label for a node id: `script:<path>` → `<path>`; asset ids
+  // (`<kind>:<path>`) are kept verbatim (slicing `script:` off them corrupts
+  // the name).
+  const idLabel = (id: string): string => (id.startsWith("script:") ? scriptPathOf(id) : id);
+
+  const dag = buildLineageDag(graph);
+  let selectedScripts: Set<string>;
+  let reachableEnds: string[] = [];
+  let droppedEnds: string[] = [];
+  if (ends.length === 0) {
+    // No bound → full read-aware downstream of start (pure readers included).
+    const all = new Set(descendants(dag, start));
+    all.add(start);
+    selectedScripts = new Set(scriptsOf(all));
+  } else {
+    const res = boundedSet(dag, start, ends);
+    reachableEnds = res.reachableEnds;
+    droppedEnds = res.droppedEnds;
+    for (const d of droppedEnds) {
+      log.warn(`end '${idLabel(d)}' is not downstream of the start — ignored.`);
+    }
+    selectedScripts = new Set(scriptsOf(res.nodes));
+  }
+
+  const { order, cyclic } = topoOrder(graph, selectedScripts);
+  if (cyclic.length > 0) {
+    log.warn(`Skipping ${cyclic.length} script(s) on a dependency cycle: ${cyclic.sort().join(", ")}`);
+  }
+
+  if (opts.json) {
+    // Surface reachable/dropped ends so a machine-readable plan reflects the
+    // same trimming the human-facing warning does — a resolved-but-unreachable
+    // `--to` must not look like a clean plan that silently runs only the start.
+    console.log(
+      JSON.stringify({
+        start: scriptPathOf(start),
+        ends: ends.map(idLabel),
+        reachableEnds: reachableEnds.map(idLabel),
+        droppedEnds: droppedEnds.map(idLabel),
+        order,
+        cyclic,
+      }),
+    );
+  }
+
+  if (order.length === 0) {
+    if (!opts.json) log.info("Nothing to run.");
+    return;
+  }
+
+  if (opts.dryRun) {
+    if (!opts.json) {
+      log.info(
+        colors.bold(`Bounded run plan — ${order.length} script${order.length === 1 ? "" : "s"}`) +
+          colors.dim(` (from ${shortName(scriptPathOf(start))})`),
+      );
+      order.forEach((p, i) => log.info(`  ${i + 1}. ${p}`));
+    }
+    return;
+  }
+
+  // Execute in topological order, stopping on the first failure.
+  for (const path of order) {
+    if (!opts.json) log.info(colors.gray(`▶ running ${path}…`));
+    const id = await wmill.runScriptByPath({
+      workspace: workspace.workspaceId,
+      path,
+      requestBody: { _wmill_skip_asset_dispatch: true },
+    });
+    const ok = await waitJob(workspace.workspaceId, id);
+    if (!ok) {
+      throw new Error(`Bounded run failed at ${path} (job ${id}).`);
+    }
+    if (!opts.json) log.info(colors.green(`  ✓ ${path}`));
+  }
+  if (!opts.json) {
+    log.info(colors.green.bold(`Bounded run complete — ${order.length} script(s) succeeded.`));
+  }
+}
+
 const command = new Command()
  .description(
    "inspect asset-driven pipelines (scripts marked `// pipeline`, wired by `// on <spec>` annotations)",
@@ -282,6 +484,23 @@ const command = new Command()
  )
  .arguments("<folder:string>")
  .option("--json", "Output the raw asset graph as JSON")
-  .action(show as any);
+  .action(show as any)
+  .command(
+    "run",
+    "run a bounded cascade: from a schedule/manual root, fan downstream up to the --to end node(s)",
+  )
+  .arguments("<folder:string>")
+  .option(
+    "--from <script:string>",
+    "Start script (short name or path). Defaults to the folder's sole schedule/manual root.",
+  )
+  .option(
+    "--to <node:string>",
+    "End node(s) to stop at — script names/paths or asset URIs (e.g. datatable://main/staged). Repeatable or comma-separated. Omit to run the full downstream.",
+    { collect: true },
+  )
+  .option("--dry-run", "Print the topological run plan without executing.")
+  .option("--json", "Output the plan as JSON (for piping to jq).")
+  .action(run as any);

 export default command;
@@ -10,4 +10,4 @@ export const WM_FORK_PREFIX = "wm-fork";
 // (e.g. utils.ts) can read it without importing main.ts and creating a circular
 // dependency (main → workspace → utils → main) that triggers a TDZ.
 // Re-exported from main.ts for backwards compatibility.
-export const VERSION = "1.737.0";
+export const VERSION = "1.739.0";
@@ -6897,6 +6897,11 @@ inspect asset-driven pipelines (scripts marked \`// pipeline\`, wired by \`// on
  - \`--json\` - Output as JSON (for piping to jq)
 - \`pipeline show <folder:string>\` - render a pipeline folder's DAG (sources, lineage, subscriptions) in the terminal
  - \`--json\` - Output the raw asset graph as JSON
+- \`pipeline run <folder:string>\` - run a bounded cascade: from a schedule/manual root, fan downstream up to the --to end node(s)
+  - \`--from <script:string>\` - Start script (short name or path). Defaults to the folder's sole schedule/manual root.
+  - \`--to <node:string>\` - End node(s) to stop at — script names/paths or asset URIs (e.g. datatable://main/staged). Repeatable or comma-separated. Omit to run the full downstream.
+  - \`--dry-run\` - Print the topological run plan without executing.
+  - \`--json\` - Output the plan as JSON (for piping to jq).

 ### protection-rules

@@ -0,0 +1,160 @@
+import { expect, test } from "bun:test";
+
+// Mirror of
+// frontend/src/lib/components/assets/AssetGraph/boundedCascade.test.ts — keep
+// the two engines in sync. (Bun project → bun:test, not Deno.)
+import {
+  type BCGraph,
+  ancestors,
+  assetUriToNodeId,
+  boundedSet,
+  buildLineageDag,
+  descendants,
+  resolveToken,
+  scriptNodeId,
+  scriptsOf,
+  topoOrder,
+  validStarts,
+} from "../src/commands/pipeline/boundedCascade.ts";
+
+type W = [script: string, asset: string];
+type S = [script: string, asset: string];
+type R = [script: string, asset: string];
+
+function graph(opts: {
+  scripts?: string[];
+  writes?: W[];
+  reads?: R[];
+  subs?: S[];
+  native?: Array<[kind: string, script: string]>;
+}): BCGraph {
+  const { scripts = [], writes = [], reads = [], subs = [], native = [] } = opts;
+  return {
+    assets: [],
+    runnables: scripts.map((p) => ({ path: p, usage_kind: "script" as const })),
+    edges: [
+      ...writes.map(([s, a]) => ({
+        runnable_kind: "script",
+        runnable_path: s,
+        asset_kind: "datatable",
+        asset_path: a,
+        access_type: "w" as const,
+      })),
+      ...reads.map(([s, a]) => ({
+        runnable_kind: "script",
+        runnable_path: s,
+        asset_kind: "datatable",
+        asset_path: a,
+        access_type: "r" as const,
+      })),
+    ],
+    triggers: [
+      ...subs.map(([s, a]) => ({
+        trigger_kind: "asset" as const,
+        asset_kind: "datatable",
+        asset_path: a,
+        runnable_kind: "script",
+        runnable_path: s,
+      })),
+      ...native.map(([kind, s]) => ({
+        trigger_kind: kind,
+        runnable_kind: "script",
+        runnable_path: s,
+      })),
+    ],
+  };
+}
+
+const sn = scriptNodeId;
+const sorted = (it: Iterable<string>) => [...it].sort();
+
+// a → x → b → y → c → z → d (linear chain through assets)
+const chain = () =>
+  graph({
+    scripts: ["a", "b", "c", "d"],
+    writes: [
+      ["a", "x"],
+      ["b", "y"],
+      ["c", "z"],
+    ],
+    subs: [
+      ["b", "x"],
+      ["c", "y"],
+      ["d", "z"],
+    ],
+  });
+
+test("boundedSet stops at a single end node", () => {
+  const res = boundedSet(buildLineageDag(chain()), sn("a"), [sn("c")]);
+  expect(sorted(scriptsOf(res.nodes))).toEqual(["a", "b", "c"]);
+  expect(res.nodes.has("datatable:z")).toBe(false);
+});
+
+test("boundedSet supports an asset as the end bound", () => {
+  const res = boundedSet(buildLineageDag(chain()), sn("a"), ["datatable:y"]);
+  expect(sorted(scriptsOf(res.nodes))).toEqual(["a", "b"]);
+});
+
+test("boundedSet drops ends not downstream of start", () => {
+  const res = boundedSet(buildLineageDag(chain()), sn("c"), [sn("a")]);
+  expect(res.droppedEnds).toEqual([sn("a")]);
+  expect([...res.nodes]).toEqual([sn("c")]);
+});
+
+test("validStarts: schedule and manual roots, not events or subscribers", () => {
+  const g = graph({
+    scripts: ["a", "sub", "sched", "kfk"],
+    writes: [["a", "x"]],
+    subs: [["sub", "x"], ["sched", "x"]],
+    native: [["schedule", "sched"], ["kafka", "kfk"]],
+  });
+  const starts = validStarts(g);
+  expect(starts.has(sn("a"))).toBe(true); // manual root
+  expect(starts.has(sn("sched"))).toBe(true); // schedule overrides subscriber
+  expect(starts.has(sn("sub"))).toBe(false); // pure subscriber
+  expect(starts.has(sn("kfk"))).toBe(false); // event-only
+});
+
+test("assetUriToNodeId maps s3 → s3object, others verbatim", () => {
+  expect(assetUriToNodeId("s3://b/k")).toBe("s3object:b/k");
+  expect(assetUriToNodeId("datatable://main/users")).toBe("datatable:main/users");
+  expect(assetUriToNodeId("nope")).toBe(undefined);
+});
+
+test("resolveToken: short name, full path, and asset URI", () => {
+  const g = graph({ scripts: ["f/p/stage"], writes: [["f/p/stage", "main/staged"]] });
+  g.assets = [{ kind: "datatable", path: "main/staged" }];
+  expect(resolveToken(g, "stage")).toBe(sn("f/p/stage"));
+  expect(resolveToken(g, "f/p/stage")).toBe(sn("f/p/stage"));
+  expect(resolveToken(g, "datatable://main/staged")).toBe("datatable:main/staged");
+  expect(resolveToken(g, "missing")).toBe(undefined);
+});
+
+test("topoOrder sorts the bounded scripts and flags cycles", () => {
+  const { order, cyclic } = topoOrder(chain(), new Set(["a", "b", "c"]));
+  expect(order).toEqual(["a", "b", "c"]);
+  expect(cyclic).toEqual([]);
+});
+
+test("descendants/ancestors exclude the start even on a cycle", () => {
+  // a → x → b → y → a (cycle): closures must not contain a.
+  const g = graph({
+    scripts: ["a", "b"],
+    writes: [["a", "x"], ["b", "y"]],
+    subs: [["b", "x"], ["a", "y"]],
+  });
+  const dag = buildLineageDag(g);
+  expect(descendants(dag, sn("a")).has(sn("a"))).toBe(false);
+  expect(ancestors(dag, sn("a")).has(sn("a"))).toBe(false);
+});
+
+test("topoOrder orders a pure reader after its producer", () => {
+  // a writes x; c only *reads* x (no `// on x`). c must run after a.
+  const g = graph({
+    scripts: ["a", "c"],
+    writes: [["a", "x"]],
+    reads: [["c", "x"]],
+  });
+  const { order } = topoOrder(g, new Set(["a", "c"]));
+  expect(order).toEqual(["a", "c"]);
+});
@@ -54,7 +54,7 @@ RUN curl -fsSL https://www.postgresql.org/media/keys/ACCC4CF8.asc | gpg --dearmo
 ENV TZ=Etc/UTC

 # Install UV
-RUN curl --proto '=https' --tlsv1.2 -LsSf https://github.com/astral-sh/uv/releases/download/0.9.25/uv-installer.sh | sh && mv /root/.local/bin/uv /usr/local/bin/uv
+RUN curl --proto '=https' --tlsv1.2 -LsSf https://github.com/astral-sh/uv/releases/download/0.11.24/uv-installer.sh | sh && mv /root/.local/bin/uv /usr/local/bin/uv

 # Preinstall python runtime to temp location (will copy with world-writable perms later)
 # --compile-bytecode precompiles the stdlib to .pyc so jobs don't recompile it on every run
@@ -54,7 +54,7 @@ RUN curl -fsSL https://www.postgresql.org/media/keys/ACCC4CF8.asc | gpg --dearmo
 ENV TZ=Etc/UTC

 # Install UV
-RUN curl --proto '=https' --tlsv1.2 -LsSf https://github.com/astral-sh/uv/releases/download/0.9.25/uv-installer.sh | sh && mv /root/.local/bin/uv /usr/local/bin/uv
+RUN curl --proto '=https' --tlsv1.2 -LsSf https://github.com/astral-sh/uv/releases/download/0.11.24/uv-installer.sh | sh && mv /root/.local/bin/uv /usr/local/bin/uv

 # Preinstall python runtime to temp location (will copy with world-writable perms later)
 # --compile-bytecode precompiles the stdlib to .pyc so jobs don't recompile it on every run
@@ -171,29 +171,141 @@ This one table drives four things at once:

 ## Reproducibility — the beyond-dbt part

-Because every materialization records the snapshot it produced, a downstream
-consumer can read the *exact* upstream snapshot its run saw:
+Because every materialization records the snapshot it produced, you can read any
+table *as of* a past version — a capability dbt has no native answer for (dbt
+models are always "whatever's in the warehouse now"). DuckLake gives us this for
+the cost of recording one integer per run, and it covers the three things people
+actually reach for: **debugging** ("what did this table look like at the failing
+run"), **rollback** (re-materialize a consumer from snapshot N), and ad-hoc
+**experimentation** on a historical state.
+
+### How it's surfaced (shipped): explicit, discoverable time-travel
+
+The version is exposed as a *user-driven* surface, not hidden plumbing. A
+consumer pins a read by writing the DuckLake clause directly:

 ```sql
-FROM dl.orders_daily AT (VERSION => $WM_UPSTREAM_SNAPSHOT)
+FROM dl.orders_daily AT (VERSION => 42)
 ```

-The cascade already threads a `trigger` blob (producer path, partition) to each
-subscriber; add the producer's captured `snapshot_id` to it, and a consumer's
-read is pinned to the upstream state at dispatch time. That makes the *whole
-pipeline* reproducible and time-travelable — something dbt has no native answer
-for (dbt models are always "whatever's in the warehouse now"). It also gives
-rollback (re-point an asset to snapshot N) and "what did this table look like at
-the failing run" debugging, for free off the same captured ids.
+The asset node's **History** tab is a master-detail view: the snapshot list (id
+ time) on the left selects the version previewed in a read-only grid on the
+right, which surfaces — and copies — the catalog-qualified
+`FROM lake.<table> AT (VERSION => n)` clause. Snapshot ids are captured automatically; the user opts
+into pinning when they want it, and the clause degrades to "latest" if removed,
+so the same script still runs standalone. Mechanically this rides on
+time-travel **reads** (`make_select_query` / `make_count_query` emit the `AT`
+clause when a `version` is threaded through the `WM_INTERNAL_DB_*` markers) plus
+a `DUCKLAKE_SNAPSHOTS` read for the history list — capabilities DuckLake already
+has, no new write path.

-This is the differentiator worth leaning on. It is not catch-up to dbt; it is a
-capability dbt structurally cannot offer, and DuckLake gives it to us at the
-cost of recording one integer per run.
+### Deferred: automatic snapshot pinning across the cascade
+
+An earlier sketch had the cascade *automatically* thread each producer's
+`snapshot_id` into the `trigger` blob and inject `AT (VERSION => $WM_UPSTREAM_SNAPSHOT)`
+into consumer reads, so a whole run is pinned to upstream state at dispatch time
+without anyone asking. This is deliberately **not** built, for three reasons:
+
+- **Not critical.** The only thing it adds over the explicit surface above is
+  *automatic per-run consistency* — protection against an upstream
+  re-materializing in the window between dispatch and a consumer reading. That
+  race only bites high-frequency event-driven cascades (rare today), and the
+  read is always a whole, ACID snapshot regardless — never corruption, just
+  "newer than the triggering version". Debugging and rollback are already
+  covered by the explicit surface.
+- **Implicit magic.** Auto-injecting an `AT` clause and stripping it on
+  standalone runs is invisible behaviour to debug when it misfires; the explicit
+  clause is inspectable.
+- **Multi-upstream ambiguity + EE coupling.** A consumer reading two ducklake
+  upstreams needs a per-ref snapshot *map* accumulated across the AND-join — and
+  the join-slot logic is EE. A single `$WM_UPSTREAM_SNAPSHOT` would silently pin
+  every read to one (the firing) producer's snapshot.
+
+If a workload ever shows the consistency race in practice, pinning can be layered
+on top — the capture and the snapshot surfacing built here are its foundation.

 It also means **we do not build SCD2 snapshots** (gap #4 in `pipelines-vs-dbt.md`):
 DuckLake time-travel is a strictly better answer for most of what dbt's
 `{% snapshot %}` is used for. One fewer engine to write.

+## Data tests (`// data_test`) — and the extensible-annotation pattern
+
+Data tests are the first dbt-parity gap closed on top of materialization, and
+the **first deliberately extensible annotation**. The design goal was not just
+"add five test types" but to establish the convention a sibling family
+(column-lineage is the next one) follows, so the annotation vocabulary stops
+being a closed hardcoded list (`pipelines-vs-dbt.md` gap #7).
+
+### Grammar
+
+```
+// data_test unique <col>
+// data_test not_null <col>
+// data_test accepted_values <col> = a,b,c
+// data_test relationships <col> -> datatable://other/asset.<col>
+// data_test <script_path>            ← escape hatch (dbt's singular test)
+```
+
+`// data_test` lines **accumulate** (every well-formed line adds one check),
+unlike the single-value annotations (`// materialize`, `// partitioned`, …)
+which are first-write-wins. Malformed lines are dropped fail-safe — a typo
+becomes an *absent* check (visible in the graph), never a mis-parsed one.
+
+The keyword is `data_test`, **not `test`** — there is an unrelated, shipped
+`// test:` CI-test annotation (`windmill_common::schema::parse_ci_test_annotation`,
+tests a script's *logic* on deploy). `data_test` tests the *data* in the
+materialized asset at run time. This mirrors dbt 1.8's own `tests:` →
+`data_tests:` rename, made for exactly this disambiguation.
+
+### The pattern: annotation → verifier
+
+The reusable shape, in three layers, each a clean extension seam:
+
+1. **Parse** (`asset_parser.rs` + `parsePipelineAnnotations.ts`, kept in
+   lockstep by the parity corpus). A `data_test` line is dispatched on a
+   **keyword head** to a typed variant (`DataTest`). A new built-in is one
+   match arm + its sub-parser; the `Custom` arm is the open fallback. A sibling
+   family reuses this head-keyword dispatch rather than adding a parallel list.
+2. **Compile** (`sql_materialize.rs::build_data_test_checks`). Each test becomes
+   a **check**: `(name, violating-row-count query)`. Built-ins differ only in
+   their count query; `Custom` supplies its own (the user's SELECT of violating
+   rows). Referenced assets (relationships) emit an `ATTACH` resolved by the
+   same transform pass as the user's own.
+3. **Execute** (`duckdb_executor.rs`). The materialize summary query embeds
+   every check's count in one `data_tests` list-of-struct column (computed in a
+   CTE, since DuckDB rejects subqueries inside struct literals), so **all tests
+   run in a single pass** against the freshly-materialized slice — no
+   abort-on-first. The worker reads the breakdown from the result and decides
+   pass/fail: any violation **fails the run** (record `Failed`, propagate up the
+   cascade) with an error listing *every* test (✓/✗ + counts); a clean run
+   returns the per-test summary so the UI can render a checklist.
+
+A new annotation family that produces post-materialize checks (or, for
+column-lineage, post-materialize *metadata reads*) plugs into the same three
+seams: add a parsed variant, emit its check/reader SQL into the summary, read
+it back in the worker. Nothing about the closed set of *today's* keywords is
+load-bearing.
+
+### Scoping decisions (v1)
+
+- **Partition scope.** When `// partitioned`, built-in checks are scoped to the
+  slice just written (`WHERE _wm_partition = <value>`), so a rerun/backfill of
+  one partition is independent of other partitions' (possibly pre-existing)
+  data. Whole-table assertions are a follow-up.
+- **Commit-then-test.** Like dbt, the write commits before tests run; a failed
+  test fails the *run* (and records `Failed`, so downstream cascade stops) but
+  does not roll back the committed snapshot. Time-travel still lets you inspect
+  exactly what failed.
+- **Custom = DuckDB SQL, server worker.** The escape hatch fetches the deployed
+  script's content (a single DuckDB `SELECT`/CTE returning the violating rows —
+  it's embedded as a subquery, so a multi-statement body is rejected with a
+  clear error) and inlines it as a check; `{partition}` is substituted and
+  `_wm_target` is in scope. Agent (Http) workers — which have no script cache —
+  get a clear error. Non-DuckDB custom tests (dispatched as sub-jobs, any
+  language) are the natural follow-up and fit the same verifier seam.
+- **Managed only.** `// materialize manual` + `// data_test` is rejected with a
+  clear error (we can't know the manual script's target alias / partition col).
+
 ## Scoping decision: DuckLake vs DataTable

 **Make DuckLake the materialization/versioning substrate; keep DataTable as the
@@ -224,8 +336,10 @@ Don't try to give both the full treatment for v1.
   `materialized_partition` rows.
 5. **Surface it** — last-materialized/snapshot/row-count on the asset node;
   missing-partition set feeds the backfill UI.
-6. *v1.x* — snapshot pinning across the cascade (`$WM_UPSTREAM_SNAPSHOT`),
-   rollback, time-travel read helper.
+6. *v1.x* — time-travel UX over the captured snapshots: a per-asset **History**
+   tab — a master-detail snapshot list + query-at-version preview that copies the
+   full `FROM lake.<table> AT (VERSION => n)` clause. Automatic cascade pinning
+   (`$WM_UPSTREAM_SNAPSHOT`) is deferred — see §"Reproducibility" for why.

 Steps 1–5 are a thin annotation+template layer plus one metadata table and one
 extra read per run. They deliver managed/incremental/versioned assets,
@@ -46,7 +46,7 @@ Asset-centric, polyglot, annotation-driven, event-aware:

 | Gap | Architectural blocker? | Verdict |
 |---|---|---|
-| Data tests | No | Pure TODO |
+| Data tests | No | **Shipped** (`// data_test`) |
 | Incremental materializations | No, but pick a philosophy | TODO with design decision |
 | Column lineage + docs site | No | Pure TODO |
 | Snapshots / SCD2 | No | New output kind |
@@ -64,10 +64,16 @@ and annotation extensibility**. The rest is execution.
 dbt: `unique`, `not_null`, `accepted_values`, custom generic tests, plus
 singular tests. Run as `SELECT` statements that pass when they return 0 rows.

-Today: nothing. Annotation parser is the natural hook —
-`// test unique col_name`, `// test not_null col_name`,
-`// test <script_path>` for custom. Pipeline runtime already handles
-failure propagation. Lowest-risk, highest-payoff item.
+**Shipped** via the `// data_test` annotation (built on materialization):
+`// data_test unique <col>`, `not_null`, `accepted_values <col> = a,b,c`,
+`relationships <col> -> <asset>.<col>`, and `// data_test <script_path>` for
+the custom escape hatch (dbt's singular test). Each compiles to a SQL
+*verifier probe* that runs against the freshly-materialized asset and raises on
+violation, riding the existing failure-propagation path. This is also the first
+*extensible* annotation — see `ducklake-materialization.md` §"Data tests" for
+the annotation→verifier pattern that column-lineage will reuse. The keyword is
+`data_test`, not `test`, to stay clear of the unrelated `// test:` CI-test
+annotation.

 ### 2. Incremental materializations

@@ -20,11 +20,11 @@
    },
    "nixpkgs": {
      "locked": {
-        "lastModified": 1764517877,
-        "narHash": "sha256-pp3uT4hHijIC8JUK5MEqeAWmParJrgBVzHLNfJDZxg4=",
+        "lastModified": 1768377964,
+        "narHash": "sha256-RU35vQnfg9NwJUviGCfMH9ChgHANoNSiRaAn4/wINT4=",
        "owner": "NixOS",
        "repo": "nixpkgs",
-        "rev": "2d293cbfa5a793b4c50d17c05ef9e385b90edf6c",
+        "rev": "cadda13afe838615fb74b0a9720905920559c535",
        "type": "github"
      },
      "original": {
@@ -1,6 +1,6 @@
 {
 	"name": "@windmill-labs/components",
-	"version": "1.737.0",
+	"version": "1.739.0",
 	"scripts": {
 		"dev": "vite dev",
 		"dev:ui-builder": "mv static/ui_builder static/ui_builder.dev-disabled 2>/dev/null || true ; trap 'mv static/ui_builder.dev-disabled static/ui_builder 2>/dev/null || true' EXIT ; vite dev",
@@ -127,6 +127,7 @@
 		"lru-cache": "^11.1.0",
 		"lucide-svelte": "^0.540.0",
 		"mdast-util-find-and-replace": "^3.0.2",
+		"mermaid": "^11.15.0",
 		"minimatch": "^10.0.1",
 		"monaco-editor": "npm:@codingame/monaco-vscode-editor-api@=25.0.0",
 		"monaco-languageclient": "10.6.0",
@@ -1,5 +1,5 @@
 {
 	"baseUrl": "https://pub-06154ed168a24e73a86ab84db6bf15d8.r2.dev",
-	"version": "ad76918",
-	"sha256": "4c940570936a391c217a054b74c691587d7f27d012ccddbf5e3b3d25e7c1fe4a"
+	"version": "062d11c",
+	"sha256": "355bf3acfb935080e4a4fa23d0509e03e4bc01aeed622db110663d7ec6c1c438"
 }
@@ -0,0 +1,51 @@
+<script lang="ts">
+	// Renders the per-test breakdown a managed `// materialize` run attaches to
+	// its result as `data_tests: [{ test, violating }]`. Shown above the raw
+	// result so "which tests ran, and which passed/failed" is clear at a glance.
+	// (On a failed run the job result is the error message — which already lists
+	// every test — so this success-path checklist and that error text together
+	// cover both outcomes.)
+	import { CheckCircle2, FlaskConical, XCircle } from 'lucide-svelte'
+
+	let { tests }: { tests: Array<{ test: string; violating: number }> } = $props()
+
+	let failed = $derived(tests.filter((t) => t.violating > 0).length)
+</script>
+
+<div
+	class="mb-2 rounded-md border bg-surface-secondary text-xs overflow-hidden {failed > 0
+		? 'border-red-300 dark:border-red-900/60'
+		: 'border-border'}"
+>
+	<div class="flex items-center gap-1.5 px-2 py-1 border-b border-border text-secondary">
+		<FlaskConical size={13} />
+		<span class="font-medium">
+			{tests.length} data test{tests.length === 1 ? '' : 's'}
+		</span>
+		<span
+			class={failed > 0
+				? 'text-red-600 dark:text-red-400'
+				: 'text-emerald-600 dark:text-emerald-400'}
+		>
+			· {failed > 0 ? `${failed} failed` : 'all passed'}
+		</span>
+	</div>
+	<ul class="divide-y divide-border">
+		{#each tests as t (t.test)}
+			<li class="flex items-center gap-1.5 px-2 py-1 font-mono">
+				{#if t.violating > 0}
+					<XCircle size={13} class="shrink-0 text-red-600 dark:text-red-400" />
+					<span class="sr-only">failed:</span>
+					<span class="text-primary">{t.test}</span>
+					<span class="text-red-600 dark:text-red-400"
+						>— {t.violating} violating row{t.violating === 1 ? '' : 's'}</span
+					>
+				{:else}
+					<CheckCircle2 size={13} class="shrink-0 text-emerald-600 dark:text-emerald-400" />
+					<span class="sr-only">passed:</span>
+					<span class="text-secondary">{t.test}</span>
+				{/if}
+			</li>
+		{/each}
+	</ul>
+</div>
@@ -20,6 +20,7 @@
 		Loader2
 	} from 'lucide-svelte'
 	import DucklakeResultPreview from './assets/AssetGraph/DucklakeResultPreview.svelte'
+	import DataTestsResult from './DataTestsResult.svelte'
 	import Portal from '$lib/components/Portal.svelte'
 	import DisplayResultControlBar from './DisplayResultControlBar.svelte'

@@ -564,10 +565,54 @@
 			resultHeaderHeight
 		)
 	})
+
+	// Per-test breakdown of a managed `// materialize` run, rendered as a
+	// checklist above the raw result. On success it rides the result
+	// (`data_tests: [{ test, violating }]`, a one-row array). On failure the job
+	// result is the error, whose message is the worker's breakdown text — parsed
+	// back into the same shape so the checklist shows on both outcomes. Both
+	// formats are produced by this repo's worker (see duckdb_executor.rs); the
+	// derivation is inert (undefined) for every other DisplayResult use.
+	let dataTests = $derived.by(() => {
+		// Success: structured column on the summary row.
+		const row = Array.isArray(result) ? (result as any)?.[0] : (result as any)
+		let dt = row?.data_tests
+		if (typeof dt === 'string') {
+			try {
+				dt = JSON.parse(dt)
+			} catch {
+				dt = undefined
+			}
+		}
+		if (
+			Array.isArray(dt) &&
+			dt.length > 0 &&
+			dt.every((x) => x && typeof x.test === 'string' && typeof x.violating === 'number')
+		) {
+			return dt as Array<{ test: string; violating: number }>
+		}
+		// Failure: parse the worker's breakdown out of the error message.
+		const msg = (result as any)?.error?.message
+		if (typeof msg === 'string' && msg.includes('data tests failed on')) {
+			const out: Array<{ test: string; violating: number }> = []
+			for (const line of msg.split('\n')) {
+				const fail = line.match(/^\s*✗\s*(.+?)\s*—\s*(\d+)\s+violating/)
+				const pass = line.match(/^\s*✓\s*(.+?)\s*$/)
+				if (fail) out.push({ test: fail[1], violating: parseInt(fail[2], 10) })
+				else if (pass) out.push({ test: pass[1], violating: 0 })
+			}
+			if (out.length > 0) return out
+		}
+		return undefined
+	})
 </script>

 <HighlightTheme />

+{#if dataTests}
+	<DataTestsResult tests={dataTests} />
+{/if}
+
 {#if result_stream && result == undefined}
 	<div class="flex flex-col w-full gap-2">
 		<div class="flex items-center gap-2 text-secondary text-xs">
@@ -30,6 +30,10 @@
 		placement?: Placement
 		usePointerDownOutside?: boolean
 		closeOnOtherDropdownOpen?: boolean
+		// When false the menu stays open after an item is selected (melt's closeOnItemClick).
+		// Consumers that keep the menu open must close it themselves where appropriate.
+		// Read once at menu creation (like `placement`); changing it after mount has no effect.
+		closeOnItemClick?: boolean
 		fixedHeight?: boolean
 		hidePopup?: boolean
 		open?: boolean
@@ -41,10 +45,18 @@
 		size?: ButtonType.UnifiedSize
 		btnText?: string
 		buttonReplacement?: import('svelte').Snippet
-		// In customMenu mode the snippet receives the melt-ui `item` action
-		// store so consumers can wrap their own rows in <MenuItem> (or
-		// `use:melt={$item}`) and get arrow-key navigation + aria wiring.
-		menu?: import('svelte').Snippet<[{ item: MenubarMenuElements['item']; close: () => void }]>
+		// In customMenu mode the snippet receives the melt-ui `item` action store
+		// (so consumers can wrap rows in <MenuItem> for arrow-key navigation + aria)
+		// and `builders` (so they can compose melt submenus, e.g. via DropdownSubmenuItem).
+		menu?: import('svelte').Snippet<
+			[
+				{
+					item: MenubarMenuElements['item']
+					close: () => void
+					builders: ReturnType<typeof createDropdownMenu>['builders']
+				}
+			]
+		>
 		maxHeight?: string | undefined
 	}

@@ -56,6 +68,7 @@
 		placement = 'bottom-end',
 		usePointerDownOutside = false,
 		closeOnOtherDropdownOpen = true,
+		closeOnItemClick = true,
 		fixedHeight = true,
 		hidePopup = false,
 		open = $bindable(false),
@@ -82,6 +95,7 @@
 		positioning: {
 			placement: untrack(() => placement)
 		},
+		closeOnItemClick: untrack(() => closeOnItemClick),
 		loop: true,
 		onOpenChange: ({ next }) => {
 			if (closeOnOtherDropdownOpen) {
@@ -176,7 +190,7 @@
 		transition:fly={{ duration: enableFlyTransition ? 100 : 0, y: -16 }}
 	>
 		{#if customMenu}
-			{@render menu?.({ item, close })}
+			{@render menu?.({ item, close, builders })}
 		{:else}
 			<div
 				class="bg-surface-tertiary dark:border w-56 origin-top-right rounded-lg shadow-lg focus:outline-none overflow-y-auto py-1"
@@ -158,11 +158,6 @@
 		preparedAssetsSqlQueries?: InferAssetsSqlQueryDetails[] | undefined
 		// To execute preview scripts with the right worker group
 		customTag?: string
-		// Opt-in: reflect external `code` prop mutations back into Monaco (see
-		// the effect below). One-way `code={...}` callers that need live
-		// external updates — e.g. the inline flow rawscript — set this. Off by
-		// default so every other caller's behavior is unchanged.
-		syncExternalCode?: boolean
 	}

 	let {
@@ -195,8 +190,7 @@
 		enablePreprocessorSnippet = false,
 		rawAppRunnableKey = undefined,
 		preparedAssetsSqlQueries,
-		customTag,
-		syncExternalCode = false
+		customTag
 	}: Props = $props()

 	$effect.pre(() => {
@@ -375,23 +369,12 @@
 			code = ncode
 		}

-		if (noHistory) {
-			editor?.setValue(ncode)
-		} else {
-			if (editor?.getModel()) {
-				// editor.setValue(ncode)
-				editor.pushUndoStop()
-
-				editor.executeEdits('set', [
-					{
-						range: editor.getModel()!.getFullModelRange(), // full range
-						text: ncode
-					}
-				])
-
-				editor.pushUndoStop()
-			}
-		}
+		// setCode is an authoritative overwrite (reset, AI apply, module switch).
+		// Cancel any in-flight keystroke debounce first: otherwise alignCodeWithEditor
+		// skips on the `timeoutModel` guard (leaving Monaco stale), and the pending
+		// updateCode later reads the old buffer and writes it back over `ncode`.
+		cancelPendingChanges()
+		alignCodeWithEditor(!noHistory)
 		// Dispatch change immediately when code actually changed. This ensures
 		// callers like the Reset button and copilot trigger on:change handlers.
 		// The debounced onDidChangeModelContent handler will no-op since code
@@ -425,6 +408,7 @@
 			return
 		}
 		code = ncode
+		lastEditorCode = ncode
 		dispatch('change', ncode)
 	}

@@ -436,12 +420,19 @@
 	 * see it. Clears the chain state so the next keystroke after this
 	 * flush is a fresh leading fire. */
 	export function flushPendingChanges(): void {
+		cancelPendingChanges()
+		updateCode()
+	}
+
+	/** Discard any in-flight keystroke debounce without materializing it, so a
+	 * deferred updateCode can't fire later. Resets chain state to a fresh leading
+	 * fire on the next keystroke. */
+	function cancelPendingChanges(): void {
 		if (timeoutModel !== undefined) {
 			clearTimeout(timeoutModel)
 			timeoutModel = undefined
 		}
 		changeChainStart = undefined
-		updateCode()
 	}

 	export function append(code: string): void {
@@ -1901,29 +1892,6 @@
 		lang = scriptLangToEditorLang(scriptLang)
 	})

-	// Opt-in (syncExternalCode): reflect external `code` prop mutations into
-	// Monaco's model. Parents that pass `code={...}` one-way (no bind) — e.g.
-	// the inline rawscript in the flow editor — otherwise mutate the prop
-	// without Monaco ever showing the change (the AI chat editing a flow
-	// module's content in a session is the motivating case). Gated off by
-	// default: Editor is sensitive and most callers either bind:code (and
-	// carry their own external-sync) or treat code as init-only, so a blanket
-	// setValue would risk clobbering them. The `getValue() !== code` guard
-	// keeps the caret intact when the change originated from typing inside
-	// Monaco (which round-trips code back via `$bindable`, re-firing this
-	// effect with `code === getValue()`).
-	let lastExternalCodeSync = code
-	$effect(() => {
-		if (!syncExternalCode) return
-		if (code === lastExternalCodeSync) return
-		lastExternalCodeSync = code
-		if (!editor) return
-		untrack(() => {
-			if (editor!.getValue() !== code) {
-				editor!.setValue(code ?? '')
-			}
-		})
-	})
 	$effect(() => {
 		filePath = computePath(path)
 	})
@@ -2011,25 +1979,50 @@
 		})
 	})

-	// External `code` prop changes should flow into the Monaco editor. The
-	// `untrack` block reads/writes Monaco without subscribing — only the
-	// prop read above is tracked — so the editor's own change handler
-	// (`updateCode`) re-running with the same value short-circuits and we
-	// don't loop.
-	$effect(() => {
-		const next = code ?? ''
+	let applyExternalCode = useDebounce(() => alignCodeWithEditor(true), 800)
+
+	// Last `code` value the editor itself produced or aligned to. Used to tell an
+	// echo (the bindable changed because the user typed — Monaco is already
+	// ahead) from a genuine external write. Without this, a typing burst longer
+	// than the debounce window would sync the lagging `code` back over newer
+	// keystrokes. Must be kept in step with every editor↔`code` sync point.
+	let lastEditorCode = code
+
+	function alignCodeWithEditor(history: boolean) {
 		const ed = editor
 		if (!ed) return
-		untrack(() => {
-			if (ed.getValue() === next) return
-			const model = ed.getModel()
-			if (!model) return
+		const next = code ?? ''
+		const value = ed.getValue()
+		const model = ed.getModel()
+		// Some keystrokes are still being debounced, don't overwrite them.
+		// When the debounce is done, updateCode will be called and the code will be aligned with the editor.
+		if (timeoutModel !== undefined) return
+		if (!model) return
+		lastEditorCode = next
+		if (value === next) return
+		if (history) {
 			ed.pushUndoStop()
 			ed.executeEdits('external', [{ range: model.getFullModelRange(), text: next }])
 			ed.pushUndoStop()
+		} else {
+			ed.setValue(next)
+		}
+	}
+
+	// External `code` prop changes should flow into the Monaco editor. Skip
+	// echoes: when `code` matches what the editor last produced (`updateCode`)
+	// or aligned to, the change came from the editor itself, so syncing back
+	// would clobber input typed since. Only genuine external writes — where
+	// `code` diverges from `lastEditorCode` — schedule a sync. The `untrack`
+	// block reads/writes Monaco without subscribing, so we don't loop.
+	$effect(() => {
+		;[code, editor]
+		if (!editor) return
+		untrack(() => {
+			if (code === lastEditorCode) return
+			applyExternalCode()
 		})
 	})
-
 	let isTsWorkerInitialized = resource([() => lang, () => initialized], async () => {
 		if (lang !== 'typescript' || !initialized) return false
 		// Use the stable model URI (computed once at mount), not filePath which changes on rename
@@ -1180,15 +1180,25 @@
 	export type FlowModuleForTimeline = {
 		id: string
 		type: FlowModuleValue['type']
+		suspend?: boolean
 	}

 	function allModulesForTimeline(
 		modules: FlowModule[],
 		expandedSubflows: Record<string, { modules: FlowModule[]; groups?: any[] }>
 	): FlowModuleForTimeline[] {
-		const ids = dfs(modules, (x) => ({ id: x.id, type: x.value.type }) as FlowModuleForTimeline, {
-			skipToolNodes: true
-		})
+		const ids = dfs(
+			modules,
+			(x) =>
+				({
+					id: x.id,
+					type: x.value.type,
+					suspend: x.suspend != undefined
+				}) as FlowModuleForTimeline,
+			{
+				skipToolNodes: true
+			}
+		)

 		function rec(
 			ids: FlowModuleForTimeline[],
@@ -1208,7 +1218,8 @@
 									fms,
 									(x) => ({
 										id: x.id.startsWith('subflow:') ? x.id : buildSubflowKey(x.id, nprefix),
-										type: x.value.type
+										type: x.value.type,
+										suspend: x.suspend != undefined
 									}),
 									{ skipToolNodes: true }
 								),
@@ -72,6 +72,66 @@
 	}

 	const barHeight = 32
+
+	// Whole approval-wait machinery below is inert unless a step actually has a suspend config,
+	// so large suspend-free flows pay nothing for the flatten/sort and the per-tick recompute.
+	const hasSuspendModule = $derived(flowModules.some((m) => m.suspend))
+
+	// Push times of every job on the timeline, ascending. Used to locate when the step
+	// that follows an approval step started — i.e. the moment the approval was granted.
+	const allCreatedAts = $derived(
+		hasSuspendModule
+			? Object.values(items ?? {})
+					.flat()
+					.map((j) => j.created_at)
+					.filter((t): t is number => t != undefined)
+					.sort((a, b) => a - b)
+			: []
+	)
+
+	// Heuristic: the grant moment is approximated by the next job pushed anywhere on the
+	// timeline. Exact for sequential flows; for an approval step inside one branch of a
+	// parallel branchall a concurrent sibling job can land first and understate the wait.
+	function nextCreatedAtAfter(t: number): number | undefined {
+		return allCreatedAts.find((c) => c > t)
+	}
+
+	// For a completed suspend/approval step, the time spent waiting for the approval is the
+	// gap between the step finishing and the next step being pushed (or now, if still waiting).
+	function approvalWait(b: {
+		started_at?: number
+		duration_ms?: number
+	}): { start: number; len: number; running: boolean } | undefined {
+		if (b.started_at == undefined || b.duration_ms == undefined) {
+			return undefined
+		}
+		const end = b.started_at + b.duration_ms
+		const next = nextCreatedAtAfter(end)
+		const waitEnd = next ?? (flowDone ? undefined : now)
+		if (waitEnd == undefined) {
+			return undefined
+		}
+		const len = waitEnd - end
+		if (len < 100) {
+			return undefined
+		}
+		return { start: end, len, running: next == undefined }
+	}
+
+	// Approval wait per module id, computed once and consumed by both the rows and the legend.
+	const approvalWaitByModule = $derived.by(() => {
+		const result: Record<string, { start: number; len: number; running: boolean }> = {}
+		for (const m of flowModules) {
+			if (!m.suspend) continue
+			const sub = (items?.[m.id] ?? []).filter((x) => x.created_at && x.started_at)
+			if (sub.length !== 1) continue
+			const aw = approvalWait(sub[0])
+			if (aw) result[m.id] = aw
+		}
+		return result
+	})
+
+	const hasApprovalWait = $derived(Object.keys(approvalWaitByModule).length > 0)
 </script>

 <OnChange
@@ -95,6 +155,12 @@
 					<div class="h-2.5 w-2.5 rounded-sm bg-blue-500/90"></div>
 					<span>Execution</span>
 				</div>
+				{#if hasApprovalWait}
+					<div class="flex gap-1.5 items-center">
+						<div class="h-2.5 w-2.5 rounded-sm bg-purple-400/80"></div>
+						<span>Approval wait</span>
+					</div>
+				{/if}
 				{#if max && min}
 					<span class="font-mono">{msToSec(max - min, 1)}s</span>
 				{/if}
@@ -113,7 +179,7 @@
 				/>
 			</div>
 		{/if}
-		{#each flowModules as { id: k, type: typ } (k)}
+		{#each flowModules as { id: k, type: typ, suspend: isSuspend } (k)}
 			{@const subItems = items?.[k]?.filter((x) => x.created_at && x.started_at)}
 			<div class="relative px-3 py-1.5">
 				<div class="flex items-center justify-between mb-0.5">
@@ -161,6 +227,7 @@
 												? 0
 												: now - b?.created_at
 										: 0}
+									{@const aw = isSuspend ? approvalWaitByModule[k] : undefined}
 									<div class="flex w-full py-0.5 items-center" {style}>
 										<TimelineBar
 											position="left"
@@ -175,7 +242,7 @@
 										/>
 										{#if b.started_at}
 											<TimelineBar
-												position={waitingLen < 100 ? 'center' : 'right'}
+												position={aw || waitingLen < 100 ? 'center' : 'right'}
 												id={b?.id}
 												{total}
 												{min}
@@ -185,6 +252,20 @@
 												running={b?.duration_ms == undefined}
 											/>
 										{/if}
+										{#if aw}
+											<TimelineBar
+												position="right"
+												id={b?.id}
+												{total}
+												{min}
+												concat
+												colorClass="bg-purple-400/80"
+												tooltip={`Waiting for approval — ${msToSec(aw.len, 1)}s`}
+												started_at={aw.start}
+												len={aw.len}
+												running={aw.running}
+											/>
+										{/if}
 									</div>
 								{:else}
 									<div class="flex w-full py-0.5"></div>
@@ -196,7 +277,6 @@
 			</div>
 		{/each}
 	</div>
-
 {:else}
 	<Loader2 class="animate-spin" />
 {/if}
@@ -12,10 +12,14 @@
 	let comparison: WorkspaceComparison | undefined = $state(undefined)
 	let error: string | undefined = $state(undefined)

-	let isFork = $derived($workspaceStore?.startsWith('wm-fork-') ?? false)
 	let currentWorkspaceData = $derived($userWorkspaces.find((w) => w.id === $workspaceStore))
 	let parentWorkspaceId = $derived(currentWorkspaceData?.parent_workspace_id)
 	let parentWorkspaceData = $derived($userWorkspaces.find((w) => w.id === parentWorkspaceId))
+	// A fork must have a parent to compare/merge against. Treating the wm-fork-
+	// prefix alone as "is a fork" renders a parentless "Fork of ()" banner when
+	// the parent linkage was dropped (e.g. by a workspace id change), so require
+	// both, matching the forks/compare page.
+	let isFork = $derived(($workspaceStore?.startsWith('wm-fork-') ?? false) && !!parentWorkspaceId)

 	// Drafts in this fork. When the fork is otherwise in sync with its parent, a
 	// user with only pending drafts should still get the draft CTA (mirrors the
@@ -1063,6 +1063,14 @@
 						<li
 							>AI chat usage (provider, model, mode, session count, message count — last 30 days)</li
 						>
+						<li
+							>resource counts (workspaces, scripts per language, flows, workflows as code, low-code
+							apps, raw apps)</li
+						>
+						<li
+							>infrastructure info (container runtime, managed database provider, database version,
+							size and cluster size, max and active connections, object storage backend)</li
+						>
 					</ul>
 					<br />For air-gapped instances, you can download the telemetry data and send it manually.
 				</div>
@@ -1101,6 +1109,10 @@
 						<li
 							>AI chat usage (provider, model, mode, session count, message count — last 30 days)</li
 						>
+						<li
+							>resource counts (workspaces, scripts per language, flows, workflows as code, low-code
+							apps, raw apps)</li
+						>
 					</ul>
 				</div>
 			{/if}
@@ -952,9 +952,7 @@
 			}
 		})
 	})
-	$effect(() => {
-		readFieldsRecursively(script)
-	})
+
 	// Mirror the draft triggers (held in a separate `triggersState` $state)
 	// back into `script.draft_triggers` so the UserDraft autosave — which
 	// deep-tracks `script` — picks them up. Pre-PR ScriptBuilder ran its own
@@ -52,6 +52,7 @@
 		Play,
 		PlayIcon,
 		Plus,
+		Target,
 		Terminal,
 		Pencil,
 		WandSparkles,
@@ -268,6 +269,7 @@
 		if (activeModuleTab === null && code !== lastSyncedCode) {
 			editorCode = code
 			lastSyncedCode = code
+			editor?.setCode(editorCode) // immediate sync, don't wait for the 800ms debounce
 			untrack(() => inferSchema(code))
 		}
 	})
@@ -1591,18 +1593,29 @@
 	let error = $derived(getError(testJob))

 	$effect(() => {
-		const options: ScriptOptions = {
-			code,
-			lang: lang as ScriptLang,
-			error,
-			args: args ?? {},
-			path,
+		;[
+			editor,
 			lastSavedCode,
 			lastDeployedCode,
 			diffMode,
-			workflowAsCode: workflowAsCodeAiContext
-		}
+			workflowAsCodeAiContext,
+			args,
+			error,
+			lang,
+			path
+		]
 		untrack(() => {
+			const options: ScriptOptions = {
+				getCode: () => code,
+				lang: lang as ScriptLang,
+				error,
+				args: args ?? {},
+				path,
+				lastSavedCode,
+				lastDeployedCode,
+				diffMode,
+				workflowAsCode: workflowAsCodeAiContext
+			}
 			aiChatManager.scriptEditorOptions = options
 			aiChatManager.scriptEditorApplyCode = async (code: string, opts?: ReviewChangesOpts) => {
 				hideDiffMode()
@@ -1900,15 +1913,18 @@
 								<div class="absolute top-1 left-2 z-10">
 									{#if testIsLoading}
 										{@render cancelTestButton('sm', 'shadow-md')}
-									{:else if (customUi?.previewPanel?.downstreamSubscribers ?? 0) > 0}
+									{:else if (customUi?.previewPanel?.downstreamSubscribers ?? 0) > 0 || customUi?.previewPanel?.onBoundedRun}
 										<!-- Split button: primary "Test" runs just this step
 									     (skips the asset-trigger cascade); the caret
 									     opens a popover with the cascade option labelled
 									     by the downstream count. The active mode is
 									     reflected in both the button label and the
 									     check-mark on the menu item so the user always
-									     knows whether the next run will fan out. -->
-										{@const downstream = customUi!.previewPanel!.downstreamSubscribers!}
+									     knows whether the next run will fan out. A
+									     pure-reader-only root has no subscriber downstream
+									     but still gets `onBoundedRun`, so the split button
+									     also opens for it (with the cascade item hidden). -->
+										{@const downstream = customUi?.previewPanel?.downstreamSubscribers ?? 0}
 										<div class="flex items-stretch shadow-md rounded-md overflow-hidden">
 											<Button
 												on:click={() => runTest()}
@@ -1971,31 +1987,55 @@
 																>
 															</div>
 														</button>
-														<button
-															type="button"
-															class="w-full text-left px-3 py-2 hover:bg-surface-hover flex items-start gap-2"
-															onclick={() => {
-																cascadeDownstream = true
-																close()
-																void runTest()
-															}}
-														>
-															<Zap
-																size={14}
-																class="mt-0.5 shrink-0 text-amber-600 dark:text-amber-400"
-															/>
-															<div class="flex flex-col min-w-0">
-																<span class="font-medium">
-																	Test + trigger {downstream} downstream {cascadeDownstream
-																		? '(current)'
-																		: ''}
-																</span>
-																<span class="text-2xs text-secondary">
-																	Let the asset-trigger cascade fan out to the {downstream}
-																	subscribed script{downstream === 1 ? '' : 's'} after this run succeeds.
-																</span>
-															</div>
-														</button>
+														{#if downstream > 0}
+															<button
+																type="button"
+																class="w-full text-left px-3 py-2 hover:bg-surface-hover flex items-start gap-2"
+																onclick={() => {
+																	cascadeDownstream = true
+																	close()
+																	void runTest()
+																}}
+															>
+																<Zap
+																	size={14}
+																	class="mt-0.5 shrink-0 text-amber-600 dark:text-amber-400"
+																/>
+																<div class="flex flex-col min-w-0">
+																	<span class="font-medium">
+																		Test + trigger {downstream} downstream {cascadeDownstream
+																			? '(current)'
+																			: ''}
+																	</span>
+																	<span class="text-2xs text-secondary">
+																		Let the asset-trigger cascade fan out to the {downstream}
+																		subscribed script{downstream === 1 ? '' : 's'} after this run succeeds.
+																	</span>
+																</div>
+															</button>
+														{/if}
+														{#if customUi?.previewPanel?.onBoundedRun}
+															<button
+																type="button"
+																class="w-full text-left px-3 py-2 hover:bg-surface-hover flex items-start gap-2 border-t"
+																onclick={() => {
+																	close()
+																	customUi!.previewPanel!.onBoundedRun!()
+																}}
+															>
+																<Target
+																	size={14}
+																	class="mt-0.5 shrink-0 text-blue-600 dark:text-blue-400"
+																/>
+																<div class="flex flex-col min-w-0">
+																	<span class="font-medium">Run downstream up to…</span>
+																	<span class="text-2xs text-secondary">
+																		Pick end node(s) on the graph, then run only the cascade between
+																		this script and them.
+																	</span>
+																</div>
+															</button>
+														{/if}
 													</div>
 												{/snippet}
 											</Popover>
@@ -10,7 +10,7 @@
 	import { base } from '$lib/base'
 	import SearchItems from './SearchItems.svelte'
 	import { page } from '$app/state'
-	import { goto as gotoUrl } from '$app/navigation'
+	import { replaceState } from '$app/navigation'
 	import Version from './Version.svelte'
 	import Uptodate from './Uptodate.svelte'
 	import InstanceSettings from './InstanceSettings.svelte'
@@ -70,7 +70,16 @@
 		const index = page.url.href.lastIndexOf('#')
 		if (index === -1) return
 		const hashRemoved = page.url.href.slice(0, index)
-		gotoUrl(hashRemoved)
+		// Strip the drawer's URL hash without a SvelteKit navigation: a `goto`
+		// here re-fires path-reactive effects on the underlying page (e.g. the
+		// script editor's load effect), wiping unsaved editor content.
+		try {
+			replaceState(hashRemoved, page.state)
+		} catch (e) {
+			// replaceState throws if the router isn't initialized yet — possible
+			// when onDestroy runs during router teardown.
+			console.error(e)
+		}
 	}

 	onDestroy(() => {
@@ -15,6 +15,10 @@
 		concat?: boolean
 		gray?: boolean
 		spacerClass?: string
+		/** Overrides the default gray/blue bar color (e.g. to mark an approval wait). */
+		colorClass?: string
+		/** Tooltip label shown on hover instead of the default job link. */
+		tooltip?: string
 	}

 	let {
@@ -27,7 +31,9 @@
 		running,
 		concat = false,
 		gray = false,
-		spacerClass = ''
+		spacerClass = '',
+		colorClass = undefined,
+		tooltip = undefined
 	}: Props = $props()
 </script>

@@ -37,20 +43,26 @@
 	{/if}
 	<Popover
 		style="width: {(len / total) * 100}%"
-		class="h-5 relative {gray
-			? 'bg-gray-300 dark:bg-gray-600'
-			: running
-				? 'bg-blue-400/90'
-				: 'bg-blue-500/90'} {position == 'left'
+		class="h-5 relative {colorClass
+			? colorClass
+			: gray
+				? 'bg-gray-300 dark:bg-gray-600'
+				: running
+					? 'bg-blue-400/90'
+					: 'bg-blue-500/90'} {position == 'left'
 			? 'rounded-l-md'
 			: position == 'right'
 				? 'rounded-r-md'
 				: 'rounded-md'} center-center text-white text-2xs whitespace-nowrap hover:outline outline-1 outline-black"
 	>
 		{#snippet text()}
-			<a href="{base}/run/{id}" class="inline-flex items-center gap-1" target="_blank"
-				>{id} <ExternalLink size={14} /></a
-			>
+			{#if tooltip}
+				<span>{tooltip}</span>
+			{:else}
+				<a href="{base}/run/{id}" class="inline-flex items-center gap-1" target="_blank"
+					>{id} <ExternalLink size={14} /></a
+				>
+			{/if}
 		{/snippet}
 		{#if len > 0}
 			{@const narrow = len / total < 0.09}
@@ -4,6 +4,9 @@ Inline diff renderer for a single workspace item. Mirrors the per-kind
 rendering that DiffDrawer does in its body (`DiffDrawer.svelte:181-271`):

 - `flow` → `<FlowDiffViewer>` (its own Graph / YAML toggle inside)
+- `raw_app_file` → `<RawAppFileDiff>` (one synthesized raw-app file item: a
+  single diff with a per-file size guard; the metadata item adds a full-app
+  YAML expand). Raw apps are exploded into these items by `rawAppDiffToItems`.
 - has `content` (scripts) → Tabs(Content | Metadata) with two Monaco diffs
 - everything else (apps, resources, variables, schedules, triggers…) →
  a single Monaco YAML diff over the metadata
@@ -18,12 +21,15 @@ doesn't reflow the parent.
 	import Tabs from './common/tabs/Tabs.svelte'
 	import Tab from './common/tabs/Tab.svelte'
 	import FlowDiffViewer from './FlowDiffViewer.svelte'
+	import RawAppFileDiff from './raw_apps/RawAppFileDiff.svelte'
+	import type { RawAppFileItem } from './raw_apps/rawAppDiffUtils'
 	import { Loader2 } from 'lucide-svelte'
 	import { cleanValueProperties, orderedYamlStringify, replaceFalseWithUndefined } from '$lib/utils'
 	import { scriptLangToEditorLang } from '$lib/scripts'

 	interface Props {
-		/** Any WorkspaceItemDiff['kind'] — used only to special-case `flow`. */
+		/** Any WorkspaceItemDiff['kind'], plus the synthetic `raw_app_file`.
+		 * `flow` and `raw_app_file` are special-cased. */
 		kind: string
 		/** Raw value from `getItemValue(kind, path, parentWorkspace)`. Undefined
 		 * for "added" items (don't exist in the parent). */
@@ -33,9 +39,11 @@ doesn't reflow the parent.
 		currentRaw?: unknown
 		/** Force unified diff (Monaco renderSideBySide=false). Default false. */
 		inlineDiff?: boolean
+		/** For `raw_app_file`: the synthesized per-file diff item to render. */
+		rawFile?: RawAppFileItem
 	}

-	let { kind, originalRaw, currentRaw, inlineDiff = false }: Props = $props()
+	let { kind, originalRaw, currentRaw, inlineDiff = false, rawFile }: Props = $props()

 	type Prepared = { lang?: string; content?: string; metadata: string }

@@ -102,6 +110,16 @@ doesn't reflow the parent.
 			{inlineDiff}
 		/>
 	</div>
+{:else if kind === 'raw_app_file' && rawFile}
+	<RawAppFileDiff
+		original={rawFile.original}
+		current={rawFile.current}
+		lang={rawFile.lang}
+		isMetadata={rawFile.isMetadata}
+		fullYamlOriginal={rawFile.fullYamlOriginal}
+		fullYamlCurrent={rawFile.fullYamlCurrent}
+		{inlineDiff}
+	/>
 {:else if hasContent}
 	<div class="flex flex-col">
 		<Tabs bind:selected={contentTab}>
--- a/Show More
+++ b/Show More