Compare commits

..
Author SHA1 Message Date
centdix 00392ba548 fix 2026-06-23 16:23:41 +02:00
200 changed files with 2348 additions and 12342 deletions
-40
View File
@@ -1,40 +0,0 @@
---
name: ai-chat
description: Guidance for improving the Windmill AI chat (copilot), especially global mode — tools, prompts, and context-window discipline. Use when editing chat tools, system prompts, or tool-result shapes under frontend/src/lib/components/copilot/chat, or when changing how the chat manages its context window.
---
## Always benchmark before and after
No context or behavior change ships without an `ai_evals` A/B on the affected mode.
Add or adjust cases for exactly what you changed — see the `ai-evals` skill for
authoring and the full run reference.
Run the affected mode **before** your change and **after**, same model(s), same cases.
## Measure the window first, and cumulative second
Optimize **`finalContextTokens`** (window occupancy — what drives overflow and
compaction), then cumulative prompt tokens.
## Context discipline
The dominant fixed cost is per-iteration overhead: the system prompt **plus every
tool schema** is re-sent on every loop iteration. So:
- **Every tool and every parameter is a permanent tax.** Justify each one and measure
it; an extra "locate" round-trip can cost more than the reads it saves. Strip dead
params rather than leaving them in the schema.
- **Tool results return the minimum.** Never echo content the model already has. The
canonical mistake: a write tool that returns the whole edited artifact right after
the model authored it — return `{ success, message }` instead. When you touch a
*shared* write helper (e.g. `finishAppDraftWrite` in `global/core.ts`), re-check
this invariant for **all** the write tools routing through it — the echo has
regressed before via a shared refactor.
## Prompts and tool descriptions are part of the surface
The system prompt and tool descriptions steer behavior as much as the tools
themselves, and are benchmarkable the same way. A description that advertises
truncation makes the model self-limit; the path-conventions block changes where
drafts land. Treat prompt/description edits as real changes and A/B them — a
pure-prompt change is a legitimate, measurable improvement.
-87
View File
@@ -1,87 +0,0 @@
---
name: ai-evals
description: Author and run black-box benchmark cases for the Windmill AI generation modes (flow/app/script/cli/global) in ai_evals/. Use when adding or changing eval cases, or when running before/after benchmarks for AI chat / copilot changes.
---
# AI evals — authoring and running benchmark cases
`ai_evals/` is a black-box benchmark runner for the Windmill AI generation modes:
`flow`, `app`, `script`, `cli`, `global`. It always tests the **current** production
prompts, tools, and guidance in this checkout. Each attempt runs the real production
path, deterministic validation, then LLM judging.
The goal is to test current production guidance with realistic user requests — **not**
to pin one exact implementation shape.
## Running benchmarks
```bash
cd ai_evals
bun install # first time; frontend modes also need `cd frontend && bun install`
bun run cli -- models # list model aliases
bun run cli -- cases global # list cases for a mode
bun run cli -- run global global-test1-script-create --model sonnet
```
Frontend modes (`flow`/`script`/`app`/`global`) route model calls through a Windmill
backend's `/api/w/<ws>/ai/proxy`, so you need **any** reachable backend:
```bash
WMILL_AI_EVAL_BACKEND_URL=http://127.0.0.1:<port> WMILL_AI_EVAL_BACKEND_WORKSPACE=integration-tests \
bun run cli -- run global <caseIds...> --models sonnet,gpt-5.5,gemini-3.1-pro-preview
```
- **Reuse an existing workspace.** CE builds cap workspaces, so temp-workspace
creation 400s ("reached workspace limit"). Always set
`WMILL_AI_EVAL_BACKEND_WORKSPACE=integration-tests` (or any existing workspace) to
reuse one. The only side effect of a run is upserting an `f/evals/ai/<provider>`
resource there.
- Provider keys live in `ai_evals/.env` and are auto-loaded by bun. The judge is a
separate Anthropic call (default `claude-sonnet-4-6`) regardless of the model under
test.
## Authoring core rules
1. Write prompts like a real user request.
2. Prefer behavior, inputs, constraints, and outcomes over internal implementation.
3. Keep deterministic validation narrow and hard.
4. Put semantic expectations in `judgeChecklist`.
5. Use `expected` fixtures only when exact structure really matters.
### Prompt writing
Prompts should sound like something a user would naturally ask. Do not write prompts
as if the user knows Windmill internals unless the case explicitly tests a power-user
workflow.
Good:
- "Create a flow that routes support requests based on customer tier."
- "Add a reset button that sets the counter back to 0."
- "Create a flow that reuses the existing greeting script instead of duplicating the logic."
Bad:
- "Use `branchone` with 3 branches and a default branch."
- "Create a `rawscript` step with this exact topology."
- "This is a benchmark harness."
### Deterministic validation
Use deterministic checks only for hard failures: missing required files; unexpected
extra files when the prompt says not to create them; syntax errors; unresolved flow
refs; missing required special modules or suspend config; obvious corruption.
Do **not** encode one preferred implementation. Bad hard checks: exact step topology
for a creation flow; exact branch structure when the prompt only asked for routing;
exact input shape when multiple reasonable shapes are acceptable.
### Judge checklist
Every non-trivial case should have a `judgeChecklist` capturing user-visible behavior
that must be present, important constraints, and key completion criteria — not
low-level implementation details unless truly required.
Good: "the flow calculates the order total with 8% tax"; "the flow reuses the existing
workspace script instead of rewriting the logic". Bad: "uses `branchone`"; "contains a
`rawscript` node".
See `ai_evals/README.md` for the full case format, fields, and fixture details.
-1
View File
@@ -1 +0,0 @@
../../../.agents/skills/ai-chat/SKILL.md
-1
View File
@@ -1 +0,0 @@
../../../.agents/skills/ai-evals/SKILL.md
+1 -1
View File
@@ -28,7 +28,7 @@ ENV PATH="${PATH}:/usr/local/go/bin"
ENV GO_PATH=/usr/local/go/bin/go
# UV
RUN curl --proto '=https' --tlsv1.2 -LsSf https://github.com/astral-sh/uv/releases/download/0.11.24/uv-installer.sh | sh && mv /usr/local/cargo/bin/uv /usr/local/bin/uv
RUN curl --proto '=https' --tlsv1.2 -LsSf https://github.com/astral-sh/uv/releases/download/0.9.25/uv-installer.sh | sh && mv /usr/local/cargo/bin/uv /usr/local/bin/uv
ENV TZ=Etc/UTC
+1 -1
View File
@@ -74,7 +74,7 @@ jobs:
- uses: astral-sh/setup-uv@v6.2.1
with:
version: "0.11.24"
version: "0.9.25"
- uses: shivammathur/setup-php@v2
with:
+1 -1
View File
@@ -62,7 +62,7 @@ jobs:
node-version: "20"
- uses: astral-sh/setup-uv@v6.2.1
with:
version: "0.11.24"
version: "0.9.25"
- uses: shivammathur/setup-php@v2
with:
php-version: "8.3"
-59
View File
@@ -1,64 +1,5 @@
# Changelog
## [1.739.0](https://github.com/windmill-labs/windmill/compare/v1.738.0...v1.739.0) (2026-06-24)
### Features
* add /compact session chat command ([#9764](https://github.com/windmill-labs/windmill/issues/9764)) ([83cc553](https://github.com/windmill-labs/windmill/commit/83cc5533ee92e59356a117eefeaf42dad23287f6))
* add session chat slash commands ([#9748](https://github.com/windmill-labs/windmill/issues/9748)) ([24b95e9](https://github.com/windmill-labs/windmill/commit/24b95e9fe12ba4abdfe1ff6e9f9fe42cb2ded011))
* **ai-chat:** add /clear session command to start a fresh conversation ([#9769](https://github.com/windmill-labs/windmill/issues/9769)) ([3fafac2](https://github.com/windmill-labs/windmill/commit/3fafac275d2100a6f89924040650cf959945d209))
* **ai-chat:** context usage gauge + unified model settings menu ([#9763](https://github.com/windmill-labs/windmill/issues/9763)) ([2e020b2](https://github.com/windmill-labs/windmill/commit/2e020b2ccc7a649a5923bff72a98f07d4fc85381))
* **apps:** show raw-app fork diffs as per-file tree items ([#9491](https://github.com/windmill-labs/windmill/issues/9491)) ([e98df38](https://github.com/windmill-labs/windmill/commit/e98df38ac43823ee85209a4b09cd70690469302d))
* **frontend:** add filter submenu to collapsed AI sessions popover ([#9757](https://github.com/windmill-labs/windmill/issues/9757)) ([3d48ba7](https://github.com/windmill-labs/windmill/commit/3d48ba7738c3d3356539b5fc44a871f6b7f9d548))
* **frontend:** restore raw app 'open preview in separate window' ([#9765](https://github.com/windmill-labs/windmill/issues/9765)) ([a116715](https://github.com/windmill-labs/windmill/commit/a116715c418c39d48a91e6c0b4484a31537dff38))
* **frontend:** show approval wait as a distinct segment in flow timeline ([#9756](https://github.com/windmill-labs/windmill/issues/9756)) ([2a70ccc](https://github.com/windmill-labs/windmill/commit/2a70ccc38675c7c2353807a4f85764a8a35224e2))
* scope AI sessions per workspace root with lifecycle reconcile ([#9734](https://github.com/windmill-labs/windmill/issues/9734)) ([42c5e7a](https://github.com/windmill-labs/windmill/commit/42c5e7a3fc9b74256de8806ec0d7b62e8bbf029c))
### Bug Fixes
* **ai-chat:** strip unclosed &lt;summary&gt; tag leaking into compaction summary ([#9750](https://github.com/windmill-labs/windmill/issues/9750)) ([250a05f](https://github.com/windmill-labs/windmill/commit/250a05f544ae397bb91af5fc83bf408cfe1c554d))
* **apps:** realign legacy raw-app drafts to raw_app draft kind ([#9761](https://github.com/windmill-labs/windmill/issues/9761)) ([288318a](https://github.com/windmill-labs/windmill/commit/288318ac269714fc03b15622dbb86b1c28268a36))
* **backend:** resolve folder_labels search_path on non-public (PG_SCHEMA) schemas ([#9758](https://github.com/windmill-labs/windmill/issues/9758)) ([f582878](https://github.com/windmill-labs/windmill/commit/f5828780fd6a8be070b2933ebd41ee6dff98a9e1))
* forbid superadmin job tokens from global user and token management ([#9715](https://github.com/windmill-labs/windmill/issues/9715)) ([043c2c0](https://github.com/windmill-labs/windmill/commit/043c2c05b7678c49faca0ccb28e5f6393567ba4d))
* **frontend:** highlight the runtime-chosen branch in flow graph viewer ([#9755](https://github.com/windmill-labs/windmill/issues/9755)) ([de6192b](https://github.com/windmill-labs/windmill/commit/de6192bec1695883a07452f7db2fb51c94dbfd43))
* **frontend:** keep #content portal target present on AI-session route ([#9754](https://github.com/windmill-labs/windmill/issues/9754)) ([5e09c50](https://github.com/windmill-labs/windmill/commit/5e09c501713ebbe05b28ce0084eca641f0dbe95c))
* **frontend:** show AI skills settings only when global mode enabled ([#9747](https://github.com/windmill-labs/windmill/issues/9747)) ([c017f7f](https://github.com/windmill-labs/windmill/commit/c017f7f8919a51292ddf01574961d1774bc1ba23))
* **frontend:** stop flow step id generation from being poisoned by non-canonical keys ([#9766](https://github.com/windmill-labs/windmill/issues/9766)) ([4dbf873](https://github.com/windmill-labs/windmill/commit/4dbf8737238ccc4dc2c67365e6d43f04f46c75b5))
* persist on-behalf-of user across app deploy paths ([#9773](https://github.com/windmill-labs/windmill/issues/9773)) ([f99781c](https://github.com/windmill-labs/windmill/commit/f99781ca5f77248206c951935cc44acfa5f072eb))
* reject symlink traversal in job-dir path validation ([#9713](https://github.com/windmill-labs/windmill/issues/9713)) ([b5bd824](https://github.com/windmill-labs/windmill/commit/b5bd8245d81b84fc14d3ea955bf1e66ac576bf37))
### Performance Improvements
* **audit:** adaptive timestamp floor for S3 audit-log export ([#9752](https://github.com/windmill-labs/windmill/issues/9752)) ([55bed4a](https://github.com/windmill-labs/windmill/commit/55bed4abcfce2a611b16054573980d2eb613ccb3))
* **monitor:** vacuum job_perms/job_result_stream right after each orphan sweep ([#9753](https://github.com/windmill-labs/windmill/issues/9753)) ([8912e21](https://github.com/windmill-labs/windmill/commit/8912e21d1571e57b5cf21b7d4d9520e20a28e70d))
## [1.738.0](https://github.com/windmill-labs/windmill/compare/v1.737.0...v1.738.0) (2026-06-23)
### Features
* add resource and infrastructure telemetry ([#9737](https://github.com/windmill-labs/windmill/issues/9737)) ([9793d01](https://github.com/windmill-labs/windmill/commit/9793d01575415963a89609a1baf2cd64f0d050cc))
* render mermaid diagrams in chat code blocks ([#9738](https://github.com/windmill-labs/windmill/issues/9738)) ([cfb9f1d](https://github.com/windmill-labs/windmill/commit/cfb9f1dbc23110ecf8f91bb3c8c81fc6e35dc09b))
### Bug Fixes
* **ai-chat:** Fix incorrect editor edits from ai chat [#1](https://github.com/windmill-labs/windmill/issues/1) ([#9741](https://github.com/windmill-labs/windmill/issues/9741)) ([fc797a3](https://github.com/windmill-labs/windmill/commit/fc797a35fe7885630c81453df0fc94769e73873a))
* allow object storage test for non-super-admins, harden on cloud ([#9739](https://github.com/windmill-labs/windmill/issues/9739)) ([24446e8](https://github.com/windmill-labs/windmill/commit/24446e80093ade349f7fbf65063d2d1cb5551c1e))
* **frontend:** debounce external code→Monaco sync in Editor ([#9743](https://github.com/windmill-labs/windmill/issues/9743)) ([29c67ce](https://github.com/windmill-labs/windmill/commit/29c67ced97bf2919584986f9d9eceb4337c34ad9))
* **frontend:** preserve editor content when closing instance settings drawer ([#9740](https://github.com/windmill-labs/windmill/issues/9740)) ([11d0e65](https://github.com/windmill-labs/windmill/commit/11d0e65f3af9a048bc1921bbdd3d676a07483a57))
* pipeline annotation false-positives from body comments ([#9736](https://github.com/windmill-labs/windmill/issues/9736)) ([984ea72](https://github.com/windmill-labs/windmill/commit/984ea728d98649b66b1cae899bdab9af3176caa7))
* preserve fork parent linkage on workspace id change ([#9716](https://github.com/windmill-labs/windmill/issues/9716)) ([cbf54d4](https://github.com/windmill-labs/windmill/commit/cbf54d4eb432638e27f67c4c8b879cbcc0291da3))
* prevent variable push from corrupting is_secret variables ([#9705](https://github.com/windmill-labs/windmill/issues/9705)) ([ba4b368](https://github.com/windmill-labs/windmill/commit/ba4b368706e95e22f346a10e5fe145b0795ac3f6))
### Performance Improvements
* **monitor:** skip protected prefix in retention delete via cross-batch watermark (WIN-2088) ([#9744](https://github.com/windmill-labs/windmill/issues/9744)) ([e90b2be](https://github.com/windmill-labs/windmill/commit/e90b2be8fade1eb78cd685890291f5a4553a6a10))
## [1.737.0](https://github.com/windmill-labs/windmill/compare/v1.736.0...v1.737.0) (2026-06-23)
+5 -5
View File
@@ -163,14 +163,14 @@ ENV PATH /usr/local/bin:/root/.local/bin:/tmp/.local/bin:$PATH
RUN apt-get update \
&& apt-get install -y --no-install-recommends netbase tzdata ca-certificates wget curl jq unzip build-essential unixodbc xmlsec1 tini gnupg \
&& apt-get install -y --no-install-recommends netbase tzdata ca-certificates wget curl jq unzip build-essential unixodbc xmlsec1 software-properties-common tini gnupg lsb-release \
&& if echo "$features" | grep -q "ee"; then apt-get install -y --no-install-recommends libsasl2-modules-gssapi-mit krb5-user; fi \
&& apt-get clean \
&& rm -rf /var/lib/apt/lists/*
# Install latest PostgreSQL client (pg_dump) from official PostgreSQL apt repository
RUN curl -fsSL https://www.postgresql.org/media/keys/ACCC4CF8.asc | gpg --dearmor -o /usr/share/keyrings/postgresql-archive-keyring.gpg \
&& echo "deb [signed-by=/usr/share/keyrings/postgresql-archive-keyring.gpg] https://apt.postgresql.org/pub/repos/apt $(. /etc/os-release; echo "$VERSION_CODENAME")-pgdg main" > /etc/apt/sources.list.d/pgdg.list \
&& echo "deb [signed-by=/usr/share/keyrings/postgresql-archive-keyring.gpg] https://apt.postgresql.org/pub/repos/apt $(lsb_release -cs)-pgdg main" > /etc/apt/sources.list.d/pgdg.list \
&& apt-get update \
&& apt-get install -y --no-install-recommends postgresql-client \
&& apt-get clean \
@@ -184,11 +184,11 @@ RUN if [ "$WITH_GIT" = "true" ]; then \
else echo 'Building the image without git'; fi;
RUN if [ "$WITH_POWERSHELL" = "true" ]; then \
if [ "$TARGETPLATFORM" = "linux/amd64" ]; then apt-get update -y && apt install libicu72 -y && wget -O 'pwsh.deb' "https://github.com/PowerShell/PowerShell/releases/download/v${POWERSHELL_VERSION}/powershell_${POWERSHELL_DEB_VERSION}.deb_amd64.deb" && apt-get clean \
if [ "$TARGETPLATFORM" = "linux/amd64" ]; then apt-get update -y && apt install libicu-dev -y && wget -O 'pwsh.deb' "https://github.com/PowerShell/PowerShell/releases/download/v${POWERSHELL_VERSION}/powershell_${POWERSHELL_DEB_VERSION}.deb_amd64.deb" && apt-get clean \
&& rm -rf /var/lib/apt/lists/* && \
dpkg --install 'pwsh.deb' && \
rm 'pwsh.deb'; \
elif [ "$TARGETPLATFORM" = "linux/arm64" ]; then apt-get update -y && apt install libicu72 -y && wget -O powershell.tar.gz "https://github.com/PowerShell/PowerShell/releases/download/v${POWERSHELL_VERSION}/powershell-${POWERSHELL_VERSION}-linux-arm64.tar.gz" && apt-get clean \
elif [ "$TARGETPLATFORM" = "linux/arm64" ]; then apt-get update -y && apt install libicu-dev -y && wget -O powershell.tar.gz "https://github.com/PowerShell/PowerShell/releases/download/v${POWERSHELL_VERSION}/powershell-${POWERSHELL_VERSION}-linux-arm64.tar.gz" && apt-get clean \
&& rm -rf /var/lib/apt/lists/* && \
mkdir -p /opt/microsoft/powershell/7 && \
tar zxf powershell.tar.gz -C /opt/microsoft/powershell/7 && \
@@ -233,7 +233,7 @@ ENV PATH="${PATH}:/usr/local/go/bin"
ENV GO_PATH=/usr/local/go/bin/go
# Install UV
RUN curl --proto '=https' --tlsv1.2 -LsSf https://github.com/astral-sh/uv/releases/download/0.11.24/uv-installer.sh | sh && mv /root/.local/bin/uv /usr/local/bin/uv
RUN curl --proto '=https' --tlsv1.2 -LsSf https://github.com/astral-sh/uv/releases/download/0.9.25/uv-installer.sh | sh && mv /root/.local/bin/uv /usr/local/bin/uv
# Preinstall python runtimes to temp build location (will copy with world-writable perms later)
# --compile-bytecode precompiles the stdlib to .pyc so jobs don't recompile it on every run
+204 -10
View File
@@ -1,14 +1,208 @@
# AI Evals
# AI Evals Authoring Guide
Black-box benchmark cases for the Windmill AI generation modes (`flow`, `app`,
`script`, `cli`, `global`).
This folder contains black-box benchmark cases for:
**Authoring and running cases is documented in the `ai-evals` skill** — load it
before adding/changing a case or running a benchmark. Claude Code reads
`.claude/skills/ai-evals/SKILL.md`; Codex and Pi read
`.agents/skills/ai-evals/SKILL.md` (same canonical file). Invoke with `/ai-evals` in
Claude Code, `$ai-evals` in Codex, or `pi --skill ai-evals`.
- `flow`
- `app`
- `script`
- `cli`
- `global`
For AI chat / copilot changes that these evals measure, see the `ai-chat` skill.
The goal is to test the current production prompts and guidance with realistic user requests, not to test one exact implementation shape.
The full case format, fields, and fixture details remain in `ai_evals/README.md`.
## Core rules
1. Write prompts like a real user request.
2. Prefer behavior, inputs, constraints, and outcomes over internal implementation details.
3. Keep deterministic validation narrow and hard.
4. Put semantic expectations in `judgeChecklist`.
5. Use `expected` fixtures only when exact structure really matters.
## Prompt writing
Prompts should sound like something a user would naturally ask.
Good:
- "Create a flow that routes support requests based on customer tier."
- "Add a reset button that sets the counter back to 0."
- "Create a flow that reuses the existing greeting script instead of duplicating the logic."
Bad:
- "Use `branchone` with 3 branches and a default branch."
- "Create a `rawscript` step with this exact topology."
- "This is a benchmark harness."
Do not write prompts as if the user knows Windmill internals unless the case is explicitly testing a power-user workflow.
## Flow-specific rules
This is the main principle you asked for:
- flow prompts should read like requests from a user who does not know the product internals
- the user should ask for behavior, not for `branchone`, `branchall`, `rawscript`, `preprocessor_module`, `failure_module`, exact graph topology, or other internal constructs
That means:
- creation cases should describe the business behavior and expected result
- modification cases may mention existing step names, because the user can see the current flow
- only mention special Windmill constructs when the case is explicitly about those constructs
Examples:
- acceptable creation prompt:
"Create a purchase approval flow that pauses for approval and asks the approver for a comment."
- avoid:
"Create a suspend step with one required event and a resume form."
For flow cases, do not fail a case just because the model chose a different valid topology.
## App-specific rules
App prompts should focus on user-visible behavior:
- what the UI should let the user do
- what should persist
- what backend behavior is needed
Avoid prompting in terms of React structure, component names, or implementation unless the case is specifically about editing an existing app.
## CLI-specific rules
CLI prompts can be more explicit about paths and file names because real CLI users often do specify them.
Still, avoid benchmark phrasing. The prompt should read like a repo task, not a harness instruction.
When relevant, ask the assistant to tell the user which `wmill` commands to run next. That is part of the benchmarked behavior.
## Global-specific rules
Global prompts should exercise workspace-level drafting behavior:
- inspecting existing scripts, flows, apps, schedules, triggers, resources, and variables when relevant
- writing AI drafts rather than saving or deploying by default
- producing coherent multi-artifact changes when the request crosses artifact boundaries
Keep deterministic validation focused on the draft contract: required draft type/path, required content snippets, forbidden draft paths, and forbidden mutating tools such as deploy/delete unless the case explicitly asks for them.
Datatable cases should set `skipJudge: true` and validate through tool-use
(`requiredToolsUsed` / `forbiddenToolsUsed`) and SQL-argument assertions
(`toolCallArgs` with `stringIncludesAnyOf`, e.g. `['select']`, `['create table']`,
`['update', 'insert into']`). Two reasons the judge is unreliable here:
- `list_datatables`, `get_datatable_table_schema`, and `exec_datatable_sql`
produce no drafts, and the global judge only sees the drafts artifact — it
scores a no-draft conversational answer as empty (same as the
`askUserQuestion` cases).
- Even a case that *does* produce a draft (a script reading the data table via
`wmill.datatable()` at runtime) is mis-judged: the judge has no datatable SDK
reference and penalizes correct `wmill.datatable()` usage as wrong. Verify the
SDK call deterministically instead — `requiredDrafts.valueIncludes: ['wmill.datatable(']`
plus forbidding `exec_datatable_sql` (keeping chat-time SQL distinct from
runtime SDK use).
`stringIncludesAnyOf` is existential over calls (at least one matching call), so a
mutation case still passes when the model mixes its UPDATE/INSERT with
verification SELECTs. The in-memory engine (`datatableSqlEngine.ts`) is stateful
within a case — writes persist, so a model that re-queries to verify its
CREATE/UPDATE sees the change and does not loop. But the engine is best-effort
(SELECT returns all rows of the referenced/first table with no WHERE/projection),
so still never assert specific returned row values. Seed data via
`workspace.datatables` in the `initial` fixture (see README).
## Deterministic validation
Use deterministic validation only for hard failures such as:
- missing required files
- unexpected extra files when the prompt says not to create them
- syntax errors
- unresolved flow refs
- missing required special modules or suspend config
- obvious artifact corruption
Do not use deterministic validation to enforce one preferred implementation for broad creation tasks.
Examples of bad hard checks:
- exact step topology for a creation flow
- exact branch structure when the prompt only asked for routing behavior
- exact input shape when multiple reasonable shapes are acceptable
## Judge checklist
Every non-trivial case should have a `judgeChecklist`.
The checklist should capture:
- the user-visible behavior that must be present
- important constraints
- key completion criteria
The checklist should not duplicate low-level implementation details unless they are truly required by the task.
Good checklist items:
- "the flow calculates the order total with 8% tax"
- "the app persists recipes appropriately for a raw Windmill app"
- "the flow reuses the existing workspace script instead of rewriting the logic"
Bad checklist items:
- "uses `branchone`"
- "contains a `rawscript` node"
## When to use `expected`
Use `expected` fixtures when the case is structure-sensitive, for example:
- exact file creation
- exact script content
- modification cases where a specific file must change in a specific way
- cases where preserving an existing structure is part of the requirement
Do not use a full `expected` artifact as the semantic oracle for broad creation tasks when multiple valid outputs should pass.
## When to use `initial`
Use `initial` when the benchmark is about:
- editing an existing artifact
- reusing existing workspace assets
- preserving existing behavior while adding a change
If the case is greenfield, prefer no `initial`.
## Case design ladder
Prefer suites that get gradually harder:
1. trivial create case
2. realistic create case
3. reuse-existing-assets case
4. modification case
5. refactor case
6. edge-case or niche product behavior
The last cases in a suite should cover unusual or product-specific behavior.
## Anti-patterns
Avoid these:
- benchmark framing in prompts
- over-specified internal topology for creation tasks
- judge checklists that just restate implementation details
- deterministic validation that encodes one preferred solution
- fixtures that are so minimal or brittle that they create false negatives
## Before adding a case
Ask:
1. Would a real user plausibly write this prompt?
2. If the model solves it in a different valid way, would the case still pass?
3. Are the hard deterministic checks only catching objectively broken output?
4. Does the `judgeChecklist` describe the real success criteria?
5. If this case fails, will the reason be understandable from the saved artifacts?
-1
View File
@@ -3,7 +3,6 @@
Create a draft Bun script at `f/evals/global/greet_user`.
It should take a string `name` input and return `Hello, ${name}!`.
Leave it as an AI draft only; do not deploy or save it.
initial: ai_evals/fixtures/frontend/global/initial/user_admin_evals_folder.json
runtime:
maxTurns: 10
validate:
-3
View File
@@ -212,9 +212,6 @@ describe("loadCases", () => {
},
],
});
expect(caseEntry?.initialPath).toContain(
"ai_evals/fixtures/frontend/global/initial/user_admin_evals_folder.json"
);
expect(caseEntry?.toolExpect).toMatchObject({
requiredToolsUsed: ["write_script"],
forbiddenToolsUsed: ["deploy_workspace_item", "delete_workspace_item"],
@@ -1,8 +0,0 @@
{
"user": {
"username": "admin",
"is_admin": true,
"folders": ["evals"],
"folders_read": ["evals"]
}
}
@@ -1,20 +0,0 @@
{
"db_name": "PostgreSQL",
"query": "SELECT SUM(pg_database_size(datname))::BIGINT AS \"v!\" FROM pg_database",
"describe": {
"columns": [
{
"ordinal": 0,
"name": "v!",
"type_info": "Int8"
}
],
"parameters": {
"Left": []
},
"nullable": [
null
]
},
"hash": "074dd26f6427f4ff97e92c35163ad042144e656adcca56a4936a2eb196d3f48c"
}
@@ -1,58 +0,0 @@
{
"db_name": "PostgreSQL",
"query": "\n SELECT language AS \"language!: _\", COUNT(*)::BIGINT AS \"count!\"\n FROM script\n WHERE archived = false AND deleted = false AND kind = 'script'\n AND (auto_kind IS NULL OR auto_kind <> 'wac')\n GROUP BY language\n ",
"describe": {
"columns": [
{
"ordinal": 0,
"name": "language!: _",
"type_info": {
"Custom": {
"name": "script_lang",
"kind": {
"Enum": [
"python3",
"deno",
"go",
"bash",
"postgresql",
"nativets",
"bun",
"mysql",
"bigquery",
"snowflake",
"graphql",
"powershell",
"mssql",
"php",
"bunnative",
"rust",
"ansible",
"csharp",
"oracledb",
"nu",
"java",
"duckdb",
"ruby",
"rlang"
]
}
}
}
},
{
"ordinal": 1,
"name": "count!",
"type_info": "Int8"
}
],
"parameters": {
"Left": []
},
"nullable": [
false,
null
]
},
"hash": "11813108dbf6b104eba968c3609c74ac5a589542d3e76b1e756a82fb19d49ee8"
}
@@ -1,29 +0,0 @@
{
"db_name": "PostgreSQL",
"query": "\n SELECT DISTINCT ON (path) path AS \"path!\", content AS \"content!\"\n FROM script\n WHERE workspace_id = $1\n AND auto_kind = 'pipeline'\n AND archived = false\n AND deleted = false\n AND ($2::text IS NULL OR path LIKE $2)\n ORDER BY path, created_at DESC\n ",
"describe": {
"columns": [
{
"ordinal": 0,
"name": "path!",
"type_info": "Varchar"
},
{
"ordinal": 1,
"name": "content!",
"type_info": "Text"
}
],
"parameters": {
"Left": [
"Text",
"Text"
]
},
"nullable": [
false,
false
]
},
"hash": "394e2598880aff8a7f4ee05c3fe748be58b6381f5fae5619d8376daefc3b21db"
}
@@ -1,20 +0,0 @@
{
"db_name": "PostgreSQL",
"query": "SELECT COUNT(*)::INT AS \"v!\" FROM pg_stat_activity",
"describe": {
"columns": [
{
"ordinal": 0,
"name": "v!",
"type_info": "Int4"
}
],
"parameters": {
"Left": []
},
"nullable": [
null
]
},
"hash": "3b439ae7af0fcbb9df8e19faf84abf590e5e94898954711d57a602e6fd8a2f84"
}
@@ -1,20 +0,0 @@
{
"db_name": "PostgreSQL",
"query": "SELECT COUNT(*)::BIGINT AS \"count!\" FROM script WHERE archived = false AND deleted = false AND auto_kind = 'wac'",
"describe": {
"columns": [
{
"ordinal": 0,
"name": "count!",
"type_info": "Int8"
}
],
"parameters": {
"Left": []
},
"nullable": [
null
]
},
"hash": "3db1c61295c284725eef9e74a8aa2bc7822d263605a445f1f4a76e58e76a3e79"
}
@@ -1,15 +0,0 @@
{
"db_name": "PostgreSQL",
"query": "UPDATE workspace SET parent_workspace_id = $1 WHERE parent_workspace_id = $2",
"describe": {
"columns": [],
"parameters": {
"Left": [
"Varchar",
"Text"
]
},
"nullable": []
},
"hash": "40a8cf5e87bb489fd172689e9a6f0f1075b878f9916145929b3cd3b1a53b777e"
}
@@ -1,28 +0,0 @@
{
"db_name": "PostgreSQL",
"query": "SELECT p.id AS \"id!\", p.deleted AS \"deleted!\"\n FROM workspace f\n JOIN workspace p ON p.id = f.parent_workspace_id\n WHERE f.id = $1",
"describe": {
"columns": [
{
"ordinal": 0,
"name": "id!",
"type_info": "Varchar"
},
{
"ordinal": 1,
"name": "deleted!",
"type_info": "Bool"
}
],
"parameters": {
"Left": [
"Text"
]
},
"nullable": [
false,
false
]
},
"hash": "42322020ff9cc7dd7ebafc1cb4122ba3d670cc36bdbc6451f29b8f22f8cff688"
}
@@ -1,20 +0,0 @@
{
"db_name": "PostgreSQL",
"query": "SELECT pg_database_size(current_database())::BIGINT AS \"v!\"",
"describe": {
"columns": [
{
"ordinal": 0,
"name": "v!",
"type_info": "Int8"
}
],
"parameters": {
"Left": []
},
"nullable": [
null
]
},
"hash": "48a5df355a2bca557a3a541cf66c8e75790b7c3dd7845359019ff645a1f7c8bf"
}
@@ -1,20 +0,0 @@
{
"db_name": "PostgreSQL",
"query": "SELECT current_setting('max_connections')::INT AS \"v!\"",
"describe": {
"columns": [
{
"ordinal": 0,
"name": "v!",
"type_info": "Int4"
}
],
"parameters": {
"Left": []
},
"nullable": [
null
]
},
"hash": "49e1f5663eed128ed956c9a50bc73a85256c0a3e5a701cc13c944e66f6402617"
}
@@ -1,20 +0,0 @@
{
"db_name": "PostgreSQL",
"query": "SELECT COUNT(*)::BIGINT AS \"count!\" FROM flow WHERE archived = false",
"describe": {
"columns": [
{
"ordinal": 0,
"name": "count!",
"type_info": "Int8"
}
],
"parameters": {
"Left": []
},
"nullable": [
null
]
},
"hash": "50490ff42fb1f2d78864d7b374d299bf8290c3b969b576cb185c8b5b0abb0265"
}
@@ -1,20 +0,0 @@
{
"db_name": "PostgreSQL",
"query": "SELECT current_setting('server_version_num')::INT AS \"v!\"",
"describe": {
"columns": [
{
"ordinal": 0,
"name": "v!",
"type_info": "Int4"
}
],
"parameters": {
"Left": []
},
"nullable": [
null
]
},
"hash": "56a98a07a2f6af4d694db05d57acbe0b55cc39d64f73a3c90c250ea670f9cdee"
}
@@ -1,20 +0,0 @@
{
"db_name": "PostgreSQL",
"query": "SELECT CASE WHEN (current_setting('is_superuser') = 'on'\n OR pg_has_role(current_user, 'pg_read_all_stats', 'USAGE'))\n AND NOT EXISTS (SELECT 1 FROM pg_prepared_xacts)\n THEN (SELECT min(xact_start) FROM pg_stat_activity WHERE xact_start IS NOT NULL)\n ELSE NULL END AS \"x\"",
"describe": {
"columns": [
{
"ordinal": 0,
"name": "x",
"type_info": "Timestamptz"
}
],
"parameters": {
"Left": []
},
"nullable": [
null
]
},
"hash": "5d84f1ede2fbe09923a36d80d7c01699bf9968a02675f9f598c97a19c6df089b"
}
@@ -1,26 +0,0 @@
{
"db_name": "PostgreSQL",
"query": "\n SELECT\n COUNT(*) FILTER (WHERE av.raw_app = false)::BIGINT AS \"low_code!\",\n COUNT(*) FILTER (WHERE av.raw_app = true)::BIGINT AS \"raw!\"\n FROM app a\n JOIN app_version av ON av.id = a.versions[array_upper(a.versions, 1)]\n ",
"describe": {
"columns": [
{
"ordinal": 0,
"name": "low_code!",
"type_info": "Int8"
},
{
"ordinal": 1,
"name": "raw!",
"type_info": "Int8"
}
],
"parameters": {
"Left": []
},
"nullable": [
null,
null
]
},
"hash": "60dc0f1fa17bd2946ba7ddf0c41fe58b8a53d071cdc83eca3092194b4a9c9174"
}
@@ -1,20 +0,0 @@
{
"db_name": "PostgreSQL",
"query": "SELECT COUNT(*)::BIGINT AS \"count!\" FROM workspace WHERE deleted = false AND id NOT LIKE 'wm-fork%'",
"describe": {
"columns": [
{
"ordinal": 0,
"name": "count!",
"type_info": "Int8"
}
],
"parameters": {
"Left": []
},
"nullable": [
null
]
},
"hash": "9170a350e1da0b33a421a119d4a5b86575c1be761de158ad664670e981524cbf"
}
@@ -1,30 +0,0 @@
{
"db_name": "PostgreSQL",
"query": "DELETE FROM v2_job_completed\n WHERE id IN (\n SELECT id FROM v2_job_completed\n WHERE completed_at <= now() - ($1::bigint::text || ' s')::interval\n AND ($3::timestamptz IS NULL OR completed_at >= $3)\n ORDER BY completed_at ASC\n LIMIT $2\n FOR UPDATE SKIP LOCKED\n )\n RETURNING id, completed_at",
"describe": {
"columns": [
{
"ordinal": 0,
"name": "id",
"type_info": "Uuid"
},
{
"ordinal": 1,
"name": "completed_at",
"type_info": "Timestamptz"
}
],
"parameters": {
"Left": [
"Int8",
"Int8",
"Timestamptz"
]
},
"nullable": [
false,
false
]
},
"hash": "a2d4a8aedb15e9faf0a2512fa4241a9e9f2d5a56e12a9d6805ca58ff40f61614"
}
@@ -1,17 +0,0 @@
{
"db_name": "PostgreSQL",
"query": "INSERT INTO workspace (id, name, owner, deleted, premium, parent_workspace_id)\n SELECT $1, $2, owner, false, premium,\n CASE WHEN $4 THEN parent_workspace_id ELSE NULL END\n FROM workspace WHERE id = $3",
"describe": {
"columns": [],
"parameters": {
"Left": [
"Varchar",
"Varchar",
"Text",
"Bool"
]
},
"nullable": []
},
"hash": "a54efa4a7466e61fd54d8fe293cb775225dcb430026cebe15ba4994ac636514d"
}
@@ -1,38 +0,0 @@
{
"db_name": "PostgreSQL",
"query": "\n SELECT\n EXISTS(SELECT 1 FROM pg_proc WHERE proname = 'aurora_version') AS \"aurora!\",\n EXISTS(SELECT 1 FROM pg_roles WHERE rolname = 'rds_superuser') AS \"rds!\",\n EXISTS(SELECT 1 FROM pg_roles WHERE rolname = 'cloudsqlsuperuser') AS \"cloudsql!\",\n EXISTS(SELECT 1 FROM pg_roles WHERE rolname IN ('azure_pg_admin', 'azuresu')) AS \"azure!\"\n ",
"describe": {
"columns": [
{
"ordinal": 0,
"name": "aurora!",
"type_info": "Bool"
},
{
"ordinal": 1,
"name": "rds!",
"type_info": "Bool"
},
{
"ordinal": 2,
"name": "cloudsql!",
"type_info": "Bool"
},
{
"ordinal": 3,
"name": "azure!",
"type_info": "Bool"
}
],
"parameters": {
"Left": []
},
"nullable": [
null,
null,
null,
null
]
},
"hash": "a90e3a1d7c7c0dfb422f44b0ed599f681f5630c024ae9a437a301f149636b0db"
}
@@ -1,31 +0,0 @@
{
"db_name": "PostgreSQL",
"query": "DELETE FROM v2_job_completed\n WHERE id IN (\n SELECT jc.id FROM v2_job_completed jc\n LEFT JOIN v2_job j ON j.id = jc.id\n WHERE jc.completed_at <= now() - ($1::bigint::text || ' s')::interval\n AND ($4::timestamptz IS NULL OR jc.completed_at >= $4)\n AND COALESCE(j.root_job, j.flow_innermost_root_job, jc.id) NOT IN (\n SELECT u FROM unnest($3::uuid[]) AS u WHERE u IS NOT NULL\n )\n ORDER BY jc.completed_at ASC\n LIMIT $2\n FOR UPDATE OF jc SKIP LOCKED\n )\n RETURNING id, completed_at",
"describe": {
"columns": [
{
"ordinal": 0,
"name": "id",
"type_info": "Uuid"
},
{
"ordinal": 1,
"name": "completed_at",
"type_info": "Timestamptz"
}
],
"parameters": {
"Left": [
"Int8",
"Int8",
"UuidArray",
"Timestamptz"
]
},
"nullable": [
false,
false
]
},
"hash": "c033a690fde04da79745e850b72fa7cfd861f1dcad88c7ea75a0a8b014ec1f75"
}
@@ -0,0 +1,24 @@
{
"db_name": "PostgreSQL",
"query": "DELETE FROM v2_job_completed\n WHERE id IN (\n SELECT jc.id FROM v2_job_completed jc\n LEFT JOIN v2_job j ON j.id = jc.id\n WHERE jc.completed_at <= now() - ($1::bigint::text || ' s')::interval\n AND COALESCE(j.root_job, j.flow_innermost_root_job, jc.id) NOT IN (\n SELECT u FROM unnest($3::uuid[]) AS u WHERE u IS NOT NULL\n )\n ORDER BY jc.completed_at ASC\n LIMIT $2\n FOR UPDATE OF jc SKIP LOCKED\n )\n RETURNING id",
"describe": {
"columns": [
{
"ordinal": 0,
"name": "id",
"type_info": "Uuid"
}
],
"parameters": {
"Left": [
"Int8",
"Int8",
"UuidArray"
]
},
"nullable": [
false
]
},
"hash": "fbe3a876efd1253d2ef086b03366b2bd117ceb6bc152d2abcd45850ff6aecff9"
}
@@ -1,29 +0,0 @@
{
"db_name": "PostgreSQL",
"query": "SELECT req.id AS \"id!\",\n (CASE\n WHEN usr.email IS NULL THEN 'deleted'\n WHEN workspace.deleted THEN 'archived'\n ELSE 'active'\n END) AS \"status!\"\n FROM unnest($1::text[]) AS req(id)\n LEFT JOIN workspace ON workspace.id = req.id\n LEFT JOIN usr ON usr.workspace_id = workspace.id AND usr.email = $2",
"describe": {
"columns": [
{
"ordinal": 0,
"name": "id!",
"type_info": "Text"
},
{
"ordinal": 1,
"name": "status!",
"type_info": "Text"
}
],
"parameters": {
"Left": [
"TextArray",
"Text"
]
},
"nullable": [
null,
null
]
},
"hash": "fc4583d1570f3a2a428bb28390ca72e61719fae68aa4b42730f9076f3bd97441"
}
+83 -82
View File
@@ -2101,9 +2101,9 @@ checksum = "613afe47fcd5fac7ccf1db93babcb082c5994d996f20b8b159f2ad1658eb5724"
[[package]]
name = "chacha20"
version = "0.10.1"
version = "0.10.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "d524456ba66e72eb8b115ff89e01e497f8e6d11d78b70b1aa13c0fbd97540a81"
checksum = "6f8d983286843e49675a4b7a2d174efe136dc93a18d69130dd18198a6c167601"
dependencies = [
"cfg-if",
"cpufeatures 0.3.0",
@@ -5712,9 +5712,9 @@ dependencies = [
[[package]]
name = "hyper-http-proxy"
version = "1.1.1"
version = "1.1.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "8021e0ae20c08eadc94d0bdafdeda66d4f0858541c146ae6e46b219bfe58497e"
checksum = "7ad4b0a1e37510028bc4ba81d0e38d239c39671b0f0ce9e02dfa93a8133f7c08"
dependencies = [
"bytes",
"futures-util",
@@ -5726,6 +5726,7 @@ dependencies = [
"hyper-util",
"native-tls",
"pin-project-lite",
"rustls-native-certs 0.7.3",
"tokio",
"tokio-native-tls",
"tokio-rustls 0.26.4",
@@ -13734,7 +13735,7 @@ dependencies = [
[[package]]
name = "windmill"
version = "1.739.0"
version = "1.737.0"
dependencies = [
"anyhow",
"async-nats",
@@ -13816,7 +13817,7 @@ dependencies = [
[[package]]
name = "windmill-ai"
version = "1.739.0"
version = "1.737.0"
dependencies = [
"async-stream",
"async-trait",
@@ -13849,7 +13850,7 @@ dependencies = [
[[package]]
name = "windmill-alerting"
version = "1.739.0"
version = "1.737.0"
dependencies = [
"axum 0.8.9",
"chrono",
@@ -13862,7 +13863,7 @@ dependencies = [
[[package]]
name = "windmill-api"
version = "1.739.0"
version = "1.737.0"
dependencies = [
"anyhow",
"argon2",
@@ -14000,7 +14001,7 @@ dependencies = [
[[package]]
name = "windmill-api-agent-workers"
version = "1.739.0"
version = "1.737.0"
dependencies = [
"axum 0.8.9",
"chrono",
@@ -14023,7 +14024,7 @@ dependencies = [
[[package]]
name = "windmill-api-assets"
version = "1.739.0"
version = "1.737.0"
dependencies = [
"axum 0.8.9",
"chrono",
@@ -14036,7 +14037,7 @@ dependencies = [
[[package]]
name = "windmill-api-auth"
version = "1.739.0"
version = "1.737.0"
dependencies = [
"anyhow",
"axum 0.8.9",
@@ -14062,7 +14063,7 @@ dependencies = [
[[package]]
name = "windmill-api-client"
version = "1.739.0"
version = "1.737.0"
dependencies = [
"reqwest 0.12.28",
"serde",
@@ -14072,7 +14073,7 @@ dependencies = [
[[package]]
name = "windmill-api-configs"
version = "1.739.0"
version = "1.737.0"
dependencies = [
"axum 0.8.9",
"chrono",
@@ -14089,7 +14090,7 @@ dependencies = [
[[package]]
name = "windmill-api-debug"
version = "1.739.0"
version = "1.737.0"
dependencies = [
"axum 0.8.9",
"base64 0.22.1",
@@ -14111,7 +14112,7 @@ dependencies = [
[[package]]
name = "windmill-api-embeddings"
version = "1.739.0"
version = "1.737.0"
dependencies = [
"anyhow",
"axum 0.8.9",
@@ -14134,7 +14135,7 @@ dependencies = [
[[package]]
name = "windmill-api-flow-conversations"
version = "1.739.0"
version = "1.737.0"
dependencies = [
"axum 0.8.9",
"chrono",
@@ -14150,7 +14151,7 @@ dependencies = [
[[package]]
name = "windmill-api-flows"
version = "1.739.0"
version = "1.737.0"
dependencies = [
"axum 0.8.9",
"chrono",
@@ -14171,7 +14172,7 @@ dependencies = [
[[package]]
name = "windmill-api-groups"
version = "1.739.0"
version = "1.737.0"
dependencies = [
"axum 0.8.9",
"chrono",
@@ -14192,7 +14193,7 @@ dependencies = [
[[package]]
name = "windmill-api-inputs"
version = "1.739.0"
version = "1.737.0"
dependencies = [
"axum 0.8.9",
"chrono",
@@ -14206,7 +14207,7 @@ dependencies = [
[[package]]
name = "windmill-api-integration-tests"
version = "1.739.0"
version = "1.737.0"
dependencies = [
"anyhow",
"async-nats",
@@ -14241,7 +14242,7 @@ dependencies = [
[[package]]
name = "windmill-api-jobs"
version = "1.739.0"
version = "1.737.0"
dependencies = [
"anyhow",
"axum 0.8.9",
@@ -14266,7 +14267,7 @@ dependencies = [
[[package]]
name = "windmill-api-npm-proxy"
version = "1.739.0"
version = "1.737.0"
dependencies = [
"axum 0.8.9",
"flate2",
@@ -14284,7 +14285,7 @@ dependencies = [
[[package]]
name = "windmill-api-openapi"
version = "1.739.0"
version = "1.737.0"
dependencies = [
"anyhow",
"axum 0.8.9",
@@ -14306,7 +14307,7 @@ dependencies = [
[[package]]
name = "windmill-api-schedule"
version = "1.739.0"
version = "1.737.0"
dependencies = [
"axum 0.8.9",
"chrono",
@@ -14326,7 +14327,7 @@ dependencies = [
[[package]]
name = "windmill-api-scripts"
version = "1.739.0"
version = "1.737.0"
dependencies = [
"axum 0.8.9",
"chrono",
@@ -14363,7 +14364,7 @@ dependencies = [
[[package]]
name = "windmill-api-settings"
version = "1.739.0"
version = "1.737.0"
dependencies = [
"anyhow",
"axum 0.8.9",
@@ -14391,7 +14392,7 @@ dependencies = [
[[package]]
name = "windmill-api-sse"
version = "1.739.0"
version = "1.737.0"
dependencies = [
"lazy_static",
"serde",
@@ -14403,7 +14404,7 @@ dependencies = [
[[package]]
name = "windmill-api-users"
version = "1.739.0"
version = "1.737.0"
dependencies = [
"argon2",
"axum 0.8.9",
@@ -14428,7 +14429,7 @@ dependencies = [
[[package]]
name = "windmill-api-workers"
version = "1.739.0"
version = "1.737.0"
dependencies = [
"axum 0.8.9",
"chrono",
@@ -14442,7 +14443,7 @@ dependencies = [
[[package]]
name = "windmill-api-workspaces"
version = "1.739.0"
version = "1.737.0"
dependencies = [
"axum 0.8.9",
"chrono",
@@ -14475,7 +14476,7 @@ dependencies = [
[[package]]
name = "windmill-audit"
version = "1.739.0"
version = "1.737.0"
dependencies = [
"chrono",
"lazy_static",
@@ -14489,7 +14490,7 @@ dependencies = [
[[package]]
name = "windmill-autoscaling"
version = "1.739.0"
version = "1.737.0"
dependencies = [
"anyhow",
"axum 0.8.9",
@@ -14508,7 +14509,7 @@ dependencies = [
[[package]]
name = "windmill-common"
version = "1.739.0"
version = "1.737.0"
dependencies = [
"aes-gcm",
"aho-corasick",
@@ -14610,7 +14611,7 @@ dependencies = [
[[package]]
name = "windmill-dep-map"
version = "1.739.0"
version = "1.737.0"
dependencies = [
"chrono",
"itertools 0.14.0",
@@ -14629,7 +14630,7 @@ dependencies = [
[[package]]
name = "windmill-git-sync"
version = "1.739.0"
version = "1.737.0"
dependencies = [
"regex",
"serde",
@@ -14644,7 +14645,7 @@ dependencies = [
[[package]]
name = "windmill-indexer"
version = "1.739.0"
version = "1.737.0"
dependencies = [
"anyhow",
"astral-tokio-tar",
@@ -14668,7 +14669,7 @@ dependencies = [
[[package]]
name = "windmill-jseval"
version = "1.739.0"
version = "1.737.0"
dependencies = [
"anyhow",
"futures",
@@ -14685,7 +14686,7 @@ dependencies = [
[[package]]
name = "windmill-macros"
version = "1.739.0"
version = "1.737.0"
dependencies = [
"itertools 0.14.0",
"lazy_static",
@@ -14701,7 +14702,7 @@ dependencies = [
[[package]]
name = "windmill-mcp"
version = "1.739.0"
version = "1.737.0"
dependencies = [
"anyhow",
"async-trait",
@@ -14722,7 +14723,7 @@ dependencies = [
[[package]]
name = "windmill-native-triggers"
version = "1.739.0"
version = "1.737.0"
dependencies = [
"anyhow",
"async-trait",
@@ -14753,7 +14754,7 @@ dependencies = [
[[package]]
name = "windmill-oauth"
version = "1.739.0"
version = "1.737.0"
dependencies = [
"anyhow",
"arc-swap",
@@ -14778,7 +14779,7 @@ dependencies = [
[[package]]
name = "windmill-object-store"
version = "1.739.0"
version = "1.737.0"
dependencies = [
"anyhow",
"async-stream",
@@ -14812,7 +14813,7 @@ dependencies = [
[[package]]
name = "windmill-operator"
version = "1.739.0"
version = "1.737.0"
dependencies = [
"anyhow",
"futures",
@@ -14830,7 +14831,7 @@ dependencies = [
[[package]]
name = "windmill-parser"
version = "1.739.0"
version = "1.737.0"
dependencies = [
"convert_case 0.6.0",
"serde",
@@ -14839,7 +14840,7 @@ dependencies = [
[[package]]
name = "windmill-parser-bash"
version = "1.739.0"
version = "1.737.0"
dependencies = [
"anyhow",
"lazy_static",
@@ -14851,7 +14852,7 @@ dependencies = [
[[package]]
name = "windmill-parser-csharp"
version = "1.739.0"
version = "1.737.0"
dependencies = [
"anyhow",
"serde_json",
@@ -14863,7 +14864,7 @@ dependencies = [
[[package]]
name = "windmill-parser-go"
version = "1.739.0"
version = "1.737.0"
dependencies = [
"anyhow",
"gosyn",
@@ -14875,7 +14876,7 @@ dependencies = [
[[package]]
name = "windmill-parser-graphql"
version = "1.739.0"
version = "1.737.0"
dependencies = [
"anyhow",
"lazy_static",
@@ -14887,7 +14888,7 @@ dependencies = [
[[package]]
name = "windmill-parser-java"
version = "1.739.0"
version = "1.737.0"
dependencies = [
"anyhow",
"serde_json",
@@ -14899,7 +14900,7 @@ dependencies = [
[[package]]
name = "windmill-parser-nu"
version = "1.739.0"
version = "1.737.0"
dependencies = [
"anyhow",
"nu-parser",
@@ -14910,7 +14911,7 @@ dependencies = [
[[package]]
name = "windmill-parser-php"
version = "1.739.0"
version = "1.737.0"
dependencies = [
"anyhow",
"itertools 0.14.0",
@@ -14921,7 +14922,7 @@ dependencies = [
[[package]]
name = "windmill-parser-py"
version = "1.739.0"
version = "1.737.0"
dependencies = [
"anyhow",
"itertools 0.14.0",
@@ -14933,7 +14934,7 @@ dependencies = [
[[package]]
name = "windmill-parser-py-asset"
version = "1.739.0"
version = "1.737.0"
dependencies = [
"anyhow",
"rustpython-ast",
@@ -14944,7 +14945,7 @@ dependencies = [
[[package]]
name = "windmill-parser-py-imports"
version = "1.739.0"
version = "1.737.0"
dependencies = [
"anyhow",
"async-recursion",
@@ -14966,7 +14967,7 @@ dependencies = [
[[package]]
name = "windmill-parser-r"
version = "1.739.0"
version = "1.737.0"
dependencies = [
"anyhow",
"serde_json",
@@ -14978,7 +14979,7 @@ dependencies = [
[[package]]
name = "windmill-parser-ruby"
version = "1.739.0"
version = "1.737.0"
dependencies = [
"anyhow",
"lazy_static",
@@ -14992,7 +14993,7 @@ dependencies = [
[[package]]
name = "windmill-parser-rust"
version = "1.739.0"
version = "1.737.0"
dependencies = [
"anyhow",
"convert_case 0.6.0",
@@ -15009,7 +15010,7 @@ dependencies = [
[[package]]
name = "windmill-parser-sql"
version = "1.739.0"
version = "1.737.0"
dependencies = [
"anyhow",
"lazy_static",
@@ -15022,7 +15023,7 @@ dependencies = [
[[package]]
name = "windmill-parser-sql-asset"
version = "1.739.0"
version = "1.737.0"
dependencies = [
"anyhow",
"serde",
@@ -15034,7 +15035,7 @@ dependencies = [
[[package]]
name = "windmill-parser-ts"
version = "1.739.0"
version = "1.737.0"
dependencies = [
"anyhow",
"lazy_static",
@@ -15052,7 +15053,7 @@ dependencies = [
[[package]]
name = "windmill-parser-ts-asset"
version = "1.739.0"
version = "1.737.0"
dependencies = [
"anyhow",
"serde-wasm-bindgen",
@@ -15068,7 +15069,7 @@ dependencies = [
[[package]]
name = "windmill-parser-wac"
version = "1.739.0"
version = "1.737.0"
dependencies = [
"anyhow",
"rustpython-ast",
@@ -15084,7 +15085,7 @@ dependencies = [
[[package]]
name = "windmill-parser-yaml"
version = "1.739.0"
version = "1.737.0"
dependencies = [
"anyhow",
"serde",
@@ -15095,7 +15096,7 @@ dependencies = [
[[package]]
name = "windmill-queue"
version = "1.739.0"
version = "1.737.0"
dependencies = [
"anyhow",
"async-recursion",
@@ -15133,7 +15134,7 @@ dependencies = [
[[package]]
name = "windmill-runtime-nativets"
version = "1.739.0"
version = "1.737.0"
dependencies = [
"anyhow",
"const_format",
@@ -15172,7 +15173,7 @@ dependencies = [
[[package]]
name = "windmill-sql-datatype-parser-wasm"
version = "1.739.0"
version = "1.737.0"
dependencies = [
"getrandom 0.3.4",
"wasm-bindgen",
@@ -15183,7 +15184,7 @@ dependencies = [
[[package]]
name = "windmill-store"
version = "1.739.0"
version = "1.737.0"
dependencies = [
"anyhow",
"async-recursion",
@@ -15217,7 +15218,7 @@ dependencies = [
[[package]]
name = "windmill-test-utils"
version = "1.739.0"
version = "1.737.0"
dependencies = [
"anyhow",
"async-trait",
@@ -15241,7 +15242,7 @@ dependencies = [
[[package]]
name = "windmill-trigger"
version = "1.739.0"
version = "1.737.0"
dependencies = [
"anyhow",
"async-trait",
@@ -15274,7 +15275,7 @@ dependencies = [
[[package]]
name = "windmill-trigger-azure"
version = "1.739.0"
version = "1.737.0"
dependencies = [
"anyhow",
"async-trait",
@@ -15307,7 +15308,7 @@ dependencies = [
[[package]]
name = "windmill-trigger-email"
version = "1.739.0"
version = "1.737.0"
dependencies = [
"anyhow",
"async-trait",
@@ -15327,7 +15328,7 @@ dependencies = [
[[package]]
name = "windmill-trigger-gcp"
version = "1.739.0"
version = "1.737.0"
dependencies = [
"anyhow",
"async-trait",
@@ -15361,7 +15362,7 @@ dependencies = [
[[package]]
name = "windmill-trigger-http"
version = "1.739.0"
version = "1.737.0"
dependencies = [
"anyhow",
"async-trait",
@@ -15397,7 +15398,7 @@ dependencies = [
[[package]]
name = "windmill-trigger-kafka"
version = "1.739.0"
version = "1.737.0"
dependencies = [
"anyhow",
"async-trait",
@@ -15420,7 +15421,7 @@ dependencies = [
[[package]]
name = "windmill-trigger-mqtt"
version = "1.739.0"
version = "1.737.0"
dependencies = [
"anyhow",
"async-trait",
@@ -15444,7 +15445,7 @@ dependencies = [
[[package]]
name = "windmill-trigger-nats"
version = "1.739.0"
version = "1.737.0"
dependencies = [
"anyhow",
"async-nats",
@@ -15468,7 +15469,7 @@ dependencies = [
[[package]]
name = "windmill-trigger-postgres"
version = "1.739.0"
version = "1.737.0"
dependencies = [
"anyhow",
"async-trait",
@@ -15503,7 +15504,7 @@ dependencies = [
[[package]]
name = "windmill-trigger-sqs"
version = "1.739.0"
version = "1.737.0"
dependencies = [
"anyhow",
"async-trait",
@@ -15531,7 +15532,7 @@ dependencies = [
[[package]]
name = "windmill-trigger-websocket"
version = "1.739.0"
version = "1.737.0"
dependencies = [
"anyhow",
"async-trait",
@@ -15556,7 +15557,7 @@ dependencies = [
[[package]]
name = "windmill-types"
version = "1.739.0"
version = "1.737.0"
dependencies = [
"anyhow",
"bitflags 2.13.0",
@@ -15575,7 +15576,7 @@ dependencies = [
[[package]]
name = "windmill-worker"
version = "1.739.0"
version = "1.737.0"
dependencies = [
"anyhow",
"async-once-cell",
@@ -15685,7 +15686,7 @@ dependencies = [
[[package]]
name = "windmill-worker-volumes"
version = "1.739.0"
version = "1.737.0"
dependencies = [
"bytes",
"futures",
+2 -2
View File
@@ -1,6 +1,6 @@
[package]
name = "windmill"
version = "1.739.0"
version = "1.737.0"
authors.workspace = true
edition.workspace = true
@@ -87,7 +87,7 @@ members = [
exclude = ["./windmill-duckdb-ffi-internal", "./parsers/windmill-parser-wasm"]
[workspace.package]
version = "1.739.0"
version = "1.737.0"
authors = ["Ruben Fiszel <ruben@windmill.dev>"]
edition = "2021"
+1 -1
View File
@@ -1 +1 @@
ed89574be9117cda5e2d7d9de02cb5db066e93e3
ac1f6f666f36141cb6ba6f8eaa614821a90464ad
@@ -1,3 +0,0 @@
-- No-op: this migration only re-pins the function's search_path. Reverting would
-- mean restoring the hardcoded `SET search_path = public`, which is the very bug
-- this repairs, so there is nothing to undo.
@@ -1,22 +0,0 @@
-- Repair instances that already applied the folder-labels migrations while the
-- function hardcoded `SET search_path = public`. On a non-public schema (PG_SCHEMA)
-- the function was pinned to `public`, so at runtime it read the wrong `folder`
-- table (or a stray public.folder) instead of the workspace's real one.
--
-- `FROM CURRENT` snapshots the migration connection's search_path (the actual
-- Windmill schema) into the function, keeping the SECURITY DEFINER injection
-- hardening. On public-schema installs this re-pins to `public`, i.e. a no-op.
-- Idempotent: redefining with the same body is harmless on already-correct installs.
CREATE OR REPLACE FUNCTION folder_labels(w_id text, item_path text) RETURNS text[]
LANGUAGE sql STABLE SECURITY DEFINER SET search_path FROM CURRENT AS $$
SELECT (
SELECT array_agg(l ORDER BY first_ord)
FROM (
SELECT u.l, min(u.ord) AS first_ord
FROM unnest(f.labels) WITH ORDINALITY AS u(l, ord)
GROUP BY u.l
) deduped
)
FROM folder f
WHERE f.workspace_id = w_id AND item_path LIKE 'f/%' AND f.name = split_part(item_path, '/', 2)
$$;
@@ -1,3 +0,0 @@
-- Irreversible data backfill: once a raw app's draft is retyped to 'raw_app' it
-- is indistinguishable from one saved as 'raw_app' by the per-kind code, so the
-- original typ='app' state cannot be reconstructed. No-op on revert.
@@ -1,35 +0,0 @@
-- The pre-per-user `DRAFT_TYPE` enum had only ('script','flow','app'): a raw
-- app's draft was therefore stored as typ='app'. The new model splits app vs
-- raw_app into distinct draft kinds chosen from the deployed app's `raw_app`
-- flag, so a raw app's pre-migration draft is invisible to the per-kind lookups
-- (editor overlay, migrate-legacy, get-for-user), which all query typ='raw_app'.
-- Realign every such draft (any owner, including the legacy NULL-email row) to
-- 'raw_app' when the deployed app at that path is a raw app.
-- Drop, don't retype, a stale 'app' row when a 'raw_app' draft already exists
-- for the same owner (the newer 'raw_app' row, saved with the per-kind code, is
-- authoritative) — retyping would collide on the draft_pkey_with_user /
-- draft_pkey_legacy partial unique indexes over (workspace_id, path, typ, email).
DELETE FROM draft d
USING app a
JOIN app_version av ON av.id = a.versions[array_upper(a.versions, 1)]
WHERE d.typ = 'app'
AND a.workspace_id = d.workspace_id
AND a.path = d.path
AND av.raw_app IS TRUE
AND EXISTS (
SELECT 1 FROM draft d2
WHERE d2.workspace_id = d.workspace_id
AND d2.path = d.path
AND d2.typ = 'raw_app'
AND d2.email IS NOT DISTINCT FROM d.email
);
UPDATE draft d
SET typ = 'raw_app'
FROM app a
JOIN app_version av ON av.id = a.versions[array_upper(a.versions, 1)]
WHERE d.typ = 'app'
AND a.workspace_id = d.workspace_id
AND a.path = d.path
AND av.raw_app IS TRUE;
+24 -24
View File
@@ -6191,7 +6191,7 @@ checksum = "712e227841d057c1ee1cd2fb22fa7e5a5461ae8e48fa2ca79ec42cfc1931183f"
[[package]]
name = "windmill-common"
version = "1.739.0"
version = "1.737.0"
dependencies = [
"aho-corasick",
"anyhow",
@@ -6272,7 +6272,7 @@ dependencies = [
[[package]]
name = "windmill-macros"
version = "1.739.0"
version = "1.737.0"
dependencies = [
"proc-macro2",
"quote",
@@ -6284,7 +6284,7 @@ dependencies = [
[[package]]
name = "windmill-parser"
version = "1.739.0"
version = "1.737.0"
dependencies = [
"convert_case",
"serde",
@@ -6293,7 +6293,7 @@ dependencies = [
[[package]]
name = "windmill-parser-bash"
version = "1.739.0"
version = "1.737.0"
dependencies = [
"anyhow",
"lazy_static",
@@ -6305,7 +6305,7 @@ dependencies = [
[[package]]
name = "windmill-parser-csharp"
version = "1.739.0"
version = "1.737.0"
dependencies = [
"anyhow",
"serde_json",
@@ -6317,7 +6317,7 @@ dependencies = [
[[package]]
name = "windmill-parser-go"
version = "1.739.0"
version = "1.737.0"
dependencies = [
"anyhow",
"gosyn",
@@ -6329,7 +6329,7 @@ dependencies = [
[[package]]
name = "windmill-parser-graphql"
version = "1.739.0"
version = "1.737.0"
dependencies = [
"anyhow",
"lazy_static",
@@ -6341,7 +6341,7 @@ dependencies = [
[[package]]
name = "windmill-parser-java"
version = "1.739.0"
version = "1.737.0"
dependencies = [
"anyhow",
"serde_json",
@@ -6353,7 +6353,7 @@ dependencies = [
[[package]]
name = "windmill-parser-nu"
version = "1.739.0"
version = "1.737.0"
dependencies = [
"anyhow",
"nu-parser",
@@ -6364,7 +6364,7 @@ dependencies = [
[[package]]
name = "windmill-parser-php"
version = "1.739.0"
version = "1.737.0"
dependencies = [
"anyhow",
"itertools 0.14.0",
@@ -6375,7 +6375,7 @@ dependencies = [
[[package]]
name = "windmill-parser-py"
version = "1.739.0"
version = "1.737.0"
dependencies = [
"anyhow",
"itertools 0.14.0",
@@ -6387,7 +6387,7 @@ dependencies = [
[[package]]
name = "windmill-parser-py-asset"
version = "1.739.0"
version = "1.737.0"
dependencies = [
"anyhow",
"rustpython-ast",
@@ -6398,7 +6398,7 @@ dependencies = [
[[package]]
name = "windmill-parser-py-imports"
version = "1.739.0"
version = "1.737.0"
dependencies = [
"anyhow",
"async-recursion",
@@ -6420,7 +6420,7 @@ dependencies = [
[[package]]
name = "windmill-parser-r"
version = "1.739.0"
version = "1.737.0"
dependencies = [
"anyhow",
"serde_json",
@@ -6432,7 +6432,7 @@ dependencies = [
[[package]]
name = "windmill-parser-ruby"
version = "1.739.0"
version = "1.737.0"
dependencies = [
"anyhow",
"lazy_static",
@@ -6446,7 +6446,7 @@ dependencies = [
[[package]]
name = "windmill-parser-rust"
version = "1.739.0"
version = "1.737.0"
dependencies = [
"anyhow",
"convert_case",
@@ -6463,7 +6463,7 @@ dependencies = [
[[package]]
name = "windmill-parser-sql"
version = "1.739.0"
version = "1.737.0"
dependencies = [
"anyhow",
"lazy_static",
@@ -6476,7 +6476,7 @@ dependencies = [
[[package]]
name = "windmill-parser-sql-asset"
version = "1.739.0"
version = "1.737.0"
dependencies = [
"anyhow",
"serde",
@@ -6488,7 +6488,7 @@ dependencies = [
[[package]]
name = "windmill-parser-ts"
version = "1.739.0"
version = "1.737.0"
dependencies = [
"anyhow",
"lazy_static",
@@ -6506,7 +6506,7 @@ dependencies = [
[[package]]
name = "windmill-parser-ts-asset"
version = "1.739.0"
version = "1.737.0"
dependencies = [
"anyhow",
"serde-wasm-bindgen",
@@ -6522,7 +6522,7 @@ dependencies = [
[[package]]
name = "windmill-parser-wac"
version = "1.739.0"
version = "1.737.0"
dependencies = [
"anyhow",
"rustpython-ast",
@@ -6538,7 +6538,7 @@ dependencies = [
[[package]]
name = "windmill-parser-wasm"
version = "1.739.0"
version = "1.737.0"
dependencies = [
"anyhow",
"getrandom 0.2.17",
@@ -6570,7 +6570,7 @@ dependencies = [
[[package]]
name = "windmill-parser-yaml"
version = "1.739.0"
version = "1.737.0"
dependencies = [
"anyhow",
"serde",
@@ -6581,7 +6581,7 @@ dependencies = [
[[package]]
name = "windmill-types"
version = "1.739.0"
version = "1.737.0"
dependencies = [
"anyhow",
"bitflags",
@@ -12,7 +12,7 @@ resolver = "2"
members = ["."]
[workspace.package]
version = "1.739.0"
version = "1.737.0"
edition = "2021"
authors = ["Ruben Fiszel <ruben@windmill.dev>"]
@@ -112,11 +112,6 @@ pub struct ParseAssetsOutput {
// Drives the worker's write-strategy + snapshot capture.
#[serde(skip_serializing_if = "Option::is_none", default)]
pub materialize: Option<MaterializeSpec>,
// `// data_test <kind> …` — data-quality assertions run against the
// materialized asset after the write commits. Accumulating (multiple
// lines allowed). Drives the worker's post-materialize verifier probes.
#[serde(skip_serializing_if = "Vec::is_empty", default)]
pub data_tests: Vec<DataTest>,
}
#[derive(Serialize, Debug, PartialEq, Clone)]
@@ -240,41 +235,6 @@ pub struct MaterializeSpec {
pub unique_key: Option<String>,
}
// `// data_test <kind> …` — a data-quality assertion run against the
// freshly-materialized asset (post DELETE+INSERT), failing the run on
// violation. The first extensible annotation family: the parser turns a
// `data_test` line into one of a known *vocabulary* of checks, and the
// runtime turns each check into a SQL "verifier" probe. A sibling annotation
// family (e.g. column-lineage) follows the same shape — a keyword head
// selecting a variant, the rest parsed per-variant — rather than growing a
// new closed list. See `docs/ducklake-materialization.md` §"Extensible
// annotations". Multiple `// data_test` lines accumulate (unlike the
// single-value annotations above, which are first-write-wins).
//
// Built-ins mirror dbt's generic data tests; `Custom` is the escape hatch
// (dbt's singular test): a DuckDB script path whose SELECT returns the
// violating rows. The keyword is `data_test` — NOT `test` — to stay clear
// of the unrelated `// test:` CI-test annotation (see
// `windmill_common::schema::parse_ci_test_annotation`), matching dbt 1.8's
// own `tests:` → `data_tests:` rename.
#[derive(Serialize, Debug, PartialEq, Clone)]
#[serde(tag = "type", rename_all = "snake_case")]
pub enum DataTest {
// `// data_test unique <col>` — no two non-NULL rows share `column`.
Unique { column: String },
// `// data_test not_null <col>` — `column` is never NULL.
NotNull { column: String },
// `// data_test accepted_values <col> = a,b,c` — every non-NULL value of
// `column` is one of `values` (comma-separated; surrounding quotes stripped).
AcceptedValues { column: String, values: Vec<String> },
// `// data_test relationships <col> -> <asset>.<refcol>` — referential
// integrity: every non-NULL `column` value exists in `to_path`'s `to_column`.
Relationships { column: String, to_kind: AssetKind, to_path: String, to_column: String },
// `// data_test <script_path>` — escape hatch: a deployed DuckDB script
// whose trailing SELECT returns the violating rows (non-empty ⇒ fail).
Custom { path: String },
}
// `// trigger any` (default) vs `// trigger all`. `Any` = OR: any trigger
// firing runs the script (current behaviour). `All` = AND: the script
// runs only once every partition-bearing input has materialized at the
@@ -306,7 +266,6 @@ pub struct PipelineAnnotations {
pub tag: Option<String>,
pub retry: Option<RetrySpec>,
pub materialize: Option<MaterializeSpec>,
pub data_tests: Vec<DataTest>,
}
impl ParseAssetsOutput {
@@ -331,7 +290,6 @@ impl ParseAssetsOutput {
tag: pipeline.tag,
retry: pipeline.retry,
materialize: pipeline.materialize,
data_tests: pipeline.data_tests,
}
}
}
@@ -529,13 +487,9 @@ fn parse_kv_opts(s: &str) -> BTreeMap<String, String> {
out
}
// Scan the leading comment header for pipeline annotations. Only the
// contiguous block of comment lines at the top of the file is considered
// (blank lines tolerated, scan stops at the first line of actual code) so
// that ordinary comments in the body can't false-positive as annotations.
// Language-agnostic: any header line whose first non-whitespace tokens are
// a comment prefix (`//`, `#`, or `--`) followed by one of the recognized
// keywords:
// Scan raw source for pipeline annotations. Language-agnostic: any line
// whose first non-whitespace tokens are a comment prefix (`//`, `#`, or
// `--`) followed by one of the recognized keywords:
// - `pipeline` → opt-in marker (must be alone on the line)
// - `on <trigger-spec>` → asset / native trigger edge (including
// the marker-only `on schedule` form)
@@ -573,9 +527,6 @@ pub fn parse_pipeline_annotations(code: &str) -> PipelineAnnotations {
for raw_line in code.lines() {
let line = raw_line.trim_start();
if line.is_empty() {
continue;
}
let rest = if let Some(r) = line.strip_prefix("//") {
r
} else if let Some(r) = line.strip_prefix("--") {
@@ -583,11 +534,7 @@ pub fn parse_pipeline_annotations(code: &str) -> PipelineAnnotations {
} else if let Some(r) = line.strip_prefix('#') {
r
} else {
// Annotations live in the leading comment header. Stop at the first
// line of actual code so comments inside the body (e.g. a regular
// `# tag ...` prose comment) can't false-positive as annotations.
// Mirrors BashAnnotations::sandbox_image / ssh_target.
break;
continue;
};
let rest = rest.trim_start();
@@ -637,14 +584,7 @@ pub fn parse_pipeline_annotations(code: &str) -> PipelineAnnotations {
if let Some(after_kw) = consume_keyword(rest, "tag") {
let name = after_kw.trim();
// Worker tags are single-word identifiers (e.g. `heavy`, `gpu`).
// A value with whitespace or beyond the `script.tag` column width
// is almost certainly a regular comment starting with "# tag ...".
if !name.is_empty()
&& !name.contains(char::is_whitespace)
&& name.len() <= 50
&& out.tag.is_none()
{
if !name.is_empty() && out.tag.is_none() {
out.tag = Some(name.to_string());
}
continue;
@@ -668,18 +608,6 @@ pub fn parse_pipeline_annotations(code: &str) -> PipelineAnnotations {
continue;
}
// `data_test` is checked before `on`/asset shorthands and is a complete
// word (so it never collides with the `// test:` CI annotation, which
// has no whitespace after `test`). Accumulates — every well-formed line
// adds a check; malformed lines are dropped (fail-safe, the missing
// check is then simply absent from the graph + run).
if let Some(after_kw) = consume_keyword(rest, "data_test") {
if let Some(spec) = parse_data_test_spec(after_kw.trim()) {
out.data_tests.push(spec);
}
continue;
}
if let Some(after_kw) = consume_keyword(rest, "on") {
let spec_text = after_kw.trim();
if spec_text.is_empty() {
@@ -756,91 +684,6 @@ fn parse_materialize_spec(s: &str) -> Option<MaterializeSpec> {
Some(MaterializeSpec { target_kind, target_path: path.to_string(), manual, append, unique_key })
}
// Parse a `// data_test <kind> …` right-hand side into one `DataTest`. The
// leading token selects the variant; the remainder is parsed per-variant.
// Anything not matching a built-in keyword is the `Custom` escape hatch — a
// single script-path token. Returns `None` for malformed input so a typo
// fails safe (the check is dropped, never silently mis-parsed).
//
// This is the extension seam: a new built-in is one match arm + its parser;
// a sibling annotation family (column-lineage) reuses the same head-keyword
// dispatch shape rather than adding a parallel closed list.
fn parse_data_test_spec(s: &str) -> Option<DataTest> {
let s = s.trim();
if s.is_empty() {
return None;
}
let mut it = s.splitn(2, char::is_whitespace);
let head = it.next()?;
let rest = it.next().unwrap_or("").trim();
match head {
"unique" => Some(DataTest::Unique { column: single_ident(rest)? }),
"not_null" => Some(DataTest::NotNull { column: single_ident(rest)? }),
"accepted_values" => parse_accepted_values(rest),
"relationships" => parse_relationships(rest),
// Custom escape hatch: the whole right-hand side must be one path token
// (`head` with no trailing content). Trailing content after a
// non-built-in head is a malformed built-in (e.g. `uniq order_id`) and
// is rejected rather than misread as a path.
_ if rest.is_empty() => Some(DataTest::Custom { path: head.to_string() }),
_ => None,
}
}
// A single bare identifier token (column name). Rejects empty / multi-token
// input. The identifier is double-quoted + escaped at codegen, so any
// character is safe here; we only enforce "exactly one token".
fn single_ident(s: &str) -> Option<String> {
let s = s.trim();
if s.is_empty() || s.split_whitespace().count() != 1 {
return None;
}
Some(s.to_string())
}
// Strip one layer of matching surrounding single or double quotes.
fn unquote(s: &str) -> &str {
let b = s.as_bytes();
if b.len() >= 2 && (b[0] == b'"' || b[0] == b'\'') && b[b.len() - 1] == b[0] {
&s[1..s.len() - 1]
} else {
s
}
}
// `<col> = a,b,c` — column, then `=`, then a comma-separated value list.
// Surrounding quotes are stripped per value; empty values are dropped; a
// value may not itself contain a comma (v1 limitation).
fn parse_accepted_values(s: &str) -> Option<DataTest> {
let (col, vals) = s.split_once('=')?;
let column = single_ident(col)?;
let values: Vec<String> = vals
.split(',')
.map(|v| unquote(v.trim()).to_string())
.filter(|v| !v.is_empty())
.collect();
if values.is_empty() {
return None;
}
Some(DataTest::AcceptedValues { column, values })
}
// `<col> -> <asset-uri>.<refcol>` — referential integrity. The referenced
// column is the segment after the final `.`; everything before it is the
// asset URI (default-syntax shorthands enabled, like `// materialize`).
fn parse_relationships(s: &str) -> Option<DataTest> {
let (col, target) = s.split_once("->")?;
let column = single_ident(col)?;
let target = target.trim();
let (asset_uri, ref_col) = target.rsplit_once('.')?;
let to_column = single_ident(ref_col)?;
let (to_kind, to_path) = parse_asset_syntax(asset_uri.trim(), true)?;
if to_path.is_empty() {
return None;
}
Some(DataTest::Relationships { column, to_kind, to_path: to_path.to_string(), to_column })
}
// Parse a `// partitioned <kind> [opts]` right-hand side. Recognized kinds:
// `daily`, `hourly`, `weekly`, `monthly` (with optional tz/format/start),
// and `dynamic key="<jsonpath>"` (plus optional format).
@@ -1231,56 +1074,6 @@ mod pipeline_annotation_tests {
assert!(out.tag.is_none());
}
#[test]
fn tag_with_whitespace_is_skipped() {
// A regular English comment starting with "# tag " must not be
// mistaken for a worker-tag annotation (worker tags are single words).
let out =
parse_pipeline_annotations("# tag this function so we remember to refactor it later");
assert!(out.tag.is_none());
}
#[test]
fn tag_too_long_is_skipped() {
let long = "x".repeat(51);
let out = parse_pipeline_annotations(&format!("// tag {long}"));
assert!(out.tag.is_none());
}
#[test]
fn annotations_in_body_are_ignored() {
// Only the leading comment header is scanned. A regular `# tag ...`
// prose comment buried in the body — the WIN-2090 false-positive that
// crashed the `script.tag` INSERT — must not be treated as an
// annotation once real code has started.
let code = concat!(
"import pandas as pd\n",
"\n",
"def main():\n",
" # tag each row with its source so downstream steps can filter\n",
" # on s3://should/not/parse\n",
" return pd.DataFrame()\n",
);
let out = parse_pipeline_annotations(code);
assert!(out.tag.is_none());
assert!(out.triggers.is_empty());
}
#[test]
fn header_allows_blank_lines_before_code() {
// Blank lines (e.g. after a shebang) don't end the header; the first
// line of real code does.
let code = concat!(
"#!/usr/bin/env python\n",
"\n",
"# tag heavy\n",
"import os\n",
"# tag light\n",
);
let out = parse_pipeline_annotations(code);
assert_eq!(out.tag.as_deref(), Some("heavy"));
}
#[test]
fn retry_count_only() {
let out = parse_pipeline_annotations("// retry 3");
@@ -1447,102 +1240,4 @@ mod pipeline_annotation_tests {
assert_eq!(m.get("b").unwrap(), "fine");
assert!(m.get("garbage").is_none());
}
#[test]
fn data_test_builtins() {
let code = concat!(
"// data_test unique order_id\n",
"// data_test not_null user_id\n",
"// data_test accepted_values status = paid,pending,refunded\n",
"// data_test relationships user_id -> datatable://prod/users.id\n",
);
let out = parse_pipeline_annotations(code);
assert_eq!(
out.data_tests,
vec![
DataTest::Unique { column: "order_id".to_string() },
DataTest::NotNull { column: "user_id".to_string() },
DataTest::AcceptedValues {
column: "status".to_string(),
values: vec![
"paid".to_string(),
"pending".to_string(),
"refunded".to_string()
],
},
DataTest::Relationships {
column: "user_id".to_string(),
to_kind: AssetKind::DataTable,
to_path: "prod/users".to_string(),
to_column: "id".to_string(),
},
]
);
}
#[test]
fn data_test_accepts_quotes_and_spacing() {
let out = parse_pipeline_annotations("// data_test accepted_values kind = \"a b\", 'c' ,d");
assert_eq!(
out.data_tests,
vec![DataTest::AcceptedValues {
column: "kind".to_string(),
values: vec!["a b".to_string(), "c".to_string(), "d".to_string()],
}]
);
}
#[test]
fn data_test_custom_escape_hatch() {
// A non-built-in single token is a custom script path; default-syntax
// asset shorthands are NOT triggered here (a path is just a path).
let out = parse_pipeline_annotations("// data_test f/tests/orders_amount_sane");
assert_eq!(
out.data_tests,
vec![DataTest::Custom { path: "f/tests/orders_amount_sane".to_string() }]
);
}
#[test]
fn data_test_relationships_ducklake_shorthand() {
let out = parse_pipeline_annotations(
"// data_test relationships sku -> ducklake://warehouse/dim_products.sku",
);
assert_eq!(
out.data_tests,
vec![DataTest::Relationships {
column: "sku".to_string(),
to_kind: AssetKind::Ducklake,
to_path: "warehouse/dim_products".to_string(),
to_column: "sku".to_string(),
}]
);
}
#[test]
fn data_test_malformed_dropped_fail_safe() {
// A misspelled built-in with trailing content is not a valid path token
// → dropped, not misread as a custom test. An empty value list, a
// missing arrow target, and a bare keyword are all dropped too.
let out = parse_pipeline_annotations(concat!(
"// data_test uniq order_id\n", // typo'd built-in + arg
"// data_test accepted_values s =\n", // no values
"// data_test relationships a -> b\n", // no `.refcol`
"// data_test unique\n", // missing column
"// data_test\n", // bare keyword
));
assert!(out.data_tests.is_empty());
}
#[test]
fn data_test_not_confused_with_ci_test_annotation() {
// `// test:` is the unrelated CI-test annotation — it must NOT be
// parsed as a data test (no whitespace after `test`, and the keyword
// is `data_test` anyway).
let out = parse_pipeline_annotations("// test: f/foo/bar\n// data_test unique id");
assert_eq!(
out.data_tests,
vec![DataTest::Unique { column: "id".to_string() }]
);
}
}
@@ -482,8 +482,7 @@ pub fn build_wrap_blocks(
partition_value_sql: &str,
partitioned: bool,
strategy: MaterializeStrategy,
tests: &[DataTestResolved],
) -> Result<Vec<String>, String> {
) -> Vec<String> {
let target_qualified = format!("{TARGET_ALIAS}.{target_table}");
let cg = MaterializeCodegen {
target_qualified: &target_qualified,
@@ -493,14 +492,6 @@ pub fn build_wrap_blocks(
partitioned,
strategy,
};
let ctx = DataTestCtx {
target_qualified: &target_qualified,
asset_path,
partition_col,
partition_value_sql,
partitioned,
};
let test_sql = build_data_test_checks(tests, &ctx)?;
let mut blocks: Vec<String> = Vec::new();
// Setup blocks come from the splitter with their `;` stripped — re-terminate
// each so that when the executor re-joins and re-splits the assembled query,
@@ -508,20 +499,15 @@ pub fn build_wrap_blocks(
// don't merge into one malformed statement.
blocks.extend(plan.setup.iter().map(|s| terminate(s)));
blocks.push(target_attach.to_string());
// Referenced-asset ATTACHes (relationships tests) — read-only, before the
// write and the summary that probes them.
blocks.extend(test_sql.attaches);
blocks.extend(cg.statements());
// The summary read carries the per-test breakdown (when any tests apply).
blocks.push(materialize_result_sql(
&target_qualified,
asset_path,
partition_col,
partition_value_sql,
partitioned,
&test_sql.checks,
));
Ok(blocks)
blocks
}
/// The trailing one-row summary the materialize run returns: the asset it
@@ -534,7 +520,6 @@ pub fn materialize_result_sql(
partition_col: &str,
partition_value_sql: &str,
partitioned: bool,
checks: &[DataTestCheck],
) -> String {
let (count_expr, partition_sel) = if partitioned {
// Row count is the slice this run wrote (the partition); `partition`
@@ -551,38 +536,10 @@ pub fn materialize_result_sql(
String::new(),
)
};
let base_cols = format!(
"'ducklake://{asset_path}' AS materialized, \
{partition_sel}{count_expr} AS rows, \
(SELECT max(snapshot_id) FROM ducklake_snapshots('{TARGET_ALIAS}')) AS snapshot_id"
);
if checks.is_empty() {
return format!("SELECT {base_cols};");
}
// Per-test breakdown. Each check's violating-count is computed once as a CTE
// column (`c0`, `c1`, …); the `data_tests` list-of-struct then references
// those columns — DuckDB rejects scalar subqueries *inside* a struct/list
// literal, hence the CTE. Names are single-quote-escaped. The result row
// carries the whole breakdown so the worker runs every test (no
// abort-on-first) and decides pass/fail itself.
let cte_cols = checks
.iter()
.enumerate()
.map(|(i, c)| format!("{} AS c{i}", c.violating))
.collect::<Vec<_>>()
.join(", ");
let list_items = checks
.iter()
.enumerate()
.map(|(i, c)| {
let name = c.name.replace('\'', "''");
format!("{{'test': '{name}', 'violating': c{i}}}")
})
.collect::<Vec<_>>()
.join(", ");
format!(
"WITH _wm_tr AS (SELECT {cte_cols}) \
SELECT {base_cols}, [{list_items}] AS data_tests FROM _wm_tr;"
"SELECT 'ducklake://{asset_path}' AS materialized, \
{partition_sel}{count_expr} AS rows, \
(SELECT max(snapshot_id) FROM ducklake_snapshots('{TARGET_ALIAS}')) AS snapshot_id;"
)
}
@@ -596,271 +553,6 @@ fn terminate(stmt: &str) -> String {
}
}
// ---------------------------------------------------------------------------
// Data tests (`// data_test`)
// ---------------------------------------------------------------------------
//
// A data test is the FIRST extensible annotation: the parser yields a
// `DataTest` from a known vocabulary, and this module turns each into a
// *check* — a `(name, violating-row-count query)` pair — that runs against the
// freshly-materialized target after the write commits. The materialize summary
// query embeds every check's count in one `data_tests` column, so all tests
// run in a single pass (no abort-on-first) and the worker, not the SQL,
// decides pass/fail and reports the full per-test breakdown.
//
// The pattern is deliberately open: a verifier is just `(name, count query)`.
// Built-ins differ only in their count query; the `Custom` escape hatch
// supplies its own (a user SELECT returning the violating rows). A sibling
// annotation family (column-lineage) can emit its own checks through the same
// `push_check` shape rather than bolting on a parallel mechanism. See
// `docs/ducklake-materialization.md`.
use crate::asset_parser::{AssetKind, DataTest};
/// Target context a data-test probe runs against — the materialized table and
/// the partition slice (when partitioned, tests are scoped to the slice just
/// written, so a rerun/backfill is independent of other partitions' data).
#[derive(Debug, Clone)]
pub struct DataTestCtx<'a> {
/// Fully-qualified materialized target, e.g. `_wm_target.orders`.
pub target_qualified: &'a str,
/// `<name>/<table>` of the target, for human-readable probe messages.
pub asset_path: &'a str,
/// Physical partition column on the managed table.
pub partition_col: &'a str,
/// SQL literal/expression for the current partition value (already escaped).
pub partition_value_sql: &'a str,
/// Whether the target is partitioned (scopes probes to the slice).
pub partitioned: bool,
}
/// A data test resolved enough to generate SQL. Built-ins carry only their
/// parsed `DataTest`; `Custom` additionally carries the fetched script body
/// (the parser crate can't fetch it — the worker does and passes it in).
#[derive(Debug, Clone)]
pub enum DataTestResolved {
BuiltIn(DataTest),
Custom { path: String, body: String },
}
/// One compiled data-test check: a human-readable `name` and a scalar SQL
/// expression (`violating`) yielding the number of rows that violate it (0 =
/// pass). The materialize summary query embeds every check's count so the
/// worker gets the whole breakdown in one result — all tests run (no
/// abort-on-first) and the worker, not the SQL, decides pass/fail.
#[derive(Debug, Clone, PartialEq, Eq)]
pub struct DataTestCheck {
pub name: String,
/// Scalar subquery yielding the violating-row count, e.g.
/// `(SELECT count(*) AS v FROM (…))`.
pub violating: String,
}
/// The SQL a set of data tests compiles to: referenced-asset `ATTACH`
/// statements (resolved by the executor's ATTACH-transform pass) and the
/// per-test checks, both in declaration order.
#[derive(Debug, Clone, Default, PartialEq, Eq)]
pub struct DataTestChecks {
pub attaches: Vec<String>,
pub checks: Vec<DataTestCheck>,
}
/// Alias prefix for a relationships test's referenced asset, attached
/// read-only alongside the target. `_wm_ref_<n>` so it never collides with the
/// user's aliases or the reserved `_wm_target`.
const REF_ALIAS_PREFIX: &str = "_wm_ref_";
// Double-quote a SQL identifier, escaping embedded quotes — so an arbitrary
// column name from an annotation can't break out of the identifier.
fn quote_ident(id: &str) -> String {
format!("\"{}\"", id.replace('"', "\"\""))
}
// Quote a possibly schema-qualified table reference (`schema.table`) by quoting
// each dotted segment independently: `main.dim_products` → `"main"."dim_products"`.
// Quoting the whole thing would make DuckDB read it as one table name containing
// a literal dot, querying the wrong table.
fn quote_qualified(name: &str) -> String {
name.split('.')
.map(quote_ident)
.collect::<Vec<_>>()
.join(".")
}
// Single-quote a SQL string literal, escaping embedded quotes.
fn quote_lit(s: &str) -> String {
format!("'{}'", s.replace('\'', "''"))
}
// The `WHERE`/`AND` fragment scoping a probe to the current partition, or
// empty when unpartitioned. `prefix` is `WHERE ` or `AND ` per call site.
fn partition_scope(ctx: &DataTestCtx, prefix: &str, table_alias: Option<&str>) -> String {
if !ctx.partitioned {
return String::new();
}
let col = match table_alias {
Some(a) => format!("{a}.{}", quote_ident(ctx.partition_col)),
None => quote_ident(ctx.partition_col),
};
format!("{prefix}{col} = {}", ctx.partition_value_sql)
}
// Record one check: its display `name` plus `count_query` (which yields a
// single-column violating-row count) wrapped as a scalar subquery.
fn push_check(out: &mut DataTestChecks, name: String, count_query: String) {
out.checks
.push(DataTestCheck { name, violating: format!("({count_query})") });
}
/// Compile resolved data tests into ATTACH statements + per-test checks for
/// `ctx`'s target. Pure: returns SQL text, executes nothing. Errors carry an
/// actionable message (e.g. a relationships target that isn't an attachable
/// table).
pub fn build_data_test_checks(
tests: &[DataTestResolved],
ctx: &DataTestCtx,
) -> Result<DataTestChecks, String> {
let t = ctx.target_qualified;
let mut out = DataTestChecks::default();
// Dedup ref attaches by (kind, name): a database can't be attached twice,
// so multiple relationships into the same db share one alias.
let mut ref_aliases: Vec<(AssetKind, String, String)> = Vec::new();
for resolved in tests {
match resolved {
DataTestResolved::BuiltIn(DataTest::Unique { column }) => {
let c = quote_ident(column);
let scope = partition_scope(ctx, " AND ", None);
let q = format!(
"SELECT count(*) AS v FROM (SELECT {c} FROM {t} WHERE {c} IS NOT NULL{scope} \
GROUP BY {c} HAVING count(*) > 1)"
);
push_check(&mut out, format!("unique({column})"), q);
}
DataTestResolved::BuiltIn(DataTest::NotNull { column }) => {
let c = quote_ident(column);
let scope = partition_scope(ctx, " AND ", None);
let q = format!("SELECT count(*) AS v FROM {t} WHERE {c} IS NULL{scope}");
push_check(&mut out, format!("not_null({column})"), q);
}
DataTestResolved::BuiltIn(DataTest::AcceptedValues { column, values }) => {
let c = quote_ident(column);
let scope = partition_scope(ctx, " AND ", None);
let list = values
.iter()
.map(|v| quote_lit(v))
.collect::<Vec<_>>()
.join(", ");
let q = format!(
"SELECT count(*) AS v FROM {t} WHERE {c} IS NOT NULL AND {c} NOT IN ({list}){scope}"
);
push_check(&mut out, format!("accepted_values({column})"), q);
}
DataTestResolved::BuiltIn(DataTest::Relationships {
column,
to_kind,
to_path,
to_column,
}) => {
let (ref_name, ref_table) = to_path.split_once('/').ok_or_else(|| {
format!("data_test relationships: target `{to_path}` must be `<name>/<table>`")
})?;
if ref_table.is_empty() {
return Err(format!(
"data_test relationships: target `{to_path}` has no table"
));
}
let scheme = match to_kind {
AssetKind::Ducklake => "ducklake",
AssetKind::DataTable => "datatable",
other => {
return Err(format!(
"data_test relationships: target kind {other:?} is not an attachable \
table (use ducklake:// or datatable://)"
))
}
};
// The materialize target's ducklake is already attached as
// `_wm_target`; a reference into that same lake must reuse it
// rather than ATTACH the same database again under a fresh alias
// (DuckDB forbids attaching one database twice). `asset_path` is
// the target's `<lake>/<table>`, so its lake is the part before
// the first `/`.
let target_lake = ctx.asset_path.split('/').next().unwrap_or("");
let alias = if *to_kind == AssetKind::Ducklake && ref_name == target_lake {
TARGET_ALIAS.to_string()
} else {
// Reuse an existing alias for the same (kind, name), else mint one.
match ref_aliases
.iter()
.find(|(k, n, _)| k == to_kind && n == ref_name)
{
Some((_, _, a)) => a.clone(),
None => {
let a = format!("{REF_ALIAS_PREFIX}{}", ref_aliases.len());
// Escape the name — it is interpolated into a
// single-quoted DuckDB literal (defense-in-depth: the
// name is deploy-time annotation content, but the parser
// places no character restriction on asset paths).
let esc_name = ref_name.replace('\'', "''");
out.attaches
.push(format!("ATTACH '{scheme}://{esc_name}' AS {a};"));
ref_aliases.push((*to_kind, ref_name.to_string(), a.clone()));
a
}
}
};
let c = quote_ident(column);
let rc = quote_ident(to_column);
// `ref_table` may be schema-qualified (`schema.table`); quote each
// segment so the dot stays a schema separator, not a literal.
let rt = quote_qualified(ref_table);
let scope = partition_scope(ctx, " AND ", Some("_wm_src"));
let q = format!(
"SELECT count(*) AS v FROM {t} _wm_src \
WHERE _wm_src.{c} IS NOT NULL{scope} \
AND NOT EXISTS (SELECT 1 FROM {alias}.{rt} _wm_ref \
WHERE _wm_ref.{rc} = _wm_src.{c})"
);
push_check(
&mut out,
format!("relationships({column} -> {to_path}.{to_column})"),
q,
);
}
// A parsed Custom must be resolved (body fetched) before codegen.
DataTestResolved::BuiltIn(DataTest::Custom { path }) => {
return Err(format!(
"data_test custom `{path}`: body not resolved before codegen (internal)"
));
}
DataTestResolved::Custom { path, body } => {
// dbt singular-test convention: the body is a *single* SELECT
// (or CTE) returning the violating rows. It is embedded as a
// subquery (`FROM (<body>)`), so a multi-statement body would
// produce invalid SQL — validate up front with an actionable
// error. It runs in the target's connection (can read
// `_wm_target` + the user's attaches); partition substitution is
// already applied by the worker.
let stmts = split_statements(body);
if stmts.is_empty() {
return Err(format!("data_test custom `{path}`: empty test body"));
}
if stmts.len() > 1 {
return Err(format!(
"data_test custom `{path}`: must be a single SELECT returning the \
violating rows (found {} statements)",
stmts.len()
));
}
let q = format!("SELECT count(*) AS v FROM ({})", stmts[0]);
push_check(&mut out, format!("custom({path})"), q);
}
}
}
Ok(out)
}
#[cfg(test)]
mod tests {
use super::*;
@@ -1100,9 +792,7 @@ mod tests {
"'2026-06-19'",
true,
MaterializeStrategy::Replace,
&[],
)
.unwrap();
);
// setup block first, then the target ATTACH, then codegen, then result.
assert!(blocks[0].starts_with("ATTACH 'ducklake://main' AS dl"));
// every setup block must be `;`-terminated so re-splitting can't merge it
@@ -1124,247 +814,4 @@ mod tests {
assert!(last.contains("WHERE _wm_partition = '2026-06-19') AS rows"));
assert!(last.contains("ducklake_snapshots('_wm_target')"));
}
// -- data tests ---------------------------------------------------------
fn ctx_partitioned() -> DataTestCtx<'static> {
DataTestCtx {
target_qualified: "_wm_target.orders",
asset_path: "analytics/orders",
partition_col: "_wm_partition",
partition_value_sql: "'2026-06-19'",
partitioned: true,
}
}
fn ctx_unpartitioned() -> DataTestCtx<'static> {
DataTestCtx { partitioned: false, ..ctx_partitioned() }
}
#[test]
fn data_test_unique_and_not_null_partition_scoped() {
let tests = vec![
DataTestResolved::BuiltIn(DataTest::Unique { column: "order_id".into() }),
DataTestResolved::BuiltIn(DataTest::NotNull { column: "user_id".into() }),
];
let sql = build_data_test_checks(&tests, &ctx_partitioned()).unwrap();
assert!(sql.attaches.is_empty());
assert_eq!(sql.checks.len(), 2);
// short, asset-free names (the asset is shown once by the breakdown).
assert_eq!(sql.checks[0].name, "unique(order_id)");
assert_eq!(sql.checks[1].name, "not_null(user_id)");
// each `violating` is a scalar count subquery.
assert!(sql.checks[0]
.violating
.starts_with("(SELECT count(*) AS v FROM"));
// unique: groups non-null keys within the slice, having count>1
assert!(sql.checks[0]
.violating
.contains("GROUP BY \"order_id\" HAVING count(*) > 1"));
assert!(sql.checks[0]
.violating
.contains("\"order_id\" IS NOT NULL AND \"_wm_partition\" = '2026-06-19'"));
// not_null: null rows in the slice
assert!(sql.checks[1]
.violating
.contains("WHERE \"user_id\" IS NULL AND \"_wm_partition\" = '2026-06-19'"));
}
#[test]
fn data_test_unpartitioned_has_no_partition_scope() {
let tests = vec![DataTestResolved::BuiltIn(DataTest::NotNull {
column: "id".into(),
})];
let sql = build_data_test_checks(&tests, &ctx_unpartitioned()).unwrap();
assert!(sql.checks[0].violating.contains("WHERE \"id\" IS NULL"));
assert!(!sql.checks[0].violating.contains("_wm_partition"));
}
#[test]
fn data_test_accepted_values_escapes_literals() {
let tests = vec![DataTestResolved::BuiltIn(DataTest::AcceptedValues {
column: "status".into(),
values: vec!["paid".into(), "o'brien".into()],
})];
let sql = build_data_test_checks(&tests, &ctx_unpartitioned()).unwrap();
assert!(sql.checks[0]
.violating
.contains("NOT IN ('paid', 'o''brien')"));
assert!(sql.checks[0].violating.contains("\"status\" IS NOT NULL"));
}
#[test]
fn data_test_relationships_attaches_ref_and_dedups() {
let tests = vec![
DataTestResolved::BuiltIn(DataTest::Relationships {
column: "user_id".into(),
to_kind: AssetKind::DataTable,
to_path: "prod/users".into(),
to_column: "id".into(),
}),
// second relationship into the SAME db reuses the alias (no 2nd attach)
DataTestResolved::BuiltIn(DataTest::Relationships {
column: "buyer_id".into(),
to_kind: AssetKind::DataTable,
to_path: "prod/buyers".into(),
to_column: "id".into(),
}),
];
let sql = build_data_test_checks(&tests, &ctx_partitioned()).unwrap();
assert_eq!(sql.attaches.len(), 1, "same db attached once");
assert_eq!(sql.attaches[0], "ATTACH 'datatable://prod' AS _wm_ref_0;");
assert!(sql.checks[0]
.violating
.contains("NOT EXISTS (SELECT 1 FROM _wm_ref_0.\"users\""));
assert!(sql.checks[1]
.violating
.contains("NOT EXISTS (SELECT 1 FROM _wm_ref_0.\"buyers\""));
assert!(sql.checks[0]
.violating
.contains("_wm_src.\"_wm_partition\" = '2026-06-19'"));
assert_eq!(
sql.checks[0].name,
"relationships(user_id -> prod/users.id)"
);
}
#[test]
fn data_test_relationships_escapes_ref_name_in_attach() {
let tests = vec![DataTestResolved::BuiltIn(DataTest::Relationships {
column: "k".into(),
to_kind: AssetKind::DataTable,
to_path: "ev'il/users".into(),
to_column: "id".into(),
})];
let sql = build_data_test_checks(&tests, &ctx_unpartitioned()).unwrap();
// single quote in the name is doubled so it can't break out of the literal
assert_eq!(sql.attaches[0], "ATTACH 'datatable://ev''il' AS _wm_ref_0;");
}
#[test]
fn data_test_relationships_same_lake_reuses_target() {
// A relationship into the SAME ducklake as the materialize target
// (asset_path = "analytics/orders") must NOT re-ATTACH it — _wm_target
// already holds that catalog; reuse it.
let tests = vec![DataTestResolved::BuiltIn(DataTest::Relationships {
column: "user_id".into(),
to_kind: AssetKind::Ducklake,
to_path: "analytics/users".into(),
to_column: "id".into(),
})];
let sql = build_data_test_checks(&tests, &ctx_unpartitioned()).unwrap();
assert!(
sql.attaches.is_empty(),
"same-lake ref must not ATTACH again"
);
assert!(sql.checks[0]
.violating
.contains("NOT EXISTS (SELECT 1 FROM _wm_target.\"users\""));
}
#[test]
fn data_test_relationships_schema_qualified_target() {
// `<lake>/<schema>.<table>` — the schema-qualified table must quote each
// segment so the dot stays a separator, not part of one identifier.
let tests = vec![DataTestResolved::BuiltIn(DataTest::Relationships {
column: "sku".into(),
to_kind: AssetKind::Ducklake,
to_path: "warehouse/main.dim_products".into(),
to_column: "sku".into(),
})];
let sql = build_data_test_checks(&tests, &ctx_unpartitioned()).unwrap();
assert_eq!(
sql.attaches[0],
"ATTACH 'ducklake://warehouse' AS _wm_ref_0;"
);
assert!(
sql.checks[0]
.violating
.contains("FROM _wm_ref_0.\"main\".\"dim_products\""),
"schema-qualified target should be quoted per segment: {}",
sql.checks[0].violating
);
}
#[test]
fn data_test_relationships_rejects_non_attachable_kind() {
let tests = vec![DataTestResolved::BuiltIn(DataTest::Relationships {
column: "k".into(),
to_kind: AssetKind::S3Object,
to_path: "bucket/file".into(),
to_column: "c".into(),
})];
assert!(build_data_test_checks(&tests, &ctx_unpartitioned()).is_err());
}
#[test]
fn data_test_custom_wraps_body() {
let tests = vec![DataTestResolved::Custom {
path: "f/tests/amount".into(),
body: "SELECT * FROM _wm_target.orders WHERE amount < 0;".into(),
}];
let sql = build_data_test_checks(&tests, &ctx_unpartitioned()).unwrap();
// trailing ; stripped, wrapped as a count subquery
assert!(sql.checks[0].violating.contains(
"SELECT count(*) AS v FROM (SELECT * FROM _wm_target.orders WHERE amount < 0)"
));
assert_eq!(sql.checks[0].name, "custom(f/tests/amount)");
}
#[test]
fn data_test_custom_rejects_multi_statement_body() {
// The body is embedded as a subquery, so a setup-then-SELECT body would
// produce invalid SQL — reject it up front with an actionable error.
let tests = vec![DataTestResolved::Custom {
path: "f/tests/amount".into(),
body: "SET threads = 1; SELECT * FROM _wm_target.orders WHERE amount < 0".into(),
}];
let err = build_data_test_checks(&tests, &ctx_unpartitioned()).unwrap_err();
assert!(err.contains("single SELECT"), "unexpected error: {err}");
}
#[test]
fn data_test_unresolved_custom_is_internal_error() {
let tests = vec![DataTestResolved::BuiltIn(DataTest::Custom {
path: "f/x".into(),
})];
assert!(build_data_test_checks(&tests, &ctx_unpartitioned()).is_err());
}
#[test]
fn materialize_result_sql_embeds_data_tests_breakdown() {
let checks = vec![
DataTestCheck {
name: "unique(order_id)".into(),
violating: "(SELECT count(*) AS v FROM q0)".into(),
},
DataTestCheck {
name: "custom(f/t)".into(),
violating: "(SELECT count(*) AS v FROM q1)".into(),
},
];
let sql = materialize_result_sql(
"_wm_target.orders",
"analytics/orders",
"_wm_partition",
"'2026-06-19'",
false,
&checks,
);
// counts computed once in a CTE, referenced by the list-of-struct.
assert!(sql.starts_with("WITH _wm_tr AS (SELECT (SELECT count(*) AS v FROM q0) AS c0,"));
assert!(sql.contains("[{'test': 'unique(order_id)', 'violating': c0}, "));
assert!(sql.contains("{'test': 'custom(f/t)', 'violating': c1}] AS data_tests"));
assert!(sql.contains("FROM _wm_tr;"));
// no tests -> plain summary, no CTE / data_tests column.
let plain = materialize_result_sql(
"_wm_target.orders",
"analytics/orders",
"_wm_partition",
"'x'",
false,
&[],
);
assert!(plain.starts_with("SELECT 'ducklake://analytics/orders' AS materialized"));
assert!(!plain.contains("data_tests"));
}
}
@@ -301,119 +301,5 @@
"tag": null,
"retry": null
}
},
{
"name": "data_test built-ins accumulate in order",
"code": "-- pipeline\n-- materialize ducklake://analytics/orders key=order_id\n-- data_test unique order_id\n-- data_test not_null user_id\n-- data_test accepted_values status = paid,pending,refunded\n-- data_test relationships user_id -> datatable://prod/users.id\nSELECT 1;",
"expected": {
"in_pipeline": true,
"asset_triggers": [],
"native_triggers": [],
"partition": null,
"freshness": null,
"tag": null,
"retry": null,
"materialize": {
"target_kind": "ducklake",
"target_path": "analytics/orders",
"unique_key": "order_id"
},
"data_tests": [
{ "type": "unique", "column": "order_id" },
{ "type": "not_null", "column": "user_id" },
{
"type": "accepted_values",
"column": "status",
"values": ["paid", "pending", "refunded"]
},
{
"type": "relationships",
"column": "user_id",
"to_kind": "datatable",
"to_path": "prod/users",
"to_column": "id"
}
]
}
},
{
"name": "data_test accepted_values strips quotes and spacing",
"code": "# data_test accepted_values kind = \"a b\", 'c' ,d\nprint(1)",
"expected": {
"in_pipeline": false,
"asset_triggers": [],
"native_triggers": [],
"partition": null,
"freshness": null,
"tag": null,
"retry": null,
"data_tests": [
{ "type": "accepted_values", "column": "kind", "values": ["a b", "c", "d"] }
]
}
},
{
"name": "data_test custom escape hatch is a script path",
"code": "// data_test f/tests/orders_amount_sane\nexport function main() {}",
"expected": {
"in_pipeline": false,
"asset_triggers": [],
"native_triggers": [],
"partition": null,
"freshness": null,
"tag": null,
"retry": null,
"data_tests": [{ "type": "custom", "path": "f/tests/orders_amount_sane" }]
}
},
{
"name": "data_test relationships with ducklake shorthand target",
"code": "// data_test relationships sku -> ducklake://warehouse/dim_products.sku\nSELECT 1;",
"expected": {
"in_pipeline": false,
"asset_triggers": [],
"native_triggers": [],
"partition": null,
"freshness": null,
"tag": null,
"retry": null,
"data_tests": [
{
"type": "relationships",
"column": "sku",
"to_kind": "ducklake",
"to_path": "warehouse/dim_products",
"to_column": "sku"
}
]
}
},
{
"name": "malformed data_test lines are dropped fail-safe",
"code": "// data_test uniq order_id\n// data_test accepted_values s =\n// data_test relationships a -> b\n// data_test unique\n// data_test\n// data_test unique id\nSELECT 1;",
"expected": {
"in_pipeline": false,
"asset_triggers": [],
"native_triggers": [],
"partition": null,
"freshness": null,
"tag": null,
"retry": null,
"data_tests": [{ "type": "unique", "column": "id" }]
}
},
{
"name": "ci test annotation is not a data test",
"code": "// test: f/foo/bar\n// data_test unique id\nSELECT 1;",
"expected": {
"in_pipeline": false,
"asset_triggers": [],
"native_triggers": [],
"partition": null,
"freshness": null,
"tag": null,
"retry": null,
"data_tests": [{ "type": "unique", "column": "id" }]
}
}
]
@@ -38,10 +38,6 @@ struct Expected {
// deserializing; only fixtures exercising materialization set it.
#[serde(default)]
materialize: Option<ExpectedMaterialize>,
// Snake_case `DataTest` serde shape (e.g. {"type":"unique","column":"x"}),
// compared against `serde_json::to_value(got.data_tests)`. Absent === [].
#[serde(default)]
data_tests: Vec<serde_json::Value>,
}
#[derive(Deserialize)]
@@ -193,12 +189,5 @@ fn pipeline_annotation_fixtures_match() {
want.is_some()
),
}
let got_tests = serde_json::to_value(&got.data_tests).expect("data_tests serialize");
assert_eq!(
got_tests,
serde_json::Value::Array(f.expected.data_tests.clone()),
"{ctx}: data tests"
);
}
}
+33 -108
View File
@@ -1324,9 +1324,6 @@ pub async fn delete_expired_items(db: &DB) -> () {
let cleanup_start = Instant::now();
let mut total_deleted = 0u64;
let mut batch_num = 0i32;
// Watermark carried across batches so each one resumes after the rows the previous batch
// already processed instead of re-scanning the (potentially undeletable) oldest prefix.
let mut completed_at_floor: Option<DateTime<Utc>> = None;
// Process batches until no more expired jobs or max batches reached
loop {
@@ -1339,17 +1336,14 @@ pub async fn delete_expired_items(db: &DB) -> () {
}
// Each batch runs in its own transaction to avoid long-running locks
let batch_result =
delete_expired_jobs_batch(db, job_retention_secs, batch_size, completed_at_floor)
.await;
let batch_result = delete_expired_jobs_batch(db, job_retention_secs, batch_size).await;
match batch_result {
Ok((deleted_count, max_completed_at)) => {
Ok(deleted_count) => {
if deleted_count == 0 {
// No more expired jobs to delete
break;
}
completed_at_floor = max_completed_at.or(completed_at_floor);
total_deleted += deleted_count as u64;
batch_num += 1;
}
@@ -1516,20 +1510,12 @@ pub async fn check_expiring_tokens(db: &DB) {
/// Delete a batch of expired jobs with LIMIT and SKIP LOCKED for high-scale environments.
/// Uses a single transaction per batch to minimize lock duration.
///
/// `completed_at_floor` is the watermark from the previous batch in the same cleanup run (the
/// max `completed_at` it deleted); pass `None` for the first batch. It is re-applied as
/// `completed_at >= floor` so the scan resumes past the rows already processed instead of
/// re-walking them (see the inline comment on the DELETE for why this matters).
///
/// Returns `(jobs deleted in this batch, max completed_at deleted)`. The caller feeds the
/// returned watermark back in as `completed_at_floor` for the next batch.
/// Returns the number of jobs deleted in this batch.
async fn delete_expired_jobs_batch(
db: &DB,
job_retention_secs: i64,
batch_size: i64,
completed_at_floor: Option<DateTime<Utc>>,
) -> error::Result<(usize, Option<DateTime<Utc>>)> {
) -> error::Result<usize> {
let mut tx = db.begin().await?;
// Fetch active ROOT job IDs that started before the retention period. We only care about
@@ -1545,70 +1531,34 @@ async fn delete_expired_jobs_batch(
.fetch_all(&mut *tx)
.await?;
// `completed_at_floor` is a watermark carried across batches within a cleanup run: it is the
// max(completed_at) deleted by the previous batch. Re-applying it as `completed_at >= floor`
// lets each batch resume after the rows the previous batch already processed instead of
// re-scanning them. This matters when the oldest rows are undeletable (children of a
// still-active root flow): without the floor the `ORDER BY completed_at ASC` scan walks that
// same protected prefix on every batch, turning a cleanup run quadratic in prefix size.
// Floor only ever skips rows the current run already deleted, was protecting, or skip-locked —
// all correctly deferred to the next run, identical to the unbounded scan's semantics.
//
// Use FOR UPDATE SKIP LOCKED to avoid contention between replicas; ORDER BY completed_at
// deletes oldest jobs first.
let (deleted_jobs, max_completed_at) = if active_root_job_ids.is_empty() {
// Common case: no old root flow is still running, so nothing is protected and the
// v2_job join (a PK lookup per candidate) is pure overhead — skip it entirely.
let rows = sqlx::query!(
"DELETE FROM v2_job_completed
WHERE id IN (
SELECT id FROM v2_job_completed
WHERE completed_at <= now() - ($1::bigint::text || ' s')::interval
AND ($3::timestamptz IS NULL OR completed_at >= $3)
ORDER BY completed_at ASC
LIMIT $2
FOR UPDATE SKIP LOCKED
)
RETURNING id, completed_at",
job_retention_secs,
batch_size,
completed_at_floor,
)
.fetch_all(&mut *tx)
.await?;
let max = rows.iter().map(|r| r.completed_at).max();
(rows.into_iter().map(|r| r.id).collect::<Vec<Uuid>>(), max)
} else {
// Active-root exclusion uses `NOT IN (SELECT ... unnest($3))` rather than `!= ALL($3)`:
// the subquery form lets the planner build a one-time hashed SubPlan and apply it as a
// filter on the ordered index scan, giving O(1) membership per candidate instead of a
// per-row linear array scan (which degrades sharply when many root jobs are active). The
// `u IS NOT NULL` guard sidesteps NOT IN's null-trap semantics ($3 holds non-null PK ids).
let rows = sqlx::query!(
"DELETE FROM v2_job_completed
WHERE id IN (
SELECT jc.id FROM v2_job_completed jc
LEFT JOIN v2_job j ON j.id = jc.id
WHERE jc.completed_at <= now() - ($1::bigint::text || ' s')::interval
AND ($4::timestamptz IS NULL OR jc.completed_at >= $4)
AND COALESCE(j.root_job, j.flow_innermost_root_job, jc.id) NOT IN (
SELECT u FROM unnest($3::uuid[]) AS u WHERE u IS NOT NULL
)
ORDER BY jc.completed_at ASC
LIMIT $2
FOR UPDATE OF jc SKIP LOCKED
)
RETURNING id, completed_at",
job_retention_secs,
batch_size,
&active_root_job_ids,
completed_at_floor,
)
.fetch_all(&mut *tx)
.await?;
let max = rows.iter().map(|r| r.completed_at).max();
(rows.into_iter().map(|r| r.id).collect::<Vec<Uuid>>(), max)
};
// Use FOR UPDATE SKIP LOCKED to avoid contention between replicas
// ORDER BY completed_at ensures we delete oldest jobs first.
// Active-root exclusion uses `NOT IN (SELECT ... unnest($3))` rather than
// `!= ALL($3)`: the subquery form lets the planner build a one-time hashed
// SubPlan and apply it as a filter on the ordered index scan, giving O(1)
// membership per candidate instead of a per-row linear array scan (which
// degrades sharply when many root jobs are active). The `u IS NOT NULL` guard
// sidesteps NOT IN's null-trap semantics ($3 holds non-null PK ids).
let deleted_jobs: Vec<Uuid> = sqlx::query_scalar!(
"DELETE FROM v2_job_completed
WHERE id IN (
SELECT jc.id FROM v2_job_completed jc
LEFT JOIN v2_job j ON j.id = jc.id
WHERE jc.completed_at <= now() - ($1::bigint::text || ' s')::interval
AND COALESCE(j.root_job, j.flow_innermost_root_job, jc.id) NOT IN (
SELECT u FROM unnest($3::uuid[]) AS u WHERE u IS NOT NULL
)
ORDER BY jc.completed_at ASC
LIMIT $2
FOR UPDATE OF jc SKIP LOCKED
)
RETURNING id",
job_retention_secs,
batch_size,
&active_root_job_ids
)
.fetch_all(&mut *tx)
.await?;
let deleted_count = deleted_jobs.len();
@@ -1668,7 +1618,7 @@ async fn delete_expired_jobs_batch(
tx.commit().await?;
Ok((deleted_count, max_completed_at))
Ok(deleted_count)
}
async fn delete_log_files_from_disk_and_store(
@@ -4429,32 +4379,9 @@ RETURNING key,job_id
// Per-statement cap keeps each delete short and lock-light; the per-cycle batch
// cap bounds total work per monitor iteration so monitor_db stays responsive.
// A large backlog drains across several iterations rather than one long delete.
//
// These sweeps anti-join the whole table to find orphans, so their cost tracks the heap's
// physical size. job_perms / job_result_stream_v2 are high-churn (one row per job, deleted
// here), so their bloat — not the query shape — is what makes the sweep slow. These sweeps run
// every monitor cycle, but the bulk vacuuming_tables() runs only ~hourly, so dead tuples pile
// up between bulk vacuums; each sweep VACUUMs its own table right after deleting (see below) to
// keep the heap near the live working set. The outer `ctid IN (SELECT ... LIMIT)` is
// deliberate: a `job_id IN (...)` rewrite adds a second scan/probe for the delete and
// benchmarks slower, so don't "simplify" it.
const ORPHAN_CLEANUP_BATCH_SIZE: u64 = 100_000;
const ORPHAN_CLEANUP_MAX_BATCHES: usize = 10;
// Reclaim the dead tuples a sweep just created so the next sweep's anti-join scans a lean heap
// instead of a bloated one. Plain VACUUM (not FULL) only takes SHARE UPDATE EXCLUSIVE, so
// concurrent reads/writes (every job create touches job_perms) keep running, and the visibility
// map lets it skip unchanged pages so repeated runs are cheap. SKIP_LOCKED means HA replicas
// don't pile up: one vacuums, the rest skip rather than queue behind it.
async fn vacuum_after_sweep(db: &DB, table: &str) {
if let Err(e) = sqlx::query(&format!("VACUUM (SKIP_LOCKED) {table}"))
.execute(db)
.await
{
tracing::warn!("Error vacuuming {table} after orphan cleanup: {e:?}");
}
}
async fn cleanup_job_perms_orphaned(db: &DB) -> error::Result<()> {
let mut total: u64 = 0;
for _ in 0..ORPHAN_CLEANUP_MAX_BATCHES {
@@ -4477,7 +4404,6 @@ async fn cleanup_job_perms_orphaned(db: &DB) -> error::Result<()> {
if total > 0 {
tracing::info!("Cleaned up {total} orphaned job_perms rows");
vacuum_after_sweep(db, "job_perms").await;
}
Ok(())
}
@@ -4509,7 +4435,6 @@ async fn cleanup_job_result_stream_orphaned_jobs(db: &DB) -> error::Result<()> {
if total > 0 {
tracing::info!("Cleaned up {total} orphaned job_result_stream_v2 rows");
vacuum_after_sweep(db, "job_result_stream_v2").await;
}
Ok(())
}
-206
View File
@@ -1,206 +0,0 @@
//! A WM_TOKEN (job JWT) running as a superadmin must not be able to perform
//! global user/token management — promotion, password reset, user creation,
//! token creation/impersonation, offboarding, or exporting the user table.
//! A non-admin `wm_deployers` member can mint
//! such a token implicitly via an app/flow `on_behalf_of`, so trusting it would
//! let them establish *persistent* superadmin. A real superadmin who needs this
//! from a script must use a dedicated superadmin API token (which only a real
//! superadmin can create), not `$WM_TOKEN`.
//!
//! The fixture provides `test@windmill.dev` (instance superadmin, token
//! `SECRET_TOKEN`) and `test2@windmill.dev` (non-superadmin, `SECRET_TOKEN_2`).
use serde_json::json;
use sqlx::{Pool, Postgres};
use windmill_common::auth::create_jwt_token;
use windmill_common::db::Authed;
use windmill_test_utils::*;
fn client() -> reqwest::Client {
reqwest::Client::new()
}
fn authed(builder: reqwest::RequestBuilder, token: &str) -> reqwest::RequestBuilder {
builder.header("Authorization", format!("Bearer {}", token))
}
/// Mint a WM_TOKEN: an internally-signed job JWT (note the `job_id` claim) for
/// `email`, exactly as a running app/flow job is issued.
async fn wm_token(email: &str, is_admin: bool) -> String {
let authed = Authed {
email: email.to_string(),
username: "runner".to_string(),
is_admin,
is_operator: false,
groups: vec![],
folders: vec![],
scopes: None,
token_prefix: None,
};
create_jwt_token(
authed,
"test-workspace",
3600,
Some(uuid::Uuid::new_v4()),
Some("app".to_string()),
None,
None,
)
.await
.expect("mint wm_token")
}
#[sqlx::test(fixtures("preserve_on_behalf_of"))]
async fn test_wm_token_cannot_manage_superadmin_users(db: Pool<Postgres>) -> anyhow::Result<()> {
initialize_tracing().await;
// The server decodes WM_TOKENs with the same in-process JWT secret, so
// setting it once lets us mint a valid one below.
set_jwt_secret().await;
let server = ApiServer::start(db.clone()).await?;
let port = server.addr.port();
let base = format!("http://localhost:{port}/api/users");
// A superadmin-capable WM_TOKEN — the exact thing a deployer obtains via an
// app on_behalf_of pointed at a superadmin.
let sa_wm = wm_token("test@windmill.dev", true).await;
// 1. Cannot mint a (superadmin) token.
let resp = authed(client().post(format!("{base}/tokens/create")), &sa_wm)
.json(&json!({}))
.send()
.await?;
assert_eq!(
resp.status(),
401,
"superadmin WM_TOKEN must not create tokens: {}",
resp.text().await?
);
// 2. Cannot impersonate (mint a token as another user).
let resp = authed(client().post(format!("{base}/tokens/impersonate")), &sa_wm)
.json(&json!({ "impersonate_email": "test2@windmill.dev" }))
.send()
.await?;
assert_eq!(
resp.status(),
401,
"superadmin WM_TOKEN must not impersonate: {}",
resp.text().await?
);
// 3. Cannot promote a user to superadmin.
let resp = authed(
client().post(format!("{base}/update/test2@windmill.dev")),
&sa_wm,
)
.json(&json!({ "is_super_admin": true }))
.send()
.await?;
assert_eq!(
resp.status(),
401,
"superadmin WM_TOKEN must not promote users: {}",
resp.text().await?
);
// 4. Cannot reset its own (the superadmin's) password.
let resp = authed(client().post(format!("{base}/setpassword")), &sa_wm)
.json(&json!({ "password": "hunter2" }))
.send()
.await?;
assert_eq!(
resp.status(),
401,
"superadmin WM_TOKEN must not reset passwords: {}",
resp.text().await?
);
// 4b. Cannot delete a user.
let resp = authed(
client().delete(format!("{base}/delete/test2@windmill.dev")),
&sa_wm,
)
.send()
.await?;
assert_eq!(
resp.status(),
401,
"superadmin WM_TOKEN must not delete users: {}",
resp.text().await?
);
// 4c. Cannot change a user's login type.
let resp = authed(
client().post(format!("{base}/set_login_type/test2@windmill.dev")),
&sa_wm,
)
.json(&json!({ "login_type": "password" }))
.send()
.await?;
assert_eq!(
resp.status(),
401,
"superadmin WM_TOKEN must not change login type: {}",
resp.text().await?
);
// 4d. Cannot offboard a global user (deletes user, tokens, password, invites,
// instance-group membership and reassigns their assets).
let resp = authed(
client().post(format!("{base}/offboard/test2@windmill.dev")),
&sa_wm,
)
.json(&json!({}))
.send()
.await?;
assert_eq!(
resp.status(),
401,
"superadmin WM_TOKEN must not offboard users: {}",
resp.text().await?
);
// 4e. Cannot export the global user table (leaks every user's password_hash).
let resp = authed(client().get(format!("{base}/export")), &sa_wm)
.send()
.await?;
assert_eq!(
resp.status(),
401,
"superadmin WM_TOKEN must not export global users: {}",
resp.text().await?
);
// 5. Escape hatch / no false positive: a real superadmin API token
// (SECRET_TOKEN, no job_id) can still create tokens.
let resp = authed(
client().post(format!("{base}/tokens/create")),
"SECRET_TOKEN",
)
.json(&json!({ "label": "ci" }))
.send()
.await?;
assert_eq!(
resp.status(),
201,
"a real superadmin token must still create tokens: {}",
resp.text().await?
);
// 6. No collateral: a non-superadmin WM_TOKEN can still create its own
// token — the guard only fires for superadmin-capable job tokens.
let user_wm = wm_token("test2@windmill.dev", false).await;
let resp = authed(client().post(format!("{base}/tokens/create")), &user_wm)
.json(&json!({ "label": "from-script" }))
.send()
.await?;
assert_eq!(
resp.status(),
201,
"non-superadmin WM_TOKEN must still create its own token: {}",
resp.text().await?
);
Ok(())
}
+2 -68
View File
@@ -450,33 +450,6 @@ struct GraphRunnableNode {
// pipeline-member visual state on the frontend.
#[serde(skip_serializing_if = "std::ops::Not::not", default)]
in_pipeline: bool,
// Annotation badges parsed from the deployed script body, so the canvas
// shows partition/freshness/tag/retry/data-test chips on *deployed* nodes
// (not only on live-edited drafts, which the frontend parses itself). Kept
// in lockstep with the TS `AssetGraphRunnableNode` fields the node renders.
#[serde(skip_serializing_if = "Option::is_none", default)]
partition_kind: Option<String>,
#[serde(skip_serializing_if = "Option::is_none", default)]
freshness: Option<String>,
#[serde(skip_serializing_if = "Option::is_none", default)]
tag: Option<String>,
#[serde(skip_serializing_if = "Option::is_none", default)]
retry: Option<windmill_common::assets::RetrySpec>,
#[serde(skip_serializing_if = "Vec::is_empty", default)]
data_tests: Vec<windmill_common::assets::DataTest>,
}
// The partition's kind word for the node badge (the full PartitionSpec carries
// tz/format/start, which the badge doesn't need).
fn partition_kind_word(kind: &windmill_common::assets::PartitionKind) -> &'static str {
use windmill_common::assets::PartitionKind::*;
match kind {
Daily => "daily",
Hourly => "hourly",
Weekly => "weekly",
Monthly => "monthly",
Dynamic { .. } => "dynamic",
}
}
// Lineage edge from parsed r/w usages. One per (runnable, asset, access_type)
@@ -669,21 +642,15 @@ async fn asset_graph(
.await?;
// Which scripts in scope are pipeline members (have `// pipeline`).
// Pipeline members + their latest deployed body, so the graph can surface
// annotation badges (partition/freshness/tag/retry/data_test) on deployed
// nodes. `DISTINCT ON (path) … ORDER BY created_at DESC` picks the newest
// non-archived version per path (a redeploy archives the prior one, but be
// defensive against transient overlaps).
let pipeline_member_paths = sqlx::query!(
r#"
SELECT DISTINCT ON (path) path AS "path!", content AS "content!"
SELECT path AS "path!"
FROM script
WHERE workspace_id = $1
AND auto_kind = 'pipeline'
AND archived = false
AND deleted = false
AND ($2::text IS NULL OR path LIKE $2)
ORDER BY path, created_at DESC
"#,
&w_id,
folder_filter.as_deref(),
@@ -714,20 +681,6 @@ async fn asset_graph(
tx.commit().await?;
// Parse each pipeline member's body once into its badge annotations, keyed
// by path, for the runnable-node construction below.
let annotations_by_path: std::collections::HashMap<
String,
windmill_common::assets::PipelineAnnotations,
> = pipeline_member_paths
.iter()
.map(|r| {
(
r.path.clone(),
windmill_common::assets::parse_pipeline_annotations(&r.content),
)
})
.collect();
let pipeline_member_script_paths: std::collections::HashSet<String> =
pipeline_member_paths.into_iter().map(|r| r.path).collect();
let existing_script_paths: std::collections::HashSet<String> =
@@ -851,26 +804,7 @@ async fn asset_graph(
.map(|(usage_kind, path)| {
let in_pipeline = usage_kind == AssetUsageKind::Script
&& pipeline_member_script_paths.contains(&path);
// Annotation badges, only for pipeline-member scripts (the only
// bodies we parsed). Gate on the runnable kind too: a flow sharing a
// path with a pipeline script must not inherit its badges.
let ann = (usage_kind == AssetUsageKind::Script)
.then(|| annotations_by_path.get(&path))
.flatten();
GraphRunnableNode {
in_pipeline,
partition_kind: ann
.and_then(|a| a.partition.as_ref())
.map(|p| partition_kind_word(&p.kind).to_string()),
freshness: ann
.and_then(|a| a.freshness.as_ref())
.map(|f| f.duration.clone()),
tag: ann.and_then(|a| a.tag.clone()),
retry: ann.and_then(|a| a.retry.clone()),
data_tests: ann.map(|a| a.data_tests.clone()).unwrap_or_default(),
path,
usage_kind,
}
GraphRunnableNode { path, usage_kind, in_pipeline }
})
.collect();
runnables.sort_by(|a, b| a.path.cmp(&b.path));
-78
View File
@@ -205,33 +205,6 @@ pub async fn require_super_admin(db: &DB, email: &str) -> error::Result<()> {
}
}
/// Forbid sensitive global user/token management when authenticated as a
/// superadmin *via a job token* (`WM_TOKEN`).
///
/// A `WM_TOKEN`'s identity is derived from an app/flow `on_behalf_of`, which a
/// non-admin `wm_deployers` member can point at a superadmin. Trusting it for
/// these operations would let them establish *persistent* superadmin (promote a
/// user, reset a superadmin's password, mint a superadmin token, ...). `job_id`
/// is set only for `WM_TOKEN`s; regular session/API tokens have it `None`, so a
/// real superadmin who needs this from a script uses a dedicated superadmin API
/// token (which only a real superadmin can create) instead of `$WM_TOKEN`.
pub async fn forbid_superadmin_job_token(
db: &DB,
email: &str,
job_id: Option<uuid::Uuid>,
) -> error::Result<()> {
if job_id.is_some() && is_super_admin_email(db, email).await? {
return Err(Error::NotAuthorized(
"This operation cannot be performed with a job token ($WM_TOKEN) that runs as a \
superadmin. If a script genuinely needs to do this, create a dedicated superadmin \
token from the User settings drawer (the 'Tokens' section), store it as a secret, \
and use that token explicitly instead of $WM_TOKEN."
.to_owned(),
));
}
Ok(())
}
pub fn check_scopes<F>(authed: &ApiAuthed, required: F) -> error::Result<()>
where
F: FnOnce() -> String,
@@ -1023,18 +996,6 @@ pub fn require_path_read_access_for_preview(
return Ok(());
};
// Reject path traversal before any privilege-based short-circuit. A Preview's
// path is request-supplied and bypasses the DB `proper_id` CHECK that deployed
// runnables get; it then flows to the worker where it builds on-disk module
// directories. A `..` segment or an absolute path could let a write escape the
// per-job dir.
if path.starts_with('/') || path.split('/').any(|seg| seg == "..") || path.contains('\0') {
return Err(Error::BadRequest(format!(
"Invalid path for preview job: {}",
path
)));
}
if authed.is_admin {
return Ok(());
}
@@ -1092,45 +1053,6 @@ mod tests {
}
}
// Regression tests for the Preview path traversal: a Preview's path skips the
// DB `proper_id` CHECK and reaches the worker, where it builds on-disk module
// dirs. Traversal must be rejected even for admins, who otherwise bypass the
// namespace/folder access check.
#[test]
fn preview_path_rejects_traversal() {
let admin = ApiAuthed { is_admin: true, username: "admin".into(), ..Default::default() };
for path in [
"u/admin/../../../../../../tmp/evil/payload",
"../../tmp/evil",
"/tmp/evil",
"u/admin/ok/../../../../etc/cron.d/x",
] {
assert!(
require_path_read_access_for_preview(&admin, &Some(path.to_string())).is_err(),
"expected traversal path to be rejected: {path}"
);
}
}
#[test]
fn preview_path_allows_legitimate_paths() {
let alice = ApiAuthed { username: "alice".into(), ..Default::default() };
assert!(require_path_read_access_for_preview(&alice, &None).is_ok());
assert!(require_path_read_access_for_preview(&alice, &Some(String::new())).is_ok());
assert!(
require_path_read_access_for_preview(&alice, &Some("u/alice/my_script".into())).is_ok()
);
let admin = ApiAuthed { is_admin: true, username: "admin".into(), ..Default::default() };
assert!(
require_path_read_access_for_preview(&admin, &Some("hub/foo/bar/baz".into())).is_ok()
);
// `..` only as a substring of a segment is a valid name, not traversal.
assert!(
require_path_read_access_for_preview(&admin, &Some("f/team/my..script".into())).is_ok()
);
}
#[test]
fn predicate_no_scopes_allows_all() {
let authed = authed_with_scopes(None);
@@ -646,20 +646,6 @@ async fn test_workspace_endpoints(db: Pool<Postgres>) -> anyhow::Result<()> {
.unwrap();
assert_eq!(resp.json::<bool>().await?, true);
// Regression: changing a fork's workspace id must preserve its parent
// linkage. Dropping it leaves a wm-fork- workspace with no parent — a
// "fork of nothing" that can no longer be compared or merged.
let parent: Option<String> =
sqlx::query_scalar("SELECT parent_workspace_id FROM workspace WHERE id = $1")
.bind("wm-fork-renamed")
.fetch_one(&db)
.await?;
assert_eq!(
parent.as_deref(),
Some("new-test-ws"),
"renamed fork must keep its parent_workspace_id"
);
// --- create_fork over an existing (active) workspace id: clear 400, not a raw SQL 500 ---
let resp = authed(client().post(format!("{new_ws_base}/create_fork")))
.json(&json!({
+38 -394
View File
@@ -256,300 +256,52 @@ pub async fn test_s3_bucket(
use bytes::Bytes;
use futures::StreamExt;
// The probe executes on the API server itself. On multi-tenant Cloud that is a shared control
// plane, so we constrain untrusted callers to remove the SSRF / credential-exfiltration /
// local-filesystem surface (see validate_object_storage_test). On self-hosted instances the
// object store usually lives on the local/private network and all authenticated users are
// trusted, so testing there stays unrestricted. Super admins keep the unrestricted path too.
let is_super_admin = is_super_admin_email(&db, &authed.email).await?;
let restrict = !is_super_admin && *CLOUD_HOSTED;
if restrict {
validate_object_storage_test(&test_s3_bucket).await?;
}
require_super_admin(&db, &authed.email).await?;
let client = build_object_store_from_settings(test_s3_bucket, Some(&db))
.await?
.store;
let run = async {
let mut list = client.list(Some(
&windmill_object_store::object_store_reexports::Path::from("".to_string()),
));
let first_file = list.next().await;
if first_file.is_some() {
if let Err(e) = first_file.as_ref().unwrap() {
tracing::error!("error listing bucket: {e:#}");
error::Error::internal_err(format!("Failed to list files in blob storage: {e:#}"));
}
tracing::info!("Listed files: {:?}", first_file.unwrap());
} else {
tracing::info!("No files in blob storage");
let mut list = client.list(Some(
&windmill_object_store::object_store_reexports::Path::from("".to_string()),
));
let first_file = list.next().await;
if first_file.is_some() {
if let Err(e) = first_file.as_ref().unwrap() {
tracing::error!("error listing bucket: {e:#}");
error::Error::internal_err(format!("Failed to list files in blob storage: {e:#}"));
}
let path = windmill_object_store::object_store_reexports::Path::from(format!(
"/test-s3-bucket-{uuid}",
uuid = uuid::Uuid::new_v4()
));
tracing::info!("Testing blob storage at path: {path}");
client
.put(
&path,
windmill_object_store::object_store_reexports::PutPayload::from_static(b"hello"),
)
.await
.map_err(|e| anyhow::anyhow!("error writing file to {path}: {e:#}"))?;
let content = client
.get(&path)
.await
.map_err(to_anyhow)?
.bytes()
.await
.map_err(to_anyhow)?;
if content != Bytes::from_static(b"hello") {
return Err(error::Error::internal_err(
"Failed to read back from blob storage".to_string(),
));
}
client.delete(&path).await.map_err(to_anyhow)?;
Ok::<String, error::Error>("Tested blob storage successfully".to_string())
};
if restrict {
// The object-store client is built with timeouts disabled, so a malicious endpoint could
// otherwise hold the API server connection open indefinitely.
tokio::time::timeout(Duration::from_secs(15), run)
.await
.map_err(|_| {
error::Error::internal_err("Object storage connectivity test timed out".to_string())
})?
tracing::info!("Listed files: {:?}", first_file.unwrap());
} else {
run.await
}
}
// Hardening for the object-storage connectivity test by an untrusted (non-super-admin) caller on
// Cloud. The probe runs on the shared API server, so without these constraints an authenticated
// user could coerce the server into connecting to arbitrary internal endpoints (SSRF), signing
// requests with the instance role (credential exfiltration), or reading/writing the server's local
// disk (filesystem object store).
#[cfg(feature = "parquet")]
async fn validate_object_storage_test(settings: &ObjectSettings) -> error::Result<()> {
fn non_empty(opt: &Option<String>) -> bool {
opt.as_ref().is_some_and(|s| !s.is_empty())
tracing::info!("No files in blob storage");
}
// Reject backends that rely on the server's identity or local filesystem, require explicit
// credentials for the rest (so the server never falls back to its own ambient credentials), and
// resolve the host the client will actually connect to. We derive the *effective* endpoint here
// — mirroring build_*_from_settings: the region/account-derived default and the virtual-hosted
// bucket prefix — rather than only validating a caller-supplied `endpoint`, so caller-controlled
// `region`/`account_name`/`bucket` cannot smuggle an internal host past the check (e.g. an empty
// endpoint with region = "@169.254.169.254/" otherwise resolves to the cloud metadata service).
let effective_endpoint: Option<String> = match settings {
ObjectSettings::Filesystem(_) => {
return Err(error::Error::NotAuthorized(
"Testing a local filesystem object store requires a super admin".to_string(),
));
}
ObjectSettings::AwsOidc(_) => {
return Err(error::Error::NotAuthorized(
"Testing OIDC-based object storage requires a super admin".to_string(),
));
}
ObjectSettings::S3(s3) => {
if !(non_empty(&s3.access_key) && non_empty(&s3.secret_key)) {
return Err(error::Error::NotAuthorized(
"Testing S3 storage without explicit credentials requires a super admin"
.to_string(),
));
}
let region = s3
.region
.clone()
.filter(|r| !r.is_empty())
.or_else(|| std::env::var("AWS_REGION").ok().filter(|r| !r.is_empty()))
.unwrap_or_else(|| "us-east-1".to_string());
let raw_endpoint = s3
.endpoint
.clone()
.filter(|e| !e.is_empty())
.or_else(|| std::env::var("S3_ENDPOINT").ok().filter(|e| !e.is_empty()))
.unwrap_or_else(|| format!("s3.{region}.amazonaws.com"));
Some(windmill_object_store::render_endpoint(
raw_endpoint,
!s3.allow_http.unwrap_or(true),
s3.port,
s3.path_style,
s3.bucket.clone().unwrap_or_default(),
))
}
ObjectSettings::Azure(azure) => {
if !non_empty(&azure.access_key) {
return Err(error::Error::NotAuthorized(
"Testing Azure storage without an explicit access key requires a super admin"
.to_string(),
));
}
Some(
azure
.endpoint
.clone()
.filter(|e| !e.is_empty())
.unwrap_or_else(|| format!("{}.blob.core.windows.net", azure.account_name)),
)
}
ObjectSettings::Gcs(gcs) => {
if gcs.service_account_key.is_empty() {
return Err(error::Error::NotAuthorized(
"Testing GCS storage without a service account key requires a super admin"
.to_string(),
));
}
// The service-account-key JSON can override the data-plane URL (`gcs_base_url`) and the
// OAuth token endpoint (`token_uri`); the GCS client connects to whatever they point at.
// Validate every http(s) URL embedded in the key. When none override it, the host stays
// the public storage.googleapis.com, so no further check is needed.
if let Ok(serde_json::Value::Object(map)) =
serde_json::from_str::<serde_json::Value>(&gcs.service_account_key)
{
for value in map.values() {
if let Some(url) = value.as_str() {
// Match how the URL parser reads the value: leading whitespace/control is
// ignored and the scheme is case-insensitive.
let url =
url.trim_start_matches(|c: char| c.is_whitespace() || c.is_control());
if strip_http_scheme(url).is_some() {
validate_public_endpoint(url).await?;
}
}
}
}
None
}
};
// Block non-public network targets (internal services, cloud metadata, loopback, ...).
if let Some(endpoint) = effective_endpoint {
validate_public_endpoint(&endpoint).await?;
}
Ok(())
}
#[cfg(feature = "parquet")]
async fn validate_public_endpoint(endpoint: &str) -> error::Result<()> {
let host = extract_host(endpoint).ok_or_else(|| {
error::Error::BadRequest(format!("Invalid object storage endpoint: {endpoint}"))
})?;
let addrs: Vec<std::net::SocketAddr> = tokio::net::lookup_host((host.as_str(), 443u16))
let path = windmill_object_store::object_store_reexports::Path::from(format!(
"/test-s3-bucket-{uuid}",
uuid = uuid::Uuid::new_v4()
));
tracing::info!("Testing blob storage at path: {path}");
client
.put(
&path,
windmill_object_store::object_store_reexports::PutPayload::from_static(b"hello"),
)
.await
.map_err(|e| {
error::Error::BadRequest(format!(
"Could not resolve object storage endpoint '{host}': {e}"
))
})?
.collect();
if addrs.is_empty() {
return Err(error::Error::BadRequest(format!(
"Could not resolve object storage endpoint '{host}'"
)));
}
// Reject if any resolved address is non-public, which also defeats the simplest DNS-rebinding
// attempts (a name resolving to both a public and a private address).
for addr in addrs {
if is_forbidden_ip(addr.ip()) {
return Err(error::Error::NotAuthorized(
"Testing object storage at a private, loopback, or link-local endpoint requires a super admin"
.to_string(),
));
}
}
Ok(())
}
// Strip a leading `http://`/`https://` scheme case-insensitively (URL schemes are
// case-insensitive), returning the remainder when one was present.
#[cfg(feature = "parquet")]
fn strip_http_scheme(s: &str) -> Option<&str> {
for scheme in ["https://", "http://"] {
let b = scheme.as_bytes();
if s.len() >= b.len() && s.as_bytes()[..b.len()].eq_ignore_ascii_case(b) {
return Some(&s[b.len()..]);
}
}
None
}
#[cfg(feature = "parquet")]
fn extract_host(endpoint: &str) -> Option<String> {
let mut s = endpoint.trim();
if let Some(rest) = strip_http_scheme(s) {
s = rest;
}
s = s.split(['/', '?', '#', '\\']).next().unwrap_or(s);
if let Some((_, rest)) = s.rsplit_once('@') {
s = rest;
}
let host = if let Some(rest) = s.strip_prefix('[') {
// IPv6 literal, e.g. [::1]:9000
rest.split(']').next().unwrap_or(rest)
} else {
// host or host:port
s.split(':').next().unwrap_or(s)
}
.trim();
if host.is_empty() {
None
} else {
Some(host.to_string())
}
}
#[cfg(feature = "parquet")]
fn is_forbidden_ip(ip: std::net::IpAddr) -> bool {
use std::net::{IpAddr, Ipv4Addr};
match ip {
IpAddr::V4(v4) => {
v4.is_loopback()
|| v4.is_private()
|| v4.is_link_local() // 169.254.0.0/16, incl. the cloud metadata endpoint
|| v4.is_unspecified()
|| v4.is_broadcast()
|| v4.is_documentation()
|| v4.is_multicast()
|| v4.octets()[0] == 0 // 0.0.0.0/8
|| (v4.octets()[0] == 100 && (v4.octets()[1] & 0xc0) == 64) // 100.64.0.0/10 CGNAT
}
IpAddr::V6(v6) => {
// Any IPv4 embedded in an IPv6 address (IPv4-mapped ::ffff:0:0/96, IPv4-compatible
// ::/96, or NAT64 64:ff9b::/96) is re-checked against the IPv4 rules, so e.g.
// 64:ff9b::169.254.169.254 cannot route to the metadata endpoint in a NAT64 network.
let seg = v6.segments();
let is_v4_compatible = seg[0..6] == [0, 0, 0, 0, 0, 0];
let is_nat64 = seg[0] == 0x0064 && seg[1] == 0xff9b && seg[2..6] == [0, 0, 0, 0];
if let Some(v4) = v6.to_ipv4_mapped() {
return is_forbidden_ip(IpAddr::V4(v4));
}
if is_v4_compatible || is_nat64 {
let embedded = Ipv4Addr::new(
(seg[6] >> 8) as u8,
(seg[6] & 0xff) as u8,
(seg[7] >> 8) as u8,
(seg[7] & 0xff) as u8,
);
if is_forbidden_ip(IpAddr::V4(embedded)) {
return true;
}
}
v6.is_loopback()
|| v6.is_unspecified()
|| v6.is_multicast()
|| (seg[0] & 0xfe00) == 0xfc00 // fc00::/7 unique local
|| (seg[0] & 0xffc0) == 0xfe80 // fe80::/10 link-local
}
.map_err(|e| anyhow::anyhow!("error writing file to {path}: {e:#}"))?;
let content = client
.get(&path)
.await
.map_err(to_anyhow)?
.bytes()
.await
.map_err(to_anyhow)?;
if content != Bytes::from_static(b"hello") {
return Err(error::Error::internal_err(
"Failed to read back from blob storage".to_string(),
));
}
client.delete(&path).await.map_err(to_anyhow)?;
Ok("Tested blob storage successfully".to_string())
}
#[cfg(feature = "parquet")]
@@ -1523,8 +1275,8 @@ async fn setup_custom_instance_pg_database_inner(
// Validate name to ensure it only contains alphanumeric characters
// Prevents SQL injection on the instance database
lazy_static::lazy_static! {
// Must start with a letter, then alphanumeric/underscore/hyphen
static ref VALID_NAME: regex::Regex = regex::Regex::new(r"^[a-zA-Z][a-zA-Z0-9_-]*$").unwrap();
// Must start with a letter, then alphanumeric/underscore
static ref VALID_NAME: regex::Regex = regex::Regex::new(r"^[a-zA-Z][a-zA-Z0-9_]*$").unwrap();
}
let dbname = dbname.trim();
if dbname.is_empty() {
@@ -1540,7 +1292,7 @@ async fn setup_custom_instance_pg_database_inner(
}
if !VALID_NAME.is_match(dbname) {
return Err(error::Error::BadRequest(
"Database name must start with a letter and contain only alphanumeric characters, underscores, or hyphens".to_string(),
"Database name must start with a letter and contain only alphanumeric characters or underscores".to_string(),
));
}
// Additional check: block PostgreSQL reserved/special names
@@ -2109,111 +1861,3 @@ mod tests {
);
}
}
#[cfg(all(test, feature = "parquet"))]
mod object_storage_test_hardening {
use super::{extract_host, is_forbidden_ip, validate_object_storage_test};
use std::net::IpAddr;
use windmill_object_store::ObjectSettings;
// IP literals (not hostnames) keep validate_public_endpoint deterministic — `lookup_host`
// parses them without any network round-trip.
fn gcs_settings(gcs_base_url: &str) -> ObjectSettings {
serde_json::from_value(serde_json::json!({
"type": "Gcs",
"bucket": "b",
"serviceAccountKey": { "gcs_base_url": gcs_base_url, "client_email": "x@y.z" }
}))
.unwrap()
}
#[tokio::test]
async fn rejects_gcs_internal_base_url() {
// gcs_base_url in the service-account key must not smuggle an internal host past the check,
// including via a mixed-case scheme (URL schemes are case-insensitive).
for url in [
"http://169.254.169.254",
"HTTP://169.254.169.254",
"Https://10.0.0.5",
] {
assert!(
validate_object_storage_test(&gcs_settings(url))
.await
.is_err(),
"{url} should be rejected"
);
}
}
#[tokio::test]
async fn allows_gcs_public_base_url() {
assert!(
validate_object_storage_test(&gcs_settings("https://8.8.8.8"))
.await
.is_ok()
);
}
fn ip(s: &str) -> IpAddr {
s.parse().unwrap()
}
#[test]
fn forbids_internal_ips() {
for s in [
"127.0.0.1", // loopback
"169.254.169.254", // cloud metadata (link-local)
"10.0.0.5", // private
"172.16.3.4", // private
"192.168.1.10", // private
"0.0.0.0", // unspecified
"100.64.0.1", // CGNAT
"::1", // IPv6 loopback
"fe80::1", // IPv6 link-local
"fc00::1", // IPv6 unique local
"::ffff:127.0.0.1", // IPv4-mapped loopback
"::ffff:169.254.169.254", // IPv4-mapped metadata
"::169.254.169.254", // IPv4-compatible metadata
"64:ff9b::169.254.169.254", // NAT64-embedded metadata
"64:ff9b::a9fe:a9fe", // NAT64-embedded metadata (hex form)
] {
assert!(is_forbidden_ip(ip(s)), "{s} should be forbidden");
}
}
#[test]
fn allows_public_ips() {
for s in ["8.8.8.8", "1.1.1.1", "52.95.110.1", "2606:4700:4700::1111"] {
assert!(!is_forbidden_ip(ip(s)), "{s} should be allowed");
}
}
#[test]
fn extracts_host_from_endpoint() {
let cases = [
("s3.amazonaws.com", Some("s3.amazonaws.com")),
("https://minio.internal:9000", Some("minio.internal")),
("http://10.0.0.5:9000/bucket", Some("10.0.0.5")),
("user:pass@host.example:443", Some("host.example")),
("[::1]:9000", Some("::1")),
("https://[fe80::1]/x", Some("fe80::1")),
("", None),
// Injection via region/bucket interpolation into the default endpoint string: the
// userinfo `@` and the path `/` must not hide the real authority from the host check.
(
"https://s3.@169.254.169.254/.amazonaws.com",
Some("169.254.169.254"),
),
(
"https://@169.254.169.254/mybucket.s3.amazonaws.com",
Some("169.254.169.254"),
),
("s3.#@169.254.169.254/x.amazonaws.com", Some("s3.")),
// Scheme is case-insensitive.
("HTTP://169.254.169.254", Some("169.254.169.254")),
];
for (input, expected) in cases {
assert_eq!(extract_host(input).as_deref(), expected, "input: {input}");
}
}
}
@@ -339,15 +339,13 @@ async fn cleanup_job_logs(
return Ok(());
}
let mut completed_at_floor: Option<DateTime<Utc>> = None;
loop {
let (deleted_count, rel_paths, max_completed_at) =
delete_expired_jobs_batch(db, retention_secs, JOB_BATCH, completed_at_floor).await?;
let (deleted_count, rel_paths) =
delete_expired_jobs_batch(db, retention_secs, JOB_BATCH).await?;
if deleted_count == 0 {
break;
}
completed_at_floor = max_completed_at.or(completed_at_floor);
let s3_paths: Vec<ObjectPath> = rel_paths
.iter()
@@ -384,8 +382,7 @@ async fn delete_expired_jobs_batch(
db: &DB,
job_retention_secs: i64,
batch_size: i64,
completed_at_floor: Option<DateTime<Utc>>,
) -> error::Result<(usize, Vec<String>, Option<DateTime<Utc>>)> {
) -> error::Result<(usize, Vec<String>)> {
let mut tx = db.begin().await?;
let active_root_job_ids: Vec<Uuid> = sqlx::query_scalar!(
@@ -398,61 +395,33 @@ async fn delete_expired_jobs_batch(
.fetch_all(&mut *tx)
.await?;
// `completed_at_floor` carries a watermark across batches so each one resumes after the rows
// the previous batch processed instead of re-scanning the (potentially undeletable) oldest
// prefix; the empty-active-roots branch skips the v2_job join entirely. See
// backend/src/monitor.rs::delete_expired_jobs_batch for the full rationale.
let (deleted_jobs, max_completed_at) = if active_root_job_ids.is_empty() {
let rows = sqlx::query!(
"DELETE FROM v2_job_completed
WHERE id IN (
SELECT id FROM v2_job_completed
WHERE completed_at <= now() - ($1::bigint::text || ' s')::interval
AND ($3::timestamptz IS NULL OR completed_at >= $3)
ORDER BY completed_at ASC
LIMIT $2
FOR UPDATE SKIP LOCKED
)
RETURNING id, completed_at",
job_retention_secs,
batch_size,
completed_at_floor,
)
.fetch_all(&mut *tx)
.await?;
let max = rows.iter().map(|r| r.completed_at).max();
(rows.into_iter().map(|r| r.id).collect::<Vec<Uuid>>(), max)
} else {
let rows = sqlx::query!(
"DELETE FROM v2_job_completed
WHERE id IN (
SELECT jc.id FROM v2_job_completed jc
LEFT JOIN v2_job j ON j.id = jc.id
WHERE jc.completed_at <= now() - ($1::bigint::text || ' s')::interval
AND ($4::timestamptz IS NULL OR jc.completed_at >= $4)
AND COALESCE(j.root_job, j.flow_innermost_root_job, jc.id) NOT IN (
SELECT u FROM unnest($3::uuid[]) AS u WHERE u IS NOT NULL
)
ORDER BY jc.completed_at ASC
LIMIT $2
FOR UPDATE OF jc SKIP LOCKED
)
RETURNING id, completed_at",
job_retention_secs,
batch_size,
&active_root_job_ids,
completed_at_floor,
)
.fetch_all(&mut *tx)
.await?;
let max = rows.iter().map(|r| r.completed_at).max();
(rows.into_iter().map(|r| r.id).collect::<Vec<Uuid>>(), max)
};
// Active-root exclusion via NOT IN (hashed SubPlan) instead of `!= ALL($3)`;
// see backend/src/monitor.rs::delete_expired_jobs_batch for the rationale.
let deleted_jobs: Vec<Uuid> = sqlx::query_scalar!(
"DELETE FROM v2_job_completed
WHERE id IN (
SELECT jc.id FROM v2_job_completed jc
LEFT JOIN v2_job j ON j.id = jc.id
WHERE jc.completed_at <= now() - ($1::bigint::text || ' s')::interval
AND COALESCE(j.root_job, j.flow_innermost_root_job, jc.id) NOT IN (
SELECT u FROM unnest($3::uuid[]) AS u WHERE u IS NOT NULL
)
ORDER BY jc.completed_at ASC
LIMIT $2
FOR UPDATE OF jc SKIP LOCKED
)
RETURNING id",
job_retention_secs,
batch_size,
&active_root_job_ids
)
.fetch_all(&mut *tx)
.await?;
let deleted_count = deleted_jobs.len();
if deleted_count == 0 {
tx.commit().await?;
return Ok((0, Vec::new(), max_completed_at));
return Ok((0, Vec::new()));
}
if let Err(e) = sqlx::query!(
@@ -502,7 +471,7 @@ async fn delete_expired_jobs_batch(
tx.commit().await?;
Ok((deleted_count, log_paths, max_completed_at))
Ok((deleted_count, log_paths))
}
/// Scan S3 under the `logs/` prefix for orphan log files and delete them.
+1 -15
View File
@@ -27,7 +27,7 @@ use axum::{
Json, Router,
};
use hyper::{header::LOCATION, StatusCode};
use windmill_api_auth::{forbid_superadmin_job_token, require_super_admin, OptJobAuthed};
use windmill_api_auth::require_super_admin;
use windmill_common::usernames::{
generate_instance_wide_unique_username, get_instance_username_or_create_pending,
};
@@ -1415,13 +1415,11 @@ async fn convert_user_to_group(
async fn update_user(
authed: ApiAuthed,
OptJobAuthed { job_id, .. }: OptJobAuthed,
Path(email_to_update): Path<String>,
Extension(db): Extension<DB>,
Json(eu): Json<EditUser>,
) -> Result<String> {
require_super_admin(&db, &authed.email).await?;
forbid_superadmin_job_token(&db, &authed.email, job_id).await?;
let mut tx = db.begin().await?;
let mut new_super_admin: Option<bool> = None;
@@ -1583,12 +1581,10 @@ async fn update_user(
async fn delete_user(
authed: ApiAuthed,
OptJobAuthed { job_id, .. }: OptJobAuthed,
Path(email_to_delete): Path<String>,
Extension(db): Extension<DB>,
) -> Result<String> {
require_super_admin(&db, &authed.email).await?;
forbid_superadmin_job_token(&db, &authed.email, job_id).await?;
let mut tx = db.begin().await?;
sqlx::query!("DELETE FROM token WHERE email = $1", &email_to_delete)
@@ -1881,11 +1877,9 @@ async fn set_login_type(
Extension(db): Extension<DB>,
Path(email): Path<String>,
authed: ApiAuthed,
OptJobAuthed { job_id, .. }: OptJobAuthed,
Json(et): Json<EditLoginType>,
) -> Result<String> {
require_super_admin(&db, &authed.email).await?;
forbid_superadmin_job_token(&db, &authed.email, job_id).await?;
let mut tx = db.begin().await?;
sqlx::query!(
@@ -2165,10 +2159,8 @@ pub async fn create_session_token<'c>(
async fn create_token(
Extension(db): Extension<DB>,
authed: ApiAuthed,
OptJobAuthed { job_id, .. }: OptJobAuthed,
Json(token_config): Json<NewToken>,
) -> Result<(StatusCode, String)> {
forbid_superadmin_job_token(&db, &authed.email, job_id).await?;
check_token_create_rate_limit(&authed.username)?;
windmill_api_auth::ensure_scopes_within_caller(&authed, token_config.scopes.as_deref())?;
@@ -2184,7 +2176,6 @@ async fn create_token(
async fn impersonate(
Extension(db): Extension<DB>,
authed: ApiAuthed,
OptJobAuthed { job_id, .. }: OptJobAuthed,
Json(new_token): Json<NewToken>,
) -> Result<(StatusCode, String)> {
use windmill_common::min_version::MIN_VERSION_SUPPORTS_TOKEN_HASH;
@@ -2198,7 +2189,6 @@ async fn impersonate(
Some(&token)
};
require_super_admin(&db, &authed.email).await?;
forbid_superadmin_job_token(&db, &authed.email, job_id).await?;
if new_token.impersonate_email.is_none() {
return Err(Error::BadRequest(
@@ -2717,10 +2707,8 @@ struct ExportedGlobalUser {
async fn export_global_users(
Extension(db): Extension<DB>,
authed: ApiAuthed,
OptJobAuthed { job_id, .. }: OptJobAuthed,
) -> JsonResult<Vec<ExportedGlobalUser>> {
require_super_admin(&db, &authed.email).await?;
forbid_superadmin_job_token(&db, &authed.email, job_id).await?;
let mut tx = db.begin().await?;
let users = sqlx::query_as!(
ExportedGlobalUser,
@@ -2756,11 +2744,9 @@ async fn export_global_users() -> JsonResult<String> {
async fn overwrite_global_users(
Extension(db): Extension<DB>,
authed: ApiAuthed,
OptJobAuthed { job_id, .. }: OptJobAuthed,
Json(users): Json<Vec<ExportedGlobalUser>>,
) -> Result<String> {
require_super_admin(&db, &authed.email).await?;
forbid_superadmin_job_token(&db, &authed.email, job_id).await?;
let mut tx = db.begin().await?;
sqlx::query!("DELETE FROM password")
.execute(&mut *tx)
@@ -717,27 +717,16 @@ async fn create_deployment_request_comment(
// ---- helpers ------------------------------------------------------------
async fn parent_of_fork(db: &DB, w_id: &str) -> Result<String> {
// Resolve the fork's parent and require it to still exist and be active. A
// parent that is archived (soft-deleted) can no longer be accessed, so a
// diff or deployment request against it targets an unreachable workspace.
let parent = sqlx::query!(
"SELECT p.id AS \"id!\", p.deleted AS \"deleted!\"
FROM workspace f
JOIN workspace p ON p.id = f.parent_workspace_id
WHERE f.id = $1",
sqlx::query_scalar!(
"SELECT parent_workspace_id FROM workspace WHERE id = $1",
w_id,
)
.fetch_optional(db)
.await?;
match parent {
None => Err(Error::BadRequest(format!(
.await?
.flatten()
.ok_or_else(|| {
Error::BadRequest(format!(
"workspace {w_id} is not a fork (no parent_workspace_id)"
))),
Some(p) if p.deleted => Err(Error::BadRequest(format!(
"parent workspace {} of fork {w_id} is archived",
p.id
))),
Some(p) => Ok(p.id),
}
))
})
}
@@ -197,7 +197,6 @@ pub fn global_service() -> Router {
.route("/list_as_superadmin", get(list_workspaces_as_super_admin))
.route("/list", get(list_workspaces))
.route("/users", get(user_workspaces))
.route("/session_workspace_status", post(session_workspace_status))
.route("/create", post(create_workspace))
.route("/create_fork", post(deprecated_create_workspace_fork))
.route("/exists", post(exists_workspace))
@@ -3624,47 +3623,6 @@ async fn user_workspaces(
Ok(Json(WorkspaceList { email, workspaces }))
}
#[derive(Deserialize)]
struct SessionWorkspaceStatusRequest {
workspace_ids: Vec<String>,
}
/// Reconciliation support for client-side AI sessions, which the backend cannot touch
/// directly. The client posts the workspace ids its sessions reference and uses the
/// per-id status to keep sessions in sync with workspace lifecycle: `deleted` (no row /
/// no access → unresolvable) drops the sessions, `archived` (soft-deleted, still a
/// member) archives them, `active` restores ones previously archived-by-workspace.
/// Archived and hard-deleted workspaces are absent from `user_workspaces`, so this is the
/// only way the client learns about a change made while it was away or on another device.
async fn session_workspace_status(
Extension(db): Extension<DB>,
ApiAuthed { email, .. }: ApiAuthed,
Json(req): Json<SessionWorkspaceStatusRequest>,
) -> JsonResult<HashMap<String, String>> {
if req.workspace_ids.len() > 1000 {
return Err(Error::BadRequest(
"Too many workspace ids (max 1000)".to_string(),
));
}
let rows = sqlx::query!(
"SELECT req.id AS \"id!\",
(CASE
WHEN usr.email IS NULL THEN 'deleted'
WHEN workspace.deleted THEN 'archived'
ELSE 'active'
END) AS \"status!\"
FROM unnest($1::text[]) AS req(id)
LEFT JOIN workspace ON workspace.id = req.id
LEFT JOIN usr ON usr.workspace_id = workspace.id AND usr.email = $2",
&req.workspace_ids[..],
email,
)
.fetch_all(&db)
.await?;
let statuses = rows.into_iter().map(|r| (r.id, r.status)).collect();
Ok(Json(statuses))
}
pub async fn check_w_id_conflict<'c>(tx: &mut Transaction<'c, Postgres>, w_id: &str) -> Result<()> {
if w_id == "global" {
return Err(windmill_common::error::Error::BadRequest(
@@ -65,22 +65,13 @@ pub(crate) async fn change_workspace_id(
old_id, rw.new_id
);
// Create new workspace with new id and name. A fork that keeps a wm-fork-
// id must carry its parent_workspace_id over, otherwise it becomes a
// parentless "fork of nothing" with no source to compare or merge against.
// A non-fork target id means the workspace is being promoted out of a fork,
// so the parent pointer is intentionally cleared.
// Create new workspace with new id and name
info!("Creating new workspace row");
let new_is_fork = rw.new_id.starts_with(WM_FORK_PREFIX);
sqlx::query!(
"INSERT INTO workspace (id, name, owner, deleted, premium, parent_workspace_id)
SELECT $1, $2, owner, false, premium,
CASE WHEN $4 THEN parent_workspace_id ELSE NULL END
FROM workspace WHERE id = $3",
"INSERT INTO workspace SELECT $1, $2, owner, false, premium FROM workspace WHERE id = $3",
&rw.new_id,
&rw.new_name,
&old_id,
new_is_fork
&old_id
)
.execute(&mut *tx)
.await?;
@@ -356,18 +347,6 @@ pub(crate) async fn change_workspace_id(
.execute(&mut *tx)
.await?;
// Re-parent child forks: any fork whose parent_workspace_id was the old id
// must follow the renamed parent to the new id, otherwise it is left
// pointing at the soft-deleted old shell (whose data has moved here).
info!("Re-parenting child forks to the new workspace id");
sqlx::query!(
"UPDATE workspace SET parent_workspace_id = $1 WHERE parent_workspace_id = $2",
&rw.new_id,
&old_id
)
.execute(&mut *tx)
.await?;
info!("Updating workspace_protection_rule table");
sqlx::query!(
"UPDATE workspace_protection_rule SET workspace_id = $1 WHERE workspace_id = $2",
+1 -34
View File
@@ -1,7 +1,7 @@
openapi: "3.0.3"
info:
version: 1.739.0
version: 1.737.0
title: Windmill API
contact:
@@ -997,39 +997,6 @@ paths:
schema:
$ref: "#/components/schemas/UserWorkspaceList"
/workspaces/session_workspace_status:
post:
summary: get the lifecycle status of workspaces referenced by client-side sessions
operationId: getSessionWorkspaceStatus
tags:
- workspace
requestBody:
required: true
content:
application/json:
schema:
type: object
properties:
workspace_ids:
type: array
items:
type: string
required:
- workspace_ids
responses:
"200":
description: map of workspace id to status (active, archived, or deleted)
content:
application/json:
schema:
type: object
additionalProperties:
type: string
enum:
- active
- archived
- deleted
/w/{workspace}/workspaces/get_as_superadmin:
get:
summary: get workspace as super admin (require to be super admin)
-6
View File
@@ -84,12 +84,6 @@ lazy_static::lazy_static! {
(20260228000000, include_str!(
"../../migrations/20260228000000_v2_job_completed_failure_index.up.sql"
).replace("CREATE INDEX", "CREATE INDEX CONCURRENTLY")),
(20260610151334, include_str!(
"../../migrations/20260610151334_folder_labels.up.sql"
).replace("SET search_path = public", "SET search_path FROM CURRENT").to_string()),
(20260614075900, include_str!(
"../../migrations/20260614075900_dedup_folder_labels.up.sql"
).replace("SET search_path = public", "SET search_path FROM CURRENT").to_string()),
].into_iter().collect();
}
+2 -4
View File
@@ -1,13 +1,13 @@
use std::collections::HashMap;
use crate::db::{ApiAuthed, OptJobAuthed};
use crate::db::ApiAuthed;
use crate::secret_backend_ext::rename_vault_secrets_with_prefix;
use axum::{
extract::{Extension, Path},
Json,
};
use serde::{Deserialize, Serialize};
use windmill_api_auth::{forbid_superadmin_job_token, require_super_admin};
use windmill_api_auth::require_super_admin;
use windmill_api_users::users::delete_workspace_user_internal;
use windmill_audit::audit_oss::audit_log;
use windmill_audit::ActionKind;
@@ -483,13 +483,11 @@ pub(crate) async fn global_offboard_preview(
pub(crate) async fn offboard_global_user(
authed: ApiAuthed,
OptJobAuthed { job_id, .. }: OptJobAuthed,
Extension(db): Extension<DB>,
Path(email): Path<String>,
Json(req): Json<GlobalOffboardRequest>,
) -> Result<Json<OffboardResponse>> {
require_super_admin(&db, &authed.email).await?;
forbid_superadmin_job_token(&db, &authed.email, job_id).await?;
let workspaces = sqlx::query!(
"SELECT workspace_id, username FROM usr WHERE email = $1",
+227 -6
View File
@@ -8,24 +8,245 @@
//! Secret backend extension for the API layer
//!
//! Backend resolution and read helpers live in
//! `windmill_common::secret_backend`; this module keeps the API-specific bulk
//! rename helper used when renaming users.
//! This module provides helper functions for integrating the SecretBackend
//! trait with variable operations in the API.
//!
//! Note: HashiCorp Vault integration requires Enterprise Edition.
//! The OSS version only supports the database backend.
#[cfg(all(feature = "private", feature = "enterprise"))]
use std::sync::Arc;
use windmill_common::{db::DB, error::Result};
#[cfg(all(feature = "private", feature = "enterprise"))]
use windmill_common::error::Error;
#[cfg(all(feature = "private", feature = "enterprise"))]
use windmill_common::secret_backend::{database::DatabaseBackend, SecretBackend};
#[cfg(all(feature = "private", feature = "enterprise"))]
use windmill_common::{
error::Error,
global_settings::{load_value_from_global_settings, SECRET_BACKEND_SETTING},
secret_backend::{
get_secret_backend, is_aws_sm_stored_value, is_azure_kv_stored_value,
is_external_stored_value, is_vault_backend_configured,
AwsSecretsManagerBackend, AwsSecretsManagerSettings, AzureKeyVaultBackend,
AzureKeyVaultSettings, SecretBackendConfig, VaultBackend, VaultSettings,
},
};
#[cfg(all(feature = "private", feature = "enterprise"))]
use tokio::sync::RwLock;
// Cached Vault backend to avoid recreating it for every request
// This enables connection pooling and avoids repeated setup overhead
#[cfg(all(feature = "private", feature = "enterprise"))]
struct CachedVaultBackend {
backend: Arc<dyn SecretBackend>,
settings: VaultSettings,
}
#[cfg(all(feature = "private", feature = "enterprise"))]
lazy_static::lazy_static! {
static ref VAULT_BACKEND_CACHE: RwLock<Option<CachedVaultBackend>> = RwLock::new(None);
}
// Cached Azure Key Vault backend
#[cfg(all(feature = "private", feature = "enterprise"))]
struct CachedAzureKvBackend {
backend: Arc<dyn SecretBackend>,
settings: AzureKeyVaultSettings,
}
#[cfg(all(feature = "private", feature = "enterprise"))]
lazy_static::lazy_static! {
static ref AZURE_KV_BACKEND_CACHE: RwLock<Option<CachedAzureKvBackend>> = RwLock::new(None);
}
// Cached AWS Secrets Manager backend
#[cfg(all(feature = "private", feature = "enterprise"))]
struct CachedAwsSmBackend {
backend: Arc<dyn SecretBackend>,
settings: AwsSecretsManagerSettings,
}
#[cfg(all(feature = "private", feature = "enterprise"))]
lazy_static::lazy_static! {
static ref AWS_SM_BACKEND_CACHE: RwLock<Option<CachedAwsSmBackend>> = RwLock::new(None);
}
/// Get the current secret backend based on global settings (EE only)
#[cfg(all(feature = "private", feature = "enterprise"))]
async fn get_secret_backend(db: &DB) -> Result<Arc<dyn SecretBackend>> {
let config = match load_value_from_global_settings(db, SECRET_BACKEND_SETTING).await? {
Some(value) => serde_json::from_value::<SecretBackendConfig>(value).unwrap_or_default(),
None => SecretBackendConfig::default(),
};
match config {
SecretBackendConfig::Database => Ok(Arc::new(DatabaseBackend::new(db.clone()))),
SecretBackendConfig::HashiCorpVault(settings) => {
get_or_create_vault_backend(db, settings).await
}
SecretBackendConfig::AzureKeyVault(settings) => {
get_or_create_azure_kv_backend(db, settings).await
}
SecretBackendConfig::AwsSecretsManager(settings) => {
get_or_create_aws_sm_backend(db, settings).await
}
}
}
/// Get a cached Vault backend or create a new one if settings changed
#[cfg(all(feature = "private", feature = "enterprise"))]
async fn get_or_create_vault_backend(
_db: &DB,
settings: VaultSettings,
) -> Result<Arc<dyn SecretBackend>> {
// Check if we have a cached backend with matching settings (read lock)
{
let cache = VAULT_BACKEND_CACHE.read().await;
if let Some(ref cached) = *cache {
if cached.settings == settings {
return Ok(cached.backend.clone());
}
}
}
// Need to create a new backend - acquire write lock
let mut cache = VAULT_BACKEND_CACHE.write().await;
// Double-check (another task may have created it while we waited)
if let Some(ref cached) = *cache {
if cached.settings == settings {
return Ok(cached.backend.clone());
}
}
// Create new backend
let backend: Arc<dyn SecretBackend> = {
#[cfg(feature = "openidconnect")]
if settings.token.is_none() {
Arc::new(VaultBackend::new_with_db(settings.clone(), _db.clone()))
} else {
Arc::new(VaultBackend::new(settings.clone()))
}
#[cfg(not(feature = "openidconnect"))]
Arc::new(VaultBackend::new(settings.clone()))
};
// Cache it
*cache = Some(CachedVaultBackend { backend: backend.clone(), settings });
Ok(backend)
}
/// Get a cached Azure Key Vault backend or create a new one if settings changed
#[cfg(all(feature = "private", feature = "enterprise"))]
async fn get_or_create_azure_kv_backend(
_db: &DB,
settings: AzureKeyVaultSettings,
) -> Result<Arc<dyn SecretBackend>> {
// Check if we have a cached backend with matching settings (read lock)
{
let cache = AZURE_KV_BACKEND_CACHE.read().await;
if let Some(ref cached) = *cache {
if cached.settings == settings {
return Ok(cached.backend.clone());
}
}
}
// Need to create a new backend - acquire write lock
let mut cache = AZURE_KV_BACKEND_CACHE.write().await;
// Double-check (another task may have created it while we waited)
if let Some(ref cached) = *cache {
if cached.settings == settings {
return Ok(cached.backend.clone());
}
}
// Create new backend
let backend: Arc<dyn SecretBackend> = Arc::new(AzureKeyVaultBackend::new(settings.clone()));
// Cache it
*cache = Some(CachedAzureKvBackend { backend: backend.clone(), settings });
Ok(backend)
}
/// Get a cached AWS SM backend or create a new one if settings changed
#[cfg(all(feature = "private", feature = "enterprise"))]
async fn get_or_create_aws_sm_backend(
_db: &DB,
settings: AwsSecretsManagerSettings,
) -> Result<Arc<dyn SecretBackend>> {
{
let cache = AWS_SM_BACKEND_CACHE.read().await;
if let Some(ref cached) = *cache {
if cached.settings == settings {
return Ok(cached.backend.clone());
}
}
}
let mut cache = AWS_SM_BACKEND_CACHE.write().await;
if let Some(ref cached) = *cache {
if cached.settings == settings {
return Ok(cached.backend.clone());
}
}
let backend: Arc<dyn SecretBackend> =
Arc::new(AwsSecretsManagerBackend::new_with_client(settings.clone()).await?);
*cache = Some(CachedAwsSmBackend { backend: backend.clone(), settings });
Ok(backend)
}
/// Check if an external secret backend is currently configured (EE only)
#[cfg(all(feature = "private", feature = "enterprise"))]
async fn is_vault_backend_configured(db: &DB) -> Result<bool> {
let config = match load_value_from_global_settings(db, SECRET_BACKEND_SETTING).await? {
Some(value) => serde_json::from_value::<SecretBackendConfig>(value).unwrap_or_default(),
None => SecretBackendConfig::default(),
};
Ok(matches!(
config,
SecretBackendConfig::HashiCorpVault(_)
| SecretBackendConfig::AzureKeyVault(_)
| SecretBackendConfig::AwsSecretsManager(_)
))
}
/// Check if a value is stored in Vault (indicated by the $vault: prefix)
#[cfg(all(feature = "private", feature = "enterprise"))]
fn is_vault_stored_value(value: &str) -> bool {
value.starts_with("$vault:")
}
/// Check if a value is stored in Azure Key Vault (indicated by the $azure_kv: prefix)
#[cfg(all(feature = "private", feature = "enterprise"))]
fn is_azure_kv_stored_value(value: &str) -> bool {
value.starts_with("$azure_kv:")
}
/// Check if a value is stored in AWS Secrets Manager
#[cfg(all(feature = "private", feature = "enterprise"))]
fn is_aws_sm_stored_value(value: &str) -> bool {
value.starts_with("$aws_sm:")
}
/// Check if a value is stored in any external secret backend
#[cfg(all(feature = "private", feature = "enterprise"))]
fn is_external_stored_value(value: &str) -> bool {
is_vault_stored_value(value) || is_azure_kv_stored_value(value) || is_aws_sm_stored_value(value)
}
/// Bulk rename secrets in Vault when a path prefix changes (e.g., user rename)
/// EE only feature.
///
+2 -10
View File
@@ -11,7 +11,7 @@ pub use windmill_api_users::users::*;
use std::sync::Arc;
use crate::db::{ApiAuthed, OptJobAuthed};
use crate::db::ApiAuthed;
use crate::secret_backend_ext::rename_vault_secrets_with_prefix;
use argon2::Argon2;
use axum::{
@@ -21,7 +21,7 @@ use axum::{
};
use hyper::StatusCode;
use serde::Deserialize;
use windmill_api_auth::{forbid_superadmin_job_token, require_super_admin};
use windmill_api_auth::require_super_admin;
use windmill_audit::audit_oss::audit_log;
use windmill_audit::ActionKind;
use windmill_common::audit::AuditAuthor;
@@ -71,13 +71,11 @@ pub fn make_unauthed_service() -> Router {
async fn create_user(
authed: ApiAuthed,
OptJobAuthed { job_id, .. }: OptJobAuthed,
Extension(db): Extension<DB>,
Extension(webhook): Extension<windmill_common::webhook::WebhookShared>,
Extension(argon2): Extension<Arc<Argon2<'_>>>,
Json(nu): Json<NewUser>,
) -> Result<(StatusCode, String)> {
forbid_superadmin_job_token(&db, &authed.email, job_id).await?;
crate::users_oss::create_user(authed, db, webhook, argon2, nu).await
}
@@ -143,10 +141,8 @@ async fn set_password(
Extension(db): Extension<DB>,
Extension(argon2): Extension<Arc<Argon2<'_>>>,
authed: ApiAuthed,
OptJobAuthed { job_id, .. }: OptJobAuthed,
Json(ep): Json<EditPassword>,
) -> Result<String> {
forbid_superadmin_job_token(&db, &authed.email, job_id).await?;
let email = authed.email.clone();
crate::users_oss::set_password(db, argon2, authed, &email, ep).await
}
@@ -156,11 +152,9 @@ async fn set_password_of_user(
Extension(argon2): Extension<Arc<Argon2<'_>>>,
Path(email): Path<String>,
authed: ApiAuthed,
OptJobAuthed { job_id, .. }: OptJobAuthed,
Json(ep): Json<EditPassword>,
) -> Result<String> {
require_super_admin(&db, &authed.email).await?;
forbid_superadmin_job_token(&db, &authed.email, job_id).await?;
crate::users_oss::set_password(db, argon2, authed, &email, ep).await
}
@@ -171,13 +165,11 @@ struct RenameUser {
async fn rename_user(
authed: ApiAuthed,
OptJobAuthed { job_id, .. }: OptJobAuthed,
Path(user_email): Path<String>,
Extension(db): Extension<DB>,
Json(ru): Json<RenameUser>,
) -> Result<String> {
require_super_admin(&db, &authed.email).await?;
forbid_superadmin_job_token(&db, &authed.email, job_id).await?;
let mut tx = db.begin().await?;
+1 -4
View File
@@ -4,10 +4,7 @@ use sqlx::{PgExecutor, Postgres, Transaction};
use crate::{error, scripts::ScriptHash};
pub use windmill_parser::asset_parser::{
parse_pipeline_annotations, DataTest, PartitionKind, PipelineAnnotations, RetrySpec,
TriggerSpec, PARTITION_TOKEN,
};
pub use windmill_parser::asset_parser::{parse_pipeline_annotations, TriggerSpec, PARTITION_TOKEN};
pub use windmill_types::assets::*;
#[derive(sqlx::Type, Debug, Clone, Copy, PartialEq)]
+3 -29
View File
@@ -529,31 +529,6 @@ mod classify_python_logging_line_tests {
}
}
#[cfg(test)]
mod validate_dbname_tests {
use super::validate_dbname;
#[test]
fn accepts_letters_digits_underscores_and_hyphens() {
assert!(validate_dbname("mydb").is_ok());
assert!(validate_dbname("my_db").is_ok());
assert!(validate_dbname("my-database").is_ok());
assert!(validate_dbname("My-Db_1").is_ok());
}
#[test]
fn rejects_invalid_names() {
// Must start with a letter (hyphen/digit/underscore leads are rejected).
assert!(validate_dbname("-db").is_err());
assert!(validate_dbname("1db").is_err());
assert!(validate_dbname("_db").is_err());
// No other special characters or whitespace.
assert!(validate_dbname("my db").is_err());
assert!(validate_dbname("my;db").is_err());
assert!(validate_dbname("").is_err());
}
}
#[derive(Serialize, Debug)]
pub struct PrepareQueryColumnInfo {
pub name: String,
@@ -837,7 +812,7 @@ impl PgDatabase {
}
/// Validate a database name to prevent SQL injection.
/// Must start with a letter, contain only alphanumeric characters, underscores, or hyphens, and be <= 63 chars.
/// Must start with a letter, contain only alphanumeric characters or underscores, and be <= 63 chars.
pub fn validate_dbname(dbname: &str) -> error::Result<()> {
let dbname = dbname.trim();
if dbname.is_empty() {
@@ -861,11 +836,10 @@ pub fn validate_dbname(dbname: &str) -> error::Result<()> {
}
if !dbname
.chars()
.all(|c| c.is_ascii_alphanumeric() || c == '_' || c == '-')
.all(|c| c.is_ascii_alphanumeric() || c == '_')
{
return Err(error::Error::BadRequest(
"Database name must contain only alphanumeric characters, underscores, or hyphens"
.to_string(),
"Database name must contain only alphanumeric characters or underscores".to_string(),
));
}
Ok(())
+17 -161
View File
@@ -172,19 +172,6 @@ pub struct SimpleColumn {
pub struct SelectOptions {
pub limit: Option<i64>,
pub offset: Option<i64>,
/// DuckLake time-travel: when set (DuckDB only), the read is pinned to this
/// catalog snapshot via `AT (VERSION => n)`. Ignored for other db types.
pub version: Option<i64>,
}
/// DuckLake time-travel suffix appended after a table name in a FROM clause.
/// `n` is a server-controlled `i64` (a snapshot id), so inlining it is
/// injection-safe. Empty string when unpinned (reads the latest snapshot).
fn duckdb_version_suffix(version: Option<i64>) -> String {
match version {
Some(v) => format!(" AT (VERSION => {})", v),
None => String::new(),
}
}
// ---------------------------------------------------------------------------
@@ -203,8 +190,6 @@ struct SelectPayload {
#[serde(rename = "fixPgIntTypes")]
fix_pg_int_types: Option<bool>,
ducklake: Option<String>,
/// DuckLake snapshot to time-travel the read to (DuckDB only).
version: Option<i64>,
}
#[derive(Deserialize)]
@@ -215,21 +200,6 @@ struct CountPayload {
#[serde(rename = "whereClause")]
where_clause: Option<String>,
ducklake: Option<String>,
/// DuckLake snapshot to time-travel the count to (DuckDB only).
version: Option<i64>,
}
/// `WM_INTERNAL_DB_DUCKLAKE_SNAPSHOTS` payload — lists the time-travel history
/// of a ducklake table. DuckLake snapshots are catalog-wide commits, so without
/// a `table` this lists every commit; with one it is scoped to snapshots where
/// that table exists (see `expand_ducklake_snapshots`).
#[derive(Deserialize)]
struct DucklakeSnapshotsPayload {
ducklake: String,
/// Schema-qualified table name (e.g. `main.events_daily`) to scope the
/// history to. Snapshots predating the table's creation are excluded — a
/// time-travel read can't target a version where the table didn't exist.
table: Option<String>,
}
#[derive(Deserialize)]
@@ -334,10 +304,6 @@ pub fn try_expand_internal_db_query(
expand_primary_key_constraint(json_str, db_type).map(ExpandedQuery::sql)
}
"SNOWFLAKE_PRIMARY_KEYS" => expand_snowflake_primary_keys(json_str).map(ExpandedQuery::sql),
// DuckLake time-travel: list a ducklake's snapshot history
"DUCKLAKE_SNAPSHOTS" => {
expand_ducklake_snapshots(json_str, db_type).map(ExpandedQuery::sql)
}
_ => Err(format!("Unknown WM_INTERNAL_DB operation: {}", op)),
};
@@ -358,8 +324,7 @@ fn expand_select(json_str: &str, db_type: DbType) -> Result<String, String> {
let payload: SelectPayload =
serde_json::from_str(json_str).map_err(|e| format!("Invalid SELECT payload: {}", e))?;
let options =
SelectOptions { limit: payload.limit, offset: payload.offset, version: payload.version };
let options = SelectOptions { limit: payload.limit, offset: payload.offset };
let breaking = payload
.fix_pg_int_types
.map(|v| BreakingFeatures { fix_pg_int_types: v });
@@ -385,47 +350,11 @@ fn expand_count(json_str: &str, db_type: DbType) -> Result<String, String> {
&payload.table,
payload.where_clause.as_deref(),
&payload.column_defs,
payload.version,
)?;
Ok(maybe_wrap_ducklake(query, payload.ducklake.as_deref()))
}
/// Expand `DUCKLAKE_SNAPSHOTS` into the catalog's time-travel history. DuckLake
/// snapshots are catalog-wide commits, so `ducklake_snapshots('dl')` (the alias
/// `maybe_wrap_ducklake` attaches) lists every version any `AT (VERSION => n)`
/// read can target, newest first.
fn expand_ducklake_snapshots(json_str: &str, db_type: DbType) -> Result<String, String> {
if db_type != DbType::Duckdb {
return Err("DUCKLAKE_SNAPSHOTS is only supported for DuckDB".to_string());
}
let payload: DucklakeSnapshotsPayload = serde_json::from_str(json_str)
.map_err(|e| format!("Invalid DUCKLAKE_SNAPSHOTS payload: {}", e))?;
// `dl` is the alias `wrap_ducklake_query` attaches and `USE`s below.
let query = match &payload.table {
// Scope to snapshots from the table's first creation onward. A DuckLake
// table created at snapshot N can't be read before N (the catalog-wide
// list would otherwise offer impossible versions). The creation snapshot
// is the earliest whose `changes.tables_created` names the table;
// COALESCE to 0 (show all) if it is never found.
Some(table) => {
let table = escape_sql_literal(table);
format!(
"SELECT snapshot_id, snapshot_time FROM ducklake_snapshots('dl') \
WHERE snapshot_id >= COALESCE((\
SELECT min(snapshot_id) FROM ducklake_snapshots('dl') \
WHERE list_contains(changes.tables_created, '{table}')), 0) \
ORDER BY snapshot_id DESC"
)
}
None => {
"SELECT snapshot_id, snapshot_time FROM ducklake_snapshots('dl') ORDER BY snapshot_id DESC"
.to_string()
}
};
Ok(maybe_wrap_ducklake(query, Some(&payload.ducklake)))
}
/// Filter columns to primary keys only; fall back to all columns if none are marked.
fn pk_columns_or_all(columns: &[ColumnDef]) -> Vec<ColumnDef> {
let pks: Vec<ColumnDef> = columns.iter().filter(|c| c.isprimarykey).cloned().collect();
@@ -1021,10 +950,9 @@ pub fn make_select_query(
);
query.push_str(&format!(
"SELECT {} FROM {}{}\n",
"SELECT {} FROM {}\n",
filtered_columns.join(", "),
quote_table_name(table, db_type),
duckdb_version_suffix(options.and_then(|o| o.version))
quote_table_name(table, db_type)
));
query.push_str(&format!(
" WHERE {} {}\n",
@@ -1049,8 +977,6 @@ pub fn make_count_query(
table: &str,
where_clause: Option<&str>,
column_defs: &[ColumnDef],
// DuckLake time-travel snapshot (DuckDB only); `None` counts the latest.
version: Option<i64>,
) -> Result<String, String> {
let where_prefix = " WHERE ";
let and_condition = " AND ";
@@ -1192,9 +1118,8 @@ pub fn make_count_query(
quicksearch_condition.push_str(" ($quicksearch = '' OR 1 = 1)");
}
query.push_str(&format!(
"SELECT COUNT(*) as count FROM {}{}",
quote_table_name(table, db_type),
duckdb_version_suffix(version)
"SELECT COUNT(*) as count FROM {}",
quote_table_name(table, db_type)
));
}
}
@@ -3073,7 +2998,7 @@ mod tests {
#[test]
fn test_select_snowflake_custom_limit() {
let cols = vec![col("id", "int")];
let opts = SelectOptions { limit: Some(50), offset: Some(10), version: None };
let opts = SelectOptions { limit: Some(50), offset: Some(10) };
let result = make_select_query(
"my_table",
&cols,
@@ -3147,7 +3072,7 @@ mod tests {
#[test]
fn test_count_postgresql_basic() {
let cols = vec![col("id", "int4"), col("name", "text")];
let result = make_count_query(DbType::Postgresql, "my_table", None, &cols, None).unwrap();
let result = make_count_query(DbType::Postgresql, "my_table", None, &cols).unwrap();
assert!(result.contains("-- $1 quicksearch (text)"));
assert!(result.contains("SELECT COUNT(*) as count FROM \"my_table\""));
@@ -3165,7 +3090,6 @@ mod tests {
"my_table",
Some("status = 'active'"),
&cols,
None,
)
.unwrap();
@@ -3181,7 +3105,7 @@ mod tests {
c.ignored = Some(true);
c
}];
let result = make_count_query(DbType::Postgresql, "my_table", None, &cols, None).unwrap();
let result = make_count_query(DbType::Postgresql, "my_table", None, &cols).unwrap();
assert!(result.contains("($1 = '' OR 1 = 1)"));
}
@@ -3192,7 +3116,7 @@ mod tests {
#[test]
fn test_count_mysql_basic() {
let cols = vec![col("id", "int"), col("name", "varchar")];
let result = make_count_query(DbType::Mysql, "my_table", None, &cols, None).unwrap();
let result = make_count_query(DbType::Mysql, "my_table", None, &cols).unwrap();
assert!(result.contains("-- :quicksearch (text)"));
assert!(result.contains("SELECT COUNT(*) as count FROM `my_table`"));
@@ -3206,7 +3130,7 @@ mod tests {
#[test]
fn test_count_mssql_basic() {
let cols = vec![col("id", "int"), col("name", "nvarchar")];
let result = make_count_query(DbType::MsSqlServer, "my_table", None, &cols, None).unwrap();
let result = make_count_query(DbType::MsSqlServer, "my_table", None, &cols).unwrap();
assert!(result.contains("SELECT COUNT(*) as count FROM [my_table]"));
assert!(result.contains("(@p1 = '' OR CONCAT([id], [name]) LIKE '%' + @p1 + '%')"));
@@ -3219,7 +3143,7 @@ mod tests {
#[test]
fn test_count_snowflake_basic() {
let cols = vec![col("id", "int"), col("name", "text")];
let result = make_count_query(DbType::Snowflake, "my_table", None, &cols, None).unwrap();
let result = make_count_query(DbType::Snowflake, "my_table", None, &cols).unwrap();
// Two quicksearch params for snowflake with visible columns
assert!(result.contains("-- ? quicksearch (text)\n-- ? quicksearch (text)"));
@@ -3234,7 +3158,7 @@ mod tests {
c.ignored = Some(true);
c
}];
let result = make_count_query(DbType::Snowflake, "my_table", None, &cols, None).unwrap();
let result = make_count_query(DbType::Snowflake, "my_table", None, &cols).unwrap();
// One quicksearch param
let param_lines: Vec<&str> = result.lines().filter(|l| l.starts_with("-- ?")).collect();
assert_eq!(param_lines.len(), 1);
@@ -3248,7 +3172,7 @@ mod tests {
#[test]
fn test_count_bigquery_basic() {
let cols = vec![col("id", "INTEGER"), col("name", "STRING")];
let result = make_count_query(DbType::Bigquery, "my_table", None, &cols, None).unwrap();
let result = make_count_query(DbType::Bigquery, "my_table", None, &cols).unwrap();
assert!(result.contains("-- @quicksearch (string)"));
assert!(result.contains("SELECT COUNT(*) as count FROM `my_table`"));
@@ -3258,7 +3182,7 @@ mod tests {
#[test]
fn test_count_bigquery_json_type() {
let cols = vec![col("id", "INTEGER"), col("data", "JSON")];
let result = make_count_query(DbType::Bigquery, "my_table", None, &cols, None).unwrap();
let result = make_count_query(DbType::Bigquery, "my_table", None, &cols).unwrap();
assert!(result.contains("TO_JSON_STRING(`data`)"));
}
@@ -3269,7 +3193,7 @@ mod tests {
#[test]
fn test_count_duckdb_basic() {
let cols = vec![col("id", "int"), col("name", "text")];
let result = make_count_query(DbType::Duckdb, "my_table", None, &cols, None).unwrap();
let result = make_count_query(DbType::Duckdb, "my_table", None, &cols).unwrap();
assert!(result.contains("-- $quicksearch (text)"));
assert!(result.contains("SELECT COUNT(*) as count FROM \"my_table\""));
@@ -3278,74 +3202,6 @@ mod tests {
);
}
// -----------------------------------------------------------------------
// DuckLake time-travel (AT VERSION) + snapshot history
// -----------------------------------------------------------------------
#[test]
fn test_select_duckdb_time_travel() {
let cols = vec![col("id", "int"), col("name", "text")];
let opts = SelectOptions { limit: None, offset: None, version: Some(42) };
let result =
make_select_query("orders", &cols, None, DbType::Duckdb, Some(&opts), None).unwrap();
// Read is pinned to the catalog snapshot via AT (VERSION => n).
assert!(result.contains("FROM \"orders\" AT (VERSION => 42)\n"));
}
#[test]
fn test_select_duckdb_no_version_unpinned() {
let cols = vec![col("id", "int")];
let result = make_select_query("orders", &cols, None, DbType::Duckdb, None, None).unwrap();
// Without a version the read targets the latest snapshot — no AT clause.
assert!(result.contains("FROM \"orders\"\n"));
assert!(!result.contains("AT (VERSION"));
}
#[test]
fn test_count_duckdb_time_travel() {
let cols = vec![col("id", "int")];
let result = make_count_query(DbType::Duckdb, "orders", None, &cols, Some(7)).unwrap();
assert!(result.contains("FROM \"orders\" AT (VERSION => 7)"));
}
#[test]
fn test_version_ignored_for_non_duckdb() {
// AT (VERSION) is DuckLake-only; other dialects must never emit it even
// if a version is somehow passed through.
let cols = vec![col("id", "int4")];
let opts = SelectOptions { limit: None, offset: None, version: Some(5) };
let result =
make_select_query("orders", &cols, None, DbType::Postgresql, Some(&opts), None)
.unwrap();
assert!(!result.contains("AT (VERSION"));
}
#[test]
fn test_expand_ducklake_snapshots() {
let json = r#"{"ducklake": "analytics"}"#;
let result = expand_ducklake_snapshots(json, DbType::Duckdb).unwrap();
assert!(result.contains("ATTACH 'ducklake://analytics' AS dl;USE dl;"));
assert!(result.contains("ducklake_snapshots('dl')"));
assert!(result.contains("ORDER BY snapshot_id DESC"));
// Unscoped: no per-table existence filter.
assert!(!result.contains("tables_created"));
}
#[test]
fn test_expand_ducklake_snapshots_scoped_to_table() {
let json = r#"{"ducklake": "analytics", "table": "main.events_daily"}"#;
let result = expand_ducklake_snapshots(json, DbType::Duckdb).unwrap();
// Scoped to snapshots from the table's first creation onward.
assert!(result.contains("list_contains(changes.tables_created, 'main.events_daily')"));
assert!(result.contains("snapshot_id >= COALESCE"));
}
#[test]
fn test_expand_ducklake_snapshots_non_duckdb_errors() {
let json = r#"{"ducklake": "analytics"}"#;
assert!(expand_ducklake_snapshots(json, DbType::Postgresql).is_err());
}
// -----------------------------------------------------------------------
// DELETE - all DB types
// -----------------------------------------------------------------------
@@ -3644,7 +3500,7 @@ mod tests {
#[test]
fn test_count_mssql_no_where() {
let cols = vec![col("id", "int")];
let result = make_count_query(DbType::MsSqlServer, "my_table", None, &cols, None).unwrap();
let result = make_count_query(DbType::MsSqlServer, "my_table", None, &cols).unwrap();
// MSSQL uses WHERE directly (no AND replacement)
assert!(result.contains("SELECT COUNT(*) as count FROM [my_table] WHERE "));
}
@@ -3652,7 +3508,7 @@ mod tests {
#[test]
fn test_count_mysql_no_where_uses_where_keyword() {
let cols = vec![col("id", "int")];
let result = make_count_query(DbType::Mysql, "my_table", None, &cols, None).unwrap();
let result = make_count_query(DbType::Mysql, "my_table", None, &cols).unwrap();
// The AND should be replaced with WHERE
assert!(result.contains("FROM `my_table` WHERE "));
assert!(!result.contains("FROM `my_table` AND "));
@@ -13,9 +13,6 @@
//! vaults like HashiCorp Vault (Enterprise Edition).
pub mod database;
pub mod resolver;
pub use resolver::*;
#[cfg(feature = "private")]
pub mod vault_ee;
@@ -1,324 +0,0 @@
/*
* Author: Ruben Fiszel
* Copyright: Windmill Labs, Inc 2024
* This file and its contents are licensed under the AGPLv3 License.
* Please see the included NOTICE for copyright information and
* LICENSE-AGPL for a copy of the license.
*/
//! Resolution of the configured secret backend.
//!
//! Lives in `windmill-common` (rather than the API/store crates) so that
//! lower-level helpers such as [`crate::variables::get_variable_or_self`] can
//! route secret reads through the configured backend. With an external backend
//! (Vault / Azure Key Vault / AWS Secrets Manager), the `variable.value` column
//! holds a `$vault:`/`$azure_kv:`/`$aws_sm:` marker rather than base64
//! ciphertext, so decrypting it directly fails — reads must go through the
//! backend instead.
//!
//! Note: external backends require Enterprise Edition. The OSS version only
//! supports the database backend.
use std::sync::Arc;
use crate::{
db::DB,
error::{Error, Result},
secret_backend::{database::DatabaseBackend, SecretBackend},
variables::{build_crypt, decrypt},
};
#[cfg(all(feature = "private", feature = "enterprise"))]
use crate::{
global_settings::{load_value_from_global_settings, SECRET_BACKEND_SETTING},
secret_backend::{
AwsSecretsManagerBackend, AwsSecretsManagerSettings, AzureKeyVaultBackend,
AzureKeyVaultSettings, SecretBackendConfig, VaultBackend, VaultSettings,
},
};
#[cfg(all(feature = "private", feature = "enterprise"))]
use tokio::sync::RwLock;
// Cached Vault backend to avoid recreating it for every request
// This enables connection pooling and avoids repeated setup overhead
#[cfg(all(feature = "private", feature = "enterprise"))]
struct CachedVaultBackend {
backend: Arc<dyn SecretBackend>,
settings: VaultSettings,
}
#[cfg(all(feature = "private", feature = "enterprise"))]
lazy_static::lazy_static! {
static ref VAULT_BACKEND_CACHE: RwLock<Option<CachedVaultBackend>> = RwLock::new(None);
}
#[cfg(all(feature = "private", feature = "enterprise"))]
struct CachedAzureKvBackend {
backend: Arc<dyn SecretBackend>,
settings: AzureKeyVaultSettings,
}
#[cfg(all(feature = "private", feature = "enterprise"))]
lazy_static::lazy_static! {
static ref AZURE_KV_BACKEND_CACHE: RwLock<Option<CachedAzureKvBackend>> = RwLock::new(None);
}
// Cached AWS Secrets Manager backend
#[cfg(all(feature = "private", feature = "enterprise"))]
struct CachedAwsSmBackend {
backend: Arc<dyn SecretBackend>,
settings: AwsSecretsManagerSettings,
}
#[cfg(all(feature = "private", feature = "enterprise"))]
lazy_static::lazy_static! {
static ref AWS_SM_BACKEND_CACHE: RwLock<Option<CachedAwsSmBackend>> = RwLock::new(None);
}
/// Get the current secret backend based on global settings
///
/// OSS: Always returns DatabaseBackend
/// EE: Returns configured backend (Database or Vault)
#[cfg(not(all(feature = "private", feature = "enterprise")))]
pub async fn get_secret_backend(db: &DB) -> Result<Arc<dyn SecretBackend>> {
Ok(Arc::new(DatabaseBackend::new(db.clone())))
}
#[cfg(all(feature = "private", feature = "enterprise"))]
pub async fn get_secret_backend(db: &DB) -> Result<Arc<dyn SecretBackend>> {
let config = match load_value_from_global_settings(db, SECRET_BACKEND_SETTING).await? {
Some(value) => serde_json::from_value::<SecretBackendConfig>(value).unwrap_or_default(),
None => SecretBackendConfig::default(),
};
match config {
SecretBackendConfig::Database => Ok(Arc::new(DatabaseBackend::new(db.clone()))),
SecretBackendConfig::HashiCorpVault(settings) => {
get_or_create_vault_backend(db, settings).await
}
SecretBackendConfig::AzureKeyVault(settings) => {
get_or_create_azure_kv_backend(db, settings).await
}
SecretBackendConfig::AwsSecretsManager(settings) => {
get_or_create_aws_sm_backend(db, settings).await
}
}
}
/// Get a cached Vault backend or create a new one if settings changed
#[cfg(all(feature = "private", feature = "enterprise"))]
async fn get_or_create_vault_backend(
_db: &DB,
settings: VaultSettings,
) -> Result<Arc<dyn SecretBackend>> {
// Check if we have a cached backend with matching settings (read lock)
{
let cache = VAULT_BACKEND_CACHE.read().await;
if let Some(ref cached) = *cache {
if cached.settings == settings {
return Ok(cached.backend.clone());
}
}
}
// Need to create a new backend - acquire write lock
let mut cache = VAULT_BACKEND_CACHE.write().await;
// Double-check (another task may have created it while we waited)
if let Some(ref cached) = *cache {
if cached.settings == settings {
return Ok(cached.backend.clone());
}
}
// Create new backend
let backend: Arc<dyn SecretBackend> = {
#[cfg(feature = "openidconnect")]
if settings.token.is_none() {
Arc::new(VaultBackend::new_with_db(settings.clone(), _db.clone()))
} else {
Arc::new(VaultBackend::new(settings.clone()))
}
#[cfg(not(feature = "openidconnect"))]
Arc::new(VaultBackend::new(settings.clone()))
};
// Cache it
*cache = Some(CachedVaultBackend { backend: backend.clone(), settings });
Ok(backend)
}
/// Get a cached Azure Key Vault backend or create a new one if settings changed
#[cfg(all(feature = "private", feature = "enterprise"))]
async fn get_or_create_azure_kv_backend(
_db: &DB,
settings: AzureKeyVaultSettings,
) -> Result<Arc<dyn SecretBackend>> {
// Check if we have a cached backend with matching settings (read lock)
{
let cache = AZURE_KV_BACKEND_CACHE.read().await;
if let Some(ref cached) = *cache {
if cached.settings == settings {
return Ok(cached.backend.clone());
}
}
}
// Need to create a new backend - acquire write lock
let mut cache = AZURE_KV_BACKEND_CACHE.write().await;
// Double-check (another task may have created it while we waited)
if let Some(ref cached) = *cache {
if cached.settings == settings {
return Ok(cached.backend.clone());
}
}
// Create new backend
let backend: Arc<dyn SecretBackend> = Arc::new(AzureKeyVaultBackend::new(settings.clone()));
// Cache it
*cache = Some(CachedAzureKvBackend { backend: backend.clone(), settings });
Ok(backend)
}
/// Get a cached AWS SM backend or create a new one if settings changed
#[cfg(all(feature = "private", feature = "enterprise"))]
async fn get_or_create_aws_sm_backend(
_db: &DB,
settings: AwsSecretsManagerSettings,
) -> Result<Arc<dyn SecretBackend>> {
{
let cache = AWS_SM_BACKEND_CACHE.read().await;
if let Some(ref cached) = *cache {
if cached.settings == settings {
return Ok(cached.backend.clone());
}
}
}
let mut cache = AWS_SM_BACKEND_CACHE.write().await;
if let Some(ref cached) = *cache {
if cached.settings == settings {
return Ok(cached.backend.clone());
}
}
let backend: Arc<dyn SecretBackend> =
Arc::new(AwsSecretsManagerBackend::new_with_client(settings.clone()).await?);
*cache = Some(CachedAwsSmBackend { backend: backend.clone(), settings });
Ok(backend)
}
/// Check if a Vault backend is currently configured
///
/// OSS: Always returns false
/// EE: Checks global settings
#[cfg(not(all(feature = "private", feature = "enterprise")))]
pub async fn is_vault_backend_configured(_db: &DB) -> Result<bool> {
Ok(false)
}
#[cfg(all(feature = "private", feature = "enterprise"))]
pub async fn is_vault_backend_configured(db: &DB) -> Result<bool> {
let config = match load_value_from_global_settings(db, SECRET_BACKEND_SETTING).await? {
Some(value) => serde_json::from_value::<SecretBackendConfig>(value).unwrap_or_default(),
None => SecretBackendConfig::default(),
};
Ok(matches!(
config,
SecretBackendConfig::HashiCorpVault(_)
| SecretBackendConfig::AzureKeyVault(_)
| SecretBackendConfig::AwsSecretsManager(_)
))
}
/// Get a secret value using the configured backend
///
/// For database backend: decrypts `encrypted_value` using the workspace key
/// For external backends (EE only): fetches from the backend at `path`,
/// ignoring `encrypted_value` (which holds only a `$...:` marker)
pub async fn get_secret_value(
db: &DB,
workspace_id: &str,
path: &str,
encrypted_value: &str,
) -> Result<String> {
let backend = get_secret_backend(db).await?;
match backend.backend_name() {
"database" => {
// Use existing database decryption
let mc = build_crypt(db, workspace_id).await?;
decrypt(&mc, encrypted_value.to_string()).map_err(|e| {
Error::internal_err(format!("Error decrypting variable {}: {}", path, e))
})
}
"hashicorp_vault" => {
// Fetch from Vault directly
backend.get_secret(workspace_id, path).await
}
"azure_key_vault" => backend.get_secret(workspace_id, path).await,
"aws_secrets_manager" => backend.get_secret(workspace_id, path).await,
_ => Err(Error::internal_err(format!(
"Unknown backend: {}",
backend.backend_name()
))),
}
}
/// Check if a value is stored in Vault (indicated by the $vault: prefix)
pub fn is_vault_stored_value(value: &str) -> bool {
value.starts_with("$vault:")
}
/// Check if a value is stored in Azure Key Vault (indicated by the $azure_kv: prefix)
pub fn is_azure_kv_stored_value(value: &str) -> bool {
value.starts_with("$azure_kv:")
}
/// Check if a value is stored in AWS Secrets Manager (indicated by the $aws_sm: prefix)
pub fn is_aws_sm_stored_value(value: &str) -> bool {
value.starts_with("$aws_sm:")
}
/// Check if a value is stored in any external secret backend
pub fn is_external_stored_value(value: &str) -> bool {
is_vault_stored_value(value) || is_azure_kv_stored_value(value) || is_aws_sm_stored_value(value)
}
#[cfg(test)]
mod tests {
use super::*;
#[test]
fn external_markers_are_detected() {
assert!(is_external_stored_value("$vault:u/admin/secret"));
assert!(is_external_stored_value("$azure_kv:u/admin/secret"));
assert!(is_external_stored_value("$aws_sm:u/admin/secret"));
}
#[test]
fn base64_ciphertext_is_not_treated_as_external() {
// A base64 magic_crypt blob must route through `decrypt`, never the
// external backend. The leading `$` is what distinguishes a marker from
// ciphertext; decrypting a marker fails with "Invalid byte 36" (`$`),
// which is the bug this gate prevents.
for v in [
"bm90LWEtbWFya2Vy",
"AAAA1234+/abcd==",
"",
"$something_else",
] {
assert!(!is_external_stored_value(v), "unexpected external: {v:?}");
}
}
}
+15 -28
View File
@@ -9,7 +9,6 @@
use crate::db::{Authable, UserDB};
use crate::error::{self, Error};
use crate::scripts::ScriptHash;
use crate::secret_backend::{get_secret_value, is_external_stored_value};
use crate::utils::WarnAfterExt;
use crate::worker::Connection;
use crate::{worker::WORKER_GROUP, BASE_URL, DB};
@@ -235,17 +234,13 @@ pub async fn get_secret_value_as_admin(
let r = if variable.is_secret {
let value = variable.value;
if !value.is_empty() {
if is_external_stored_value(&value) {
get_secret_value(db, w_id, &variable.path, &value).await?
} else {
let mc = build_crypt(db, w_id).await?;
decrypt(&mc, value).map_err(|e| {
crate::error::Error::internal_err(format!(
"Error decrypting variable {}: {}",
variable.path, e
))
})?
}
let mc = build_crypt(db, w_id).await?;
decrypt(&mc, value).map_err(|e| {
crate::error::Error::internal_err(format!(
"Error decrypting variable {}: {}",
variable.path, e
))
})?
} else {
"".to_string()
}
@@ -551,14 +546,10 @@ pub async fn get_variable_or_self(
if let Some(record) = record {
let mut value = record.value;
if record.is_secret {
if is_external_stored_value(&value) {
value = get_secret_value(db, w_id, &path, &value).await?;
} else {
let mc = build_crypt(db, w_id).await?;
value = decrypt(&mc, value).map_err(|e| {
Error::internal_err(format!("Error decrypting variable {}: {}", path, e))
})?;
}
let mc = build_crypt(db, w_id).await?;
value = decrypt(&mc, value).map_err(|e| {
Error::internal_err(format!("Error decrypting variable {}: {}", path, e))
})?;
}
Ok(value)
@@ -600,14 +591,10 @@ pub async fn get_variable_or_self_as<T: Authable + Sync>(
if let Some(record) = record {
let mut value = record.value;
if record.is_secret {
if is_external_stored_value(&value) {
value = get_secret_value(db, w_id, &var_path, &value).await?;
} else {
let mc = build_crypt(db, w_id).await?;
value = decrypt(&mc, value).map_err(|e| {
Error::internal_err(format!("Error decrypting variable {}: {}", var_path, e))
})?;
}
let mc = build_crypt(db, w_id).await?;
value = decrypt(&mc, value).map_err(|e| {
Error::internal_err(format!("Error decrypting variable {}: {}", var_path, e))
})?;
}
Ok(value)
+2 -97
View File
@@ -693,6 +693,8 @@ pub fn is_allowed_file_location(job_dir: &str, user_defined_path: &str) -> error
let full_path = job_dir.join(&user_path);
// let normalized_job_dir = std::fs::canonicalize(job_dir)?;
// let normalized_full_path = std::fs::canonicalize(&full_path)?;
let normalized_job_dir = normalize_path(job_dir);
let normalized_full_path = normalize_path(&full_path);
@@ -704,36 +706,6 @@ pub fn is_allowed_file_location(job_dir: &str, user_defined_path: &str) -> error
.into());
}
// The lexical check above cannot see symlinks: a symlink planted inside the
// job dir - e.g. by an earlier Ansible `git_repos` clone whose tracked
// content includes one - would let a later `git clone` or file write follow
// it out of the job dir while still passing the textual `starts_with` check.
// Walk the *normalized* relative path (`..`/`.` already collapsed) so each
// step matches the real on-disk resolution, and reject any existing component
// that is a symlink. Walking the raw user path would drift on an in-bounds
// `..` (e.g. `foo/../link`, which normalizes back inside the job dir) and miss
// the real symlinked component. Not-yet-existing components are safe: a path
// that does not exist cannot itself be a symlink.
let relative = normalized_full_path
.strip_prefix(&normalized_job_dir)
.unwrap_or(&normalized_full_path);
let mut current = normalized_job_dir.clone();
for component in relative.components() {
if let Component::Normal(c) = component {
current.push(c);
if std::fs::symlink_metadata(&current)
.map(|m| m.file_type().is_symlink())
.unwrap_or(false)
{
return Err(std::io::Error::new(
std::io::ErrorKind::PermissionDenied,
"Path traverses a symlink, which is not allowed.",
)
.into());
}
}
}
Ok(normalized_full_path)
}
@@ -2856,71 +2828,4 @@ mod tests {
let _ = std::fs::remove_dir_all(&base);
}
#[test]
fn test_is_allowed_file_location_allows_plain_relative() {
let base = std::env::temp_dir().join(format!("wm_allowed_loc_ok_{}", uuid::Uuid::new_v4()));
let job_dir = base.join("job");
std::fs::create_dir_all(&job_dir).unwrap();
let job_dir_str = job_dir.to_str().unwrap();
let out = is_allowed_file_location(job_dir_str, "repo/sub/playbook.yml").unwrap();
assert_eq!(out, normalize_path(&job_dir.join("repo/sub/playbook.yml")));
let _ = std::fs::remove_dir_all(&base);
}
#[test]
fn test_is_allowed_file_location_rejects_parent_and_absolute() {
let base =
std::env::temp_dir().join(format!("wm_allowed_loc_esc_{}", uuid::Uuid::new_v4()));
let job_dir = base.join("job");
std::fs::create_dir_all(&job_dir).unwrap();
let job_dir_str = job_dir.to_str().unwrap();
assert!(is_allowed_file_location(job_dir_str, "../escape").is_err());
assert!(is_allowed_file_location(job_dir_str, "a/../../escape").is_err());
assert!(is_allowed_file_location(job_dir_str, "/etc/passwd").is_err());
let _ = std::fs::remove_dir_all(&base);
}
// Regression for GHSA-v934-cvpf-6fjw: a symlink planted inside the job dir
// (e.g. by an earlier `git_repos` clone) must not let a later target traverse
// it out of the job dir, even though the lexical path stays "inside".
#[cfg(unix)]
#[test]
fn test_is_allowed_file_location_rejects_symlink_traversal() {
let base =
std::env::temp_dir().join(format!("wm_allowed_loc_symlink_{}", uuid::Uuid::new_v4()));
let job_dir = base.join("job");
std::fs::create_dir_all(&job_dir).unwrap();
// Stand-in for the shared cache dir living outside the job dir.
let outside = base.join("outside");
std::fs::create_dir_all(&outside).unwrap();
let job_dir_str = job_dir.to_str().unwrap();
// Plant `job/repo` -> `../outside`, as a malicious first clone would.
let planted = job_dir.join("repo");
std::os::unix::fs::symlink(&outside, &planted).unwrap();
// Both the symlink itself and any path traversing it are rejected.
assert!(is_allowed_file_location(job_dir_str, "repo").is_err());
assert!(is_allowed_file_location(job_dir_str, "repo/payload").is_err());
assert!(is_allowed_file_location(job_dir_str, "repo/sub/payload").is_err());
// An in-bounds `..` must not bypass the check: `foo/../repo/payload`
// normalizes back to `repo/payload` and still traverses the symlink.
assert!(is_allowed_file_location(job_dir_str, "foo/../repo/payload").is_err());
std::fs::create_dir(job_dir.join("real")).unwrap();
assert!(is_allowed_file_location(job_dir_str, "real/../repo/payload").is_err());
// A dangling symlink (target does not exist yet) is still caught:
// `symlink_metadata` does not follow the link.
let dangling = job_dir.join("dangling");
std::os::unix::fs::symlink(base.join("nonexistent"), &dangling).unwrap();
assert!(is_allowed_file_location(job_dir_str, "dangling/payload").is_err());
let _ = std::fs::remove_dir_all(&base);
}
}
+4 -9
View File
@@ -8,7 +8,6 @@ use strum::AsRefStr;
use crate::{
error::{self, to_anyhow, Error, Result},
get_database_url,
secret_backend::{get_secret_value, is_external_stored_value},
utils::get_custom_pg_instance_password,
variables::{build_crypt, decrypt},
PgDatabase, DB,
@@ -726,14 +725,10 @@ async fn transform_json_unchecked(
.await
.map_err(to_anyhow)?;
let value = if is_secret {
if is_external_stored_value(&value) {
get_secret_value(db, w_id, &s[5..], &value).await?
} else {
let mc = build_crypt(&db, &w_id).await?;
decrypt(&mc, value).map_err(|e| {
Error::internal_err(format!("Error decrypting variable {}: {}", &s, e))
})?
}
let mc = build_crypt(&db, &w_id).await?;
decrypt(&mc, value).map_err(|e| {
Error::internal_err(format!("Error decrypting variable {}: {}", &s, e))
})?
} else {
value
};
@@ -6,27 +6,269 @@
* LICENSE-AGPL for a copy of the license.
*/
//! Secret backend extension for the store layer
//! Secret backend extension for the API layer
//!
//! Write-side helpers for integrating the SecretBackend trait with variable
//! operations. Backend resolution and read helpers live in
//! `windmill_common::secret_backend` (so lower-level crates can resolve secrets
//! too) and are re-exported here for existing callers.
//! This module provides helper functions for integrating the SecretBackend
//! trait with variable operations in the API.
//!
//! Note: HashiCorp Vault integration requires Enterprise Edition.
//! The OSS version only supports the database backend.
use std::sync::Arc;
use windmill_common::{
db::DB,
error::{Error, Result},
variables::{build_crypt, encrypt},
secret_backend::{database::DatabaseBackend, SecretBackend},
variables::{build_crypt, decrypt, encrypt},
};
pub use windmill_common::secret_backend::{
get_secret_backend, get_secret_value, is_aws_sm_stored_value, is_azure_kv_stored_value,
is_external_stored_value, is_vault_backend_configured, is_vault_stored_value,
#[cfg(all(feature = "private", feature = "enterprise"))]
use windmill_common::{
global_settings::{load_value_from_global_settings, SECRET_BACKEND_SETTING},
secret_backend::{
AwsSecretsManagerBackend, AwsSecretsManagerSettings, AzureKeyVaultBackend,
AzureKeyVaultSettings, SecretBackendConfig, VaultBackend, VaultSettings,
},
};
#[cfg(all(feature = "private", feature = "enterprise"))]
use tokio::sync::RwLock;
// Cached Vault backend to avoid recreating it for every request
// This enables connection pooling and avoids repeated setup overhead
#[cfg(all(feature = "private", feature = "enterprise"))]
struct CachedVaultBackend {
backend: Arc<dyn SecretBackend>,
settings: VaultSettings,
}
#[cfg(all(feature = "private", feature = "enterprise"))]
lazy_static::lazy_static! {
static ref VAULT_BACKEND_CACHE: RwLock<Option<CachedVaultBackend>> = RwLock::new(None);
}
#[cfg(all(feature = "private", feature = "enterprise"))]
struct CachedAzureKvBackend {
backend: Arc<dyn SecretBackend>,
settings: AzureKeyVaultSettings,
}
#[cfg(all(feature = "private", feature = "enterprise"))]
lazy_static::lazy_static! {
static ref AZURE_KV_BACKEND_CACHE: RwLock<Option<CachedAzureKvBackend>> = RwLock::new(None);
}
// Cached AWS Secrets Manager backend
#[cfg(all(feature = "private", feature = "enterprise"))]
struct CachedAwsSmBackend {
backend: Arc<dyn SecretBackend>,
settings: AwsSecretsManagerSettings,
}
#[cfg(all(feature = "private", feature = "enterprise"))]
lazy_static::lazy_static! {
static ref AWS_SM_BACKEND_CACHE: RwLock<Option<CachedAwsSmBackend>> = RwLock::new(None);
}
/// Get the current secret backend based on global settings
///
/// OSS: Always returns DatabaseBackend
/// EE: Returns configured backend (Database or Vault)
#[cfg(not(all(feature = "private", feature = "enterprise")))]
pub async fn get_secret_backend(db: &DB) -> Result<Arc<dyn SecretBackend>> {
Ok(Arc::new(DatabaseBackend::new(db.clone())))
}
#[cfg(all(feature = "private", feature = "enterprise"))]
pub async fn get_secret_backend(db: &DB) -> Result<Arc<dyn SecretBackend>> {
let config = match load_value_from_global_settings(db, SECRET_BACKEND_SETTING).await? {
Some(value) => serde_json::from_value::<SecretBackendConfig>(value).unwrap_or_default(),
None => SecretBackendConfig::default(),
};
match config {
SecretBackendConfig::Database => Ok(Arc::new(DatabaseBackend::new(db.clone()))),
SecretBackendConfig::HashiCorpVault(settings) => {
get_or_create_vault_backend(db, settings).await
}
SecretBackendConfig::AzureKeyVault(settings) => {
get_or_create_azure_kv_backend(db, settings).await
}
SecretBackendConfig::AwsSecretsManager(settings) => {
get_or_create_aws_sm_backend(db, settings).await
}
}
}
/// Get a cached Vault backend or create a new one if settings changed
#[cfg(all(feature = "private", feature = "enterprise"))]
async fn get_or_create_vault_backend(
_db: &DB,
settings: VaultSettings,
) -> Result<Arc<dyn SecretBackend>> {
// Check if we have a cached backend with matching settings (read lock)
{
let cache = VAULT_BACKEND_CACHE.read().await;
if let Some(ref cached) = *cache {
if cached.settings == settings {
return Ok(cached.backend.clone());
}
}
}
// Need to create a new backend - acquire write lock
let mut cache = VAULT_BACKEND_CACHE.write().await;
// Double-check (another task may have created it while we waited)
if let Some(ref cached) = *cache {
if cached.settings == settings {
return Ok(cached.backend.clone());
}
}
// Create new backend
let backend: Arc<dyn SecretBackend> = {
#[cfg(feature = "openidconnect")]
if settings.token.is_none() {
Arc::new(VaultBackend::new_with_db(settings.clone(), _db.clone()))
} else {
Arc::new(VaultBackend::new(settings.clone()))
}
#[cfg(not(feature = "openidconnect"))]
Arc::new(VaultBackend::new(settings.clone()))
};
// Cache it
*cache = Some(CachedVaultBackend { backend: backend.clone(), settings });
Ok(backend)
}
/// Get a cached Azure Key Vault backend or create a new one if settings changed
#[cfg(all(feature = "private", feature = "enterprise"))]
async fn get_or_create_azure_kv_backend(
_db: &DB,
settings: AzureKeyVaultSettings,
) -> Result<Arc<dyn SecretBackend>> {
// Check if we have a cached backend with matching settings (read lock)
{
let cache = AZURE_KV_BACKEND_CACHE.read().await;
if let Some(ref cached) = *cache {
if cached.settings == settings {
return Ok(cached.backend.clone());
}
}
}
// Need to create a new backend - acquire write lock
let mut cache = AZURE_KV_BACKEND_CACHE.write().await;
// Double-check (another task may have created it while we waited)
if let Some(ref cached) = *cache {
if cached.settings == settings {
return Ok(cached.backend.clone());
}
}
// Create new backend
let backend: Arc<dyn SecretBackend> = Arc::new(AzureKeyVaultBackend::new(settings.clone()));
// Cache it
*cache = Some(CachedAzureKvBackend { backend: backend.clone(), settings });
Ok(backend)
}
/// Get a cached AWS SM backend or create a new one if settings changed
#[cfg(all(feature = "private", feature = "enterprise"))]
async fn get_or_create_aws_sm_backend(
_db: &DB,
settings: AwsSecretsManagerSettings,
) -> Result<Arc<dyn SecretBackend>> {
{
let cache = AWS_SM_BACKEND_CACHE.read().await;
if let Some(ref cached) = *cache {
if cached.settings == settings {
return Ok(cached.backend.clone());
}
}
}
let mut cache = AWS_SM_BACKEND_CACHE.write().await;
if let Some(ref cached) = *cache {
if cached.settings == settings {
return Ok(cached.backend.clone());
}
}
let backend: Arc<dyn SecretBackend> =
Arc::new(AwsSecretsManagerBackend::new_with_client(settings.clone()).await?);
*cache = Some(CachedAwsSmBackend { backend: backend.clone(), settings });
Ok(backend)
}
/// Check if a Vault backend is currently configured
///
/// OSS: Always returns false
/// EE: Checks global settings
#[cfg(not(all(feature = "private", feature = "enterprise")))]
pub async fn is_vault_backend_configured(_db: &DB) -> Result<bool> {
Ok(false)
}
#[cfg(all(feature = "private", feature = "enterprise"))]
pub async fn is_vault_backend_configured(db: &DB) -> Result<bool> {
let config = match load_value_from_global_settings(db, SECRET_BACKEND_SETTING).await? {
Some(value) => serde_json::from_value::<SecretBackendConfig>(value).unwrap_or_default(),
None => SecretBackendConfig::default(),
};
Ok(matches!(
config,
SecretBackendConfig::HashiCorpVault(_)
| SecretBackendConfig::AzureKeyVault(_)
| SecretBackendConfig::AwsSecretsManager(_)
))
}
/// Get a secret value using the configured backend
///
/// For database backend: decrypts using workspace key
/// For vault backend (EE only): fetches from Vault directly
pub async fn get_secret_value(
db: &DB,
workspace_id: &str,
path: &str,
encrypted_value: &str,
) -> Result<String> {
let backend = get_secret_backend(db).await?;
match backend.backend_name() {
"database" => {
// Use existing database decryption
let mc = build_crypt(db, workspace_id).await?;
decrypt(&mc, encrypted_value.to_string()).map_err(|e| {
Error::internal_err(format!("Error decrypting variable {}: {}", path, e))
})
}
"hashicorp_vault" => {
// Fetch from Vault directly
backend.get_secret(workspace_id, path).await
}
"azure_key_vault" => backend.get_secret(workspace_id, path).await,
"aws_secrets_manager" => backend.get_secret(workspace_id, path).await,
_ => Err(Error::internal_err(format!(
"Unknown backend: {}",
backend.backend_name()
))),
}
}
/// Store a secret value using the configured backend
///
/// For database backend: encrypts using workspace key and returns encrypted value
@@ -170,6 +412,26 @@ pub async fn delete_secret_from_backend(db: &DB, workspace_id: &str, path: &str)
}
}
/// Check if a value is stored in Vault (indicated by the $vault: prefix)
pub fn is_vault_stored_value(value: &str) -> bool {
value.starts_with("$vault:")
}
/// Check if a value is stored in Azure Key Vault (indicated by the $azure_kv: prefix)
pub fn is_azure_kv_stored_value(value: &str) -> bool {
value.starts_with("$azure_kv:")
}
/// Check if a value is stored in AWS Secrets Manager (indicated by the $aws_sm: prefix)
pub fn is_aws_sm_stored_value(value: &str) -> bool {
value.starts_with("$aws_sm:")
}
/// Check if a value is stored in any external secret backend
pub fn is_external_stored_value(value: &str) -> bool {
is_vault_stored_value(value) || is_azure_kv_stored_value(value) || is_aws_sm_stored_value(value)
}
/// Rename a secret in Vault when a variable path changes (EE only)
#[cfg(not(all(feature = "private", feature = "enterprise")))]
pub async fn rename_vault_secret(
+31 -349
View File
@@ -39,47 +39,6 @@ struct MaterializeExec {
asset_kind: windmill_common::assets::AssetKind,
asset_path: String,
partition: String,
// Number of `// data_test` checks the codegen embedded. Enforcement recovers
// the per-test outcomes from the summary row; if it recovers fewer than this
// (e.g. an FFI serialization change drops the column), we fail loud rather
// than silently pass declared-but-unverified tests.
n_data_tests: usize,
}
// Fetch and validate a custom data-test script's body. v1 custom tests are
// DuckDB scripts holding a single SELECT/CTE that returns the violating rows
// (dbt's singular-test convention); the worker embeds that query as a subquery
// check in the materialize connection (the single-statement constraint is
// enforced in sql_materialize.rs). Server workers only — agent (Http) workers
// have no script cache to read deployed content from.
async fn fetch_custom_test_body(conn: &Connection, w_id: &str, path: &str) -> Result<String> {
let Connection::Sql(db) = conn else {
return Err(Error::ExecutionErr(format!(
"data_test custom `{path}`: custom tests require a server worker (not supported on \
agent workers in v1)"
)));
};
let hash = windmill_common::get_latest_script_hash(db, path, w_id)
.await?
.ok_or_else(|| {
Error::ExecutionErr(format!(
"data_test custom `{path}`: no deployed script found at this path"
))
})?;
let content =
crate::get_script_content_by_hash(&windmill_common::scripts::ScriptHash(hash), w_id, conn)
.await?;
if !matches!(
content.language,
Some(windmill_common::scripts::ScriptLang::DuckDb)
) {
return Err(Error::ExecutionErr(format!(
"data_test custom `{path}`: must be a DuckDB script returning the violating rows \
(got language {:?})",
content.language
)));
}
Ok(content.content)
}
// If `query` declares `// materialize <ducklake>`, return what to record plus,
@@ -87,47 +46,22 @@ async fn fetch_custom_test_body(conn: &Connection, w_id: &str, path: &str) -> Re
// mode the script writes its own DDL, so the rewrite is `None`). The rewritten
// SQL contains a synthetic `ATTACH 'ducklake://<name>' AS _wm_target` that the
// normal ATTACH-transform pass resolves to real credentials — the same path as
// the user's own ATTACH. `// data_test` lines append verifier probes that run
// against the freshly-materialized target and raise (failing the run) on
// violation. Returns `None` when there is no materialize annotation or the
// target isn't a ducklake (only ducklake is materialized in v1).
// the user's own ATTACH. Returns `None` when there is no materialize annotation
// or the target isn't a ducklake (only ducklake is materialized in v1).
fn build_materialized_query(
query: &str,
partition_value: Option<&str>,
// Custom (`// data_test <path>`) test bodies, pre-fetched by the caller
// (`fetch_custom_test_bodies`) so this stays pure/sync and unit-testable —
// the DB read is the only thing that needs a connection. Keyed by script path.
custom_test_bodies: &std::collections::HashMap<String, String>,
) -> Result<Option<(Option<String>, MaterializeExec)>> {
use windmill_parser::asset_parser::{
parse_pipeline_annotations, AssetKind as PAssetKind, DataTest,
};
use windmill_parser::asset_parser::{parse_pipeline_annotations, AssetKind as PAssetKind};
use windmill_parser::sql_materialize::{
build_wrap_blocks, DataTestResolved, MaterializeStrategy, TARGET_ALIAS,
build_wrap_blocks, classify_wrap, MaterializeStrategy, TARGET_ALIAS,
};
let ann = parse_pipeline_annotations(query);
let has_tests = !ann.data_tests.is_empty();
let Some(m) = ann.materialize else {
// Data tests run *against the materialized asset*; without a
// `// materialize` target there is nothing to test. Fail loudly rather
// than silently skip the declared checks.
if has_tests {
return Err(Error::ExecutionErr(
"data_test: requires a `// materialize` target — data tests run against the \
materialized asset"
.to_string(),
));
}
return Ok(None);
};
if m.target_kind != PAssetKind::Ducklake {
if has_tests {
return Err(Error::ExecutionErr(
"data_test: only `ducklake://` materialization targets support data tests in v1"
.to_string(),
));
}
return Ok(None);
}
let partitioned = ann.partition.is_some();
@@ -152,36 +86,10 @@ fn build_materialized_query(
asset_kind: windmill_common::assets::AssetKind::Ducklake,
asset_path: m.target_path.clone(),
partition: partition.clone(),
n_data_tests: ann.data_tests.len(),
};
// `{partition}` → escaped SQL literal substitution, applied to the managed
// SELECT, its setup, and any custom-test body so a partitioned test can
// filter by the active slice. Always a complete `'…'` literal (with `'`
// doubled) whether or not the author quoted it, so a run caller can't break
// out and alter statement boundaries. The pre-quoted `'{partition}'` form is
// matched first so it doesn't become `''…''`. No-op when unpartitioned.
let lit = format!("'{}'", partition.replace('\'', "''"));
let substitute = |s: &str| -> String {
if !partitioned {
return s.to_string();
}
let tok = windmill_common::assets::PARTITION_TOKEN;
let quoted_tok = format!("'{tok}'");
s.replace(&quoted_tok, &lit).replace(tok, &lit)
};
if m.manual {
// Escape hatch: the script owns its DDL. We can't reliably attach the
// managed target or know the partition column it wrote, so data tests
// are not generated for manual mode in v1.
if has_tests {
return Err(Error::ExecutionErr(
"data_test: not supported with `// materialize manual` in v1 — use managed \
`// materialize`"
.to_string(),
));
}
// Escape hatch: the script owns its DDL; we only record state.
return Ok(Some((None, meta)));
}
if table.is_empty() {
@@ -190,10 +98,24 @@ fn build_materialized_query(
m.target_path
)));
}
let mut plan = classify_wrap_or_err(query)?;
plan.output = substitute(&plan.output);
for s in plan.setup.iter_mut() {
*s = substitute(s);
let mut plan = classify_wrap(query).map_err(|e| Error::ExecutionErr(e.message()))?;
// Resolve the `{partition}` token (same token `// on` asset URIs use) to the
// current partition value everywhere in the managed script, so a partitioned
// materialize can filter its source by the active slice, e.g.
// `WHERE day = {partition}`. The token is always replaced by a *complete*
// escaped SQL literal (`'…'` with `'` doubled) whether or not the author
// quoted it — so a run caller can't pass metacharacters that break out of
// the literal and alter statement boundaries. The pre-quoted form
// `'{partition}'` is matched first so it doesn't become `''…''`. Only
// meaningful when partitioned.
if partitioned {
let lit = format!("'{}'", partition.replace('\'', "''"));
let tok = windmill_common::assets::PARTITION_TOKEN;
let quoted_tok = format!("'{tok}'");
plan.output = plan.output.replace(&quoted_tok, &lit).replace(tok, &lit);
for s in plan.setup.iter_mut() {
*s = s.replace(&quoted_tok, &lit).replace(tok, &lit);
}
}
let strategy = if m.append {
MaterializeStrategy::Append
@@ -204,30 +126,8 @@ fn build_materialized_query(
};
// Inline the partition as an escaped SQL literal (DuckLake has no bind for
// the partition column in our generated DDL).
let pval = lit.clone();
let pval = format!("'{}'", partition.replace('\'', "''"));
let synthetic_attach = format!("ATTACH 'ducklake://{ducklake_name}' AS {TARGET_ALIAS};");
// Resolve data tests (fetch + partition-substitute custom bodies) so codegen
// can embed every check's violating-row count in the materialize summary.
// The summary then carries the full per-test breakdown back to the worker,
// which runs them all and decides pass/fail (no abort-on-first). Empty when
// there are no `// data_test` lines.
let mut resolved = Vec::with_capacity(ann.data_tests.len());
for test in &ann.data_tests {
match test {
DataTest::Custom { path } => {
let raw = custom_test_bodies.get(path).ok_or_else(|| {
Error::ExecutionErr(format!(
"data_test custom `{path}`: body not fetched before codegen (internal)"
))
})?;
resolved
.push(DataTestResolved::Custom { path: path.clone(), body: substitute(raw) });
}
other => resolved.push(DataTestResolved::BuiltIn(other.clone())),
}
}
let blocks = build_wrap_blocks(
&plan,
&synthetic_attach,
@@ -237,44 +137,10 @@ fn build_materialized_query(
&pval,
partitioned,
strategy,
&resolved,
)
.map_err(Error::ExecutionErr)?;
);
Ok(Some((Some(blocks.join("\n")), meta)))
}
// Fetch the deployed body of every `// data_test <path>` custom test declared in
// `query`, keyed by path, so the sync `build_materialized_query` can splice them
// in. The DB read is the only part of materialize codegen that needs a
// connection; isolating it here keeps the codegen pure and unit-testable.
// Server workers only (`fetch_custom_test_body` errors on agent workers). Empty
// when there are no custom tests.
async fn fetch_custom_test_bodies(
query: &str,
conn: &Connection,
w_id: &str,
) -> Result<std::collections::HashMap<String, String>> {
use windmill_parser::asset_parser::{parse_pipeline_annotations, DataTest};
let ann = parse_pipeline_annotations(query);
let mut bodies = std::collections::HashMap::new();
for test in &ann.data_tests {
if let DataTest::Custom { path } = test {
if !bodies.contains_key(path) {
let body = fetch_custom_test_body(conn, w_id, path).await?;
bodies.insert(path.clone(), body);
}
}
}
Ok(bodies)
}
// classify_wrap with the spec's actionable message turned into an executor error.
fn classify_wrap_or_err(query: &str) -> Result<windmill_parser::sql_materialize::WrapPlan> {
windmill_parser::sql_materialize::classify_wrap(query)
.map_err(|e| Error::ExecutionErr(e.message()))
}
// Pull a named i64 field (`snapshot_id` / `rows`) out of the trailing summary
// read — which in wrap mode is the job result. Shape-tolerant (object / array /
// nested), returns None if absent (literal mode, or capture failed).
@@ -290,75 +156,6 @@ fn extract_i64(result: &RawValue, field: &str) -> Option<i64> {
find(&serde_json::from_str::<Value>(result.get()).ok()?, field)
}
// One data test's outcome as carried by the materialize summary's `data_tests`
// column: its display name and how many rows violated it (0 = pass).
struct DataTestOutcome {
name: String,
violating: i64,
}
// Pull the per-test breakdown out of the materialize summary result. The
// `data_tests` column is a DuckLake list-of-struct `[{test, violating}, …]`;
// the FFI may surface it as a nested JSON array or as a JSON string, so accept
// both. Returns empty when there are no tests (the column is absent).
fn extract_data_tests(result: &RawValue) -> Vec<DataTestOutcome> {
fn collect(v: &Value, out: &mut Vec<DataTestOutcome>) {
if let Value::Array(arr) = v {
for item in arr {
if let Value::Object(o) = item {
if let Some(Value::String(name)) = o.get("test") {
let violating = o
.get("violating")
.and_then(|x| x.as_i64().or_else(|| x.as_f64().map(|f| f as i64)))
.unwrap_or(0);
out.push(DataTestOutcome { name: name.clone(), violating });
}
}
}
}
}
fn find_field(v: &Value) -> Option<&Value> {
match v {
Value::Object(o) => o.get("data_tests"),
Value::Array(a) => a.iter().find_map(find_field),
_ => None,
}
}
let mut out = Vec::new();
let Ok(root) = serde_json::from_str::<Value>(result.get()) else {
return out;
};
match find_field(&root) {
Some(arr @ Value::Array(_)) => collect(arr, &mut out),
// FFI serialized the list-of-struct as a JSON string — parse it.
Some(Value::String(s)) => {
if let Ok(parsed) = serde_json::from_str::<Value>(s) {
collect(&parsed, &mut out);
}
}
_ => {}
}
out
}
// Render the full pass/fail breakdown for a failed data-test run — every test,
// not just the first failure, so the user sees the whole picture in one place.
fn format_data_test_breakdown(asset_path: &str, tests: &[DataTestOutcome]) -> String {
let failed = tests.iter().filter(|t| t.violating > 0).count();
let mut lines = vec![format!(
"data tests failed on {asset_path} ({failed}/{} failed):",
tests.len()
)];
for t in tests {
if t.violating > 0 {
lines.push(format!("{}{} violating row(s)", t.name, t.violating));
} else {
lines.push(format!("{}", t.name));
}
}
lines.join("\n")
}
// Best-effort record of a materialization outcome. On a Sql connection it writes
// the row directly; on an agent worker (Http, no direct DB) it posts to the API
// so state lands the same way. Never fails the job — a lost row degrades the
@@ -453,12 +250,8 @@ pub async fn do_duckdb(
.and_then(|a| a.0.get(windmill_common::partition::PARTITION_ARG))
.and_then(|rv| serde_json::from_str::<String>(rv.get()).ok())
.filter(|s| !s.is_empty());
let materialize = if query.contains("materialize") || query.contains("data_test") {
// Custom-test bodies need a DB read; fetch them first so the codegen
// itself stays pure/sync.
let custom_test_bodies =
fetch_custom_test_bodies(query, conn, &job.workspace_id).await?;
build_materialized_query(query, partition_value.as_deref(), &custom_test_bodies)?
let materialize = if query.contains("materialize") {
build_materialized_query(query, partition_value.as_deref())?
} else {
None
};
@@ -478,19 +271,6 @@ pub async fn do_duckdb(
}
_ => query,
};
// Managed materialize generates its own trailing summary row (asset /
// rows / snapshot_id / data_tests), and data-test enforcement below reads
// the `data_tests` column off that row. The row shape is ours, not the
// user's — so force the full-last-row strategy regardless of any
// `// result_collection` annotation, which would otherwise reshape it
// (e.g. a scalar mode drops every column but the first) and silently
// bypass test enforcement.
let collection_strategy = if matches!(&materialize, Some((Some(_), _))) {
SqlResultCollectionStrategy::LastStatementAllRows
} else {
collection_strategy
};
let mut job_args = build_args_values(job, client, conn).await?;
let reserved_variables =
@@ -647,55 +427,9 @@ pub async fn do_duckdb(
if let Some((_, meta)) = &materialize {
// In wrap mode the job result is the summary read (snapshot_id +
// rows + the per-test breakdown); in literal mode there is none.
// rows); in literal mode there is none, so both stay None.
let snapshot_id = extract_i64(&result, "snapshot_id");
let row_count = extract_i64(&result, "rows");
// Data tests all ran (every check counted in one query); decide
// pass/fail here. Any violation fails the run — the write is already
// committed (like dbt), so the slice is recorded `Failed` and the
// cascade stops. The error lists *every* test so the user sees the
// whole picture, not just the first failure.
let tests = extract_data_tests(&result);
// Defense-in-depth: codegen embedded `n_data_tests` checks, so the
// summary row must carry that many outcomes. Recovering fewer means
// the `data_tests` column was dropped/reshaped before we read it —
// fail loud rather than silently pass unverified tests.
if tests.len() < meta.n_data_tests {
let msg = format!(
"data tests on {}: expected {} test outcome(s) but recovered {} from the \
result aborting to avoid a silent pass",
meta.asset_path,
meta.n_data_tests,
tests.len()
);
record_mat(
conn,
&job.workspace_id,
job.id,
meta,
windmill_common::materialization::MaterializationStatus::Failed,
snapshot_id,
row_count,
Some(&msg),
)
.await;
return Err(Error::ExecutionErr(msg));
}
if tests.iter().any(|t| t.violating > 0) {
let breakdown = format_data_test_breakdown(&meta.asset_path, &tests);
record_mat(
conn,
&job.workspace_id,
job.id,
meta,
windmill_common::materialization::MaterializationStatus::Failed,
snapshot_id,
row_count,
Some(&breakdown),
)
.await;
return Err(Error::ExecutionErr(breakdown));
}
record_mat(
conn,
&job.workspace_id,
@@ -1399,11 +1133,9 @@ mod tests {
assert_eq!(file_arg.otyp.as_deref(), Some("s3object"));
// The wrapped query still references `$file`, so the parsed sig binds it.
// No custom data tests here, so no fetched bodies are needed.
let (rewritten, _) =
build_materialized_query(script, None, &std::collections::HashMap::new())
.expect("materialize builds")
.expect("materialize present");
let (rewritten, _) = build_materialized_query(script, None)
.expect("materialize builds")
.expect("materialize present");
let rewritten = rewritten.expect("managed mode rewrites the query");
assert!(
rewritten.contains("$file"),
@@ -1772,54 +1504,4 @@ mod tests {
let serialized = serde_json::to_string(&arg).unwrap();
assert!(serialized.contains("\"json_value\":{\"key\":\"value\"}"));
}
fn raw(s: &str) -> Box<RawValue> {
serde_json::from_str(s).unwrap()
}
#[test]
fn extract_data_tests_parses_nested_array() {
// The real result shape: an array of one summary row carrying a nested
// `data_tests` array (how the FFI serialises the list-of-struct).
let r = raw(
r#"[{"rows":3,"snapshot_id":17,"materialized":"ducklake://a/b",
"data_tests":[{"test":"unique(order_id)","violating":0},
{"test":"accepted_values(status)","violating":2}]}]"#,
);
let out = extract_data_tests(&r);
assert_eq!(out.len(), 2);
assert_eq!(out[0].name, "unique(order_id)");
assert_eq!(out[0].violating, 0);
assert_eq!(out[1].name, "accepted_values(status)");
assert_eq!(out[1].violating, 2);
}
#[test]
fn extract_data_tests_handles_string_encoded_and_absent() {
// Fallback: some serialisations surface the list-of-struct as a JSON string.
let s = raw(r#"{"data_tests":"[{\"test\":\"not_null(x)\",\"violating\":1}]"}"#);
let out = extract_data_tests(&s);
assert_eq!(out.len(), 1);
assert_eq!(out[0].name, "not_null(x)");
assert_eq!(out[0].violating, 1);
// Absent column (no tests) -> empty, no panic.
assert!(extract_data_tests(&raw(r#"[{"rows":3}]"#)).is_empty());
}
#[test]
fn format_data_test_breakdown_lists_all_with_marks() {
let tests = vec![
DataTestOutcome { name: "unique(order_id)".into(), violating: 1 },
DataTestOutcome { name: "not_null(user_id)".into(), violating: 0 },
DataTestOutcome { name: "accepted_values(status)".into(), violating: 2 },
];
let msg = format_data_test_breakdown("analytics/orders", &tests);
assert_eq!(
msg,
"data tests failed on analytics/orders (2/3 failed):\n \
unique(order_id) 1 violating row(s)\n \
not_null(user_id)\n \
accepted_values(status) 2 violating row(s)"
);
}
}
+6 -114
View File
@@ -37,8 +37,8 @@ use windmill_common::{
scripts::ScriptLang,
utils::calculate_hash,
worker::{
copy_dir_recursively, is_allowed_file_location, pad_string, split_python_requirements,
write_file, Connection, PyVAlias, PythonAnnotations, WORKER_CONFIG,
copy_dir_recursively, pad_string, split_python_requirements, write_file, Connection,
PyVAlias, PythonAnnotations, WORKER_CONFIG,
},
};
@@ -664,16 +664,10 @@ pub fn compute_python_module_dir(script_path: &str) -> String {
.replace("-", "_")
.replace("@", ".");
if dirs_full.len() > 0 {
let dirs = dirs_full.strip_prefix("/").unwrap_or(&dirs_full);
// This directory is appended to job_dir and written to. Neutralize any
// `.`/`..` segment so the result stays a relative path inside job_dir: a
// Preview path is request-supplied and skips the DB `proper_id` CHECK that
// deployed runnables get, and the `@`->`.` rewrite above can also turn a
// segment like `@.` into `..`.
dirs.split('/')
.map(|seg| if seg == "." || seg == ".." { "_" } else { seg })
.collect::<Vec<_>>()
.join("/")
dirs_full
.strip_prefix("/")
.unwrap_or(&dirs_full)
.to_string()
} else {
"tmp".to_string()
}
@@ -1674,10 +1668,6 @@ async fn prepare_wrapper(
last
};
let module_dir = format!("{}/{}", job_dir, dirs);
// Defense-in-depth: `dirs`/`last` derive from the (request-supplied for
// previews) script path. compute_python_module_dir already neutralizes `..`,
// but assert containment here too so the write can never escape job_dir.
is_allowed_file_location(job_dir, &format!("{dirs}/{last}.py"))?;
tokio::fs::create_dir_all(format!("{module_dir}/")).await?;
let _ = write_file(&module_dir, &format!("{last}.py"), inner_content)?;
@@ -2367,35 +2357,6 @@ async fn verify_wheel_record(venv_p: &str) -> Result<(), String> {
}
}
lazy_static::lazy_static! {
/// Per-target-directory install locks. The wheel cache
/// (`ROOT_CACHE_DIR/python_<v>/<pkg>==<ver>`) is shared by every job on a
/// worker, and `uv pip install --reinstall --target <dir>` transiently
/// empties that directory while it runs. Two jobs installing the same
/// package concurrently would clobber each other's files — and a job that
/// imports from the dir mid-reinstall hits a flaky `ModuleNotFoundError`
/// (e.g. wmill's `httpx` vanishing during a WAC run). We serialize installs
/// per target dir and re-check the `.valid.windmill` marker once the lock is
/// held, so a waiter reuses the freshly populated cache instead of
/// reinstalling over it.
static ref PY_INSTALL_LOCKS: std::sync::Mutex<HashMap<String, Arc<tokio::sync::Mutex<()>>>> =
std::sync::Mutex::new(HashMap::new());
}
/// Returns the install lock guarding `venv_p`, creating it on first use.
fn py_install_lock_for(venv_p: &str) -> Arc<tokio::sync::Mutex<()>> {
let mut locks = PY_INSTALL_LOCKS.lock().unwrap();
locks
.entry(venv_p.to_string())
.or_insert_with(|| Arc::new(tokio::sync::Mutex::new(())))
.clone()
}
/// A venv dir is usable iff its install wrote the `.valid.windmill` marker.
async fn is_venv_valid(venv_p: &str) -> bool {
metadata(format!("{venv_p}/.valid.windmill")).await.is_ok()
}
/// uv pip install, include cached or pull from S3
pub async fn handle_python_reqs(
requirements: Vec<String>,
@@ -2755,34 +2716,6 @@ pub async fn handle_python_reqs(
);
let start = std::time::Instant::now();
// Serialize population of this shared cache dir against other jobs.
// `--reinstall` (and the S3 tar extraction below) rewrite `venv_p` in
// place, so a concurrent installer must not run while another job is
// still populating it.
let install_lock = py_install_lock_for(&venv_p);
let _install_guard = install_lock.lock().await;
// Another job may have finished the install while we waited on the
// lock — reuse its result instead of reinstalling over a dir other
// jobs may now be importing from.
if is_venv_valid(&venv_p).await {
print_success(
false,
false,
&job_id,
&w_id,
&req,
req_tl,
counter_arc,
total_to_install,
start,
&conn,
)
.await;
pids.lock().await.get_mut(i).and_then(|e| e.take());
return Ok(());
}
#[cfg(all(feature = "enterprise", feature = "parquet"))]
if is_not_pro {
if let Some(os) = windmill_object_store::get_object_store().await {
@@ -3369,36 +3302,6 @@ pub async fn start_worker(
mod tests {
use super::*;
#[tokio::test]
async fn py_install_lock_serializes_same_target_only() {
// The shared wheel cache race fix relies on: same target dir -> same lock
// (serialized installs), different target dir -> independent locks. If this
// keying ever regresses, concurrent `--reinstall` could clobber a dir a job
// is importing from, reintroducing the flaky ModuleNotFoundError.
let a1 = py_install_lock_for("/cache/python_3_12/wmill==1.0.0");
let a2 = py_install_lock_for("/cache/python_3_12/wmill==1.0.0");
let b = py_install_lock_for("/cache/python_3_12/httpx==0.27.0");
assert!(
Arc::ptr_eq(&a1, &a2),
"same target dir must share one install lock"
);
assert!(
!Arc::ptr_eq(&a1, &b),
"different target dirs must use independent install locks"
);
let _held = a1.lock().await;
assert!(
a2.try_lock().is_err(),
"installs into the same target dir must be mutually exclusive"
);
assert!(
b.try_lock().is_ok(),
"installs into different target dirs must not block each other"
);
}
#[test]
fn test_compute_python_module_dir_nested_path() {
assert_eq!(
@@ -3454,17 +3357,6 @@ mod tests {
assert_eq!(compute_python_module_dir("f/in/script"), "f/_in");
}
#[test]
fn test_compute_python_module_dir_neutralizes_traversal() {
// A Preview path skips the DB `proper_id` CHECK, so it can carry `..`.
// `..`/`.` segments must be neutralized so the dir stays inside job_dir.
let dirs = compute_python_module_dir("u/x/../../../../tmp/evil/payload");
assert!(!dirs.split('/').any(|s| s == ".." || s == "."));
assert_eq!(dirs, "u/x/_/_/_/_/tmp/evil");
// The `@`->`.` rewrite must not be able to synthesize a `..` segment.
assert_eq!(compute_python_module_dir("u/@./script"), "u/_");
}
#[test]
fn test_compute_py_codegen_basic_args() {
let code = "def main(x: str, y: int):\n return x\n";
+1 -1
View File
@@ -2,7 +2,7 @@ import { sleep } from "https://deno.land/x/sleep@v1.2.1/mod.ts";
import * as windmill from "https://deno.land/x/windmill@v1.174.0/mod.ts";
import * as api from "https://deno.land/x/windmill@v1.174.0/windmill-api/index.ts";
export const VERSION = "v1.739.0";
export const VERSION = "v1.737.0";
export async function login(email: string, password: string): Promise<string> {
return await windmill.UserService.login({
-241
View File
@@ -1,241 +0,0 @@
// Bounded-cascade graph engine for `wmill pipeline run --to`.
//
// MIRROR of frontend/src/lib/components/assets/AssetGraph/boundedCascade.ts —
// the two have no shared import path (frontend is a separate package), so keep
// them in sync. The grammar is intentionally tiny: there is no dbt-style
// `--select` string. The user names a start (a schedule / manual root) and one
// or more end nodes; the run is the "path between" them:
//
// descendants(start) ∩ (ancestors(ends) ends) {start}
//
// Node ids: assets `${kind}:${path}` (e.g. `datatable:main/raw`); runnables
// `script:${path}`. Operates on the asset-graph payload shape used by
// pipeline.ts.
export type BCGraph = {
runnables: { path: string; usage_kind: "script" | "flow" | "job" }[];
assets: { kind: string; path: string }[];
edges: {
runnable_kind: string;
runnable_path: string;
asset_kind: string;
asset_path: string;
access_type?: "r" | "w" | "rw";
}[];
triggers: (
| {
trigger_kind: "asset";
asset_kind: string;
asset_path: string;
runnable_kind: string;
runnable_path: string;
}
| { trigger_kind: string; runnable_kind: string; runnable_path: string }
)[];
};
export const SCRIPT_PREFIX = "script:";
export const scriptNodeId = (path: string): string => `${SCRIPT_PREFIX}${path}`;
export const isScriptNode = (id: string): boolean => id.startsWith(SCRIPT_PREFIX);
export const scriptPathOf = (id: string): string => id.slice(SCRIPT_PREFIX.length);
const assetNodeId = (kind: string, path: string): string => `${kind}:${path}`;
// Native trigger kinds that fan out per-event — never bounded-run starts.
// `webhook` / `data_upload` have no trigger row in the graph payload, so a root
// whose only entry is one of those reads as a manual root.
const EVENT_TRIGGER_KINDS = new Set([
"kafka",
"mqtt",
"nats",
"postgres",
"sqs",
"gcp",
"email",
]);
/** Resolve an asset URI (`datatable://x`, `s3://b/k`, …) to its node id. */
export function assetUriToNodeId(uri: string): string | undefined {
const m = uri.match(/^([a-z0-9_]+):\/\/(.+)$/i);
if (!m) return undefined;
const prefix = m[1].toLowerCase();
const kind = prefix === "s3" ? "s3object" : prefix;
return `${kind}:${m[2]}`;
}
export type LineageDag = {
down: Map<string, Set<string>>;
up: Map<string, Set<string>>;
nodes: Set<string>;
};
function addEdge(dag: LineageDag, a: string, b: string) {
if (a === b) return;
dag.nodes.add(a);
dag.nodes.add(b);
(dag.down.get(a) ?? dag.down.set(a, new Set()).get(a)!).add(b);
(dag.up.get(b) ?? dag.up.set(b, new Set()).get(b)!).add(a);
}
/** Unified upstream→downstream lineage DAG over scripts assets. */
export function buildLineageDag(g: BCGraph): LineageDag {
const dag: LineageDag = { down: new Map(), up: new Map(), nodes: new Set() };
for (const r of g.runnables ?? []) {
if (r.usage_kind === "script") dag.nodes.add(scriptNodeId(r.path));
}
for (const a of g.assets ?? []) dag.nodes.add(assetNodeId(a.kind, a.path));
for (const e of g.edges ?? []) {
if (e.runnable_kind !== "script") continue;
const aid = assetNodeId(e.asset_kind, e.asset_path);
const access = e.access_type ?? "r";
if (access === "w" || access === "rw") {
addEdge(dag, scriptNodeId(e.runnable_path), aid); // producer
} else if (access === "r") {
addEdge(dag, aid, scriptNodeId(e.runnable_path)); // pure reader
}
}
for (const t of g.triggers ?? []) {
if (t.trigger_kind !== "asset" || t.runnable_kind !== "script") continue;
const at = t as Extract<BCGraph["triggers"][number], { trigger_kind: "asset" }>;
addEdge(dag, assetNodeId(at.asset_kind, at.asset_path), scriptNodeId(at.runnable_path));
}
return dag;
}
function closure(adj: Map<string, Set<string>>, start: string): Set<string> {
const seen = new Set<string>();
const queue = [start];
while (queue.length > 0) {
const cur = queue.shift()!;
for (const n of adj.get(cur) ?? []) {
if (seen.has(n)) continue;
seen.add(n);
queue.push(n);
}
}
// A cycle back to `start` would have re-added it; the contract excludes
// the node itself.
seen.delete(start);
return seen;
}
export const descendants = (dag: LineageDag, n: string): Set<string> => closure(dag.down, n);
export const ancestors = (dag: LineageDag, n: string): Set<string> => closure(dag.up, n);
export type BoundedResult = {
nodes: Set<string>;
reachableEnds: string[];
droppedEnds: string[];
};
/** Path-between node set for `start` and `ends` (inclusive). */
export function boundedSet(dag: LineageDag, start: string, ends: string[]): BoundedResult {
const downSet = new Set(descendants(dag, start));
downSet.add(start);
const reachableEnds = ends.filter((e) => downSet.has(e));
const droppedEnds = ends.filter((e) => !downSet.has(e));
if (reachableEnds.length === 0) {
return { nodes: new Set([start]), reachableEnds, droppedEnds };
}
const upClosure = new Set<string>();
for (const e of reachableEnds) {
upClosure.add(e);
for (const a of ancestors(dag, e)) upClosure.add(a);
}
const nodes = new Set<string>();
for (const n of downSet) if (upClosure.has(n)) nodes.add(n);
nodes.add(start);
return { nodes, reachableEnds, droppedEnds };
}
/** Script node ids eligible to start a bounded run. */
export function validStarts(g: BCGraph): Set<string> {
const subscribers = new Set<string>();
const scheduleScripts = new Set<string>();
const eventScripts = new Set<string>();
for (const t of g.triggers ?? []) {
if (t.runnable_kind !== "script") continue;
if (t.trigger_kind === "asset") subscribers.add(t.runnable_path);
else if (t.trigger_kind === "schedule") scheduleScripts.add(t.runnable_path);
else if (EVENT_TRIGGER_KINDS.has(t.trigger_kind)) eventScripts.add(t.runnable_path);
}
const out = new Set<string>();
for (const r of g.runnables ?? []) {
if (r.usage_kind !== "script") continue;
const p = r.path;
if (scheduleScripts.has(p)) out.add(scriptNodeId(p));
else if (!subscribers.has(p) && !eventScripts.has(p)) out.add(scriptNodeId(p));
}
return out;
}
/** Project a node-id set to the script paths it contains. */
export function scriptsOf(nodes: Iterable<string>): string[] {
const out: string[] = [];
for (const id of nodes) if (isScriptNode(id)) out.push(scriptPathOf(id));
return out;
}
/**
* Resolve a CLI `--to` / `--from` token to a node id, or undefined if it
* matches nothing. Asset URIs (`kind://path`) resolve to the asset node; a bare
* token matches a runnable by exact path or by short (last-segment) name.
*/
export function resolveToken(g: BCGraph, token: string): string | undefined {
if (token.includes("://")) {
const id = assetUriToNodeId(token);
return id && g.assets.some((a) => `${a.kind}:${a.path}` === id) ? id : undefined;
}
const scripts = (g.runnables ?? []).filter((r) => r.usage_kind === "script");
const exact = scripts.find((r) => r.path === token);
if (exact) return scriptNodeId(exact.path);
const byShort = scripts.filter((r) => (r.path.split("/").pop() ?? r.path) === token);
return byShort.length === 1 ? scriptNodeId(byShort[0].path) : undefined;
}
/**
* Topological order of `scripts` over the in-set producersubscriber edges
* (assets collapsed). Scripts on a cycle are returned in `cyclic` and excluded
* from `order`. Serial-run friendly: every script comes after its in-set
* upstreams.
*/
export function topoOrder(
g: BCGraph,
scripts: Set<string>,
): { order: string[]; cyclic: string[] } {
const dag = buildLineageDag(g);
const down = new Map<string, Set<string>>();
const indegree = new Map<string, number>();
for (const s of scripts) indegree.set(s, 0);
// One-hop (through a single asset) script→script edges, restricted to the set.
for (const s of scripts) {
const sid = scriptNodeId(s);
const oneHop = new Set<string>();
for (const asset of dag.down.get(sid) ?? []) {
for (const sub of dag.down.get(asset) ?? []) {
if (isScriptNode(sub)) {
const p = scriptPathOf(sub);
if (p !== s && scripts.has(p)) oneHop.add(p);
}
}
}
if (oneHop.size > 0) {
down.set(s, oneHop);
for (const p of oneHop) indegree.set(p, (indegree.get(p) ?? 0) + 1);
}
}
const ready = [...scripts].filter((s) => (indegree.get(s) ?? 0) === 0);
const remaining = new Map(indegree);
const order: string[] = [];
while (ready.length > 0) {
const n = ready.shift()!;
order.push(n);
for (const p of down.get(n) ?? []) {
const d = (remaining.get(p) ?? 0) - 1;
remaining.set(p, d);
if (d === 0) ready.push(p);
}
}
const orderedSet = new Set(order);
const cyclic = [...scripts].filter((s) => !orderedSet.has(s));
return { order, cyclic };
}
+1 -220
View File
@@ -8,18 +8,6 @@ import { requireLogin } from "../../core/auth.ts";
import { resolveWorkspace } from "../../core/context.ts";
import * as log from "../../core/log.ts";
import { GlobalOptions } from "../../types.ts";
import {
type BCGraph,
boundedSet,
buildLineageDag,
descendants,
resolveToken,
scriptNodeId,
scriptPathOf,
scriptsOf,
topoOrder,
validStarts,
} from "./boundedCascade.ts";
// Mirrors the asset-graph endpoint payload (backend/windmill-api-assets).
// TODO: the checked-in generated client (cli/gen, last regenerated 2025-04)
@@ -281,196 +269,6 @@ async function show(
console.log(lines.join("\n"));
}
// Poll a launched job to a terminal state. Modest fixed cadence; capped so a
// wedged job can't hang the CLI forever.
async function waitJob(workspace: string, id: string): Promise<boolean> {
const MAX_RETRIES = 6000; // ~10min at 100ms
for (let i = 0; i < MAX_RETRIES; i++) {
try {
const r = await wmill.getCompletedJobResultMaybe({
workspace,
id,
getStarted: false,
});
// A completed job without an explicit `success: true` is a failure
// (mirrors the frontend `waitJobTerminal`): the cascade only advances on
// a confirmed success.
if (r.completed) return r.success === true;
} catch {
// transient — retry
}
await new Promise((res) => setTimeout(res, 100));
}
throw new Error(`Timed out waiting for job ${id}`);
}
// Bounded-cascade run: start at a schedule / manual root, fan downstream, but
// stop at the `--to` end node(s). Scripts run in topological order; each is
// launched with `_wmill_skip_asset_dispatch` so the CLI owns the whole closure
// (the backend dispatcher never double-fires the deployed part). With no
// `--to`, runs the full read-aware downstream of `--from` (every descendant in
// the lineage DAG, pure readers included — broader than the canvas cascade,
// which dispatches subscribers only).
async function run(
opts: GlobalOptions & {
from?: string;
to?: string[];
dryRun?: boolean;
json?: boolean;
},
folder: string,
) {
if (opts.json) log.setSilent(true);
const workspace = await resolveWorkspace(opts);
await requireLogin(opts);
const f = folder.replace(/^f\//, "").replace(/\/$/, "");
const graph = await apiGet<BCGraph>(
`/w/${workspace.workspaceId}/assets/graph?folder=${encodeURIComponent(f)}&asset_kinds=${ASSET_KINDS}`,
);
// Resolve the start: explicit --from (must be a valid start) or the folder's
// sole valid start.
const starts = validStarts(graph);
let start: string;
if (opts.from) {
const resolved = resolveToken(graph, opts.from);
if (!resolved) {
// Distinguish "no match" from "ambiguous short name" (resolveToken
// returns undefined for both) so the hint is actionable.
const matches = graph.runnables.filter(
(r) => r.usage_kind === "script" && (r.path.split("/").pop() ?? r.path) === opts.from,
);
if (matches.length > 1) {
throw new Error(
`--from '${opts.from}' matches multiple scripts (${matches.map((r) => r.path).sort().join(", ")}) — pass the full path.`,
);
}
throw new Error(`--from '${opts.from}' matched no script in f/${f}.`);
}
if (!starts.has(resolved)) {
throw new Error(
`--from '${opts.from}' is not a valid bounded-run start. Starts must be schedule-triggered or manual roots; row-backed event triggers (kafka/mqtt/nats/postgres/sqs/gcp/email) fan out per-event and can't be bounded.`,
);
}
start = resolved;
} else if (starts.size === 1) {
start = [...starts][0];
} else if (starts.size === 0) {
throw new Error(
`No schedule or manual root in f/${f} to start a bounded run from.`,
);
} else {
throw new Error(
`f/${f} has ${starts.size} possible starts — pass --from <script>. Candidates: ${scriptsOf(starts).sort().join(", ")}`,
);
}
// Resolve --to end node(s): split on comma, resolve each. An unresolved or
// ambiguous token is a hard error — silently ignoring it would run a
// different subset than the user asked for.
const toTokens = (opts.to ?? []).flatMap((t) => t.split(",")).map((t) => t.trim()).filter(
Boolean,
);
const ends: string[] = [];
const unresolved: string[] = [];
for (const tok of toTokens) {
const id = resolveToken(graph, tok);
if (!id) unresolved.push(tok);
else ends.push(id);
}
if (unresolved.length > 0) {
const details = unresolved.map((tok) => {
const matches = graph.runnables.filter(
(r) => r.usage_kind === "script" && (r.path.split("/").pop() ?? r.path) === tok,
);
return matches.length > 1
? `'${tok}' (ambiguous: ${matches.map((r) => r.path).sort().join(", ")} — use the full path)`
: `'${tok}' (no match in f/${f})`;
});
throw new Error(`--to could not be resolved: ${details.join("; ")}.`);
}
// Readable label for a node id: `script:<path>` → `<path>`; asset ids
// (`<kind>:<path>`) are kept verbatim (slicing `script:` off them corrupts
// the name).
const idLabel = (id: string): string => (id.startsWith("script:") ? scriptPathOf(id) : id);
const dag = buildLineageDag(graph);
let selectedScripts: Set<string>;
let reachableEnds: string[] = [];
let droppedEnds: string[] = [];
if (ends.length === 0) {
// No bound → full read-aware downstream of start (pure readers included).
const all = new Set(descendants(dag, start));
all.add(start);
selectedScripts = new Set(scriptsOf(all));
} else {
const res = boundedSet(dag, start, ends);
reachableEnds = res.reachableEnds;
droppedEnds = res.droppedEnds;
for (const d of droppedEnds) {
log.warn(`end '${idLabel(d)}' is not downstream of the start — ignored.`);
}
selectedScripts = new Set(scriptsOf(res.nodes));
}
const { order, cyclic } = topoOrder(graph, selectedScripts);
if (cyclic.length > 0) {
log.warn(`Skipping ${cyclic.length} script(s) on a dependency cycle: ${cyclic.sort().join(", ")}`);
}
if (opts.json) {
// Surface reachable/dropped ends so a machine-readable plan reflects the
// same trimming the human-facing warning does — a resolved-but-unreachable
// `--to` must not look like a clean plan that silently runs only the start.
console.log(
JSON.stringify({
start: scriptPathOf(start),
ends: ends.map(idLabel),
reachableEnds: reachableEnds.map(idLabel),
droppedEnds: droppedEnds.map(idLabel),
order,
cyclic,
}),
);
}
if (order.length === 0) {
if (!opts.json) log.info("Nothing to run.");
return;
}
if (opts.dryRun) {
if (!opts.json) {
log.info(
colors.bold(`Bounded run plan — ${order.length} script${order.length === 1 ? "" : "s"}`) +
colors.dim(` (from ${shortName(scriptPathOf(start))})`),
);
order.forEach((p, i) => log.info(` ${i + 1}. ${p}`));
}
return;
}
// Execute in topological order, stopping on the first failure.
for (const path of order) {
if (!opts.json) log.info(colors.gray(`▶ running ${path}`));
const id = await wmill.runScriptByPath({
workspace: workspace.workspaceId,
path,
requestBody: { _wmill_skip_asset_dispatch: true },
});
const ok = await waitJob(workspace.workspaceId, id);
if (!ok) {
throw new Error(`Bounded run failed at ${path} (job ${id}).`);
}
if (!opts.json) log.info(colors.green(`${path}`));
}
if (!opts.json) {
log.info(colors.green.bold(`Bounded run complete — ${order.length} script(s) succeeded.`));
}
}
const command = new Command()
.description(
"inspect asset-driven pipelines (scripts marked `// pipeline`, wired by `// on <spec>` annotations)",
@@ -484,23 +282,6 @@ const command = new Command()
)
.arguments("<folder:string>")
.option("--json", "Output the raw asset graph as JSON")
.action(show as any)
.command(
"run",
"run a bounded cascade: from a schedule/manual root, fan downstream up to the --to end node(s)",
)
.arguments("<folder:string>")
.option(
"--from <script:string>",
"Start script (short name or path). Defaults to the folder's sole schedule/manual root.",
)
.option(
"--to <node:string>",
"End node(s) to stop at — script names/paths or asset URIs (e.g. datatable://main/staged). Repeatable or comma-separated. Omit to run the full downstream.",
{ collect: true },
)
.option("--dry-run", "Print the topological run plan without executing.")
.option("--json", "Output the plan as JSON (for piping to jq).")
.action(run as any);
.action(show as any);
export default command;
+1 -1
View File
@@ -10,4 +10,4 @@ export const WM_FORK_PREFIX = "wm-fork";
// (e.g. utils.ts) can read it without importing main.ts and creating a circular
// dependency (main → workspace → utils → main) that triggers a TDZ.
// Re-exported from main.ts for backwards compatibility.
export const VERSION = "1.739.0";
export const VERSION = "1.737.0";
-5
View File
@@ -6897,11 +6897,6 @@ inspect asset-driven pipelines (scripts marked \`// pipeline\`, wired by \`// on
- \`--json\` - Output as JSON (for piping to jq)
- \`pipeline show <folder:string>\` - render a pipeline folder's DAG (sources, lineage, subscriptions) in the terminal
- \`--json\` - Output the raw asset graph as JSON
- \`pipeline run <folder:string>\` - run a bounded cascade: from a schedule/manual root, fan downstream up to the --to end node(s)
- \`--from <script:string>\` - Start script (short name or path). Defaults to the folder's sole schedule/manual root.
- \`--to <node:string>\` - End node(s) to stop at — script names/paths or asset URIs (e.g. datatable://main/staged). Repeatable or comma-separated. Omit to run the full downstream.
- \`--dry-run\` - Print the topological run plan without executing.
- \`--json\` - Output the plan as JSON (for piping to jq).
### protection-rules
@@ -1,160 +0,0 @@
import { expect, test } from "bun:test";
// Mirror of
// frontend/src/lib/components/assets/AssetGraph/boundedCascade.test.ts — keep
// the two engines in sync. (Bun project → bun:test, not Deno.)
import {
type BCGraph,
ancestors,
assetUriToNodeId,
boundedSet,
buildLineageDag,
descendants,
resolveToken,
scriptNodeId,
scriptsOf,
topoOrder,
validStarts,
} from "../src/commands/pipeline/boundedCascade.ts";
type W = [script: string, asset: string];
type S = [script: string, asset: string];
type R = [script: string, asset: string];
function graph(opts: {
scripts?: string[];
writes?: W[];
reads?: R[];
subs?: S[];
native?: Array<[kind: string, script: string]>;
}): BCGraph {
const { scripts = [], writes = [], reads = [], subs = [], native = [] } = opts;
return {
assets: [],
runnables: scripts.map((p) => ({ path: p, usage_kind: "script" as const })),
edges: [
...writes.map(([s, a]) => ({
runnable_kind: "script",
runnable_path: s,
asset_kind: "datatable",
asset_path: a,
access_type: "w" as const,
})),
...reads.map(([s, a]) => ({
runnable_kind: "script",
runnable_path: s,
asset_kind: "datatable",
asset_path: a,
access_type: "r" as const,
})),
],
triggers: [
...subs.map(([s, a]) => ({
trigger_kind: "asset" as const,
asset_kind: "datatable",
asset_path: a,
runnable_kind: "script",
runnable_path: s,
})),
...native.map(([kind, s]) => ({
trigger_kind: kind,
runnable_kind: "script",
runnable_path: s,
})),
],
};
}
const sn = scriptNodeId;
const sorted = (it: Iterable<string>) => [...it].sort();
// a → x → b → y → c → z → d (linear chain through assets)
const chain = () =>
graph({
scripts: ["a", "b", "c", "d"],
writes: [
["a", "x"],
["b", "y"],
["c", "z"],
],
subs: [
["b", "x"],
["c", "y"],
["d", "z"],
],
});
test("boundedSet stops at a single end node", () => {
const res = boundedSet(buildLineageDag(chain()), sn("a"), [sn("c")]);
expect(sorted(scriptsOf(res.nodes))).toEqual(["a", "b", "c"]);
expect(res.nodes.has("datatable:z")).toBe(false);
});
test("boundedSet supports an asset as the end bound", () => {
const res = boundedSet(buildLineageDag(chain()), sn("a"), ["datatable:y"]);
expect(sorted(scriptsOf(res.nodes))).toEqual(["a", "b"]);
});
test("boundedSet drops ends not downstream of start", () => {
const res = boundedSet(buildLineageDag(chain()), sn("c"), [sn("a")]);
expect(res.droppedEnds).toEqual([sn("a")]);
expect([...res.nodes]).toEqual([sn("c")]);
});
test("validStarts: schedule and manual roots, not events or subscribers", () => {
const g = graph({
scripts: ["a", "sub", "sched", "kfk"],
writes: [["a", "x"]],
subs: [["sub", "x"], ["sched", "x"]],
native: [["schedule", "sched"], ["kafka", "kfk"]],
});
const starts = validStarts(g);
expect(starts.has(sn("a"))).toBe(true); // manual root
expect(starts.has(sn("sched"))).toBe(true); // schedule overrides subscriber
expect(starts.has(sn("sub"))).toBe(false); // pure subscriber
expect(starts.has(sn("kfk"))).toBe(false); // event-only
});
test("assetUriToNodeId maps s3 → s3object, others verbatim", () => {
expect(assetUriToNodeId("s3://b/k")).toBe("s3object:b/k");
expect(assetUriToNodeId("datatable://main/users")).toBe("datatable:main/users");
expect(assetUriToNodeId("nope")).toBe(undefined);
});
test("resolveToken: short name, full path, and asset URI", () => {
const g = graph({ scripts: ["f/p/stage"], writes: [["f/p/stage", "main/staged"]] });
g.assets = [{ kind: "datatable", path: "main/staged" }];
expect(resolveToken(g, "stage")).toBe(sn("f/p/stage"));
expect(resolveToken(g, "f/p/stage")).toBe(sn("f/p/stage"));
expect(resolveToken(g, "datatable://main/staged")).toBe("datatable:main/staged");
expect(resolveToken(g, "missing")).toBe(undefined);
});
test("topoOrder sorts the bounded scripts and flags cycles", () => {
const { order, cyclic } = topoOrder(chain(), new Set(["a", "b", "c"]));
expect(order).toEqual(["a", "b", "c"]);
expect(cyclic).toEqual([]);
});
test("descendants/ancestors exclude the start even on a cycle", () => {
// a → x → b → y → a (cycle): closures must not contain a.
const g = graph({
scripts: ["a", "b"],
writes: [["a", "x"], ["b", "y"]],
subs: [["b", "x"], ["a", "y"]],
});
const dag = buildLineageDag(g);
expect(descendants(dag, sn("a")).has(sn("a"))).toBe(false);
expect(ancestors(dag, sn("a")).has(sn("a"))).toBe(false);
});
test("topoOrder orders a pure reader after its producer", () => {
// a writes x; c only *reads* x (no `// on x`). c must run after a.
const g = graph({
scripts: ["a", "c"],
writes: [["a", "x"]],
reads: [["c", "x"]],
});
const { order } = topoOrder(g, new Set(["a", "c"]));
expect(order).toEqual(["a", "c"]);
});
+1 -1
View File
@@ -54,7 +54,7 @@ RUN curl -fsSL https://www.postgresql.org/media/keys/ACCC4CF8.asc | gpg --dearmo
ENV TZ=Etc/UTC
# Install UV
RUN curl --proto '=https' --tlsv1.2 -LsSf https://github.com/astral-sh/uv/releases/download/0.11.24/uv-installer.sh | sh && mv /root/.local/bin/uv /usr/local/bin/uv
RUN curl --proto '=https' --tlsv1.2 -LsSf https://github.com/astral-sh/uv/releases/download/0.9.25/uv-installer.sh | sh && mv /root/.local/bin/uv /usr/local/bin/uv
# Preinstall python runtime to temp location (will copy with world-writable perms later)
# --compile-bytecode precompiles the stdlib to .pyc so jobs don't recompile it on every run
+1 -1
View File
@@ -54,7 +54,7 @@ RUN curl -fsSL https://www.postgresql.org/media/keys/ACCC4CF8.asc | gpg --dearmo
ENV TZ=Etc/UTC
# Install UV
RUN curl --proto '=https' --tlsv1.2 -LsSf https://github.com/astral-sh/uv/releases/download/0.11.24/uv-installer.sh | sh && mv /root/.local/bin/uv /usr/local/bin/uv
RUN curl --proto '=https' --tlsv1.2 -LsSf https://github.com/astral-sh/uv/releases/download/0.9.25/uv-installer.sh | sh && mv /root/.local/bin/uv /usr/local/bin/uv
# Preinstall python runtime to temp location (will copy with world-writable perms later)
# --compile-bytecode precompiles the stdlib to .pyc so jobs don't recompile it on every run
+15 -129
View File
@@ -171,141 +171,29 @@ This one table drives four things at once:
## Reproducibility — the beyond-dbt part
Because every materialization records the snapshot it produced, you can read any
table *as of* a past version — a capability dbt has no native answer for (dbt
models are always "whatever's in the warehouse now"). DuckLake gives us this for
the cost of recording one integer per run, and it covers the three things people
actually reach for: **debugging** ("what did this table look like at the failing
run"), **rollback** (re-materialize a consumer from snapshot N), and ad-hoc
**experimentation** on a historical state.
### How it's surfaced (shipped): explicit, discoverable time-travel
The version is exposed as a *user-driven* surface, not hidden plumbing. A
consumer pins a read by writing the DuckLake clause directly:
Because every materialization records the snapshot it produced, a downstream
consumer can read the *exact* upstream snapshot its run saw:
```sql
FROM dl.orders_daily AT (VERSION => 42)
FROM dl.orders_daily AT (VERSION => $WM_UPSTREAM_SNAPSHOT)
```
The asset node's **History** tab is a master-detail view: the snapshot list (id
+ time) on the left selects the version previewed in a read-only grid on the
right, which surfaces — and copies — the catalog-qualified
`FROM lake.<table> AT (VERSION => n)` clause. Snapshot ids are captured automatically; the user opts
into pinning when they want it, and the clause degrades to "latest" if removed,
so the same script still runs standalone. Mechanically this rides on
time-travel **reads** (`make_select_query` / `make_count_query` emit the `AT`
clause when a `version` is threaded through the `WM_INTERNAL_DB_*` markers) plus
a `DUCKLAKE_SNAPSHOTS` read for the history list — capabilities DuckLake already
has, no new write path.
The cascade already threads a `trigger` blob (producer path, partition) to each
subscriber; add the producer's captured `snapshot_id` to it, and a consumer's
read is pinned to the upstream state at dispatch time. That makes the *whole
pipeline* reproducible and time-travelable — something dbt has no native answer
for (dbt models are always "whatever's in the warehouse now"). It also gives
rollback (re-point an asset to snapshot N) and "what did this table look like at
the failing run" debugging, for free off the same captured ids.
### Deferred: automatic snapshot pinning across the cascade
An earlier sketch had the cascade *automatically* thread each producer's
`snapshot_id` into the `trigger` blob and inject `AT (VERSION => $WM_UPSTREAM_SNAPSHOT)`
into consumer reads, so a whole run is pinned to upstream state at dispatch time
without anyone asking. This is deliberately **not** built, for three reasons:
- **Not critical.** The only thing it adds over the explicit surface above is
*automatic per-run consistency* — protection against an upstream
re-materializing in the window between dispatch and a consumer reading. That
race only bites high-frequency event-driven cascades (rare today), and the
read is always a whole, ACID snapshot regardless — never corruption, just
"newer than the triggering version". Debugging and rollback are already
covered by the explicit surface.
- **Implicit magic.** Auto-injecting an `AT` clause and stripping it on
standalone runs is invisible behaviour to debug when it misfires; the explicit
clause is inspectable.
- **Multi-upstream ambiguity + EE coupling.** A consumer reading two ducklake
upstreams needs a per-ref snapshot *map* accumulated across the AND-join — and
the join-slot logic is EE. A single `$WM_UPSTREAM_SNAPSHOT` would silently pin
every read to one (the firing) producer's snapshot.
If a workload ever shows the consistency race in practice, pinning can be layered
on top — the capture and the snapshot surfacing built here are its foundation.
This is the differentiator worth leaning on. It is not catch-up to dbt; it is a
capability dbt structurally cannot offer, and DuckLake gives it to us at the
cost of recording one integer per run.
It also means **we do not build SCD2 snapshots** (gap #4 in `pipelines-vs-dbt.md`):
DuckLake time-travel is a strictly better answer for most of what dbt's
`{% snapshot %}` is used for. One fewer engine to write.
## Data tests (`// data_test`) — and the extensible-annotation pattern
Data tests are the first dbt-parity gap closed on top of materialization, and
the **first deliberately extensible annotation**. The design goal was not just
"add five test types" but to establish the convention a sibling family
(column-lineage is the next one) follows, so the annotation vocabulary stops
being a closed hardcoded list (`pipelines-vs-dbt.md` gap #7).
### Grammar
```
// data_test unique <col>
// data_test not_null <col>
// data_test accepted_values <col> = a,b,c
// data_test relationships <col> -> datatable://other/asset.<col>
// data_test <script_path> ← escape hatch (dbt's singular test)
```
`// data_test` lines **accumulate** (every well-formed line adds one check),
unlike the single-value annotations (`// materialize`, `// partitioned`, …)
which are first-write-wins. Malformed lines are dropped fail-safe — a typo
becomes an *absent* check (visible in the graph), never a mis-parsed one.
The keyword is `data_test`, **not `test`** — there is an unrelated, shipped
`// test:` CI-test annotation (`windmill_common::schema::parse_ci_test_annotation`,
tests a script's *logic* on deploy). `data_test` tests the *data* in the
materialized asset at run time. This mirrors dbt 1.8's own `tests:`
`data_tests:` rename, made for exactly this disambiguation.
### The pattern: annotation → verifier
The reusable shape, in three layers, each a clean extension seam:
1. **Parse** (`asset_parser.rs` + `parsePipelineAnnotations.ts`, kept in
lockstep by the parity corpus). A `data_test` line is dispatched on a
**keyword head** to a typed variant (`DataTest`). A new built-in is one
match arm + its sub-parser; the `Custom` arm is the open fallback. A sibling
family reuses this head-keyword dispatch rather than adding a parallel list.
2. **Compile** (`sql_materialize.rs::build_data_test_checks`). Each test becomes
a **check**: `(name, violating-row-count query)`. Built-ins differ only in
their count query; `Custom` supplies its own (the user's SELECT of violating
rows). Referenced assets (relationships) emit an `ATTACH` resolved by the
same transform pass as the user's own.
3. **Execute** (`duckdb_executor.rs`). The materialize summary query embeds
every check's count in one `data_tests` list-of-struct column (computed in a
CTE, since DuckDB rejects subqueries inside struct literals), so **all tests
run in a single pass** against the freshly-materialized slice — no
abort-on-first. The worker reads the breakdown from the result and decides
pass/fail: any violation **fails the run** (record `Failed`, propagate up the
cascade) with an error listing *every* test (✓/✗ + counts); a clean run
returns the per-test summary so the UI can render a checklist.
A new annotation family that produces post-materialize checks (or, for
column-lineage, post-materialize *metadata reads*) plugs into the same three
seams: add a parsed variant, emit its check/reader SQL into the summary, read
it back in the worker. Nothing about the closed set of *today's* keywords is
load-bearing.
### Scoping decisions (v1)
- **Partition scope.** When `// partitioned`, built-in checks are scoped to the
slice just written (`WHERE _wm_partition = <value>`), so a rerun/backfill of
one partition is independent of other partitions' (possibly pre-existing)
data. Whole-table assertions are a follow-up.
- **Commit-then-test.** Like dbt, the write commits before tests run; a failed
test fails the *run* (and records `Failed`, so downstream cascade stops) but
does not roll back the committed snapshot. Time-travel still lets you inspect
exactly what failed.
- **Custom = DuckDB SQL, server worker.** The escape hatch fetches the deployed
script's content (a single DuckDB `SELECT`/CTE returning the violating rows —
it's embedded as a subquery, so a multi-statement body is rejected with a
clear error) and inlines it as a check; `{partition}` is substituted and
`_wm_target` is in scope. Agent (Http) workers — which have no script cache —
get a clear error. Non-DuckDB custom tests (dispatched as sub-jobs, any
language) are the natural follow-up and fit the same verifier seam.
- **Managed only.** `// materialize manual` + `// data_test` is rejected with a
clear error (we can't know the manual script's target alias / partition col).
## Scoping decision: DuckLake vs DataTable
**Make DuckLake the materialization/versioning substrate; keep DataTable as the
@@ -336,10 +224,8 @@ Don't try to give both the full treatment for v1.
`materialized_partition` rows.
5. **Surface it** — last-materialized/snapshot/row-count on the asset node;
missing-partition set feeds the backfill UI.
6. *v1.x*time-travel UX over the captured snapshots: a per-asset **History**
tab — a master-detail snapshot list + query-at-version preview that copies the
full `FROM lake.<table> AT (VERSION => n)` clause. Automatic cascade pinning
(`$WM_UPSTREAM_SNAPSHOT`) is deferred — see §"Reproducibility" for why.
6. *v1.x*snapshot pinning across the cascade (`$WM_UPSTREAM_SNAPSHOT`),
rollback, time-travel read helper.
Steps 15 are a thin annotation+template layer plus one metadata table and one
extra read per run. They deliver managed/incremental/versioned assets,
+5 -11
View File
@@ -46,7 +46,7 @@ Asset-centric, polyglot, annotation-driven, event-aware:
| Gap | Architectural blocker? | Verdict |
|---|---|---|
| Data tests | No | **Shipped** (`// data_test`) |
| Data tests | No | Pure TODO |
| Incremental materializations | No, but pick a philosophy | TODO with design decision |
| Column lineage + docs site | No | Pure TODO |
| Snapshots / SCD2 | No | New output kind |
@@ -64,16 +64,10 @@ and annotation extensibility**. The rest is execution.
dbt: `unique`, `not_null`, `accepted_values`, custom generic tests, plus
singular tests. Run as `SELECT` statements that pass when they return 0 rows.
**Shipped** via the `// data_test` annotation (built on materialization):
`// data_test unique <col>`, `not_null`, `accepted_values <col> = a,b,c`,
`relationships <col> -> <asset>.<col>`, and `// data_test <script_path>` for
the custom escape hatch (dbt's singular test). Each compiles to a SQL
*verifier probe* that runs against the freshly-materialized asset and raises on
violation, riding the existing failure-propagation path. This is also the first
*extensible* annotation — see `ducklake-materialization.md` §"Data tests" for
the annotation→verifier pattern that column-lineage will reuse. The keyword is
`data_test`, not `test`, to stay clear of the unrelated `// test:` CI-test
annotation.
Today: nothing. Annotation parser is the natural hook —
`// test unique col_name`, `// test not_null col_name`,
`// test <script_path>` for custom. Pipeline runtime already handles
failure propagation. Lowest-risk, highest-payoff item.
### 2. Incremental materializations
Generated
+3 -3
View File
@@ -20,11 +20,11 @@
},
"nixpkgs": {
"locked": {
"lastModified": 1768377964,
"narHash": "sha256-RU35vQnfg9NwJUviGCfMH9ChgHANoNSiRaAn4/wINT4=",
"lastModified": 1764517877,
"narHash": "sha256-pp3uT4hHijIC8JUK5MEqeAWmParJrgBVzHLNfJDZxg4=",
"owner": "NixOS",
"repo": "nixpkgs",
"rev": "cadda13afe838615fb74b0a9720905920559c535",
"rev": "2d293cbfa5a793b4c50d17c05ef9e385b90edf6c",
"type": "github"
},
"original": {
+30 -689
View File
File diff suppressed because it is too large Load Diff
+1 -2
View File
@@ -1,6 +1,6 @@
{
"name": "@windmill-labs/components",
"version": "1.739.0",
"version": "1.737.0",
"scripts": {
"dev": "vite dev",
"dev:ui-builder": "mv static/ui_builder static/ui_builder.dev-disabled 2>/dev/null || true ; trap 'mv static/ui_builder.dev-disabled static/ui_builder 2>/dev/null || true' EXIT ; vite dev",
@@ -127,7 +127,6 @@
"lru-cache": "^11.1.0",
"lucide-svelte": "^0.540.0",
"mdast-util-find-and-replace": "^3.0.2",
"mermaid": "^11.15.0",
"minimatch": "^10.0.1",
"monaco-editor": "npm:@codingame/monaco-vscode-editor-api@=25.0.0",
"monaco-languageclient": "10.6.0",
+2 -2
View File
@@ -1,5 +1,5 @@
{
"baseUrl": "https://pub-06154ed168a24e73a86ab84db6bf15d8.r2.dev",
"version": "062d11c",
"sha256": "355bf3acfb935080e4a4fa23d0509e03e4bc01aeed622db110663d7ec6c1c438"
"version": "ad76918",
"sha256": "4c940570936a391c217a054b74c691587d7f27d012ccddbf5e3b3d25e7c1fe4a"
}
@@ -1,51 +0,0 @@
<script lang="ts">
// Renders the per-test breakdown a managed `// materialize` run attaches to
// its result as `data_tests: [{ test, violating }]`. Shown above the raw
// result so "which tests ran, and which passed/failed" is clear at a glance.
// (On a failed run the job result is the error message — which already lists
// every test — so this success-path checklist and that error text together
// cover both outcomes.)
import { CheckCircle2, FlaskConical, XCircle } from 'lucide-svelte'
let { tests }: { tests: Array<{ test: string; violating: number }> } = $props()
let failed = $derived(tests.filter((t) => t.violating > 0).length)
</script>
<div
class="mb-2 rounded-md border bg-surface-secondary text-xs overflow-hidden {failed > 0
? 'border-red-300 dark:border-red-900/60'
: 'border-border'}"
>
<div class="flex items-center gap-1.5 px-2 py-1 border-b border-border text-secondary">
<FlaskConical size={13} />
<span class="font-medium">
{tests.length} data test{tests.length === 1 ? '' : 's'}
</span>
<span
class={failed > 0
? 'text-red-600 dark:text-red-400'
: 'text-emerald-600 dark:text-emerald-400'}
>
· {failed > 0 ? `${failed} failed` : 'all passed'}
</span>
</div>
<ul class="divide-y divide-border">
{#each tests as t (t.test)}
<li class="flex items-center gap-1.5 px-2 py-1 font-mono">
{#if t.violating > 0}
<XCircle size={13} class="shrink-0 text-red-600 dark:text-red-400" />
<span class="sr-only">failed:</span>
<span class="text-primary">{t.test}</span>
<span class="text-red-600 dark:text-red-400"
>— {t.violating} violating row{t.violating === 1 ? '' : 's'}</span
>
{:else}
<CheckCircle2 size={13} class="shrink-0 text-emerald-600 dark:text-emerald-400" />
<span class="sr-only">passed:</span>
<span class="text-secondary">{t.test}</span>
{/if}
</li>
{/each}
</ul>
</div>
@@ -20,7 +20,6 @@
Loader2
} from 'lucide-svelte'
import DucklakeResultPreview from './assets/AssetGraph/DucklakeResultPreview.svelte'
import DataTestsResult from './DataTestsResult.svelte'
import Portal from '$lib/components/Portal.svelte'
import DisplayResultControlBar from './DisplayResultControlBar.svelte'
@@ -565,54 +564,10 @@
resultHeaderHeight
)
})
// Per-test breakdown of a managed `// materialize` run, rendered as a
// checklist above the raw result. On success it rides the result
// (`data_tests: [{ test, violating }]`, a one-row array). On failure the job
// result is the error, whose message is the worker's breakdown text — parsed
// back into the same shape so the checklist shows on both outcomes. Both
// formats are produced by this repo's worker (see duckdb_executor.rs); the
// derivation is inert (undefined) for every other DisplayResult use.
let dataTests = $derived.by(() => {
// Success: structured column on the summary row.
const row = Array.isArray(result) ? (result as any)?.[0] : (result as any)
let dt = row?.data_tests
if (typeof dt === 'string') {
try {
dt = JSON.parse(dt)
} catch {
dt = undefined
}
}
if (
Array.isArray(dt) &&
dt.length > 0 &&
dt.every((x) => x && typeof x.test === 'string' && typeof x.violating === 'number')
) {
return dt as Array<{ test: string; violating: number }>
}
// Failure: parse the worker's breakdown out of the error message.
const msg = (result as any)?.error?.message
if (typeof msg === 'string' && msg.includes('data tests failed on')) {
const out: Array<{ test: string; violating: number }> = []
for (const line of msg.split('\n')) {
const fail = line.match(/^\s*✗\s*(.+?)\s*—\s*(\d+)\s+violating/)
const pass = line.match(/^\s*✓\s*(.+?)\s*$/)
if (fail) out.push({ test: fail[1], violating: parseInt(fail[2], 10) })
else if (pass) out.push({ test: pass[1], violating: 0 })
}
if (out.length > 0) return out
}
return undefined
})
</script>
<HighlightTheme />
{#if dataTests}
<DataTestsResult tests={dataTests} />
{/if}
{#if result_stream && result == undefined}
<div class="flex flex-col w-full gap-2">
<div class="flex items-center gap-2 text-secondary text-xs">
+5 -19
View File
@@ -30,10 +30,6 @@
placement?: Placement
usePointerDownOutside?: boolean
closeOnOtherDropdownOpen?: boolean
// When false the menu stays open after an item is selected (melt's closeOnItemClick).
// Consumers that keep the menu open must close it themselves where appropriate.
// Read once at menu creation (like `placement`); changing it after mount has no effect.
closeOnItemClick?: boolean
fixedHeight?: boolean
hidePopup?: boolean
open?: boolean
@@ -45,18 +41,10 @@
size?: ButtonType.UnifiedSize
btnText?: string
buttonReplacement?: import('svelte').Snippet
// In customMenu mode the snippet receives the melt-ui `item` action store
// (so consumers can wrap rows in <MenuItem> for arrow-key navigation + aria)
// and `builders` (so they can compose melt submenus, e.g. via DropdownSubmenuItem).
menu?: import('svelte').Snippet<
[
{
item: MenubarMenuElements['item']
close: () => void
builders: ReturnType<typeof createDropdownMenu>['builders']
}
]
>
// In customMenu mode the snippet receives the melt-ui `item` action
// store so consumers can wrap their own rows in <MenuItem> (or
// `use:melt={$item}`) and get arrow-key navigation + aria wiring.
menu?: import('svelte').Snippet<[{ item: MenubarMenuElements['item']; close: () => void }]>
maxHeight?: string | undefined
}
@@ -68,7 +56,6 @@
placement = 'bottom-end',
usePointerDownOutside = false,
closeOnOtherDropdownOpen = true,
closeOnItemClick = true,
fixedHeight = true,
hidePopup = false,
open = $bindable(false),
@@ -95,7 +82,6 @@
positioning: {
placement: untrack(() => placement)
},
closeOnItemClick: untrack(() => closeOnItemClick),
loop: true,
onOpenChange: ({ next }) => {
if (closeOnOtherDropdownOpen) {
@@ -190,7 +176,7 @@
transition:fly={{ duration: enableFlyTransition ? 100 : 0, y: -16 }}
>
{#if customMenu}
{@render menu?.({ item, close, builders })}
{@render menu?.({ item, close })}
{:else}
<div
class="bg-surface-tertiary dark:border w-56 origin-top-right rounded-lg shadow-lg focus:outline-none overflow-y-auto py-1"
+82 -53
View File
@@ -158,6 +158,11 @@
preparedAssetsSqlQueries?: InferAssetsSqlQueryDetails[] | undefined
// To execute preview scripts with the right worker group
customTag?: string
// Opt-in: reflect external `code` prop mutations back into Monaco (see
// the effect below). One-way `code={...}` callers that need live
// external updates — e.g. the inline flow rawscript — set this. Off by
// default so every other caller's behavior is unchanged.
syncExternalCode?: boolean
}
let {
@@ -190,7 +195,8 @@
enablePreprocessorSnippet = false,
rawAppRunnableKey = undefined,
preparedAssetsSqlQueries,
customTag
customTag,
syncExternalCode = false
}: Props = $props()
$effect.pre(() => {
@@ -362,6 +368,15 @@
divEl?.classList.add('hidden')
}
// Mirrors the value Monaco's model currently holds, as last synced through
// `code`. The external-`code`→model effect below reflects only when `code`
// diverges from this sentinel, i.e. when `code` was set by an outside writer
// (draft load, template reset) rather than echoed back from the model. Writes
// that go straight to the model (AI chat apply, collab) update `code` via the
// debounced change handler, which keeps this sentinel in lockstep — so the
// effect never reflects a stale `code` over a model that moved ahead.
let lastReflectedCode = code
export function setCode(ncode: string, noHistory: boolean = false): void {
// Track whether the code actually changed before updating.
const changed = code != ncode
@@ -369,12 +384,26 @@
code = ncode
}
// setCode is an authoritative overwrite (reset, AI apply, module switch).
// Cancel any in-flight keystroke debounce first: otherwise alignCodeWithEditor
// skips on the `timeoutModel` guard (leaving Monaco stale), and the pending
// updateCode later reads the old buffer and writes it back over `ncode`.
cancelPendingChanges()
alignCodeWithEditor(!noHistory)
if (noHistory) {
editor?.setValue(ncode)
} else {
if (editor?.getModel()) {
// editor.setValue(ncode)
editor.pushUndoStop()
editor.executeEdits('set', [
{
range: editor.getModel()!.getFullModelRange(), // full range
text: ncode
}
])
editor.pushUndoStop()
}
}
// The model now holds `ncode`; record it so the reflect effect treats this
// as already-synced and doesn't write it back.
lastReflectedCode = ncode
// Dispatch change immediately when code actually changed. This ensures
// callers like the Reset button and copilot trigger on:change handlers.
// The debounced onDidChangeModelContent handler will no-op since code
@@ -408,7 +437,10 @@
return
}
code = ncode
lastEditorCode = ncode
// `code` was just echoed from the model, so keep the sentinel aligned —
// this is what prevents the reflect effect from racing the model during a
// burst of in-editor edits (e.g. an AI chat apply).
lastReflectedCode = ncode
dispatch('change', ncode)
}
@@ -420,19 +452,12 @@
* see it. Clears the chain state so the next keystroke after this
* flush is a fresh leading fire. */
export function flushPendingChanges(): void {
cancelPendingChanges()
updateCode()
}
/** Discard any in-flight keystroke debounce without materializing it, so a
* deferred updateCode can't fire later. Resets chain state to a fresh leading
* fire on the next keystroke. */
function cancelPendingChanges(): void {
if (timeoutModel !== undefined) {
clearTimeout(timeoutModel)
timeoutModel = undefined
}
changeChainStart = undefined
updateCode()
}
export function append(code: string): void {
@@ -1892,6 +1917,29 @@
lang = scriptLangToEditorLang(scriptLang)
})
// Opt-in (syncExternalCode): reflect external `code` prop mutations into
// Monaco's model. Parents that pass `code={...}` one-way (no bind) — e.g.
// the inline rawscript in the flow editor — otherwise mutate the prop
// without Monaco ever showing the change (the AI chat editing a flow
// module's content in a session is the motivating case). Gated off by
// default: Editor is sensitive and most callers either bind:code (and
// carry their own external-sync) or treat code as init-only, so a blanket
// setValue would risk clobbering them. The `getValue() !== code` guard
// keeps the caret intact when the change originated from typing inside
// Monaco (which round-trips code back via `$bindable`, re-firing this
// effect with `code === getValue()`).
let lastExternalCodeSync = code
$effect(() => {
if (!syncExternalCode) return
if (code === lastExternalCodeSync) return
lastExternalCodeSync = code
if (!editor) return
untrack(() => {
if (editor!.getValue() !== code) {
editor!.setValue(code ?? '')
}
})
})
$effect(() => {
filePath = computePath(path)
})
@@ -1979,50 +2027,31 @@
})
})
let applyExternalCode = useDebounce(() => alignCodeWithEditor(true), 800)
// Last `code` value the editor itself produced or aligned to. Used to tell an
// echo (the bindable changed because the user typed — Monaco is already
// ahead) from a genuine external write. Without this, a typing burst longer
// than the debounce window would sync the lagging `code` back over newer
// keystrokes. Must be kept in step with every editor↔`code` sync point.
let lastEditorCode = code
function alignCodeWithEditor(history: boolean) {
// External `code` prop changes should flow into the Monaco editor. The
// `untrack` block reads/writes Monaco without subscribing — only the
// prop read above is tracked — so the editor's own change handler
// (`updateCode`) re-running with the same value short-circuits and we
// don't loop.
$effect(() => {
const next = code ?? ''
const ed = editor
if (!ed) return
const next = code ?? ''
const value = ed.getValue()
const model = ed.getModel()
// Some keystrokes are still being debounced, don't overwrite them.
// When the debounce is done, updateCode will be called and the code will be aligned with the editor.
if (timeoutModel !== undefined) return
if (!model) return
lastEditorCode = next
if (value === next) return
if (history) {
// Only reflect genuine external `code` changes. When `code` merely echoed a
// model edit (typing, AI chat apply, collab), `lastReflectedCode` already
// matches and we skip — otherwise a debounced echo could overwrite a model
// that has since moved further ahead, reverting the newer edit.
if (code === lastReflectedCode) return
lastReflectedCode = code
untrack(() => {
if (ed.getValue() === next) return
const model = ed.getModel()
if (!model) return
ed.pushUndoStop()
ed.executeEdits('external', [{ range: model.getFullModelRange(), text: next }])
ed.pushUndoStop()
} else {
ed.setValue(next)
}
}
// External `code` prop changes should flow into the Monaco editor. Skip
// echoes: when `code` matches what the editor last produced (`updateCode`)
// or aligned to, the change came from the editor itself, so syncing back
// would clobber input typed since. Only genuine external writes — where
// `code` diverges from `lastEditorCode` — schedule a sync. The `untrack`
// block reads/writes Monaco without subscribing, so we don't loop.
$effect(() => {
;[code, editor]
if (!editor) return
untrack(() => {
if (code === lastEditorCode) return
applyExternalCode()
})
})
let isTsWorkerInitialized = resource([() => lang, () => initialized], async () => {
if (lang !== 'typescript' || !initialized) return false
// Use the stable model URI (computed once at mount), not filePath which changes on rename
@@ -1180,25 +1180,15 @@
export type FlowModuleForTimeline = {
id: string
type: FlowModuleValue['type']
suspend?: boolean
}
function allModulesForTimeline(
modules: FlowModule[],
expandedSubflows: Record<string, { modules: FlowModule[]; groups?: any[] }>
): FlowModuleForTimeline[] {
const ids = dfs(
modules,
(x) =>
({
id: x.id,
type: x.value.type,
suspend: x.suspend != undefined
}) as FlowModuleForTimeline,
{
skipToolNodes: true
}
)
const ids = dfs(modules, (x) => ({ id: x.id, type: x.value.type }) as FlowModuleForTimeline, {
skipToolNodes: true
})
function rec(
ids: FlowModuleForTimeline[],
@@ -1218,8 +1208,7 @@
fms,
(x) => ({
id: x.id.startsWith('subflow:') ? x.id : buildSubflowKey(x.id, nprefix),
type: x.value.type,
suspend: x.suspend != undefined
type: x.value.type
}),
{ skipToolNodes: true }
),
@@ -72,66 +72,6 @@
}
const barHeight = 32
// Whole approval-wait machinery below is inert unless a step actually has a suspend config,
// so large suspend-free flows pay nothing for the flatten/sort and the per-tick recompute.
const hasSuspendModule = $derived(flowModules.some((m) => m.suspend))
// Push times of every job on the timeline, ascending. Used to locate when the step
// that follows an approval step started — i.e. the moment the approval was granted.
const allCreatedAts = $derived(
hasSuspendModule
? Object.values(items ?? {})
.flat()
.map((j) => j.created_at)
.filter((t): t is number => t != undefined)
.sort((a, b) => a - b)
: []
)
// Heuristic: the grant moment is approximated by the next job pushed anywhere on the
// timeline. Exact for sequential flows; for an approval step inside one branch of a
// parallel branchall a concurrent sibling job can land first and understate the wait.
function nextCreatedAtAfter(t: number): number | undefined {
return allCreatedAts.find((c) => c > t)
}
// For a completed suspend/approval step, the time spent waiting for the approval is the
// gap between the step finishing and the next step being pushed (or now, if still waiting).
function approvalWait(b: {
started_at?: number
duration_ms?: number
}): { start: number; len: number; running: boolean } | undefined {
if (b.started_at == undefined || b.duration_ms == undefined) {
return undefined
}
const end = b.started_at + b.duration_ms
const next = nextCreatedAtAfter(end)
const waitEnd = next ?? (flowDone ? undefined : now)
if (waitEnd == undefined) {
return undefined
}
const len = waitEnd - end
if (len < 100) {
return undefined
}
return { start: end, len, running: next == undefined }
}
// Approval wait per module id, computed once and consumed by both the rows and the legend.
const approvalWaitByModule = $derived.by(() => {
const result: Record<string, { start: number; len: number; running: boolean }> = {}
for (const m of flowModules) {
if (!m.suspend) continue
const sub = (items?.[m.id] ?? []).filter((x) => x.created_at && x.started_at)
if (sub.length !== 1) continue
const aw = approvalWait(sub[0])
if (aw) result[m.id] = aw
}
return result
})
const hasApprovalWait = $derived(Object.keys(approvalWaitByModule).length > 0)
</script>
<OnChange
@@ -155,12 +95,6 @@
<div class="h-2.5 w-2.5 rounded-sm bg-blue-500/90"></div>
<span>Execution</span>
</div>
{#if hasApprovalWait}
<div class="flex gap-1.5 items-center">
<div class="h-2.5 w-2.5 rounded-sm bg-purple-400/80"></div>
<span>Approval wait</span>
</div>
{/if}
{#if max && min}
<span class="font-mono">{msToSec(max - min, 1)}s</span>
{/if}
@@ -179,7 +113,7 @@
/>
</div>
{/if}
{#each flowModules as { id: k, type: typ, suspend: isSuspend } (k)}
{#each flowModules as { id: k, type: typ } (k)}
{@const subItems = items?.[k]?.filter((x) => x.created_at && x.started_at)}
<div class="relative px-3 py-1.5">
<div class="flex items-center justify-between mb-0.5">
@@ -227,7 +161,6 @@
? 0
: now - b?.created_at
: 0}
{@const aw = isSuspend ? approvalWaitByModule[k] : undefined}
<div class="flex w-full py-0.5 items-center" {style}>
<TimelineBar
position="left"
@@ -242,7 +175,7 @@
/>
{#if b.started_at}
<TimelineBar
position={aw || waitingLen < 100 ? 'center' : 'right'}
position={waitingLen < 100 ? 'center' : 'right'}
id={b?.id}
{total}
{min}
@@ -252,20 +185,6 @@
running={b?.duration_ms == undefined}
/>
{/if}
{#if aw}
<TimelineBar
position="right"
id={b?.id}
{total}
{min}
concat
colorClass="bg-purple-400/80"
tooltip={`Waiting for approval — ${msToSec(aw.len, 1)}s`}
started_at={aw.start}
len={aw.len}
running={aw.running}
/>
{/if}
</div>
{:else}
<div class="flex w-full py-0.5"></div>
@@ -277,6 +196,7 @@
</div>
{/each}
</div>
{:else}
<Loader2 class="animate-spin" />
{/if}
@@ -12,14 +12,10 @@
let comparison: WorkspaceComparison | undefined = $state(undefined)
let error: string | undefined = $state(undefined)
let isFork = $derived($workspaceStore?.startsWith('wm-fork-') ?? false)
let currentWorkspaceData = $derived($userWorkspaces.find((w) => w.id === $workspaceStore))
let parentWorkspaceId = $derived(currentWorkspaceData?.parent_workspace_id)
let parentWorkspaceData = $derived($userWorkspaces.find((w) => w.id === parentWorkspaceId))
// A fork must have a parent to compare/merge against. Treating the wm-fork-
// prefix alone as "is a fork" renders a parentless "Fork of ()" banner when
// the parent linkage was dropped (e.g. by a workspace id change), so require
// both, matching the forks/compare page.
let isFork = $derived(($workspaceStore?.startsWith('wm-fork-') ?? false) && !!parentWorkspaceId)
// Drafts in this fork. When the fork is otherwise in sync with its parent, a
// user with only pending drafts should still get the draft CTA (mirrors the
@@ -1063,14 +1063,6 @@
<li
>AI chat usage (provider, model, mode, session count, message count — last 30 days)</li
>
<li
>resource counts (workspaces, scripts per language, flows, workflows as code, low-code
apps, raw apps)</li
>
<li
>infrastructure info (container runtime, managed database provider, database version,
size and cluster size, max and active connections, object storage backend)</li
>
</ul>
<br />For air-gapped instances, you can download the telemetry data and send it manually.
</div>
@@ -1109,10 +1101,6 @@
<li
>AI chat usage (provider, model, mode, session count, message count — last 30 days)</li
>
<li
>resource counts (workspaces, scripts per language, flows, workflows as code, low-code
apps, raw apps)</li
>
</ul>
</div>
{/if}
@@ -952,7 +952,9 @@
}
})
})
$effect(() => {
readFieldsRecursively(script)
})
// Mirror the draft triggers (held in a separate `triggersState` $state)
// back into `script.draft_triggers` so the UserDraft autosave — which
// deep-tracks `script` — picks them up. Pre-PR ScriptBuilder ran its own
+36 -76
View File
@@ -52,7 +52,6 @@
Play,
PlayIcon,
Plus,
Target,
Terminal,
Pencil,
WandSparkles,
@@ -269,7 +268,6 @@
if (activeModuleTab === null && code !== lastSyncedCode) {
editorCode = code
lastSyncedCode = code
editor?.setCode(editorCode) // immediate sync, don't wait for the 800ms debounce
untrack(() => inferSchema(code))
}
})
@@ -1593,29 +1591,18 @@
let error = $derived(getError(testJob))
$effect(() => {
;[
editor,
const options: ScriptOptions = {
code,
lang: lang as ScriptLang,
error,
args: args ?? {},
path,
lastSavedCode,
lastDeployedCode,
diffMode,
workflowAsCodeAiContext,
args,
error,
lang,
path
]
workflowAsCode: workflowAsCodeAiContext
}
untrack(() => {
const options: ScriptOptions = {
getCode: () => code,
lang: lang as ScriptLang,
error,
args: args ?? {},
path,
lastSavedCode,
lastDeployedCode,
diffMode,
workflowAsCode: workflowAsCodeAiContext
}
aiChatManager.scriptEditorOptions = options
aiChatManager.scriptEditorApplyCode = async (code: string, opts?: ReviewChangesOpts) => {
hideDiffMode()
@@ -1913,18 +1900,15 @@
<div class="absolute top-1 left-2 z-10">
{#if testIsLoading}
{@render cancelTestButton('sm', 'shadow-md')}
{:else if (customUi?.previewPanel?.downstreamSubscribers ?? 0) > 0 || customUi?.previewPanel?.onBoundedRun}
{:else if (customUi?.previewPanel?.downstreamSubscribers ?? 0) > 0}
<!-- Split button: primary "Test" runs just this step
(skips the asset-trigger cascade); the caret
opens a popover with the cascade option labelled
by the downstream count. The active mode is
reflected in both the button label and the
check-mark on the menu item so the user always
knows whether the next run will fan out. A
pure-reader-only root has no subscriber downstream
but still gets `onBoundedRun`, so the split button
also opens for it (with the cascade item hidden). -->
{@const downstream = customUi?.previewPanel?.downstreamSubscribers ?? 0}
knows whether the next run will fan out. -->
{@const downstream = customUi!.previewPanel!.downstreamSubscribers!}
<div class="flex items-stretch shadow-md rounded-md overflow-hidden">
<Button
on:click={() => runTest()}
@@ -1987,55 +1971,31 @@
>
</div>
</button>
{#if downstream > 0}
<button
type="button"
class="w-full text-left px-3 py-2 hover:bg-surface-hover flex items-start gap-2"
onclick={() => {
cascadeDownstream = true
close()
void runTest()
}}
>
<Zap
size={14}
class="mt-0.5 shrink-0 text-amber-600 dark:text-amber-400"
/>
<div class="flex flex-col min-w-0">
<span class="font-medium">
Test + trigger {downstream} downstream {cascadeDownstream
? '(current)'
: ''}
</span>
<span class="text-2xs text-secondary">
Let the asset-trigger cascade fan out to the {downstream}
subscribed script{downstream === 1 ? '' : 's'} after this run succeeds.
</span>
</div>
</button>
{/if}
{#if customUi?.previewPanel?.onBoundedRun}
<button
type="button"
class="w-full text-left px-3 py-2 hover:bg-surface-hover flex items-start gap-2 border-t"
onclick={() => {
close()
customUi!.previewPanel!.onBoundedRun!()
}}
>
<Target
size={14}
class="mt-0.5 shrink-0 text-blue-600 dark:text-blue-400"
/>
<div class="flex flex-col min-w-0">
<span class="font-medium">Run downstream up to…</span>
<span class="text-2xs text-secondary">
Pick end node(s) on the graph, then run only the cascade between
this script and them.
</span>
</div>
</button>
{/if}
<button
type="button"
class="w-full text-left px-3 py-2 hover:bg-surface-hover flex items-start gap-2"
onclick={() => {
cascadeDownstream = true
close()
void runTest()
}}
>
<Zap
size={14}
class="mt-0.5 shrink-0 text-amber-600 dark:text-amber-400"
/>
<div class="flex flex-col min-w-0">
<span class="font-medium">
Test + trigger {downstream} downstream {cascadeDownstream
? '(current)'
: ''}
</span>
<span class="text-2xs text-secondary">
Let the asset-trigger cascade fan out to the {downstream}
subscribed script{downstream === 1 ? '' : 's'} after this run succeeds.
</span>
</div>
</button>
</div>
{/snippet}
</Popover>
@@ -10,7 +10,7 @@
import { base } from '$lib/base'
import SearchItems from './SearchItems.svelte'
import { page } from '$app/state'
import { replaceState } from '$app/navigation'
import { goto as gotoUrl } from '$app/navigation'
import Version from './Version.svelte'
import Uptodate from './Uptodate.svelte'
import InstanceSettings from './InstanceSettings.svelte'
@@ -70,16 +70,7 @@
const index = page.url.href.lastIndexOf('#')
if (index === -1) return
const hashRemoved = page.url.href.slice(0, index)
// Strip the drawer's URL hash without a SvelteKit navigation: a `goto`
// here re-fires path-reactive effects on the underlying page (e.g. the
// script editor's load effect), wiping unsaved editor content.
try {
replaceState(hashRemoved, page.state)
} catch (e) {
// replaceState throws if the router isn't initialized yet — possible
// when onDestroy runs during router teardown.
console.error(e)
}
gotoUrl(hashRemoved)
}
onDestroy(() => {
+9 -21
View File
@@ -15,10 +15,6 @@
concat?: boolean
gray?: boolean
spacerClass?: string
/** Overrides the default gray/blue bar color (e.g. to mark an approval wait). */
colorClass?: string
/** Tooltip label shown on hover instead of the default job link. */
tooltip?: string
}
let {
@@ -31,9 +27,7 @@
running,
concat = false,
gray = false,
spacerClass = '',
colorClass = undefined,
tooltip = undefined
spacerClass = ''
}: Props = $props()
</script>
@@ -43,26 +37,20 @@
{/if}
<Popover
style="width: {(len / total) * 100}%"
class="h-5 relative {colorClass
? colorClass
: gray
? 'bg-gray-300 dark:bg-gray-600'
: running
? 'bg-blue-400/90'
: 'bg-blue-500/90'} {position == 'left'
class="h-5 relative {gray
? 'bg-gray-300 dark:bg-gray-600'
: running
? 'bg-blue-400/90'
: 'bg-blue-500/90'} {position == 'left'
? 'rounded-l-md'
: position == 'right'
? 'rounded-r-md'
: 'rounded-md'} center-center text-white text-2xs whitespace-nowrap hover:outline outline-1 outline-black"
>
{#snippet text()}
{#if tooltip}
<span>{tooltip}</span>
{:else}
<a href="{base}/run/{id}" class="inline-flex items-center gap-1" target="_blank"
>{id} <ExternalLink size={14} /></a
>
{/if}
<a href="{base}/run/{id}" class="inline-flex items-center gap-1" target="_blank"
>{id} <ExternalLink size={14} /></a
>
{/snippet}
{#if len > 0}
{@const narrow = len / total < 0.09}
@@ -4,9 +4,6 @@ Inline diff renderer for a single workspace item. Mirrors the per-kind
rendering that DiffDrawer does in its body (`DiffDrawer.svelte:181-271`):
- `flow` → `<FlowDiffViewer>` (its own Graph / YAML toggle inside)
- `raw_app_file` → `<RawAppFileDiff>` (one synthesized raw-app file item: a
single diff with a per-file size guard; the metadata item adds a full-app
YAML expand). Raw apps are exploded into these items by `rawAppDiffToItems`.
- has `content` (scripts) → Tabs(Content | Metadata) with two Monaco diffs
- everything else (apps, resources, variables, schedules, triggers…) →
a single Monaco YAML diff over the metadata
@@ -21,15 +18,12 @@ doesn't reflow the parent.
import Tabs from './common/tabs/Tabs.svelte'
import Tab from './common/tabs/Tab.svelte'
import FlowDiffViewer from './FlowDiffViewer.svelte'
import RawAppFileDiff from './raw_apps/RawAppFileDiff.svelte'
import type { RawAppFileItem } from './raw_apps/rawAppDiffUtils'
import { Loader2 } from 'lucide-svelte'
import { cleanValueProperties, orderedYamlStringify, replaceFalseWithUndefined } from '$lib/utils'
import { scriptLangToEditorLang } from '$lib/scripts'
interface Props {
/** Any WorkspaceItemDiff['kind'], plus the synthetic `raw_app_file`.
* `flow` and `raw_app_file` are special-cased. */
/** Any WorkspaceItemDiff['kind'] — used only to special-case `flow`. */
kind: string
/** Raw value from `getItemValue(kind, path, parentWorkspace)`. Undefined
* for "added" items (don't exist in the parent). */
@@ -39,11 +33,9 @@ doesn't reflow the parent.
currentRaw?: unknown
/** Force unified diff (Monaco renderSideBySide=false). Default false. */
inlineDiff?: boolean
/** For `raw_app_file`: the synthesized per-file diff item to render. */
rawFile?: RawAppFileItem
}
let { kind, originalRaw, currentRaw, inlineDiff = false, rawFile }: Props = $props()
let { kind, originalRaw, currentRaw, inlineDiff = false }: Props = $props()
type Prepared = { lang?: string; content?: string; metadata: string }
@@ -110,16 +102,6 @@ doesn't reflow the parent.
{inlineDiff}
/>
</div>
{:else if kind === 'raw_app_file' && rawFile}
<RawAppFileDiff
original={rawFile.original}
current={rawFile.current}
lang={rawFile.lang}
isMetadata={rawFile.isMetadata}
fullYamlOriginal={rawFile.fullYamlOriginal}
fullYamlCurrent={rawFile.fullYamlCurrent}
{inlineDiff}
/>
{:else if hasContent}
<div class="flex flex-col">
<Tabs bind:selected={contentTab}>

Some files were not shown because too many files have changed in this diff Show More