starred/windmill
1
0
Fork 0
mirror of https://github.com/windmill-labs/windmill.git synced 2026-10-18 16:03:24 +00:00
Code Issues Packages Projects Releases Wiki Activity

Merge remote-tracking branch 'origin/main' into change-68b704f7

Browse Source 
# Conflicts:
#	backend/ee-repo-ref.txt
#	cli/src/commands/sync/sync.ts
This commit is contained in:
Diego Imbert
2026-07-06 09:35:54 +02:00
parent 121e1a2906 a6c0b3756b
commit 896aae6794
257 changed files with 18318 additions and 2418 deletions
Download Patch File Download Diff File Expand all files Collapse all files
AGENTS.md
+2 -1
View File
@@ -24,7 +24,8 @@ Open-source platform for internal tools, workflows, API integrations, background
## Dev Environment
- **Backend**: `cargo run` from `backend/` (API at http://localhost:8000)
- **DuckDB local jobs**: before running DuckDB scripts locally, build the FFI shared library with `cd backend/windmill-duckdb-ffi-internal && ./build_dev.sh`. Re-run it after clean builds or when `backend/target/debug/libwindmill_duckdb_ffi_internal.*` is missing.
- **DuckDB local jobs**: before running DuckDB scripts locally, build the FFI shared library with `cd backend/windmill-duckdb-ffi-internal && ./build_dev.sh`. Re-run it after clean builds or when `backend/target/debug/libwindmill_duckdb_ffi_internal.*` is missing. The bundled DuckDB compile (~2min) is cached in a per-user dir shared across worktrees, so a fresh worktree reuses it and the build is near-instant.
- **Data pipelines (DuckLake) from source**: a plain `cargo run` (even `--features quickjs`) advertises a `duckdb` worker tag but **cannot** execute DuckDB scripts and has **no** working S3 proxy (DuckLake writes 404). Build CE DuckLake with `cargo run --features quickjs,duckdb,parquet,private` (add `,python` for Python scripts, `,enterprise,license` for EE) **and** build the FFI (bullet above). See `backend/CLAUDE.md` → "Running data pipelines (DuckLake) from source" for the exact feature sets and the two feature-gate gotchas.
- **Frontend**: `REMOTE=http://localhost:8000 npm run dev` from `frontend/` (port 3000+)
- **DB**: `psql postgres://postgres:changeme@localhost:5432/windmill`
- **Login**: `admin@windmill.dev` / `changeme`
CHANGELOG.md
+44
View File
@@ -1,5 +1,49 @@
# Changelog
## [1.749.0](https://github.com/windmill-labs/windmill/compare/v1.748.0...v1.749.0) (2026-07-05)
### Features
* **pipelines:** mid-DAG selective execution (dbt `model+`) for pipeline runs ([#9945](https://github.com/windmill-labs/windmill/issues/9945)) ([2d3a773](https://github.com/windmill-labs/windmill/commit/2d3a77344104a587548f23f1b614ceffd52a5778))
* **pipelines:** partition run-arg picker + first-run setup signpost ([#9943](https://github.com/windmill-labs/windmill/issues/9943)) ([475b072](https://github.com/windmill-labs/windmill/commit/475b072987b33d50111f5251a5f69f4245f930ae))
* **pipelines:** self-teaching custom data_test errors + scaffold ([#9937](https://github.com/windmill-labs/windmill/issues/9937)) ([0ad174f](https://github.com/windmill-labs/windmill/commit/0ad174fa490e17eeb26280b5bdfd62956dfed9ff))
### Bug Fixes
* **cli:** macro-library parity in --local pipeline graph + read-only run --dry-run ([#9942](https://github.com/windmill-labs/windmill/issues/9942)) ([e3f4303](https://github.com/windmill-labs/windmill/commit/e3f43033cafcdb5df253aeb55ce93e599b2584d2))
* **datatable:** self-teaching error for unresolved datatable:// references ([#9941](https://github.com/windmill-labs/windmill/issues/9941)) ([55451db](https://github.com/windmill-labs/windmill/commit/55451db009e3060c21948ece2c97e102a3c9b171))
* **object-storage:** remove 20-file bucket-browser listing cap in CE ([#9935](https://github.com/windmill-labs/windmill/issues/9935)) ([22452ce](https://github.com/windmill-labs/windmill/commit/22452ce54034a9bea8f7d48946818fd148b938c0))
* **pipelines:** dedup guard for keyed merge + deploy-time SCD2 validation ([#9936](https://github.com/windmill-labs/windmill/issues/9936)) ([52ce805](https://github.com/windmill-labs/windmill/commit/52ce805f619747af4f998cde7819a164c754205a))
* **pipelines:** link SCD2 <dim>_current view to its producer across all graph surfaces ([#9933](https://github.com/windmill-labs/windmill/issues/9933)) ([574d3ac](https://github.com/windmill-labs/windmill/commit/574d3ac9ff5015b5d3f53040c9d4dfbfd161a076))
* **pipelines:** order data_test relationships refs before the tested script in a cascade ([#9934](https://github.com/windmill-labs/windmill/issues/9934)) ([46be39d](https://github.com/windmill-labs/windmill/commit/46be39dfb7fbfb2b70e61819d6065b45810c41c9))
* **pipelines:** pipeline-level run control, tables label, data-test rollback + fork badges ([#9944](https://github.com/windmill-labs/windmill/issues/9944)) ([6ae8dd3](https://github.com/windmill-labs/windmill/commit/6ae8dd37b1de930ab17344cebf7c28385c6cfdba))
* rebuild windows bun loader main.ts filter from forward-slash cdir ([#9946](https://github.com/windmill-labs/windmill/issues/9946)) ([a582e04](https://github.com/windmill-labs/windmill/commit/a582e04bf40cf685f88bceaf88e3d24bde3d420a))
## [1.748.0](https://github.com/windmill-labs/windmill/compare/v1.747.0...v1.748.0) (2026-07-05)
### Features
* **ai-agent:** support reasoning effort in AI agent workflow steps ([#9886](https://github.com/windmill-labs/windmill/issues/9886)) ([a368d49](https://github.com/windmill-labs/windmill/commit/a368d49bd8786a2dca6771f2051f1d44d1b2363d))
* **ducklake:** scheduled lake maintenance (expiry, compaction, orphan cleanup) ([#9916](https://github.com/windmill-labs/windmill/issues/9916)) ([3352150](https://github.com/windmill-labs/windmill/commit/33521505dbc34f22b575d21fda1cc76d698a8840))
* **pipelines:** asset freshness — fresh/stale badge (CE) + watchdog (EE) ([#9909](https://github.com/windmill-labs/windmill/issues/9909)) ([5d7fb6d](https://github.com/windmill-labs/windmill/commit/5d7fb6deca3e02e89d77e5d3856483beb8b8bfeb))
* **pipelines:** capture violating-row samples for data tests ([#9919](https://github.com/windmill-labs/windmill/issues/9919)) ([d4b4374](https://github.com/windmill-labs/windmill/commit/d4b4374de8f8a7875b050c16d1236fcd0355812b))
* **pipelines:** fork data environments for ducklake materialization (dev data) ([#9915](https://github.com/windmill-labs/windmill/issues/9915)) ([39eb9de](https://github.com/windmill-labs/windmill/commit/39eb9de1bce400109c130a081807e40e995ae068))
* **pipelines:** on_schema_change write guardrails + data_test deploy validation ([#9930](https://github.com/windmill-labs/windmill/issues/9930)) ([377c02e](https://github.com/windmill-labs/windmill/commit/377c02ec47389e64b7ef5cbbae0de05df648266d))
* **pipelines:** record upstream snapshot ids on cascade-dispatched jobs ([#9910](https://github.com/windmill-labs/windmill/issues/9910)) ([af36498](https://github.com/windmill-labs/windmill/commit/af36498432e643108308e1c03b5d986d0f0f8888))
* **pipelines:** schema contracts — save-time consumer checks vs captured schemas ([#9917](https://github.com/windmill-labs/windmill/issues/9917)) ([42e11c6](https://github.com/windmill-labs/windmill/commit/42e11c6570b62ffaa86598438fa8ddf462c4035f))
* **pipeline:** write-audit-publish for materialization data tests ([#9911](https://github.com/windmill-labs/windmill/issues/9911)) ([dce247c](https://github.com/windmill-labs/windmill/commit/dce247c6d2678a2c95bd728027e17ae3965638e2))
* **sdk:** enforce s3:// URIs for string S3 params + ingestion (EL) docs ([#9912](https://github.com/windmill-labs/windmill/issues/9912)) ([5ad2de9](https://github.com/windmill-labs/windmill/commit/5ad2de91a26b312bf27124ceca16ef331621bde8))
### Bug Fixes
* **cli:** pipeline + workspace UX batch (init/bind stub, run errors, macro libs, lock-job report, upgrade errors) ([#9929](https://github.com/windmill-labs/windmill/issues/9929)) ([28a6b08](https://github.com/windmill-labs/windmill/commit/28a6b086c842105298f236baa0a61868f71a5eb1))
* **cli:** publish all windmill-parser-wasm-* deps so local pipeline graph keeps write edges ([#9926](https://github.com/windmill-labs/windmill/issues/9926)) ([744a759](https://github.com/windmill-labs/windmill/commit/744a7597edaf3ca9a7fd2b21a34fb33457913a64))
* **pipelines:** activity-axis label clarity + select failed node on cascade failure ([#9931](https://github.com/windmill-labs/windmill/issues/9931)) ([5769b60](https://github.com/windmill-labs/windmill/commit/5769b6036cf14b0cb424c5b3d9d878c600a5652e))
## [1.747.0](https://github.com/windmill-labs/windmill/compare/v1.746.0...v1.747.0) (2026-07-03)
backend/.sqlx/query-089d7bc7acdbb97cf477159e111bc7e9ee85289ff5c52af43166928337c257e7.json
Generated
+2 -1
View File
@@ -35,7 +35,8 @@
"ci_test",
"github",
"azure",
"asset"
"asset",
"freshness"
]
}
}
backend/.sqlx/query-0e22295e40bfc173d85f0a962d491f3afd87eeea5e4e8d935b1b86eb02f061e7.json
Generated
+15
View File
@@ -0,0 +1,15 @@
{
"db_name": "PostgreSQL",
"query": "DELETE FROM pipeline_freshness_state WHERE workspace_id = $1 AND script_path = $2",
"describe": {
"columns": [],
"parameters": {
"Left": [
"Text",
"Text"
]
},
"nullable": []
},
"hash": "0e22295e40bfc173d85f0a962d491f3afd87eeea5e4e8d935b1b86eb02f061e7"
}
backend/.sqlx/query-0f141ca6a58901dee1ddcf25694d76a4e5702c0324a27ef3a5740ef5d98d8946.json
Generated
+31
View File
@@ -0,0 +1,31 @@
{
"db_name": "PostgreSQL",
"query": "SELECT\n schedule.path, t.jobs FROM schedule,\n LATERAL(SELECT ARRAY(\n SELECT json_build_object('id', id, 'success', status = 'success', 'duration_ms', duration_ms)\n FROM v2_job_completed c JOIN v2_job j USING (id)\n WHERE trigger_kind = 'schedule'\n AND trigger = schedule.path\n AND c.workspace_id = $1\n AND j.workspace_id = $1\n AND parent_job IS NULL AND runnable_path = schedule.script_path\n AND status <> 'skipped'\n ORDER BY completed_at DESC\n LIMIT 20\n ) AS jobs) t\n WHERE workspace_id = $1 AND NOT starts_with(schedule.path, $4)\n ORDER BY edited_at DESC\n LIMIT $2 OFFSET $3",
"describe": {
"columns": [
{
"ordinal": 0,
"name": "path",
"type_info": "Varchar"
},
{
"ordinal": 1,
"name": "jobs",
"type_info": "JsonArray"
}
],
"parameters": {
"Left": [
"Text",
"Int8",
"Int8",
"Text"
]
},
"nullable": [
false,
null
]
},
"hash": "0f141ca6a58901dee1ddcf25694d76a4e5702c0324a27ef3a5740ef5d98d8946"
}
backend/.sqlx/query-142ae6939440654da21ebb06acf838563f28e832e37e5c165329441d8471acdb.json
Generated
+16
View File
@@ -0,0 +1,16 @@
{
"db_name": "PostgreSQL",
"query": "INSERT INTO schedule (\n workspace_id, path, edited_by, edited_at, schedule, enabled, script_path,\n args, extra_perms, is_flow, email, error, timezone, on_failure,\n on_recovery, on_failure_times, on_failure_exact, on_failure_extra_args,\n on_recovery_times, on_recovery_extra_args, ws_error_handler_muted, retry,\n summary, no_flow_overlap, tag, paused_until, on_success, on_success_extra_args,\n cron_version, description, dynamic_skip, permissioned_as, labels\n )\n SELECT\n $1, path, edited_by, edited_at, schedule, FALSE, script_path,\n args, extra_perms, is_flow, email, error, timezone, on_failure,\n on_recovery, on_failure_times, on_failure_exact, on_failure_extra_args,\n on_recovery_times, on_recovery_extra_args, ws_error_handler_muted, retry,\n summary, no_flow_overlap, tag, paused_until, on_success, on_success_extra_args,\n cron_version, description, dynamic_skip, permissioned_as, labels\n FROM schedule WHERE workspace_id = $2 AND path NOT LIKE $3",
"describe": {
"columns": [],
"parameters": {
"Left": [
"Varchar",
"Text",
"Text"
]
},
"nullable": []
},
"hash": "142ae6939440654da21ebb06acf838563f28e832e37e5c165329441d8471acdb"
}
backend/.sqlx/query-212553c83e4dcdc6d045eb2fe2dadbb2860ce52d37a56b2861de1215260ecff8.json
Generated
+4 -2
View File
@@ -39,7 +39,8 @@
"ci_test",
"github",
"azure",
"asset"
"asset",
"freshness"
]
}
}
@@ -77,7 +78,8 @@
"ci_test",
"github",
"azure",
"asset"
"asset",
"freshness"
]
}
}
backend/.sqlx/query-256a302afef987857016f2fa636d15474599734bc46ede6930650e2cba5d8a46.json
Generated
+15
View File
@@ -0,0 +1,15 @@
{
"db_name": "PostgreSQL",
"query": "INSERT INTO script_trigger (workspace_id, runnable_kind, runnable_path, trigger_kind, trigger_ref, join_all, debounce_s, retry_count, retry_delay_s)\n SELECT $2, runnable_kind, runnable_path, trigger_kind, trigger_ref, join_all, debounce_s, retry_count, retry_delay_s\n FROM script_trigger WHERE workspace_id = $1",
"describe": {
"columns": [],
"parameters": {
"Left": [
"Text",
"Varchar"
]
},
"nullable": []
},
"hash": "256a302afef987857016f2fa636d15474599734bc46ede6930650e2cba5d8a46"
}
backend/.sqlx/query-28ec31481bdaae8b0512d21d074da5d913caa8d935b33ad8f21336dedefd1c23.json
Generated
+15
View File
@@ -0,0 +1,15 @@
{
"db_name": "PostgreSQL",
"query": "DELETE FROM schedule WHERE workspace_id = $1 AND path = ANY($2)",
"describe": {
"columns": [],
"parameters": {
"Left": [
"Text",
"TextArray"
]
},
"nullable": []
},
"hash": "28ec31481bdaae8b0512d21d074da5d913caa8d935b33ad8f21336dedefd1c23"
}
backend/.sqlx/query-333a703a53cc760842b047ad420efef25885a9bca64bb2ba4bb173bc19dcbdba.json
Generated
+23
View File
@@ -0,0 +1,23 @@
{
"db_name": "PostgreSQL",
"query": "SELECT path FROM schedule WHERE workspace_id = $1 AND path LIKE $2",
"describe": {
"columns": [
{
"ordinal": 0,
"name": "path",
"type_info": "Varchar"
}
],
"parameters": {
"Left": [
"Text",
"Text"
]
},
"nullable": [
false
]
},
"hash": "333a703a53cc760842b047ad420efef25885a9bca64bb2ba4bb173bc19dcbdba"
}
backend/.sqlx/query-36da136cf9b9554702e3b3ec364368a6862e35350ae43383fb2335e2c82a9419.json
Generated
+29
View File
@@ -0,0 +1,29 @@
{
"db_name": "PostgreSQL",
"query": "SELECT DISTINCT path AS \"asset_path!\", usage_path AS \"producer_path!\"\n FROM asset\n WHERE workspace_id = $1 AND kind = 'ducklake' AND path = ANY($2)\n AND usage_kind = 'script' AND usage_access_type IN ('w', 'rw')",
"describe": {
"columns": [
{
"ordinal": 0,
"name": "asset_path!",
"type_info": "Varchar"
},
{
"ordinal": 1,
"name": "producer_path!",
"type_info": "Varchar"
}
],
"parameters": {
"Left": [
"Text",
"TextArray"
]
},
"nullable": [
false,
false
]
},
"hash": "36da136cf9b9554702e3b3ec364368a6862e35350ae43383fb2335e2c82a9419"
}
backend/.sqlx/query-3c84781704b84b8a927ecce5a3fcb3adcf0175d0a71013f2497397c1c8ccc619.json
Generated
+2 -1
View File
@@ -128,7 +128,8 @@
"ci_test",
"github",
"azure",
"asset"
"asset",
"freshness"
]
}
}
backend/.sqlx/query-3ea82f16050318a0b3b87c12beb66b49254036ca93f7a6415ceb4e316ed64eed.json
Generated
+29
View File
@@ -0,0 +1,29 @@
{
"db_name": "PostgreSQL",
"query": "\n SELECT p.path AS \"path!\",\n (SELECT c.completed_at\n FROM v2_job j\n JOIN v2_job_completed c ON c.id = j.id\n WHERE j.workspace_id = $1\n AND j.runnable_path = p.path\n AND j.parent_job IS NULL\n -- No 'singlestepflow': flows may share a script's path, and\n -- a same-path flow run must not read as the script being\n -- fresh (false-fresh). Script retries land as native\n -- 'script' jobs; only the rare flow-wrapper fallback is\n -- missed, which errs stale. Kept in lockstep with the\n -- freshness watchdog's queries (freshness_watchdog_ee).\n AND j.kind IN ('script', 'preview')\n AND c.status = 'success'\n ORDER BY j.created_at DESC\n LIMIT 1) AS last_success_at\n FROM unnest($2::text[]) AS p(path)\n ",
"describe": {
"columns": [
{
"ordinal": 0,
"name": "path!",
"type_info": "Text"
},
{
"ordinal": 1,
"name": "last_success_at",
"type_info": "Timestamptz"
}
],
"parameters": {
"Left": [
"Text",
"TextArray"
]
},
"nullable": [
null,
null
]
},
"hash": "3ea82f16050318a0b3b87c12beb66b49254036ca93f7a6415ceb4e316ed64eed"
}
backend/.sqlx/query-3eeecc4bb26163cf6c2c19bc7c2950a1563bd516e730aaa9d0f1501ea8cea7f9.json
Generated
+58
View File
@@ -0,0 +1,58 @@
{
"db_name": "PostgreSQL",
"query": "SELECT ducklake_name AS \"ducklake_name!\", metadata_schema AS \"metadata_schema!\",\n catalog AS \"catalog!\", storage AS \"storage!\",\n storage_ref AS \"storage_ref!\", data_path AS \"data_path!\",\n schema_dropped AS \"schema_dropped!\"\n FROM fork_ducklake_namespace WHERE workspace_id = $1",
"describe": {
"columns": [
{
"ordinal": 0,
"name": "ducklake_name!",
"type_info": "Varchar"
},
{
"ordinal": 1,
"name": "metadata_schema!",
"type_info": "Varchar"
},
{
"ordinal": 2,
"name": "catalog!",
"type_info": "Text"
},
{
"ordinal": 3,
"name": "storage!",
"type_info": "Text"
},
{
"ordinal": 4,
"name": "storage_ref!",
"type_info": "Text"
},
{
"ordinal": 5,
"name": "data_path!",
"type_info": "Text"
},
{
"ordinal": 6,
"name": "schema_dropped!",
"type_info": "Bool"
}
],
"parameters": {
"Left": [
"Text"
]
},
"nullable": [
false,
false,
false,
false,
false,
false,
false
]
},
"hash": "3eeecc4bb26163cf6c2c19bc7c2950a1563bd516e730aaa9d0f1501ea8cea7f9"
}
backend/.sqlx/query-431a5eb2eb5a0b57e096cc78f68eb5a0665c90c4926f1afb689d6cd16bd5b722.json
Generated
+29
View File
@@ -0,0 +1,29 @@
{
"db_name": "PostgreSQL",
"query": "SELECT DISTINCT ON (path) path, content\n FROM script\n WHERE workspace_id = $1 AND path = ANY($2)\n AND archived = false AND deleted = false\n ORDER BY path, created_at DESC",
"describe": {
"columns": [
{
"ordinal": 0,
"name": "path",
"type_info": "Varchar"
},
{
"ordinal": 1,
"name": "content",
"type_info": "Text"
}
],
"parameters": {
"Left": [
"Text",
"TextArray"
]
},
"nullable": [
false,
false
]
},
"hash": "431a5eb2eb5a0b57e096cc78f68eb5a0665c90c4926f1afb689d6cd16bd5b722"
}
backend/.sqlx/query-45a539e61a3c35098fb6d59c3d5de8b6fcebe83002c74647666e3a6f5b8490dd.json
Generated
+23
View File
@@ -0,0 +1,23 @@
{
"db_name": "PostgreSQL",
"query": "\n SELECT EXISTS(\n SELECT 1 FROM v2_job_queue q\n JOIN v2_job j ON j.id = q.id\n WHERE q.workspace_id = $1\n AND j.runnable_path = $2\n AND j.parent_job IS NULL\n AND j.kind IN ('script', 'preview')\n AND (q.running = true OR q.scheduled_for <= now())\n ) AS \"in_flight!\"\n ",
"describe": {
"columns": [
{
"ordinal": 0,
"name": "in_flight!",
"type_info": "Bool"
}
],
"parameters": {
"Left": [
"Text",
"Text"
]
},
"nullable": [
null
]
},
"hash": "45a539e61a3c35098fb6d59c3d5de8b6fcebe83002c74647666e3a6f5b8490dd"
}
backend/.sqlx/query-45c73bd8742601753ad947c88b65ff3ae182ff7c0631d4fa44b67b11a133498f.json
Generated
+17
View File
@@ -0,0 +1,17 @@
{
"db_name": "PostgreSQL",
"query": "\n UPDATE pipeline_freshness_state\n SET attempts = attempts + 1,\n last_push_at = now(),\n next_attempt_at = now()\n + (LEAST($3::bigint, $4::bigint * (1::bigint << LEAST(attempts + 1, 20)))::text\n || ' seconds')::interval\n WHERE workspace_id = $1 AND script_path = $2 AND next_attempt_at <= now()\n ",
"describe": {
"columns": [],
"parameters": {
"Left": [
"Text",
"Text",
"Int8",
"Int8"
]
},
"nullable": []
},
"hash": "45c73bd8742601753ad947c88b65ff3ae182ff7c0631d4fa44b67b11a133498f"
}
backend/.sqlx/query-4a43d4df6c5b2e8dda4308dcb88c23caf312ec377dd91e5307f00d3fb8ec325d.json
Generated
+2 -1
View File
@@ -80,7 +80,8 @@
"ci_test",
"github",
"azure",
"asset"
"asset",
"freshness"
]
}
}
backend/.sqlx/query-5323858a017814206179d35fdb708c547742851975163550ec4f3823057e91fe.json
Generated
+20
View File
@@ -0,0 +1,20 @@
{
"db_name": "PostgreSQL",
"query": "INSERT INTO schedule (\n workspace_id, path, schedule, timezone, edited_by, script_path,\n is_flow, enabled, email, permissioned_as, summary, tag, cron_version\n ) VALUES ($1, $2, $3, 'Etc/UTC', $4, $2, false, true, $5, $6, $7, 'duckdb', 'v2')\n ON CONFLICT (workspace_id, path) DO UPDATE SET\n schedule = EXCLUDED.schedule,\n timezone = EXCLUDED.timezone,\n edited_by = EXCLUDED.edited_by,\n edited_at = now(),\n script_path = EXCLUDED.script_path,\n is_flow = false,\n enabled = true,\n email = EXCLUDED.email,\n permissioned_as = EXCLUDED.permissioned_as,\n summary = EXCLUDED.summary,\n tag = EXCLUDED.tag,\n cron_version = EXCLUDED.cron_version,\n error = NULL",
"describe": {
"columns": [],
"parameters": {
"Left": [
"Varchar",
"Varchar",
"Varchar",
"Varchar",
"Varchar",
"Varchar",
"Varchar"
]
},
"nullable": []
},
"hash": "5323858a017814206179d35fdb708c547742851975163550ec4f3823057e91fe"
}
backend/.sqlx/query-53a1ec83aedcec2cdf6495637464e6219434de4d2f9ae5e484ed8ce5da426109.json
Generated
+16
View File
@@ -0,0 +1,16 @@
{
"db_name": "PostgreSQL",
"query": "DELETE FROM schedule WHERE workspace_id = $1 AND path = ANY($2) AND path != ALL($3)",
"describe": {
"columns": [],
"parameters": {
"Left": [
"Text",
"TextArray",
"TextArray"
]
},
"nullable": []
},
"hash": "53a1ec83aedcec2cdf6495637464e6219434de4d2f9ae5e484ed8ce5da426109"
}
backend/.sqlx/query-572b80f7b16fd5d1a05f8fe9bbb07667fc1b5e802dc2465ffd13178a042e4f89.json
Generated
+24
View File
@@ -0,0 +1,24 @@
{
"db_name": "PostgreSQL",
"query": "\n SELECT EXISTS(\n SELECT 1 FROM v2_job j\n JOIN v2_job_completed c ON c.id = j.id\n WHERE j.workspace_id = $1\n AND j.runnable_path = $2\n AND j.parent_job IS NULL\n AND j.kind IN ('script', 'preview')\n AND c.status = 'success'\n AND c.completed_at > now() - ($3::bigint::text || ' seconds')::interval\n ) AS \"fresh!\"\n ",
"describe": {
"columns": [
{
"ordinal": 0,
"name": "fresh!",
"type_info": "Bool"
}
],
"parameters": {
"Left": [
"Text",
"Text",
"Int8"
]
},
"nullable": [
null
]
},
"hash": "572b80f7b16fd5d1a05f8fe9bbb07667fc1b5e802dc2465ffd13178a042e4f89"
}
backend/.sqlx/query-5e0569092c559a1b19f636fae68f6d740a1720f7746c01f9ea693c70271ff55c.json
Generated
+17
View File
@@ -0,0 +1,17 @@
{
"db_name": "PostgreSQL",
"query": "\n INSERT INTO pipeline_freshness_state\n (workspace_id, script_path, attempts, last_push_at, next_attempt_at)\n VALUES ($1, $2, 1, now(),\n now() + (LEAST($3::bigint, $4::bigint * 2)::text || ' seconds')::interval)\n ON CONFLICT (workspace_id, script_path) DO NOTHING\n ",
"describe": {
"columns": [],
"parameters": {
"Left": [
"Varchar",
"Varchar",
"Int8",
"Int8"
]
},
"nullable": []
},
"hash": "5e0569092c559a1b19f636fae68f6d740a1720f7746c01f9ea693c70271ff55c"
}
backend/.sqlx/query-63b0f39e3ea2b8d01d4c055258b6fc3dc4e9e1f237ba7c86cbdd70140bc55541.json
Generated
+36
View File
@@ -0,0 +1,36 @@
{
"db_name": "PostgreSQL",
"query": "\n WITH ancestor AS (\n SELECT wid, ord FROM unnest($1::text[]) WITH ORDINALITY AS a(wid, ord)\n ), anc_mat AS (\n -- Per table, the nearest ancestor (lowest ord) that materialized it.\n SELECT DISTINCT ON (mp.asset_path) mp.asset_path, a.wid, a.ord\n FROM materialized_partition mp\n JOIN ancestor a ON a.wid = mp.workspace_id\n WHERE mp.asset_kind = 'ducklake' AND mp.status = 'materialized'\n AND split_part(mp.asset_path, '/', 1) = $3 AND mp.asset_path LIKE '%/%'\n ORDER BY mp.asset_path, a.ord\n ), fork_mat AS (\n -- Fork-OWNED assets: anything whose physical table exists in the fork\n -- namespace, not just clean materializations. A committed write whose data\n -- tests failed afterwards records status='failed' WITH a snapshot — its table\n -- is real, and a defer view emitted over it would silently yield to it\n -- (CREATE VIEW IF NOT EXISTS) while claiming the read defers to the parent.\n SELECT DISTINCT asset_path FROM materialized_partition\n WHERE workspace_id = $2 AND asset_kind = 'ducklake'\n AND (status = 'materialized' OR snapshot_id IS NOT NULL)\n ), latest_schema AS (\n SELECT DISTINCT ON (workspace_id, asset_path) workspace_id, asset_path, columns\n FROM materialized_asset_schema\n WHERE workspace_id = ANY($1) AND asset_kind = 'ducklake'\n ORDER BY workspace_id, asset_path, version DESC\n )\n SELECT am.asset_path AS \"asset_path!\",\n am.ord AS \"ord!\",\n COALESCE(EXISTS (\n SELECT 1 FROM jsonb_array_elements(ls.columns) e\n WHERE e->>'name' = 'is_current'\n ), false) AS \"has_current!\"\n FROM anc_mat am\n LEFT JOIN latest_schema ls\n ON ls.asset_path = am.asset_path AND ls.workspace_id = am.wid\n WHERE am.asset_path NOT IN (SELECT asset_path FROM fork_mat)\n ORDER BY am.asset_path\n ",
"describe": {
"columns": [
{
"ordinal": 0,
"name": "asset_path!",
"type_info": "Varchar"
},
{
"ordinal": 1,
"name": "ord!",
"type_info": "Int8"
},
{
"ordinal": 2,
"name": "has_current!",
"type_info": "Bool"
}
],
"parameters": {
"Left": [
"TextArray",
"Text",
"Text"
]
},
"nullable": [
false,
null,
null
]
},
"hash": "63b0f39e3ea2b8d01d4c055258b6fc3dc4e9e1f237ba7c86cbdd70140bc55541"
}
backend/.sqlx/query-67e25a7c19ea0ffaf7ea5303fcd04af5a7eb488c76f783e690af0c2153b1d6a8.json
Generated
+2 -1
View File
@@ -161,7 +161,8 @@
"ci_test",
"github",
"azure",
"asset"
"asset",
"freshness"
]
}
}
backend/.sqlx/query-6ca5cbe50df89e6eead890758e34a6ae5091c37ec901975c3d564fd817a405b7.json
Generated
+23
View File
@@ -0,0 +1,23 @@
{
"db_name": "PostgreSQL",
"query": "UPDATE workspace_settings\n SET ducklake = jsonb_set(ducklake, ARRAY['ducklakes', $2, 'fork_behavior'], '\"shared\"')\n WHERE workspace_id = $1 AND ducklake->'ducklakes' ? $2\n RETURNING 1 AS \"one!\"",
"describe": {
"columns": [
{
"ordinal": 0,
"name": "one!",
"type_info": "Int4"
}
],
"parameters": {
"Left": [
"Text",
"Text"
]
},
"nullable": [
null
]
},
"hash": "6ca5cbe50df89e6eead890758e34a6ae5091c37ec901975c3d564fd817a405b7"
}
backend/.sqlx/query-6d81570849980f316b08487a03f6017386cf2767c953e0d152e7983a00218c6f.json
Generated
+15
View File
@@ -0,0 +1,15 @@
{
"db_name": "PostgreSQL",
"query": "UPDATE pipeline_freshness_state SET next_attempt_at = now() - interval '1 second'\n WHERE workspace_id = $1 AND script_path = $2",
"describe": {
"columns": [],
"parameters": {
"Left": [
"Text",
"Text"
]
},
"nullable": []
},
"hash": "6d81570849980f316b08487a03f6017386cf2767c953e0d152e7983a00218c6f"
}
backend/.sqlx/query-76af9b978fccef61f50a5eab335d4e427342fb78c66980984aefa7b8c7592ecb.json
Generated
+22
View File
@@ -0,0 +1,22 @@
{
"db_name": "PostgreSQL",
"query": "SELECT ducklake->'ducklakes' FROM workspace_settings WHERE workspace_id = $1",
"describe": {
"columns": [
{
"ordinal": 0,
"name": "?column?",
"type_info": "Jsonb"
}
],
"parameters": {
"Left": [
"Text"
]
},
"nullable": [
null
]
},
"hash": "76af9b978fccef61f50a5eab335d4e427342fb78c66980984aefa7b8c7592ecb"
}
backend/.sqlx/query-7cd0b27f4b4a20b05ac876861b934128c3a6ade024d6662379857905a7447a2a.json
Generated
+32
View File
@@ -0,0 +1,32 @@
{
"db_name": "PostgreSQL",
"query": "\n SELECT DISTINCT ON (workspace_id, path)\n workspace_id AS \"workspace_id!\", path AS \"path!\", content AS \"content!\"\n FROM script\n WHERE auto_kind = 'pipeline'\n AND archived = false\n AND deleted = false\n AND content ILIKE '%freshness%'\n -- Workspace archival stops all execution but leaves script rows\n -- intact for unarchival; without this the watchdog would keep\n -- resurrecting runs in a workspace the admin shut down.\n AND EXISTS (SELECT 1 FROM workspace w\n WHERE w.id = script.workspace_id AND w.deleted = false)\n ORDER BY workspace_id, path, created_at DESC\n ",
"describe": {
"columns": [
{
"ordinal": 0,
"name": "workspace_id!",
"type_info": "Varchar"
},
{
"ordinal": 1,
"name": "path!",
"type_info": "Varchar"
},
{
"ordinal": 2,
"name": "content!",
"type_info": "Text"
}
],
"parameters": {
"Left": []
},
"nullable": [
false,
false,
false
]
},
"hash": "7cd0b27f4b4a20b05ac876861b934128c3a6ade024d6662379857905a7447a2a"
}
backend/.sqlx/query-81d1c56c369c5328d947f1ad21da995a8a555e4a9c2d1387dd8cee9c81efd40b.json
Generated
+15
View File
@@ -0,0 +1,15 @@
{
"db_name": "PostgreSQL",
"query": "\n DELETE FROM pipeline_freshness_state s\n WHERE NOT EXISTS (\n SELECT 1 FROM unnest($1::text[], $2::text[]) AS w(workspace_id, script_path)\n WHERE w.workspace_id = s.workspace_id AND w.script_path = s.script_path\n )\n ",
"describe": {
"columns": [],
"parameters": {
"Left": [
"TextArray",
"TextArray"
]
},
"nullable": []
},
"hash": "81d1c56c369c5328d947f1ad21da995a8a555e4a9c2d1387dd8cee9c81efd40b"
}
backend/.sqlx/query-5e50ba0ae27b09a3ea1530c223e5039aa631bb5b0993ad09bc9a1381f6715f19.json → backend/.sqlx/query-853230da671371fd8bbeeb4c87f8c20a9e310092ae787efe4b0657399c799424.json
Generated
+2 -2
View File
@@ -1,6 +1,6 @@
{
"db_name": "PostgreSQL",
"query": "INSERT INTO materialized_partition\n (workspace_id, asset_kind, asset_path, partition, status,\n snapshot_id, row_count, job_id, materialized_at, error)\n VALUES ($1, $2, $3, $4, $5, $6, $7, $8, now(), $9)\n ON CONFLICT (workspace_id, asset_kind, asset_path, partition)\n DO UPDATE SET status = EXCLUDED.status,\n snapshot_id = EXCLUDED.snapshot_id,\n row_count = EXCLUDED.row_count,\n job_id = EXCLUDED.job_id,\n materialized_at = now(),\n error = EXCLUDED.error",
"query": "INSERT INTO materialized_partition\n (workspace_id, asset_kind, asset_path, partition, status,\n snapshot_id, row_count, job_id, materialized_at, error)\n VALUES ($1, $2, $3, $4, $5, $6, $7, $8, now(), $9)\n ON CONFLICT (workspace_id, asset_kind, asset_path, partition)\n DO UPDATE SET status = EXCLUDED.status,\n -- A failed run records no snapshot, but must not erase the last\n -- committed one: a physical table from an earlier commit (or from a\n -- committed write whose data tests then failed) still exists, and\n -- fork defer/graph state keys on that evidence.\n snapshot_id = COALESCE(EXCLUDED.snapshot_id, materialized_partition.snapshot_id),\n row_count = EXCLUDED.row_count,\n job_id = EXCLUDED.job_id,\n materialized_at = now(),\n error = EXCLUDED.error",
"describe": {
"columns": [],
"parameters": {
@@ -43,5 +43,5 @@
},
"nullable": []
},
"hash": "5e50ba0ae27b09a3ea1530c223e5039aa631bb5b0993ad09bc9a1381f6715f19"
"hash": "853230da671371fd8bbeeb4c87f8c20a9e310092ae787efe4b0657399c799424"
}
backend/.sqlx/query-8756a8100d6fd6ecb64ef73d6f2c8453d53c973dce61cc8ace7d278773c7506f.json
Generated
+31
View File
@@ -0,0 +1,31 @@
{
"db_name": "PostgreSQL",
"query": "SELECT\n schedule.path, t.jobs FROM schedule,\n LATERAL(SELECT ARRAY(\n SELECT json_build_object('id', id, 'success', status = 'success', 'duration_ms', duration_ms)\n FROM v2_job_completed c JOIN v2_job j USING (id)\n WHERE trigger_kind = 'schedule'\n AND trigger = schedule.path\n AND c.workspace_id = $1\n AND j.workspace_id = $1\n AND parent_job IS NULL AND runnable_path = schedule.script_path\n AND status <> 'skipped'\n ORDER BY completed_at DESC\n LIMIT 20\n ) AS jobs) t\n WHERE workspace_id = $1 AND schedule.path NOT LIKE $4\n ORDER BY edited_at DESC\n LIMIT $2 OFFSET $3",
"describe": {
"columns": [
{
"ordinal": 0,
"name": "path",
"type_info": "Varchar"
},
{
"ordinal": 1,
"name": "jobs",
"type_info": "JsonArray"
}
],
"parameters": {
"Left": [
"Text",
"Int8",
"Int8",
"Text"
]
},
"nullable": [
false,
null
]
},
"hash": "8756a8100d6fd6ecb64ef73d6f2c8453d53c973dce61cc8ace7d278773c7506f"
}
backend/.sqlx/query-8837a126698d924eb9014cfee0e6afc85a8350706fe694935b25ef46a847409e.json
Generated
+22
View File
@@ -0,0 +1,22 @@
{
"db_name": "PostgreSQL",
"query": "\n SELECT ws.ducklake->'ducklakes' AS ducklake_name\n FROM workspace_settings ws\n WHERE ws.workspace_id = $1\n ",
"describe": {
"columns": [
{
"ordinal": 0,
"name": "ducklake_name",
"type_info": "Jsonb"
}
],
"parameters": {
"Left": [
"Text"
]
},
"nullable": [
null
]
},
"hash": "8837a126698d924eb9014cfee0e6afc85a8350706fe694935b25ef46a847409e"
}
backend/.sqlx/query-8a1e2e59c05b0f67e32c517a0c46e1a9307335b0e7a27613cd690e99812a99f3.json
Generated
+23
View File
@@ -0,0 +1,23 @@
{
"db_name": "PostgreSQL",
"query": "SELECT path FROM schedule WHERE workspace_id = $1 AND starts_with(path, $2)",
"describe": {
"columns": [
{
"ordinal": 0,
"name": "path",
"type_info": "Varchar"
}
],
"parameters": {
"Left": [
"Text",
"Text"
]
},
"nullable": [
false
]
},
"hash": "8a1e2e59c05b0f67e32c517a0c46e1a9307335b0e7a27613cd690e99812a99f3"
}
backend/.sqlx/query-8dec884a4c3b2d105afe22d1d7a01fcab837629fd3c1a09cf82b21641a890444.json
Generated
+29
View File
@@ -0,0 +1,29 @@
{
"db_name": "PostgreSQL",
"query": "\n -- Fork rows also count when a snapshot ever committed (a failed run\n -- preserves it): the physical table exists, so reads hit the FORK's\n -- data — showing 'deferred' would misstate what a query returns.\n -- Ancestor rows still require a clean materialization.\n SELECT DISTINCT asset_path AS \"asset_path!\", workspace_id AS \"workspace_id!\"\n FROM materialized_partition\n WHERE (workspace_id = $1 OR workspace_id = ANY($2))\n AND asset_kind = 'ducklake'\n AND (status = 'materialized'\n OR (workspace_id = $1 AND snapshot_id IS NOT NULL))\n ",
"describe": {
"columns": [
{
"ordinal": 0,
"name": "asset_path!",
"type_info": "Varchar"
},
{
"ordinal": 1,
"name": "workspace_id!",
"type_info": "Varchar"
}
],
"parameters": {
"Left": [
"Text",
"TextArray"
]
},
"nullable": [
false,
false
]
},
"hash": "8dec884a4c3b2d105afe22d1d7a01fcab837629fd3c1a09cf82b21641a890444"
}
backend/.sqlx/query-9ecb404e46a4eac55f977f05a3afbafe5dc3cdecc17a3d5a7476b160c1b6e7e1.json
Generated
+2 -1
View File
@@ -35,7 +35,8 @@
"ci_test",
"github",
"azure",
"asset"
"asset",
"freshness"
]
}
}
backend/.sqlx/query-a0faa5b042bac00bcd09b925bcf548e2783c2de7e92ba0ff7e6f4637419fcf7d.json
Generated
+16
View File
@@ -0,0 +1,16 @@
{
"db_name": "PostgreSQL",
"query": "DELETE FROM schedule WHERE workspace_id = $1 AND path LIKE $2 AND path != ALL($3)",
"describe": {
"columns": [],
"parameters": {
"Left": [
"Text",
"Text",
"TextArray"
]
},
"nullable": []
},
"hash": "a0faa5b042bac00bcd09b925bcf548e2783c2de7e92ba0ff7e6f4637419fcf7d"
}
backend/.sqlx/query-a4b6371d33206010b2f3ffd2b09e33244fe8ab9a803248fc23f334034d24aad4.json
Generated
+2 -1
View File
@@ -191,7 +191,8 @@
"ci_test",
"github",
"azure",
"asset"
"asset",
"freshness"
]
}
}
backend/.sqlx/query-a7f0ed0081e98a9861ea4b081b8a14576509147e51a86a95a2cc786bee3199d9.json
Generated
+16
View File
@@ -0,0 +1,16 @@
{
"db_name": "PostgreSQL",
"query": "DELETE FROM schedule WHERE workspace_id = $1 AND starts_with(path, $2) AND path != ALL($3)",
"describe": {
"columns": [],
"parameters": {
"Left": [
"Text",
"Text",
"TextArray"
]
},
"nullable": []
},
"hash": "a7f0ed0081e98a9861ea4b081b8a14576509147e51a86a95a2cc786bee3199d9"
}
backend/.sqlx/query-b10e21ec9edf2d22ff956fdcd085bd601b7dcabdf70e85543455cb4fc3986d74.json
Generated
+23
View File
@@ -0,0 +1,23 @@
{
"db_name": "PostgreSQL",
"query": "SELECT attempts FROM pipeline_freshness_state WHERE workspace_id = $1 AND script_path = $2",
"describe": {
"columns": [
{
"ordinal": 0,
"name": "attempts",
"type_info": "Int4"
}
],
"parameters": {
"Left": [
"Text",
"Text"
]
},
"nullable": [
false
]
},
"hash": "b10e21ec9edf2d22ff956fdcd085bd601b7dcabdf70e85543455cb4fc3986d74"
}
backend/.sqlx/query-b3771b690c5966272b1f42c9965bb6a8f961c119516e4c33dc928cd3b4f4edbc.json
Generated
+2 -1
View File
@@ -166,7 +166,8 @@
"ci_test",
"github",
"azure",
"asset"
"asset",
"freshness"
]
}
}
backend/.sqlx/query-bc6f566bc958923045dc3e1ff1639a26f649c960a4ee8d0a584884a236129d43.json
Generated
+15
View File
@@ -0,0 +1,15 @@
{
"db_name": "PostgreSQL",
"query": "INSERT INTO pipeline_freshness_state (workspace_id, script_path, attempts) VALUES ($1, $2, 3)",
"describe": {
"columns": [],
"parameters": {
"Left": [
"Varchar",
"Varchar"
]
},
"nullable": []
},
"hash": "bc6f566bc958923045dc3e1ff1639a26f649c960a4ee8d0a584884a236129d43"
}
backend/.sqlx/query-bcfa34cf80abea05f0c24883b9e77429c51e6166c414bcc5ce2e97fac25bcd77.json
Generated
+2 -1
View File
@@ -80,7 +80,8 @@
"ci_test",
"github",
"azure",
"asset"
"asset",
"freshness"
]
}
}
backend/.sqlx/query-be6d2c92a62b7b284651c45af809746147aa9b8d0a81642a7b7cb4738a0cad66.json
Generated
+2 -1
View File
@@ -111,7 +111,8 @@
"ci_test",
"github",
"azure",
"asset"
"asset",
"freshness"
]
}
}
backend/.sqlx/query-c6c186eb3a108e1f695bf5730c23ef4ab1ccfdc4ed33904e83e5130ac0bf5896.json
Generated
+41
View File
@@ -0,0 +1,41 @@
{
"db_name": "PostgreSQL",
"query": "SELECT DISTINCT ON (asset_path)\n asset_path, version, columns AS \"columns: Json<Vec<SchemaColumn>>\", captured_at\n FROM materialized_asset_schema\n WHERE workspace_id = $1 AND asset_kind = 'ducklake' AND asset_path = ANY($2)\n ORDER BY asset_path, version DESC",
"describe": {
"columns": [
{
"ordinal": 0,
"name": "asset_path",
"type_info": "Varchar"
},
{
"ordinal": 1,
"name": "version",
"type_info": "Int8"
},
{
"ordinal": 2,
"name": "columns: Json<Vec<SchemaColumn>>",
"type_info": "Jsonb"
},
{
"ordinal": 3,
"name": "captured_at",
"type_info": "Timestamptz"
}
],
"parameters": {
"Left": [
"Text",
"TextArray"
]
},
"nullable": [
false,
false,
false,
false
]
},
"hash": "c6c186eb3a108e1f695bf5730c23ef4ab1ccfdc4ed33904e83e5130ac0bf5896"
}
backend/.sqlx/query-c832ebc7ff0be446bf36628f6e06f59d0fbe8df62e0bd0be636d7cf70dcf95f0.json
Generated
+16
View File
@@ -0,0 +1,16 @@
{
"db_name": "PostgreSQL",
"query": "INSERT INTO schedule (\n workspace_id, path, edited_by, edited_at, schedule, enabled, script_path,\n args, extra_perms, is_flow, email, error, timezone, on_failure,\n on_recovery, on_failure_times, on_failure_exact, on_failure_extra_args,\n on_recovery_times, on_recovery_extra_args, ws_error_handler_muted, retry,\n summary, no_flow_overlap, tag, paused_until, on_success, on_success_extra_args,\n cron_version, description, dynamic_skip, permissioned_as, labels\n )\n SELECT\n $1, path, edited_by, edited_at, schedule, FALSE, script_path,\n args, extra_perms, is_flow, email, error, timezone, on_failure,\n on_recovery, on_failure_times, on_failure_exact, on_failure_extra_args,\n on_recovery_times, on_recovery_extra_args, ws_error_handler_muted, retry,\n summary, no_flow_overlap, tag, paused_until, on_success, on_success_extra_args,\n cron_version, description, dynamic_skip, permissioned_as, labels\n FROM schedule WHERE workspace_id = $2 AND NOT starts_with(path, $3)",
"describe": {
"columns": [],
"parameters": {
"Left": [
"Varchar",
"Text",
"Text"
]
},
"nullable": []
},
"hash": "c832ebc7ff0be446bf36628f6e06f59d0fbe8df62e0bd0be636d7cf70dcf95f0"
}
backend/.sqlx/query-cbbc6a894b6421d2cd49e59b7ed650fb861aa19a009d6dfe9937304057f2b91f.json
Generated
+22
View File
@@ -0,0 +1,22 @@
{
"db_name": "PostgreSQL",
"query": "\n WITH RECURSIVE chain AS (\n SELECT id, parent_workspace_id, 0 AS depth\n FROM workspace WHERE id = $1\n UNION ALL\n SELECT w.id, w.parent_workspace_id, chain.depth + 1\n FROM workspace w\n JOIN chain ON w.id = chain.parent_workspace_id\n WHERE chain.depth < 20\n )\n SELECT id AS \"id!\" FROM chain WHERE depth > 0 ORDER BY depth\n ",
"describe": {
"columns": [
{
"ordinal": 0,
"name": "id!",
"type_info": "Varchar"
}
],
"parameters": {
"Left": [
"Text"
]
},
"nullable": [
null
]
},
"hash": "cbbc6a894b6421d2cd49e59b7ed650fb861aa19a009d6dfe9937304057f2b91f"
}
backend/.sqlx/query-cd492743a27a158c9f5ff1fdb50a48ebfed85ad8800e293945eadc874d7e42f3.json
Generated
+19
View File
@@ -0,0 +1,19 @@
{
"db_name": "PostgreSQL",
"query": "UPDATE fork_ducklake_namespace SET schema_dropped = true\n WHERE workspace_id = $1 AND ducklake_name = $2 AND catalog = $3\n AND storage = $4 AND storage_ref = $5 AND data_path = $6",
"describe": {
"columns": [],
"parameters": {
"Left": [
"Text",
"Text",
"Text",
"Text",
"Text",
"Text"
]
},
"nullable": []
},
"hash": "cd492743a27a158c9f5ff1fdb50a48ebfed85ad8800e293945eadc874d7e42f3"
}
backend/.sqlx/query-d1142126245cdaf5d01dc67ab2488958b335f6a24d1e93641eab6aaac86301f7.json
Generated
+14
View File
@@ -0,0 +1,14 @@
{
"db_name": "PostgreSQL",
"query": "UPDATE workspace_settings\n SET ducklake = jsonb_set(ducklake, '{ducklakes}', (\n SELECT COALESCE(jsonb_object_agg(key, value - 'fork_behavior'), '{}'::jsonb)\n FROM jsonb_each(ducklake->'ducklakes')\n ))\n WHERE workspace_id = $1 AND jsonb_typeof(ducklake->'ducklakes') = 'object'",
"describe": {
"columns": [],
"parameters": {
"Left": [
"Text"
]
},
"nullable": []
},
"hash": "d1142126245cdaf5d01dc67ab2488958b335f6a24d1e93641eab6aaac86301f7"
}
backend/.sqlx/query-d1a1f64cb6b14c561c53c44910cad8bc7a2d226a9f22418782bcc66e46d0f706.json
Generated
+15
View File
@@ -0,0 +1,15 @@
{
"db_name": "PostgreSQL",
"query": "INSERT INTO pipeline_freshness_state (workspace_id, script_path) VALUES ($1, $2)",
"describe": {
"columns": [],
"parameters": {
"Left": [
"Varchar",
"Varchar"
]
},
"nullable": []
},
"hash": "d1a1f64cb6b14c561c53c44910cad8bc7a2d226a9f22418782bcc66e46d0f706"
}
backend/.sqlx/query-d41ea93fd58381b89e151c965eae1ea2fe96a1b94f5a92953fb1c1642d15c016.json
Generated
+2 -1
View File
@@ -111,7 +111,8 @@
"ci_test",
"github",
"azure",
"asset"
"asset",
"freshness"
]
}
}
backend/.sqlx/query-d4211392e174a0e8f89c7fcebdf120e5b0f629f9f04e08a2982df33ff23ac7a9.json
Generated
+2 -1
View File
@@ -251,7 +251,8 @@
"ci_test",
"github",
"azure",
"asset"
"asset",
"freshness"
]
}
}
backend/.sqlx/query-caf4dc1046769f410d1277cdd3747763e0d674e8a1d9f3512629cb5a9e5a3439.json → backend/.sqlx/query-dc60d43814e2b5220db648677dbb3656e27f1c962c2f0438e0cc06530661fbf8.json
Generated
+3 -4
View File
@@ -1,17 +1,16 @@
{
"db_name": "PostgreSQL",
"query": "\n SELECT ws.datatable->'datatables'->$2 AS config\n FROM workspace_settings ws\n WHERE ws.workspace_id = $1\n ",
"query": "\n SELECT ws.datatable->'datatables' AS datatables\n FROM workspace_settings ws\n WHERE ws.workspace_id = $1\n ",
"describe": {
"columns": [
{
"ordinal": 0,
"name": "config",
"name": "datatables",
"type_info": "Jsonb"
}
],
"parameters": {
"Left": [
"Text",
"Text"
]
},
@@ -19,5 +18,5 @@
null
]
},
"hash": "caf4dc1046769f410d1277cdd3747763e0d674e8a1d9f3512629cb5a9e5a3439"
"hash": "dc60d43814e2b5220db648677dbb3656e27f1c962c2f0438e0cc06530661fbf8"
}
backend/.sqlx/query-e27cc16414c181a055bf0666eb0cb16a356cb6be5c48f95b5ba200e7e6ab0efd.json
Generated
+20
View File
@@ -0,0 +1,20 @@
{
"db_name": "PostgreSQL",
"query": "INSERT INTO fork_ducklake_namespace\n (workspace_id, ducklake_name, metadata_schema, catalog, storage, storage_ref, data_path)\n VALUES ($1, $2, $3, $4, $5, $6, $7)\n ON CONFLICT (workspace_id, ducklake_name, catalog, storage, storage_ref, data_path)\n DO UPDATE SET schema_dropped = false",
"describe": {
"columns": [],
"parameters": {
"Left": [
"Varchar",
"Varchar",
"Varchar",
"Text",
"Text",
"Text",
"Text"
]
},
"nullable": []
},
"hash": "e27cc16414c181a055bf0666eb0cb16a356cb6be5c48f95b5ba200e7e6ab0efd"
}
backend/.sqlx/query-e4d71278fb80126a7a9da73f1889352d4d1e3cb3a8a08f1c9c03055a1cab1235.json
Generated
+2 -1
View File
@@ -191,7 +191,8 @@
"ci_test",
"github",
"azure",
"asset"
"asset",
"freshness"
]
}
}
backend/.sqlx/query-e50bce4bba01c90171973862bcfa088b5c0c1ad601146ff1c57d4a3ddc30556a.json
Generated
+34
View File
@@ -0,0 +1,34 @@
{
"db_name": "PostgreSQL",
"query": "SELECT runnable_path AS \"runnable_path!\", created_by AS \"created_by!\",\n args AS \"args: sqlx::types::Json<serde_json::Value>\"\n FROM v2_job\n WHERE workspace_id = $1 AND trigger_kind = 'freshness'\n ORDER BY created_at",
"describe": {
"columns": [
{
"ordinal": 0,
"name": "runnable_path!",
"type_info": "Varchar"
},
{
"ordinal": 1,
"name": "created_by!",
"type_info": "Varchar"
},
{
"ordinal": 2,
"name": "args: sqlx::types::Json<serde_json::Value>",
"type_info": "Jsonb"
}
],
"parameters": {
"Left": [
"Text"
]
},
"nullable": [
true,
false,
true
]
},
"hash": "e50bce4bba01c90171973862bcfa088b5c0c1ad601146ff1c57d4a3ddc30556a"
}
backend/.sqlx/query-f26cde14890cc754e04341264c4dba01f2f3a02bb36a8d7c20e1990dc81e0472.json
Generated
+22
View File
@@ -0,0 +1,22 @@
{
"db_name": "PostgreSQL",
"query": "SELECT COUNT(*) AS \"count!\" FROM pipeline_freshness_state WHERE workspace_id = $1",
"describe": {
"columns": [
{
"ordinal": 0,
"name": "count!",
"type_info": "Int8"
}
],
"parameters": {
"Left": [
"Text"
]
},
"nullable": [
null
]
},
"hash": "f26cde14890cc754e04341264c4dba01f2f3a02bb36a8d7c20e1990dc81e0472"
}
backend/.sqlx/query-f3fef6964a211872b01c984b19192da8ee231c772f88dd9145f01d7cf3b8f8c1.json
Generated
+15
View File
@@ -0,0 +1,15 @@
{
"db_name": "PostgreSQL",
"query": "INSERT INTO asset (workspace_id, path, kind, usage_access_type, usage_path, usage_kind, columns)\n SELECT $2, path, kind, usage_access_type, usage_path, usage_kind, columns\n FROM asset WHERE workspace_id = $1 AND usage_kind IN ('script', 'flow')",
"describe": {
"columns": [],
"parameters": {
"Left": [
"Text",
"Varchar"
]
},
"nullable": []
},
"hash": "f3fef6964a211872b01c984b19192da8ee231c772f88dd9145f01d7cf3b8f8c1"
}
backend/.sqlx/query-fbd1b02b9bbf86e17d6ee0da3e9b134780e55a521a5cee8eb657b67bbd4b2588.json
Generated
+19
View File
@@ -0,0 +1,19 @@
{
"db_name": "PostgreSQL",
"query": "DELETE FROM fork_ducklake_namespace\n WHERE workspace_id = $1 AND ducklake_name = $2 AND catalog = $3\n AND storage = $4 AND storage_ref = $5 AND data_path = $6",
"describe": {
"columns": [],
"parameters": {
"Left": [
"Text",
"Text",
"Text",
"Text",
"Text",
"Text"
]
},
"nullable": []
},
"hash": "fbd1b02b9bbf86e17d6ee0da3e9b134780e55a521a5cee8eb657b67bbd4b2588"
}
backend/CLAUDE.md
+39
View File
@@ -11,6 +11,45 @@
cd backend/windmill-duckdb-ffi-internal && ./build_dev.sh
```
Re-run after clean builds or when `target/debug/libwindmill_duckdb_ffi_internal.*` is missing.
The bundled DuckDB compile (~2min) is cached in a per-user dir shared across
worktrees (keyed by the crate's `Cargo.lock`), so a fresh worktree reuses it and
the build is near-instant — you don't pay the full compile per worktree. Editing
the FFI crate's own source falls back to an isolated per-worktree `./target`.
- **Running data pipelines (DuckLake) from source**: see the section below — a plain build
advertises the `duckdb` tag but cannot execute DuckDB scripts and has no working S3 proxy.
## Running data pipelines (DuckLake) from source
DuckLake pipelines need **both** the right cargo features **and** the prebuilt DuckDB FFI. A
plain `cargo run` (or `cargo run --features quickjs`) does **not** suffice, and the failure modes
are silent-ish, so agents lose time. Verify feature names against `backend/Cargo.toml` `[features]`.
**Feature sets** (run from `backend/`):
| Goal | Command |
|---|---|
| CE DuckLake (DuckDB scripts + S3 proxy) | `cargo run --features quickjs,duckdb,parquet,private` |
| + Python scripts | add `,python` |
| EE features (WAP, partitioning, forks, …) | add `,enterprise,license` |
`enterprise` already pulls in `license`, but list both when you want the license-gated paths.
`quickjs` is for JS eval, not DuckLake per se — keep it if your baseline build had it.
**Before running any DuckDB script**, build the FFI (see the bullet above):
`cd backend/windmill-duckdb-ffi-internal && ./build_dev.sh`.
**Two gotchas that a wrong feature set produces:**
1. **`duckdb` tag advertised, feature missing.** The `duckdb` worker tag is in the *unconditional*
default tag list (`windmill-common/src/worker.rs`, `DEFAULT_TAGS`), so a worker advertises it even
without the `duckdb` feature. Jobs then dispatch but fail at execution with
`"Duck DB requires the duckdb feature to be enabled"` (`windmill-worker/src/worker.rs`). Fix:
compile with `--features duckdb`.
2. **DuckLake writes 404 (no S3 proxy).** The workspace S3 proxy (`/w/{ws}/s3_proxy/*`) that
DuckLake uses for reads/writes only mounts the real service under
`#[cfg(all(feature = "private", feature = "parquet"))]` (`windmill-api/src/s3_proxy_oss.rs`);
otherwise it's an empty router and every proxied request 404s. Fix: compile with **both**
`private` and `parquet`.
## Cloud vs self-hosted gating
backend/Cargo.lock
Generated
+112 -100
View File
@@ -2057,9 +2057,9 @@ dependencies = [
[[package]]
name = "cc"
version = "1.2.65"
version = "1.2.66"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "e228eec9be7c17ccb640b59b36a5cd805ea2a564a4c5e162c2f659fea30d3b96"
checksum = "f5d6cac793997bd970000024b2934968efe83b382de4fdcf4fcb46b6ee4ad996"
dependencies = [
"find-msvc-tools",
"jobserver",
@@ -5072,9 +5072,11 @@ source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "300e883d756b2e4ec94e02791f39b04b522276138852cfc41d9fb7e904106099"
dependencies = [
"cfg-if",
"js-sys",
"libc",
"r-efi 6.0.0",
"rand_core 0.10.1",
"wasm-bindgen",
]
[[package]]
@@ -7563,9 +7565,9 @@ dependencies = [
[[package]]
name = "num-bigint"
version = "0.4.7"
version = "0.4.8"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "c863e9ab5e7bf9c99ba75e1050f1e4d624ae87ed3532d6238ffbdc7b585dbbe6"
checksum = "c89e69e7e0f03bea5ef08013795c25018e101932225a656383bd384495ecc367"
dependencies = [
"num-integer",
"num-traits",
@@ -8361,9 +8363,9 @@ checksum = "9b4f627cb1b25917193a259e49bdad08f671f8d9708acfd5fe0a8c1455d87220"
[[package]]
name = "pest"
version = "2.8.6"
version = "2.8.7"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "e0848c601009d37dfa3430c4666e147e49cdcf1b92ecd3e63657d8a5f19da662"
checksum = "47627dd7305c6a2d6c8c6bcd24c5a4c17dbbf425f4f9c5313e724b38fc9782e9"
dependencies = [
"memchr",
"ucd-trie",
@@ -8371,9 +8373,9 @@ dependencies = [
[[package]]
name = "pest_derive"
version = "2.8.6"
version = "2.8.7"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "11f486f1ea21e6c10ed15d5a7c77165d0ee443402f0780849d1768e7d9d6fe77"
checksum = "4b4254325ecad416ab689e27ba51da03ba01a9632bc6e108f5fe7c3c4ad29d58"
dependencies = [
"pest",
"pest_generator",
@@ -8381,9 +8383,9 @@ dependencies = [
[[package]]
name = "pest_generator"
version = "2.8.6"
version = "2.8.7"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "8040c4647b13b210a963c1ed407c1ff4fdfa01c31d6d2a098218702e6664f94f"
checksum = "6c4c0e91ead7a8f7acecbca6f003fc2e8282b1dbe2dd9c9d2f16aba42995e0a7"
dependencies = [
"pest",
"pest_meta",
@@ -8394,12 +8396,11 @@ dependencies = [
[[package]]
name = "pest_meta"
version = "2.8.6"
version = "2.8.7"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "89815c69d36021a140146f26659a81d6c2afa33d216d736dd4be5381a7362220"
checksum = "f9744bc48116fee06334924bb5f2bad41eed5e89bd26e29b0b799f9a3f82c210"
dependencies = [
"pest",
"sha2 0.10.9",
]
[[package]]
@@ -8995,15 +8996,16 @@ dependencies = [
[[package]]
name = "quinn-proto"
version = "0.11.15"
version = "0.11.16"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "4fcb935c5bec503c2f0e306bdd3e58bb9029dcb14fa8d9ac76e3a5256ac0763e"
checksum = "2f4bfc015262b9df63c8845072ce59068853ff5872180c2ce2f13038b970e560"
dependencies = [
"aws-lc-rs",
"bytes",
"getrandom 0.3.4",
"getrandom 0.4.3",
"lru-slab",
"rand 0.9.0",
"rand 0.10.2",
"rand_pcg",
"ring 0.17.14",
"rustc-hash 2.1.3",
"rustls 0.23.35",
@@ -9017,16 +9019,16 @@ dependencies = [
[[package]]
name = "quinn-udp"
version = "0.5.14"
version = "0.5.15"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "addec6a0dcad8a8d96a771f815f0eaf55f9d1805756410b39f5fa81332574cbd"
checksum = "35a133f956daabe89a61a685c2649f13d82d5aa4bd5d12d1277e1072a21c0694"
dependencies = [
"cfg_aliases",
"libc",
"once_cell",
"socket2 0.6.4",
"tracing",
"windows-sys 0.60.2",
"windows-sys 0.61.2",
]
[[package]]
@@ -9184,6 +9186,15 @@ dependencies = [
"rand_core 0.5.1",
]
[[package]]
name = "rand_pcg"
version = "0.10.2"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "caa0f4137e1c0a72f4c651489402276c8e8e1cf081f3b0ba156d2cbeef09e86a"
dependencies = [
"rand_core 0.10.1",
]
[[package]]
name = "raw-cpuid"
version = "11.6.0"
@@ -13322,9 +13333,9 @@ checksum = "7fcfc827f90e53a02eaef5e535ee14266c1d569214c6aa70133a624d8a3164ba"
[[package]]
name = "utf8-width"
version = "0.1.8"
version = "0.1.9"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "1292c0d970b54115d14f2492fe0170adf21d68a1de108eebc51c1df4f346a091"
checksum = "159a7cadce548703edd50d24069bc294c5415ecab0a480e0cd1ca06d112dc94a"
[[package]]
name = "utf8_iter"
@@ -13735,7 +13746,7 @@ dependencies = [
[[package]]
name = "windmill"
version = "1.747.0"
version = "1.749.0"
dependencies = [
"anyhow",
"async-nats",
@@ -13817,7 +13828,7 @@ dependencies = [
[[package]]
name = "windmill-ai"
version = "1.747.0"
version = "1.749.0"
dependencies = [
"async-stream",
"async-trait",
@@ -13850,7 +13861,7 @@ dependencies = [
[[package]]
name = "windmill-alerting"
version = "1.747.0"
version = "1.749.0"
dependencies = [
"axum 0.8.9",
"chrono",
@@ -13863,7 +13874,7 @@ dependencies = [
[[package]]
name = "windmill-api"
version = "1.747.0"
version = "1.749.0"
dependencies = [
"anyhow",
"argon2",
@@ -14001,7 +14012,7 @@ dependencies = [
[[package]]
name = "windmill-api-agent-workers"
version = "1.747.0"
version = "1.749.0"
dependencies = [
"axum 0.8.9",
"chrono",
@@ -14024,7 +14035,7 @@ dependencies = [
[[package]]
name = "windmill-api-assets"
version = "1.747.0"
version = "1.749.0"
dependencies = [
"axum 0.8.9",
"chrono",
@@ -14039,7 +14050,7 @@ dependencies = [
[[package]]
name = "windmill-api-auth"
version = "1.747.0"
version = "1.749.0"
dependencies = [
"anyhow",
"axum 0.8.9",
@@ -14065,7 +14076,7 @@ dependencies = [
[[package]]
name = "windmill-api-client"
version = "1.747.0"
version = "1.749.0"
dependencies = [
"reqwest 0.12.28",
"serde",
@@ -14075,7 +14086,7 @@ dependencies = [
[[package]]
name = "windmill-api-configs"
version = "1.747.0"
version = "1.749.0"
dependencies = [
"axum 0.8.9",
"chrono",
@@ -14092,7 +14103,7 @@ dependencies = [
[[package]]
name = "windmill-api-debug"
version = "1.747.0"
version = "1.749.0"
dependencies = [
"axum 0.8.9",
"base64 0.22.1",
@@ -14114,7 +14125,7 @@ dependencies = [
[[package]]
name = "windmill-api-embeddings"
version = "1.747.0"
version = "1.749.0"
dependencies = [
"anyhow",
"axum 0.8.9",
@@ -14137,7 +14148,7 @@ dependencies = [
[[package]]
name = "windmill-api-flow-conversations"
version = "1.747.0"
version = "1.749.0"
dependencies = [
"axum 0.8.9",
"chrono",
@@ -14153,7 +14164,7 @@ dependencies = [
[[package]]
name = "windmill-api-flows"
version = "1.747.0"
version = "1.749.0"
dependencies = [
"axum 0.8.9",
"chrono",
@@ -14174,7 +14185,7 @@ dependencies = [
[[package]]
name = "windmill-api-groups"
version = "1.747.0"
version = "1.749.0"
dependencies = [
"axum 0.8.9",
"chrono",
@@ -14195,7 +14206,7 @@ dependencies = [
[[package]]
name = "windmill-api-inputs"
version = "1.747.0"
version = "1.749.0"
dependencies = [
"axum 0.8.9",
"chrono",
@@ -14209,7 +14220,7 @@ dependencies = [
[[package]]
name = "windmill-api-integration-tests"
version = "1.747.0"
version = "1.749.0"
dependencies = [
"anyhow",
"async-nats",
@@ -14244,7 +14255,7 @@ dependencies = [
[[package]]
name = "windmill-api-jobs"
version = "1.747.0"
version = "1.749.0"
dependencies = [
"anyhow",
"axum 0.8.9",
@@ -14269,7 +14280,7 @@ dependencies = [
[[package]]
name = "windmill-api-npm-proxy"
version = "1.747.0"
version = "1.749.0"
dependencies = [
"axum 0.8.9",
"flate2",
@@ -14287,7 +14298,7 @@ dependencies = [
[[package]]
name = "windmill-api-openapi"
version = "1.747.0"
version = "1.749.0"
dependencies = [
"anyhow",
"axum 0.8.9",
@@ -14309,7 +14320,7 @@ dependencies = [
[[package]]
name = "windmill-api-schedule"
version = "1.747.0"
version = "1.749.0"
dependencies = [
"axum 0.8.9",
"chrono",
@@ -14329,7 +14340,7 @@ dependencies = [
[[package]]
name = "windmill-api-scripts"
version = "1.747.0"
version = "1.749.0"
dependencies = [
"axum 0.8.9",
"chrono",
@@ -14366,7 +14377,7 @@ dependencies = [
[[package]]
name = "windmill-api-settings"
version = "1.747.0"
version = "1.749.0"
dependencies = [
"anyhow",
"axum 0.8.9",
@@ -14394,7 +14405,7 @@ dependencies = [
[[package]]
name = "windmill-api-sse"
version = "1.747.0"
version = "1.749.0"
dependencies = [
"lazy_static",
"serde",
@@ -14406,7 +14417,7 @@ dependencies = [
[[package]]
name = "windmill-api-users"
version = "1.747.0"
version = "1.749.0"
dependencies = [
"argon2",
"axum 0.8.9",
@@ -14431,7 +14442,7 @@ dependencies = [
[[package]]
name = "windmill-api-workers"
version = "1.747.0"
version = "1.749.0"
dependencies = [
"axum 0.8.9",
"chrono",
@@ -14445,10 +14456,11 @@ dependencies = [
[[package]]
name = "windmill-api-workspaces"
version = "1.747.0"
version = "1.749.0"
dependencies = [
"axum 0.8.9",
"chrono",
"futures",
"hex",
"http 1.4.2",
"hyper 1.10.1",
@@ -14479,7 +14491,7 @@ dependencies = [
[[package]]
name = "windmill-audit"
version = "1.747.0"
version = "1.749.0"
dependencies = [
"chrono",
"lazy_static",
@@ -14493,7 +14505,7 @@ dependencies = [
[[package]]
name = "windmill-autoscaling"
version = "1.747.0"
version = "1.749.0"
dependencies = [
"anyhow",
"axum 0.8.9",
@@ -14512,7 +14524,7 @@ dependencies = [
[[package]]
name = "windmill-common"
version = "1.747.0"
version = "1.749.0"
dependencies = [
"aes-gcm",
"aho-corasick",
@@ -14614,7 +14626,7 @@ dependencies = [
[[package]]
name = "windmill-dep-map"
version = "1.747.0"
version = "1.749.0"
dependencies = [
"chrono",
"itertools 0.14.0",
@@ -14633,7 +14645,7 @@ dependencies = [
[[package]]
name = "windmill-git-sync"
version = "1.747.0"
version = "1.749.0"
dependencies = [
"regex",
"serde",
@@ -14648,7 +14660,7 @@ dependencies = [
[[package]]
name = "windmill-indexer"
version = "1.747.0"
version = "1.749.0"
dependencies = [
"anyhow",
"astral-tokio-tar",
@@ -14672,7 +14684,7 @@ dependencies = [
[[package]]
name = "windmill-jseval"
version = "1.747.0"
version = "1.749.0"
dependencies = [
"anyhow",
"futures",
@@ -14689,7 +14701,7 @@ dependencies = [
[[package]]
name = "windmill-macros"
version = "1.747.0"
version = "1.749.0"
dependencies = [
"itertools 0.14.0",
"lazy_static",
@@ -14705,7 +14717,7 @@ dependencies = [
[[package]]
name = "windmill-mcp"
version = "1.747.0"
version = "1.749.0"
dependencies = [
"anyhow",
"async-trait",
@@ -14726,7 +14738,7 @@ dependencies = [
[[package]]
name = "windmill-native-triggers"
version = "1.747.0"
version = "1.749.0"
dependencies = [
"anyhow",
"async-trait",
@@ -14757,7 +14769,7 @@ dependencies = [
[[package]]
name = "windmill-oauth"
version = "1.747.0"
version = "1.749.0"
dependencies = [
"anyhow",
"arc-swap",
@@ -14782,7 +14794,7 @@ dependencies = [
[[package]]
name = "windmill-object-store"
version = "1.747.0"
version = "1.749.0"
dependencies = [
"anyhow",
"async-stream",
@@ -14816,7 +14828,7 @@ dependencies = [
[[package]]
name = "windmill-operator"
version = "1.747.0"
version = "1.749.0"
dependencies = [
"anyhow",
"futures",
@@ -14834,7 +14846,7 @@ dependencies = [
[[package]]
name = "windmill-parser"
version = "1.747.0"
version = "1.749.0"
dependencies = [
"convert_case 0.6.0",
"serde",
@@ -14843,7 +14855,7 @@ dependencies = [
[[package]]
name = "windmill-parser-bash"
version = "1.747.0"
version = "1.749.0"
dependencies = [
"anyhow",
"lazy_static",
@@ -14855,7 +14867,7 @@ dependencies = [
[[package]]
name = "windmill-parser-csharp"
version = "1.747.0"
version = "1.749.0"
dependencies = [
"anyhow",
"serde_json",
@@ -14867,7 +14879,7 @@ dependencies = [
[[package]]
name = "windmill-parser-go"
version = "1.747.0"
version = "1.749.0"
dependencies = [
"anyhow",
"gosyn",
@@ -14879,7 +14891,7 @@ dependencies = [
[[package]]
name = "windmill-parser-graphql"
version = "1.747.0"
version = "1.749.0"
dependencies = [
"anyhow",
"lazy_static",
@@ -14891,7 +14903,7 @@ dependencies = [
[[package]]
name = "windmill-parser-java"
version = "1.747.0"
version = "1.749.0"
dependencies = [
"anyhow",
"serde_json",
@@ -14903,7 +14915,7 @@ dependencies = [
[[package]]
name = "windmill-parser-nu"
version = "1.747.0"
version = "1.749.0"
dependencies = [
"anyhow",
"nu-parser",
@@ -14914,7 +14926,7 @@ dependencies = [
[[package]]
name = "windmill-parser-php"
version = "1.747.0"
version = "1.749.0"
dependencies = [
"anyhow",
"itertools 0.14.0",
@@ -14925,7 +14937,7 @@ dependencies = [
[[package]]
name = "windmill-parser-py"
version = "1.747.0"
version = "1.749.0"
dependencies = [
"anyhow",
"itertools 0.14.0",
@@ -14937,7 +14949,7 @@ dependencies = [
[[package]]
name = "windmill-parser-py-asset"
version = "1.747.0"
version = "1.749.0"
dependencies = [
"anyhow",
"rustpython-ast",
@@ -14948,7 +14960,7 @@ dependencies = [
[[package]]
name = "windmill-parser-py-imports"
version = "1.747.0"
version = "1.749.0"
dependencies = [
"anyhow",
"async-recursion",
@@ -14970,7 +14982,7 @@ dependencies = [
[[package]]
name = "windmill-parser-r"
version = "1.747.0"
version = "1.749.0"
dependencies = [
"anyhow",
"serde_json",
@@ -14982,7 +14994,7 @@ dependencies = [
[[package]]
name = "windmill-parser-ruby"
version = "1.747.0"
version = "1.749.0"
dependencies = [
"anyhow",
"lazy_static",
@@ -14996,7 +15008,7 @@ dependencies = [
[[package]]
name = "windmill-parser-rust"
version = "1.747.0"
version = "1.749.0"
dependencies = [
"anyhow",
"convert_case 0.6.0",
@@ -15013,7 +15025,7 @@ dependencies = [
[[package]]
name = "windmill-parser-sql"
version = "1.747.0"
version = "1.749.0"
dependencies = [
"anyhow",
"lazy_static",
@@ -15026,7 +15038,7 @@ dependencies = [
[[package]]
name = "windmill-parser-sql-asset"
version = "1.747.0"
version = "1.749.0"
dependencies = [
"anyhow",
"serde",
@@ -15038,7 +15050,7 @@ dependencies = [
[[package]]
name = "windmill-parser-ts"
version = "1.747.0"
version = "1.749.0"
dependencies = [
"anyhow",
"lazy_static",
@@ -15056,7 +15068,7 @@ dependencies = [
[[package]]
name = "windmill-parser-ts-asset"
version = "1.747.0"
version = "1.749.0"
dependencies = [
"anyhow",
"serde-wasm-bindgen",
@@ -15072,7 +15084,7 @@ dependencies = [
[[package]]
name = "windmill-parser-wac"
version = "1.747.0"
version = "1.749.0"
dependencies = [
"anyhow",
"rustpython-ast",
@@ -15088,7 +15100,7 @@ dependencies = [
[[package]]
name = "windmill-parser-yaml"
version = "1.747.0"
version = "1.749.0"
dependencies = [
"anyhow",
"serde",
@@ -15099,7 +15111,7 @@ dependencies = [
[[package]]
name = "windmill-queue"
version = "1.747.0"
version = "1.749.0"
dependencies = [
"anyhow",
"async-recursion",
@@ -15138,7 +15150,7 @@ dependencies = [
[[package]]
name = "windmill-runtime-nativets"
version = "1.747.0"
version = "1.749.0"
dependencies = [
"anyhow",
"const_format",
@@ -15177,7 +15189,7 @@ dependencies = [
[[package]]
name = "windmill-sql-datatype-parser-wasm"
version = "1.747.0"
version = "1.749.0"
dependencies = [
"getrandom 0.3.4",
"wasm-bindgen",
@@ -15188,7 +15200,7 @@ dependencies = [
[[package]]
name = "windmill-store"
version = "1.747.0"
version = "1.749.0"
dependencies = [
"anyhow",
"async-recursion",
@@ -15222,7 +15234,7 @@ dependencies = [
[[package]]
name = "windmill-test-utils"
version = "1.747.0"
version = "1.749.0"
dependencies = [
"anyhow",
"async-trait",
@@ -15246,7 +15258,7 @@ dependencies = [
[[package]]
name = "windmill-trigger"
version = "1.747.0"
version = "1.749.0"
dependencies = [
"anyhow",
"async-trait",
@@ -15279,7 +15291,7 @@ dependencies = [
[[package]]
name = "windmill-trigger-azure"
version = "1.747.0"
version = "1.749.0"
dependencies = [
"anyhow",
"async-trait",
@@ -15312,7 +15324,7 @@ dependencies = [
[[package]]
name = "windmill-trigger-email"
version = "1.747.0"
version = "1.749.0"
dependencies = [
"anyhow",
"async-trait",
@@ -15332,7 +15344,7 @@ dependencies = [
[[package]]
name = "windmill-trigger-gcp"
version = "1.747.0"
version = "1.749.0"
dependencies = [
"anyhow",
"async-trait",
@@ -15366,7 +15378,7 @@ dependencies = [
[[package]]
name = "windmill-trigger-http"
version = "1.747.0"
version = "1.749.0"
dependencies = [
"anyhow",
"async-trait",
@@ -15402,7 +15414,7 @@ dependencies = [
[[package]]
name = "windmill-trigger-kafka"
version = "1.747.0"
version = "1.749.0"
dependencies = [
"anyhow",
"async-trait",
@@ -15425,7 +15437,7 @@ dependencies = [
[[package]]
name = "windmill-trigger-mqtt"
version = "1.747.0"
version = "1.749.0"
dependencies = [
"anyhow",
"async-trait",
@@ -15449,7 +15461,7 @@ dependencies = [
[[package]]
name = "windmill-trigger-nats"
version = "1.747.0"
version = "1.749.0"
dependencies = [
"anyhow",
"async-nats",
@@ -15473,7 +15485,7 @@ dependencies = [
[[package]]
name = "windmill-trigger-postgres"
version = "1.747.0"
version = "1.749.0"
dependencies = [
"anyhow",
"async-trait",
@@ -15508,7 +15520,7 @@ dependencies = [
[[package]]
name = "windmill-trigger-sqs"
version = "1.747.0"
version = "1.749.0"
dependencies = [
"anyhow",
"async-trait",
@@ -15536,7 +15548,7 @@ dependencies = [
[[package]]
name = "windmill-trigger-websocket"
version = "1.747.0"
version = "1.749.0"
dependencies = [
"anyhow",
"async-trait",
@@ -15561,7 +15573,7 @@ dependencies = [
[[package]]
name = "windmill-types"
version = "1.747.0"
version = "1.749.0"
dependencies = [
"anyhow",
"bitflags 2.13.0",
@@ -15580,7 +15592,7 @@ dependencies = [
[[package]]
name = "windmill-worker"
version = "1.747.0"
version = "1.749.0"
dependencies = [
"anyhow",
"async-once-cell",
@@ -15690,7 +15702,7 @@ dependencies = [
[[package]]
name = "windmill-worker-volumes"
version = "1.747.0"
version = "1.749.0"
dependencies = [
"bytes",
"futures",
backend/Cargo.toml
+2 -2
View File
@@ -1,6 +1,6 @@
[package]
name = "windmill"
version = "1.747.0"
version = "1.749.0"
authors.workspace = true
edition.workspace = true
@@ -87,7 +87,7 @@ members = [
exclude = ["./windmill-duckdb-ffi-internal", "./parsers/windmill-parser-wasm"]
[workspace.package]
version = "1.747.0"
version = "1.749.0"
authors = ["Ruben Fiszel <ruben@windmill.dev>"]
edition = "2021"
backend/ee-repo-ref.txt
+1 -1
View File
@@ -1 +1 @@
d36555eb8891becdb66f2135684387367450ed1d
6c287041cd7edd4a77a4bc07ad0e156cec32cce4
backend/migrations/20260703165613_pipeline_freshness_watchdog.down.sql
+5
View File
@@ -0,0 +1,5 @@
-- Postgres has no ALTER TYPE ... DROP VALUE for enums. The 'freshness' value
-- stays even on rollback, consistent with prior job_trigger_kind additions
-- (see 20260510174213_asset_trigger_dispatch).
DROP INDEX IF EXISTS idx_script_pipeline_freshness_scan;
DROP TABLE IF EXISTS pipeline_freshness_state;
backend/migrations/20260703165613_pipeline_freshness_watchdog.up.sql
+35
View File
@@ -0,0 +1,35 @@
-- Attribution for runs pushed by the pipeline freshness watchdog (the EE
-- background loop that re-runs a `// freshness`-annotated producer whose
-- output aged past its window).
ALTER TYPE job_trigger_kind ADD VALUE IF NOT EXISTS 'freshness';
-- Per-(workspace, script) watchdog state: exponential-backoff bookkeeping so
-- a persistently failing producer isn't re-pushed on every scan tick, and an
-- atomic claim so concurrent servers can't double-push in the same tick
-- (claim = the UPDATE/INSERT that advances next_attempt_at; only the winner
-- pushes). Rows exist only while a script is stale — observing it fresh (or
-- its annotation gone) deletes the row, resetting the backoff.
CREATE TABLE pipeline_freshness_state (
workspace_id VARCHAR(50) NOT NULL REFERENCES workspace (id) ON DELETE CASCADE,
script_path VARCHAR(510) NOT NULL,
attempts INTEGER NOT NULL DEFAULT 1,
last_push_at TIMESTAMPTZ NOT NULL DEFAULT now(),
next_attempt_at TIMESTAMPTZ NOT NULL DEFAULT now(),
PRIMARY KEY (workspace_id, script_path)
);
-- Written only by the server monitor loop on the raw (non-RLS) pool, but
-- granted like every other app table so a future user-transaction reader
-- doesn't hit the recurring missing-GRANT class of bug.
GRANT ALL ON pipeline_freshness_state TO windmill_user;
GRANT ALL ON pipeline_freshness_state TO windmill_admin;
-- The watchdog's ~60s candidate scan (latest deployed pipeline members)
-- filters on this exact predicate and orders by (workspace_id, path,
-- created_at DESC); without a matching partial index it seq-scans the whole
-- script-version heap on every tick, on instances that mostly have zero
-- pipeline scripts. (idx_script_pipeline_path is text_pattern_ops for
-- prefix LIKE — it can't serve this ordering.)
CREATE INDEX idx_script_pipeline_freshness_scan
ON script (workspace_id, path, created_at DESC)
WHERE auto_kind = 'pipeline' AND archived = false AND deleted = false;
backend/migrations/20260703170745_fork_ducklake_namespace.down.sql
+1
View File
@@ -0,0 +1 @@
DROP TABLE IF EXISTS fork_ducklake_namespace;
backend/migrations/20260703170745_fork_ducklake_namespace.up.sql
+51
View File
@@ -0,0 +1,51 @@
-- Registry of ducklake namespaces provisioned for fork/dev workspaces. One row per
-- (fork workspace, lake name): records the exact catalog metadata schema and data
-- sub-path the fork's jobs attach to, so fork deletion can drop the pg schema and
-- delete the S3 prefix deterministically (the row is written on first resolution,
-- before any physical state exists).
--
-- Deliberately NO foreign key to workspace(id): rows are the durable cleanup ledger and
-- must OUTLIVE the workspace row when physical cleanup fails after the delete commits
-- (unreachable catalog, storage outage) — a CASCADE would erase the only record of the
-- orphaned namespace, letting a recreated same-id fork silently reattach stale tables.
-- Rows are deleted explicitly after each successful cleanup; leftover rows for a reused
-- id are retried at fork creation, which refuses to proceed while they cannot be cleaned.
CREATE TABLE fork_ducklake_namespace (
workspace_id VARCHAR(50) NOT NULL,
ducklake_name VARCHAR(255) NOT NULL,
metadata_schema VARCHAR(63) NOT NULL,
-- Canonical identity of the catalog database the metadata schema lives in
-- (`<resource_type>:<resource_path>`, e.g. `instance:wm_ducklake` or
-- `postgres:u/admin/pg`). Cleanup connects to THIS catalog, not whatever the fork's
-- settings point at by then — a drifted catalog resource must not make cleanup drop a
-- schema in the wrong database and orphan the real one.
catalog TEXT NOT NULL,
-- Named workspace storage holding the fork's data files; '' = the default storage
-- (part of the PK, which cannot hold NULL).
storage TEXT NOT NULL DEFAULT '',
-- The storage's RESOLVED identity at registration time (`<lfs type>:<resource path or
-- root path>`, e.g. `s3:u/admin/minio` or `filesystem:/data/lfs`; '' = unknown). Cleanup
-- deletes the fork prefix from THIS storage, not whatever the logical name points at by
-- then — repointing a storage after attach must not orphan the original fork data (or
-- delete a colliding prefix from the new one).
storage_ref TEXT NOT NULL DEFAULT '',
-- The fork namespace's data path within that storage (a bucket-root
-- `__wm_forks/<fork wid>/…` prefix).
data_path TEXT NOT NULL,
created_at TIMESTAMPTZ NOT NULL DEFAULT now(),
-- Cleanup phase state: true once the metadata schema has been dropped but data files (or
-- the row delete) still failed. Later retries then skip the schema phase entirely — they
-- need NO catalog credentials, which may be gone for good with the deleted fork's
-- resources. Reset to false whenever a live fork re-registers the row (re-attaching
-- recreates the schema).
schema_dropped BOOLEAN NOT NULL DEFAULT false,
-- One row per physical location EVER attached: if the fork's lake settings drift
-- (catalog/storage/path change), later attaches add rows rather than replace them, so
-- cleanup covers every location the fork wrote, not just the first.
PRIMARY KEY (workspace_id, ducklake_name, catalog, storage, storage_ref, data_path)
);
-- Resolution runs under user_db transactions (SET LOCAL ROLE) in API contexts, so the
-- windmill roles need explicit grants (default privileges don't apply to app-created tables).
GRANT ALL ON fork_ducklake_namespace TO windmill_user;
GRANT ALL ON fork_ducklake_namespace TO windmill_admin;
backend/parsers/windmill-parser-py-asset/src/lib.rs
+83 -13
View File
@@ -319,12 +319,24 @@ fn dict_str_value(dict: &rustpython_ast::ExprDict, name: &str) -> Option<String>
/// `write_s3_file` to a canonical asset path, mirroring `windmill-parser-ts-asset`:
/// `S3Object(s3="<key>", storage="<bucket>"?)` — or the equivalent dict literal —
/// maps to the URI `s3://<bucket>/<key>` (empty bucket for default storage, i.e.
/// `s3:///<key>`), and a bare `"s3://bucket/key"` string is passed through.
/// `s3:///<key>`), and a `"s3://bucket/key"` URI string is passed through.
/// String args mirror the runtime `parse_s3_object` contract exactly: only a
/// `s3://<storage>/<key>` URI with a non-empty key is valid — any other
/// string (bare key, `s3://x`, empty key) raises at run time, so recording an
/// edge for it would be a phantom node for a call that can only error.
/// The resulting URI is fed through `parse_asset_syntax` so the stored path
/// matches the TS object form and the `# on s3:///…` trigger form exactly.
fn s3_object_arg_path(expr: &Expr) -> Option<String> {
let uri = match expr {
Expr::Constant(ExprConstant { value: Constant::Str(s), .. }) => s.clone(),
Expr::Constant(ExprConstant { value: Constant::Str(s), .. }) => {
match s
.strip_prefix("s3://")
.and_then(|rest| rest.split_once('/'))
{
Some((_, key)) if !key.is_empty() => s.clone(),
_ => return None,
}
}
Expr::Call(call) => {
// `S3Object(...)` imported directly or as `wmill.S3Object(...)`
let func_name = call
@@ -371,13 +383,47 @@ def main():
s.map_err(|e| e.to_string()),
Ok(vec![ParseAssetsResult {
kind: AssetKind::S3Object,
path: "/test.csv".to_string(),
path: "test.csv".to_string(),
access_type: Some(R),
columns: None,
},])
);
}
#[test]
fn test_py_asset_parser_bare_string_no_asset() {
// A plain (non-`s3://`) string is rejected by the runtime
// `parse_s3_object`, so the parser must not record a phantom asset
// for a call that can only error.
let input = r#"
import wmill
def main():
wmill.write_s3_file("pipelines/etl/out.jsonl", b"")
"#;
let s = parse_assets(input).map(|o| o.assets);
assert_eq!(s.map_err(|e| e.to_string()), Ok(vec![]));
}
#[test]
fn test_py_asset_parser_invalid_uri_no_write_edge() {
// `s3://x` (no key part) and `s3://bucket/` (empty key) are rejected
// by the runtime `parse_s3_object` — same rule for the SDK-arg path:
// no R/W edge. The generic URI-literal scan may still record them as
// ambiguous (`access_type: None`) assets, like any `s3://…` string
// constant anywhere in a script.
let input = r#"
import wmill
def main():
wmill.write_s3_file("s3://broken", b"")
wmill.write_s3_file("s3://bucket/", b"")
"#;
let assets = parse_assets(input).expect("parse").assets;
assert!(
assets.iter().all(|a| a.access_type.is_none()),
"invalid URIs must not produce R/W edges: {assets:?}"
);
}
#[test]
fn test_py_asset_parser_write_s3_object_constructor() {
// The SDK signature is `write_s3_file(s3object: S3Object | str, ...)` and
@@ -395,7 +441,31 @@ def main():
s.map_err(|e| e.to_string()),
Ok(vec![ParseAssetsResult {
kind: AssetKind::S3Object,
path: "/analytics/x.csv".to_string(),
path: "analytics/x.csv".to_string(),
access_type: Some(W),
columns: None,
},])
);
}
#[test]
fn test_py_write_key_matches_duckdb_read_key() {
// Cross-language lineage: this write records `exports/x`, the same path a
// DuckDB `read_csv('s3://exports/x')` resolves to (see
// windmill-parser-sql-asset `test_duckdb_read_key_matches_sdk_write_key`),
// so the producer and consumer connect in the pipeline graph.
let input = r#"
import wmill
from wmill import S3Object
def main():
wmill.write_s3_file(S3Object(s3="exports/x"), b"content")
"#;
let s = parse_assets(input).map(|o| o.assets);
assert_eq!(
s.map_err(|e| e.to_string()),
Ok(vec![ParseAssetsResult {
kind: AssetKind::S3Object,
path: "exports/x".to_string(),
access_type: Some(W),
columns: None,
},])
@@ -438,7 +508,7 @@ def main():
s.map_err(|e| e.to_string()),
Ok(vec![ParseAssetsResult {
kind: AssetKind::S3Object,
path: "/dir/in.csv".to_string(),
path: "dir/in.csv".to_string(),
access_type: Some(R),
columns: None,
},])
@@ -461,14 +531,14 @@ def main():
Ok(vec![
ParseAssetsResult {
kind: AssetKind::S3Object,
path: "/out.json".to_string(),
access_type: Some(W),
path: "mybucket/dir/in.csv".to_string(),
access_type: Some(R),
columns: None,
},
ParseAssetsResult {
kind: AssetKind::S3Object,
path: "mybucket/dir/in.csv".to_string(),
access_type: Some(R),
path: "out.json".to_string(),
access_type: Some(W),
columns: None,
},
])
@@ -494,25 +564,25 @@ def main():
Ok(vec![
ParseAssetsResult {
kind: AssetKind::S3Object,
path: "/pipelines/km_real/enriched.json".to_string(),
path: "pipelines/km_real/enriched.json".to_string(),
access_type: Some(W),
columns: None,
},
ParseAssetsResult {
kind: AssetKind::S3Object,
path: "/pipelines/km_real/raw_events.json".to_string(),
path: "pipelines/km_real/raw_events.json".to_string(),
access_type: Some(W),
columns: None,
},
ParseAssetsResult {
kind: AssetKind::S3Object,
path: "/pipelines/km_real/report.json".to_string(),
path: "pipelines/km_real/report.json".to_string(),
access_type: Some(W),
columns: None,
},
ParseAssetsResult {
kind: AssetKind::S3Object,
path: "/pipelines/km_real/summary.json".to_string(),
path: "pipelines/km_real/summary.json".to_string(),
access_type: Some(W),
columns: None,
},
backend/parsers/windmill-parser-sql-asset/src/asset_parser.rs
+38 -14
View File
@@ -1145,13 +1145,13 @@ mod tests {
Ok(vec![
ParseAssetsResult {
kind: AssetKind::S3Object,
path: "/a.parquet".to_string(),
path: "a.parquet".to_string(),
access_type: Some(R),
columns: None
},
ParseAssetsResult {
kind: AssetKind::S3Object,
path: "/c.parquet".to_string(),
path: "c.parquet".to_string(),
access_type: Some(W),
columns: None
},
@@ -1165,6 +1165,30 @@ mod tests {
);
}
#[test]
fn test_duckdb_read_key_matches_sdk_write_key() {
// Cross-language lineage: a TS `writeS3File({ s3: "exports/x" })` or
// Python `write_s3_file(S3Object(s3="exports/x"))` records the asset path
// `exports/x` (default storage). A DuckDB reader of the same object must
// resolve to the identical path so the graph connects the producer and
// consumer — both the bare `s3://exports/x` and the triple-slash
// `s3:///exports/x` default-storage form must yield `exports/x`.
for uri in ["s3://exports/x", "s3:///exports/x"] {
let input = format!("SELECT * FROM read_csv('{uri}');");
let assets = parse_assets(&input).expect("parse").assets;
assert_eq!(
assets,
vec![ParseAssetsResult {
kind: AssetKind::S3Object,
path: "exports/x".to_string(),
access_type: Some(R),
columns: None
}],
"DuckDB read of {uri} must resolve to the SDK write key"
);
}
}
#[test]
fn test_copy_target_echoed_in_select_stays_write_only() {
// The trailing summary SELECT merely mentions the COPY target — it
@@ -1178,7 +1202,7 @@ mod tests {
s.map_err(|e| e.to_string()),
Ok(vec![ParseAssetsResult {
kind: AssetKind::S3Object,
path: "/out.csv".to_string(),
path: "out.csv".to_string(),
access_type: Some(W),
columns: None
}])
@@ -1195,7 +1219,7 @@ mod tests {
s.map_err(|e| e.to_string()),
Ok(vec![ParseAssetsResult {
kind: AssetKind::S3Object,
path: "/referenced.csv".to_string(),
path: "referenced.csv".to_string(),
access_type: Some(R),
columns: None
}])
@@ -1215,7 +1239,7 @@ mod tests {
s.map_err(|e| e.to_string()),
Ok(vec![ParseAssetsResult {
kind: AssetKind::S3Object,
path: "/data.csv".to_string(),
path: "data.csv".to_string(),
access_type: Some(RW),
columns: None
}])
@@ -1235,7 +1259,7 @@ mod tests {
s.map_err(|e| e.to_string()),
Ok(vec![ParseAssetsResult {
kind: AssetKind::S3Object,
path: "/data.parquet".to_string(),
path: "data.parquet".to_string(),
access_type: Some(RW),
columns: None
}])
@@ -1253,13 +1277,13 @@ mod tests {
Ok(vec![
ParseAssetsResult {
kind: AssetKind::S3Object,
path: "/a.parquet".to_string(),
path: "a.parquet".to_string(),
access_type: Some(R),
columns: None
},
ParseAssetsResult {
kind: AssetKind::S3Object,
path: "/b.parquet".to_string(),
path: "b.parquet".to_string(),
access_type: Some(R),
columns: None
}
@@ -1279,7 +1303,7 @@ mod tests {
s.map_err(|e| e.to_string()),
Ok(vec![ParseAssetsResult {
kind: AssetKind::S3Object,
path: "/data.parquet".to_string(),
path: "data.parquet".to_string(),
access_type: Some(RW),
columns: None
}])
@@ -1961,7 +1985,7 @@ mod tests {
assert_eq!(result.len(), 1);
assert_eq!(result[0].kind, AssetKind::S3Object);
assert_eq!(result[0].path, "/example_file.parquet");
assert_eq!(result[0].path, "example_file.parquet");
assert_eq!(result[0].access_type, Some(R));
let columns = result[0].columns.as_ref().expect("Should have columns");
@@ -1992,7 +2016,7 @@ mod tests {
assert_eq!(result.len(), 1);
assert_eq!(result[0].kind, AssetKind::S3Object);
assert_eq!(result[0].path, "/example_file.parquet");
assert_eq!(result[0].path, "example_file.parquet");
assert!(result[0].columns.is_none());
}
@@ -2005,7 +2029,7 @@ mod tests {
assert_eq!(result.len(), 1);
assert_eq!(result[0].kind, AssetKind::S3Object);
assert_eq!(result[0].path, "/example_file.parquet");
assert_eq!(result[0].path, "example_file.parquet");
let columns = result[0].columns.as_ref().expect("Should have columns");
assert_eq!(columns.get("col1"), Some(&R));
@@ -2024,7 +2048,7 @@ mod tests {
assert_eq!(result.len(), 2);
assert!(result.iter().any(|a| {
a.path == "/file1.parquet"
a.path == "file1.parquet"
&& a.columns.as_ref().map_or(false, |c| c.contains_key("col1"))
}));
assert!(result.iter().any(|a| {
@@ -2057,7 +2081,7 @@ mod tests {
assert_eq!(result.len(), 1);
assert_eq!(result[0].kind, AssetKind::S3Object);
assert_eq!(result[0].path, "/test.parquet");
assert_eq!(result[0].path, "test.parquet");
assert_eq!(result[0].access_type, Some(R));
let columns = result[0].columns.as_ref().expect("Should have columns");
backend/parsers/windmill-parser-ts-asset/src/lib.rs
+82 -9
View File
@@ -341,12 +341,25 @@ fn object_str_prop(obj: &ObjectLit, name: &str) -> Option<String> {
/// `writeS3File` to a canonical asset path, mirroring the runtime
/// `parseS3Object`: an object `{ s3: "<key>", storage?: "<bucket>" }` maps to
/// the URI `s3://<bucket>/<key>` (empty bucket for default storage, i.e.
/// `s3:///<key>`), and a bare `"s3://bucket/key"` string is passed through.
/// `s3:///<key>`), and a `"s3://bucket/key"` URI string is passed through.
/// String args mirror the runtime `parseS3Object` contract exactly: only a
/// `s3://<storage>/<key>` URI with a non-empty key is valid — any other
/// string (bare key, `s3://x`, empty key) throws at run time, so recording an
/// edge for it would be a phantom node for a call that can only error.
/// The resulting URI is fed through `parse_asset_syntax` so the stored path
/// matches the `// on s3:///…` trigger form exactly.
fn s3_object_arg_path(arg: &Expr) -> Option<String> {
let uri = match arg {
Expr::Lit(Lit::Str(s)) => s.value.to_string(),
Expr::Lit(Lit::Str(s)) => {
let v = s.value.to_string();
match v
.strip_prefix("s3://")
.and_then(|rest| rest.split_once('/'))
{
Some((_, key)) if !key.is_empty() => v,
_ => return None,
}
}
Expr::Object(obj) => {
let key = object_str_prop(obj, "s3")?;
let storage = object_str_prop(obj, "storage").unwrap_or_default();
@@ -420,7 +433,7 @@ mod tests {
s.map(|r| r.assets).map_err(|e| e.to_string()),
Ok(vec![ParseAssetsResult {
kind: AssetKind::S3Object,
path: "/test.csv".to_string(),
path: "test.csv".to_string(),
access_type: Some(R),
columns: None,
},])
@@ -448,7 +461,31 @@ mod tests {
s.map(|r| r.assets).map_err(|e| e.to_string()),
Ok(vec![ParseAssetsResult {
kind: AssetKind::S3Object,
path: "/pipelines/km_real/raw_events.json".to_string(),
path: "pipelines/km_real/raw_events.json".to_string(),
access_type: Some(W),
columns: None,
},])
);
}
#[test]
fn test_ts_write_key_matches_duckdb_read_key() {
// Cross-language lineage: this write records `exports/x`, the same path a
// DuckDB `read_csv('s3://exports/x')` resolves to (see
// windmill-parser-sql-asset `test_duckdb_read_key_matches_sdk_write_key`),
// so the producer and consumer connect in the pipeline graph.
let input = r#"
import * as wmill from "windmill-client"
export async function main() {
await wmill.writeS3File({ s3: "exports/x" }, "[]")
}
"#;
let s = parse_assets(input);
assert_eq!(
s.map(|r| r.assets).map_err(|e| e.to_string()),
Ok(vec![ParseAssetsResult {
kind: AssetKind::S3Object,
path: "exports/x".to_string(),
access_type: Some(W),
columns: None,
},])
@@ -477,6 +514,42 @@ mod tests {
);
}
#[test]
fn test_ts_asset_parser_bare_string_no_asset() {
// A plain (non-`s3://`) string is rejected by the runtime
// `parseS3Object`, so the parser must not record a phantom asset for
// a call that can only error.
let input = r#"
import * as wmill from "windmill-client"
export async function main() {
await wmill.writeS3File("pipelines/etl/out.jsonl", "[]")
}
"#;
let s = parse_assets(input);
assert_eq!(s.map(|r| r.assets).map_err(|e| e.to_string()), Ok(vec![]));
}
#[test]
fn test_ts_asset_parser_invalid_uri_no_write_edge() {
// `s3://x` (no key part) and `s3://bucket/` (empty key) are rejected
// by the runtime `parseS3Object` — same rule for the SDK-arg path:
// no R/W edge. The generic URI-literal scan may still record them as
// ambiguous (`access_type: None`) assets, like any `s3://…` string
// constant anywhere in a script.
let input = r#"
import * as wmill from "windmill-client"
export async function main() {
await wmill.writeS3File("s3://broken", "[]")
await wmill.writeS3File("s3://bucket/", "[]")
}
"#;
let assets = parse_assets(input).expect("parse").assets;
assert!(
assets.iter().all(|a| a.access_type.is_none()),
"invalid URIs must not produce R/W edges: {assets:?}"
);
}
#[test]
fn test_ts_asset_parser_multiple_s3_object_writes() {
// Mirrors the f/km/r_seed shape: several direct object-form writes in
@@ -497,25 +570,25 @@ mod tests {
Ok(vec![
ParseAssetsResult {
kind: AssetKind::S3Object,
path: "/pipelines/km_real/enriched.json".to_string(),
path: "pipelines/km_real/enriched.json".to_string(),
access_type: Some(W),
columns: None,
},
ParseAssetsResult {
kind: AssetKind::S3Object,
path: "/pipelines/km_real/raw_events.json".to_string(),
path: "pipelines/km_real/raw_events.json".to_string(),
access_type: Some(W),
columns: None,
},
ParseAssetsResult {
kind: AssetKind::S3Object,
path: "/pipelines/km_real/report.json".to_string(),
path: "pipelines/km_real/report.json".to_string(),
access_type: Some(W),
columns: None,
},
ParseAssetsResult {
kind: AssetKind::S3Object,
path: "/pipelines/km_real/summary.json".to_string(),
path: "pipelines/km_real/summary.json".to_string(),
access_type: Some(W),
columns: None,
},
@@ -536,7 +609,7 @@ mod tests {
s.map(|r| r.assets).map_err(|e| e.to_string()),
Ok(vec![ParseAssetsResult {
kind: AssetKind::S3Object,
path: "/out.json".to_string(),
path: "out.json".to_string(),
access_type: Some(W),
columns: None,
},])
backend/parsers/windmill-parser-wasm/Cargo.lock
Generated
+24 -24
View File
@@ -6191,7 +6191,7 @@ checksum = "712e227841d057c1ee1cd2fb22fa7e5a5461ae8e48fa2ca79ec42cfc1931183f"
[[package]]
name = "windmill-common"
version = "1.747.0"
version = "1.749.0"
dependencies = [
"aho-corasick",
"anyhow",
@@ -6272,7 +6272,7 @@ dependencies = [
[[package]]
name = "windmill-macros"
version = "1.747.0"
version = "1.749.0"
dependencies = [
"proc-macro2",
"quote",
@@ -6284,7 +6284,7 @@ dependencies = [
[[package]]
name = "windmill-parser"
version = "1.747.0"
version = "1.749.0"
dependencies = [
"convert_case",
"serde",
@@ -6293,7 +6293,7 @@ dependencies = [
[[package]]
name = "windmill-parser-bash"
version = "1.747.0"
version = "1.749.0"
dependencies = [
"anyhow",
"lazy_static",
@@ -6305,7 +6305,7 @@ dependencies = [
[[package]]
name = "windmill-parser-csharp"
version = "1.747.0"
version = "1.749.0"
dependencies = [
"anyhow",
"serde_json",
@@ -6317,7 +6317,7 @@ dependencies = [
[[package]]
name = "windmill-parser-go"
version = "1.747.0"
version = "1.749.0"
dependencies = [
"anyhow",
"gosyn",
@@ -6329,7 +6329,7 @@ dependencies = [
[[package]]
name = "windmill-parser-graphql"
version = "1.747.0"
version = "1.749.0"
dependencies = [
"anyhow",
"lazy_static",
@@ -6341,7 +6341,7 @@ dependencies = [
[[package]]
name = "windmill-parser-java"
version = "1.747.0"
version = "1.749.0"
dependencies = [
"anyhow",
"serde_json",
@@ -6353,7 +6353,7 @@ dependencies = [
[[package]]
name = "windmill-parser-nu"
version = "1.747.0"
version = "1.749.0"
dependencies = [
"anyhow",
"nu-parser",
@@ -6364,7 +6364,7 @@ dependencies = [
[[package]]
name = "windmill-parser-php"
version = "1.747.0"
version = "1.749.0"
dependencies = [
"anyhow",
"itertools 0.14.0",
@@ -6375,7 +6375,7 @@ dependencies = [
[[package]]
name = "windmill-parser-py"
version = "1.747.0"
version = "1.749.0"
dependencies = [
"anyhow",
"itertools 0.14.0",
@@ -6387,7 +6387,7 @@ dependencies = [
[[package]]
name = "windmill-parser-py-asset"
version = "1.747.0"
version = "1.749.0"
dependencies = [
"anyhow",
"rustpython-ast",
@@ -6398,7 +6398,7 @@ dependencies = [
[[package]]
name = "windmill-parser-py-imports"
version = "1.747.0"
version = "1.749.0"
dependencies = [
"anyhow",
"async-recursion",
@@ -6420,7 +6420,7 @@ dependencies = [
[[package]]
name = "windmill-parser-r"
version = "1.747.0"
version = "1.749.0"
dependencies = [
"anyhow",
"serde_json",
@@ -6432,7 +6432,7 @@ dependencies = [
[[package]]
name = "windmill-parser-ruby"
version = "1.747.0"
version = "1.749.0"
dependencies = [
"anyhow",
"lazy_static",
@@ -6446,7 +6446,7 @@ dependencies = [
[[package]]
name = "windmill-parser-rust"
version = "1.747.0"
version = "1.749.0"
dependencies = [
"anyhow",
"convert_case",
@@ -6463,7 +6463,7 @@ dependencies = [
[[package]]
name = "windmill-parser-sql"
version = "1.747.0"
version = "1.749.0"
dependencies = [
"anyhow",
"lazy_static",
@@ -6476,7 +6476,7 @@ dependencies = [
[[package]]
name = "windmill-parser-sql-asset"
version = "1.747.0"
version = "1.749.0"
dependencies = [
"anyhow",
"serde",
@@ -6488,7 +6488,7 @@ dependencies = [
[[package]]
name = "windmill-parser-ts"
version = "1.747.0"
version = "1.749.0"
dependencies = [
"anyhow",
"lazy_static",
@@ -6506,7 +6506,7 @@ dependencies = [
[[package]]
name = "windmill-parser-ts-asset"
version = "1.747.0"
version = "1.749.0"
dependencies = [
"anyhow",
"serde-wasm-bindgen",
@@ -6522,7 +6522,7 @@ dependencies = [
[[package]]
name = "windmill-parser-wac"
version = "1.747.0"
version = "1.749.0"
dependencies = [
"anyhow",
"rustpython-ast",
@@ -6538,7 +6538,7 @@ dependencies = [
[[package]]
name = "windmill-parser-wasm"
version = "1.747.0"
version = "1.749.0"
dependencies = [
"anyhow",
"getrandom 0.2.17",
@@ -6570,7 +6570,7 @@ dependencies = [
[[package]]
name = "windmill-parser-yaml"
version = "1.747.0"
version = "1.749.0"
dependencies = [
"anyhow",
"serde",
@@ -6581,7 +6581,7 @@ dependencies = [
[[package]]
name = "windmill-types"
version = "1.747.0"
version = "1.749.0"
dependencies = [
"anyhow",
"bitflags",
backend/parsers/windmill-parser-wasm/Cargo.toml
+1 -1
View File
@@ -12,7 +12,7 @@ resolver = "2"
members = ["."]
[workspace.package]
version = "1.747.0"
version = "1.749.0"
edition = "2021"
authors = ["Ruben Fiszel <ruben@windmill.dev>"]
backend/parsers/windmill-parser/src/asset_parser.rs
+420 -5
View File
@@ -77,10 +77,12 @@ pub struct ParseAssetsOutput {
#[serde(skip_serializing_if = "Option::is_none", default)]
pub partition: Option<PartitionSpec>,
// `// freshness <duration>` — SLA stating outputs must be at most
// `duration` old. Active backstop: when no other trigger has fired the
// script within the window, a watchdog re-runs it. Distinct from
// schedule (which is producer cadence); freshness is consumer SLA and
// applies regardless of which trigger last fired.
// `duration` old. Drives passive monitoring in CE (the asset graph
// colors the node's badge fresh/stale against its last successful run)
// and the enterprise watchdog (windmill-queue `freshness_watchdog`),
// which re-runs a stale unpartitioned producer. Distinct from schedule
// (which is producer cadence); freshness is consumer SLA and applies
// regardless of which trigger last fired.
#[serde(skip_serializing_if = "Option::is_none", default)]
pub freshness: Option<FreshnessSpec>,
// `// trigger all` → AND join barrier; default (`any`) = OR (current
@@ -250,6 +252,8 @@ pub struct RetrySpec {
// history (`valid_from`/`valid_to`/`is_current`). The leading keyword `scd2` is a
// recognized alias for `history`. `deletes=close` (scd2 only) also closes a key
// that disappears from the snapshot; default leaves absent keys current.
// `on_schema_change=ignore` suppresses downstream schema-contract warnings for
// the produced asset (save-time metadata only; default `warn`).
#[derive(Serialize, Debug, PartialEq, Clone)]
pub struct MaterializeSpec {
pub target_kind: AssetKind,
@@ -274,6 +278,117 @@ pub struct MaterializeSpec {
// `hard_deletes=close`). Default (false) leaves absent keys current.
#[serde(skip_serializing_if = "std::ops::Not::not", default)]
pub close_deleted: bool,
// `on_schema_change=warn|ignore|fail|sync` governs two orthogonal things:
// • Save-time contract warnings (gap #2b): consumers referencing columns
// the captured schema no longer has warn by default; only `ignore`
// suppresses those warnings (`warn`/`fail`/`sync` all keep them).
// • Run-time write guardrails for the persist-and-mutate strategies
// (partitioned replace, merge, append), where the table schema is fixed
// at first CREATE and the write is positional — a renamed/added/removed
// SELECT column silently lands in the wrong column. `warn` logs the
// drift and proceeds positionally; `fail` aborts before mutating; `sync`
// ALTERs the table to match and writes by name. Whole-table replace and
// scd2 are unaffected. See `sql_materialize.rs`.
#[serde(skip_serializing_if = "OnSchemaChange::is_warn", default)]
pub on_schema_change: OnSchemaChange,
}
impl MaterializeSpec {
/// The `<target>_current` SCD2 companion view this managed materialize also
/// produces, or `None` when it isn't a managed scd2 target. Managed scd2
/// creates the base table *and* a `<dim>_current` "latest row per key" view
/// each run (see `sql_materialize.rs`); `manual` mode owns its own DDL and
/// short-circuits before that codegen, so it produces no companion.
pub fn scd2_current_target(&self) -> Option<(AssetKind, String)> {
(self.scd2 && !self.manual)
.then(|| (self.target_kind, format!("{}_current", self.target_path)))
}
/// Every asset this managed materialize produces: the base table, plus — for
/// managed scd2 — the `<target>_current` companion view. The producer's
/// trailing `SELECT` doesn't express these writes (the runtime generates the
/// DDL), so this is the single source of truth every graph surface (deploy
/// asset rows, the CLI `--local` graph, and the frontend live graph) uses to
/// link reads of the base *and* the `_current` view back to this producer.
pub fn write_targets(&self) -> Vec<(AssetKind, String)> {
let mut targets = vec![(self.target_kind, self.target_path.clone())];
targets.extend(self.scd2_current_target());
targets
}
}
// dbt's `on_schema_change`, covering both the save-time contract check and the
// run-time write guardrail for the positional persist-and-mutate strategies:
// • `warn` (default): surface consumer contract warnings; at write time, log
// the drift loudly and proceed with the positional write against the fixed
// table schema.
// • `ignore`: suppress consumer contract warnings; at write time, no guard
// (the pre-guardrail behaviour).
// • `fail`: keep contract warnings; at write time, abort the run before
// mutating when the SELECT's column *set* diverges from the table's.
// • `sync`: keep contract warnings; at write time, ALTER the table to match
// the SELECT (add/drop columns) and INSERT BY NAME.
// `warn`/`fail` drift detection is name-set based (added/removed columns), which
// is what the positional persist-and-mutate INSERT can misalign on. It does NOT
// flag a pure *reorder* of same-named columns: `SELECT b, a` into a `(a, b)`
// table has an identical column set, so `fail` does not abort and the positional
// INSERT swaps the values. Reorder-safety is exactly what `sync` provides
// (INSERT BY NAME maps by name), so a SELECT whose column order is not pinned to
// the table's should use `sync`, not `fail`. (An ordered-list comparison would
// close this, but a false positive there would abort a correctly-aligned write,
// so the guard stays on the set difference.)
// `fail`/`sync` only affect partitioned replace, merge and append; whole-table
// replace already rebuilds each run, and scd2 has no positional write — for an
// scd2 target `sync` degrades to `warn` (no write-time effect; deploy-time
// rejection is out of scope here).
#[derive(Serialize, Debug, PartialEq, Eq, Clone, Copy, Default)]
#[serde(rename_all = "lowercase")]
pub enum OnSchemaChange {
#[default]
Warn,
Ignore,
Fail,
Sync,
}
impl OnSchemaChange {
pub fn is_warn(&self) -> bool {
matches!(self, OnSchemaChange::Warn)
}
}
impl MaterializeSpec {
/// Deploy-time validation of the option combination against the script's
/// partitioning, returning a human-facing error for combinations the
/// runtime cannot honor. Called at save (`create_script_internal`) so a
/// misconfigured script is rejected up front, and again in the DuckDB
/// executor as a safety net for preview/test runs that never deploy. Both
/// checks are SCD2-specific and inert for `manual` mode (which owns its DDL
/// and ignores the reconciliation strategy). `partitioned` is whether the
/// script declares `// partitioned`.
pub fn validate(&self, partitioned: bool) -> Result<(), String> {
if self.manual || !self.scd2 {
return Ok(());
}
// SCD2 needs a natural key to identify an entity across versions.
if self.unique_key.as_deref().map_or(true, str::is_empty) {
return Err(
"materialize scd2: requires a natural key — add `key=<col>` (e.g. \
`// materialize ducklake://<name>/<table> key=id history`)"
.to_string(),
);
}
// SCD2's diff/close/open shape has no partition-scoped form in v1.
if partitioned {
return Err(
"materialize scd2: `// partitioned` is not supported with scd2 in v1 — remove \
`// partitioned`, or drop `history`/`scd2` to materialize the partition without \
history"
.to_string(),
);
}
Ok(())
}
}
// `// data_test <kind> …` — a data-quality assertion run against the
@@ -522,6 +637,24 @@ pub fn parse_asset_syntax(s: &str, enable_default_syntax: bool) -> Option<(Asset
for (prefix, kind) in ASSET_KINDS.iter() {
if s.starts_with(prefix) {
let path = &s[prefix.len()..];
// Canonicalize S3 keys to a single asset identity. The SDK object
// form (`{ s3: "key" }` / `S3Object(s3="key")`, default storage)
// resolves to `s3:///key`, whose path is `/key`, while DuckDB
// `s3://key` and `// on s3://key` yield the bare `key`. Strip every
// leading slash so the triple-slash default-storage form and the
// `s3://storage/key` form share one path — otherwise a TS/Python
// writer and a DuckDB reader of the same object become disconnected
// nodes in the pipeline graph. Stripping ALL leading slashes (not
// just one) keeps the identity stable through URI reconstruction:
// `trigger_spec_to_row` rebuilds `s3://<path>`, so a canonical path
// must never itself start with `/` or the rebuilt ref would parse
// back to a different key. Only leading slashes are touched, so
// Hive-partition keys (`s3://b/y=2024/f.parquet`) are untouched.
let path = if matches!(kind, AssetKind::S3Object) {
path.trim_start_matches('/')
} else {
path
};
return Some((*kind, path));
}
}
@@ -637,7 +770,7 @@ fn parse_kv_opts(s: &str) -> BTreeMap<String, String> {
// - `on <trigger-spec>` → asset / native trigger edge (including
// the marker-only `on schedule` form)
// - `partitioned <kind> [opts]` → partition declaration
// - `freshness <duration>` → SLA / active backstop
// - `freshness <duration>` → SLA window (badge + EE watchdog)
// - `tag <name>` → worker-tag override (annotation wins
// over UI-set value at deploy)
// - `retry <count> [<delay>]` → cascade-only retry policy
@@ -839,6 +972,39 @@ pub fn parse_pipeline_annotations(code: &str) -> PipelineAnnotations {
out
}
// Count `// data_test <…>` lines in the leading comment header whose right-hand
// side fails to parse into a check. `parse_pipeline_annotations` drops these
// fail-safe (a malformed line just yields no check), which is the wrong default
// for a data-quality assertion: a typo silently disables the test. The deploy
// path uses this count to warn. Same leading-block boundary as the parser (stop
// at the first non-comment line) so a body comment can't be miscounted, and the
// same `parse_data_test_spec` grammar so "malformed" means exactly what the
// parser rejects — no second grammar to drift.
pub fn count_malformed_data_tests(code: &str) -> usize {
let mut malformed = 0;
for raw_line in code.lines() {
let line = raw_line.trim_start();
if line.is_empty() {
continue;
}
let rest = if let Some(r) = line.strip_prefix("//") {
r
} else if let Some(r) = line.strip_prefix("--") {
r
} else if let Some(r) = line.strip_prefix('#') {
r
} else {
break;
};
if let Some(after_kw) = consume_keyword(rest.trim_start(), "data_test") {
if parse_data_test_spec(after_kw.trim()).is_none() {
malformed += 1;
}
}
}
malformed
}
// Parse a `// retry <count> [<delay>]` right-hand side. `<count>` is a
// non-negative decimal; `<delay>` is an optional raw duration string left
// for `parse_duration_secs` to validate at deploy. A bare zero count (or
@@ -915,6 +1081,15 @@ fn parse_materialize_spec(s: &str) -> Option<MaterializeSpec> {
// `deletes=close` (scd2 only) opts into hard-delete-close; any other value
// (or absence) keeps the soft-delete default.
let close_deleted = opts.get("deletes").map(|v| v == "close").unwrap_or(false);
// `on_schema_change=ignore|fail|sync`; any other value (or absence) keeps
// the `warn` default, fail-safe like `deletes=` above (a typo must never
// silently disable the guardrail).
let on_schema_change = match opts.get("on_schema_change").map(String::as_str) {
Some("ignore") => OnSchemaChange::Ignore,
Some("fail") => OnSchemaChange::Fail,
Some("sync") => OnSchemaChange::Sync,
_ => OnSchemaChange::Warn,
};
Some(MaterializeSpec {
target_kind,
target_path: path.to_string(),
@@ -924,6 +1099,7 @@ fn parse_materialize_spec(s: &str) -> Option<MaterializeSpec> {
scd2,
track,
close_deleted,
on_schema_change,
})
}
@@ -1123,6 +1299,83 @@ fn parse_trigger_spec(s: &str) -> Option<TriggerSpec> {
mod pipeline_annotation_tests {
use super::*;
#[test]
fn s3_key_normalization_unifies_uri_forms() {
// A TS/Python SDK write of `{ s3: "exports/x" }` (default storage)
// resolves to the URI `s3:///exports/x`, while a DuckDB read of
// `s3://exports/x` and the `// on s3://exports/x` trigger form yield the
// bare `exports/x`. All three must canonicalize to one asset key so
// the writer and reader connect in the pipeline graph.
let sdk_write = parse_asset_syntax("s3:///exports/x", false);
let duckdb_read = parse_asset_syntax("s3://exports/x", false);
assert_eq!(sdk_write, Some((AssetKind::S3Object, "exports/x")));
assert_eq!(duckdb_read, Some((AssetKind::S3Object, "exports/x")));
assert_eq!(sdk_write, duckdb_read);
// The `// on` trigger annotation goes through the same function.
assert_eq!(
parse_asset_syntax("s3:///exports/x", true),
parse_asset_syntax("s3://exports/x", true)
);
// Explicit-storage form is unaffected (no leading slash to strip).
assert_eq!(
parse_asset_syntax("s3://mybucket/exports/x", false),
Some((AssetKind::S3Object, "mybucket/exports/x"))
);
// Hive-partition keys and nested paths under default storage are
// preserved verbatim (only leading slashes are stripped).
assert_eq!(
parse_asset_syntax("s3:///t/year=2024/month=01/f.parquet", false),
Some((AssetKind::S3Object, "t/year=2024/month=01/f.parquet"))
);
// Every leading slash is stripped so a canonical S3 path never starts
// with `/`. `S3Object(s3="/x")` resolves to the quad-slash URI
// `s3:////x`; the identity must be the bare `x` (not `/x`) so the ref
// that `trigger_spec_to_row` rebuilds round-trips back to it.
assert_eq!(
parse_asset_syntax("s3:////x", false),
Some((AssetKind::S3Object, "x"))
);
assert_eq!(
parse_asset_syntax("s3://///deep///", false),
Some((AssetKind::S3Object, "deep///"))
);
// Non-S3 kinds keep their leading slash (their paths are workspace-
// relative and the slash is significant).
assert_eq!(
parse_asset_syntax("res://f/foo", false),
Some((AssetKind::Resource, "f/foo"))
);
assert_eq!(
parse_asset_syntax("ducklake://analytics/orders", false),
Some((AssetKind::Ducklake, "analytics/orders"))
);
}
#[test]
fn s3_explicit_storage_aliases_default_storage_nested_key() {
// Accepted tradeoff of one canonical key: the explicit-storage form
// `s3://storage/key` and the default-storage nested-key form
// `s3:///storage/key` collapse to the same node `storage/key`, even
// though they name different objects. This is a best-effort lineage
// graph that does not split the first segment as a storage name; the
// collision only happens when a storage config is named to match a
// default-storage prefix. Pinned so the aliasing is intentional, not a
// latent surprise.
assert_eq!(
parse_asset_syntax("s3://mybucket/x", false),
parse_asset_syntax("s3:///mybucket/x", false)
);
assert_eq!(
parse_asset_syntax("s3://mybucket/x", false),
Some((AssetKind::S3Object, "mybucket/x"))
);
}
#[test]
fn bare_pipeline_marker() {
let out = parse_pipeline_annotations("// pipeline\nconsole.log('hi')");
@@ -1606,6 +1859,62 @@ mod pipeline_annotation_tests {
assert!(!m.close_deleted);
}
#[test]
fn materialize_scd2_write_targets_include_current_view() {
// A managed scd2 materialize produces the base table AND the
// `<dim>_current` companion view, so the producer must be recorded as
// writing both (reads of the view otherwise resolve to an orphan asset).
let out = parse_pipeline_annotations(
"// materialize ducklake://main/dim_customers key=id history",
);
let m = out.materialize.expect("materialize");
assert_eq!(
m.write_targets(),
vec![
(AssetKind::Ducklake, "main/dim_customers".to_string()),
(
AssetKind::Ducklake,
"main/dim_customers_current".to_string()
),
]
);
assert_eq!(
m.scd2_current_target(),
Some((
AssetKind::Ducklake,
"main/dim_customers_current".to_string()
))
);
}
#[test]
fn materialize_non_scd2_write_targets_are_base_only() {
// A plain merge (no `history`) creates no companion view — only the base.
let out = parse_pipeline_annotations("// materialize ducklake://main/dim_customers key=id");
let m = out.materialize.expect("materialize");
assert_eq!(
m.write_targets(),
vec![(AssetKind::Ducklake, "main/dim_customers".to_string())]
);
assert_eq!(m.scd2_current_target(), None);
}
#[test]
fn materialize_manual_scd2_has_no_companion_view() {
// `manual` mode owns its own DDL and never creates the `_current` view,
// so registering it would be a false producer edge.
let out = parse_pipeline_annotations(
"// materialize manual ducklake://main/dim_customers key=id history",
);
let m = out.materialize.expect("materialize");
assert!(m.manual && m.scd2);
assert_eq!(m.scd2_current_target(), None);
assert_eq!(
m.write_targets(),
vec![(AssetKind::Ducklake, "main/dim_customers".to_string())]
);
}
#[test]
fn materialize_scd2_deletes_close_opt() {
let out = parse_pipeline_annotations(
@@ -1621,6 +1930,93 @@ mod pipeline_annotation_tests {
assert!(!out.materialize.expect("materialize").close_deleted);
}
#[test]
fn materialize_validate_scd2_requires_key() {
// scd2 without `key=` is rejected at deploy (was a run-time error).
let m = parse_pipeline_annotations("// materialize ducklake://a/dim history")
.materialize
.expect("materialize");
assert!(m.scd2 && m.unique_key.is_none());
let err = m.validate(false).expect_err("scd2 without key must fail");
assert!(err.contains("requires a natural key"));
// with a key it validates
let m = parse_pipeline_annotations("// materialize ducklake://a/dim key=id history")
.materialize
.expect("materialize");
assert!(m.validate(false).is_ok());
}
#[test]
fn materialize_validate_scd2_rejects_partitioned() {
// scd2 + `// partitioned` has no v1 form — rejected at deploy.
let m = parse_pipeline_annotations("// materialize ducklake://a/dim key=id history")
.materialize
.expect("materialize");
let err = m.validate(true).expect_err("scd2 + partitioned must fail");
assert!(err.contains("`// partitioned` is not supported with scd2"));
// unpartitioned scd2 is fine
assert!(m.validate(false).is_ok());
}
#[test]
fn materialize_validate_non_scd2_and_manual_are_inert() {
// Non-scd2 strategies are unconstrained by these checks, partitioned or not.
let m = parse_pipeline_annotations("// materialize ducklake://a/orders key=id")
.materialize
.expect("materialize");
assert!(m.validate(true).is_ok());
assert!(m.validate(false).is_ok());
// `manual` owns its DDL and ignores the strategy — never rejected here,
// even with a partitioned scd2-looking combo.
let m = parse_pipeline_annotations("// materialize manual ducklake://a/dim history")
.materialize
.expect("materialize");
assert!(m.manual && m.scd2);
assert!(m.validate(true).is_ok());
}
#[test]
fn materialize_on_schema_change_opt() {
let out = parse_pipeline_annotations(
"// materialize ducklake://a/orders on_schema_change=ignore",
);
let m = out.materialize.expect("materialize");
assert_eq!(m.on_schema_change, OnSchemaChange::Ignore);
// default is warn
let out = parse_pipeline_annotations("// materialize ducklake://a/orders");
assert_eq!(
out.materialize.expect("materialize").on_schema_change,
OnSchemaChange::Warn
);
// fail + sync parse to their own variants
let out =
parse_pipeline_annotations("// materialize ducklake://a/orders on_schema_change=fail");
assert_eq!(
out.materialize.expect("materialize").on_schema_change,
OnSchemaChange::Fail
);
let out =
parse_pipeline_annotations("// materialize ducklake://a/orders on_schema_change=sync");
assert_eq!(
out.materialize.expect("materialize").on_schema_change,
OnSchemaChange::Sync
);
// unknown/junk value keeps the warn default (fail-safe, like `deletes=`)
let out =
parse_pipeline_annotations("// materialize ducklake://a/orders on_schema_change=bogus");
assert_eq!(
out.materialize.expect("materialize").on_schema_change,
OnSchemaChange::Warn
);
// composes with other opts
let out = parse_pipeline_annotations(
"// materialize ducklake://a/dim key=id history on_schema_change=ignore",
);
let m = out.materialize.expect("materialize");
assert!(m.scd2);
assert_eq!(m.on_schema_change, OnSchemaChange::Ignore);
}
#[test]
fn materialize_key_without_history_is_plain_merge() {
let out = parse_pipeline_annotations("// materialize ducklake://a/dim key=id");
@@ -1950,4 +2346,23 @@ mod pipeline_annotation_tests {
));
assert!(out.column_lineage.is_empty());
}
#[test]
fn count_malformed_data_tests_counts_only_broken_header_lines() {
let n = count_malformed_data_tests(concat!(
"-- data_test not_null id\n", // valid
"-- data_test unique id\n", // valid
"-- data_test accepted_values status paid\n", // malformed: missing `=`
"-- data_test relationships cust ducklake\n", // malformed: no `->`
"-- data_test\n", // malformed: bare keyword
"-- data_test f/tests/custom\n", // valid: custom script path
"SELECT 1 -- data_test not_a_test\n", // body line: not counted
));
assert_eq!(n, 3);
// A clean header has zero.
assert_eq!(
count_malformed_data_tests("-- data_test unique id\nSELECT 1"),
0
);
}
}
backend/parsers/windmill-parser/src/sql_materialize.rs
+881 -124
View File
File diff suppressed because it is too large Load Diff
backend/parsers/windmill-parser/tests/fixtures/pipeline_annotations.json
Vendored
+67
View File
@@ -310,6 +310,43 @@
}
}
},
{
"name": "materialize on_schema_change=ignore opt",
"code": "// pipeline\n// materialize ducklake://analytics/orders on_schema_change=ignore\nSELECT 1;",
"expected": {
"in_pipeline": true,
"asset_triggers": [],
"native_triggers": [],
"partition": null,
"freshness": null,
"tag": null,
"retry": null,
"materialize": {
"target_kind": "ducklake",
"target_path": "analytics/orders",
"on_schema_change": "ignore"
}
}
},
{
"name": "materialize on_schema_change unknown value keeps warn default (fail-safe)",
"code": "// materialize ducklake://analytics/orders key=id on_schema_change=bogus\nSELECT 1;",
"expected": {
"in_pipeline": false,
"asset_triggers": [],
"native_triggers": [],
"partition": null,
"freshness": null,
"tag": null,
"retry": null,
"materialize": {
"target_kind": "ducklake",
"target_path": "analytics/orders",
"unique_key": "id",
"on_schema_change": "warn"
}
}
},
{
"name": "materialize key without history is plain merge (SCD1, not scd2)",
"code": "// materialize ducklake://analytics/dim key=id\nSELECT 1;",
@@ -702,5 +739,35 @@
"tag": null,
"retry": null
}
},
{
"name": "s3 triple-slash default-storage trigger canonicalizes to bare key",
"code": "// pipeline\n// on s3:///exports/x\nexport function main() {}",
"expected": {
"in_pipeline": true,
"asset_triggers": [
"s3object:exports/x"
],
"native_triggers": [],
"partition": null,
"freshness": null,
"tag": null,
"retry": null
}
},
{
"name": "s3 quad-slash trigger strips all leading slashes to the bare key",
"code": "// pipeline\n// on s3:////x\nexport function main() {}",
"expected": {
"in_pipeline": true,
"asset_triggers": [
"s3object:x"
],
"native_triggers": [],
"partition": null,
"freshness": null,
"tag": null,
"retry": null
}
}
]
backend/parsers/windmill-parser/tests/pipeline_annotations_parity.rs
+17
View File
@@ -71,6 +71,13 @@ struct ExpectedMaterialize {
track: Vec<String>,
#[serde(default)]
close_deleted: bool,
// "warn" | "ignore"; absent === "warn" (the default).
#[serde(default = "default_on_schema_change")]
on_schema_change: String,
}
fn default_on_schema_change() -> String {
"warn".to_string()
}
#[derive(Deserialize)]
@@ -209,6 +216,16 @@ fn pipeline_annotation_fixtures_match() {
m.close_deleted, e.close_deleted,
"{ctx}: materialize close_deleted"
);
let osc = match m.on_schema_change {
windmill_parser::asset_parser::OnSchemaChange::Warn => "warn",
windmill_parser::asset_parser::OnSchemaChange::Ignore => "ignore",
windmill_parser::asset_parser::OnSchemaChange::Fail => "fail",
windmill_parser::asset_parser::OnSchemaChange::Sync => "sync",
};
assert_eq!(
osc, e.on_schema_change,
"{ctx}: materialize on_schema_change"
);
}
(got, want) => panic!(
"{ctx}: materialize mismatch — got {:?}, want present={}",
backend/src/monitor.rs
+31
View File
@@ -178,6 +178,14 @@ lazy_static::lazy_static! {
.and_then(|x| x.parse::<bool>().ok())
.unwrap_or(false);
// Ops kill switch for the pipeline freshness watchdog (a background
// pusher — being able to stop it without a redeploy matters more than
// for read-only monitors).
pub static ref DISABLE_FRESHNESS_WATCHDOG: bool = std::env::var("DISABLE_FRESHNESS_WATCHDOG")
.ok()
.and_then(|x| x.parse::<bool>().ok())
.unwrap_or(false);
pub static ref WORKERS_NAMES: Arc<RwLock<Vec<String>>> = Arc::new(RwLock::new(Vec::new()));
static ref QUEUE_COUNT_TAGS: Arc<RwLock<Vec<String>>> = Arc::new(RwLock::new(Vec::new()));
@@ -2994,6 +3002,28 @@ pub async fn monitor_db(
}
};
// run every ~60s (2 iterations * 30s). Enterprise feature: the active
// `// freshness` backstop lives in windmill-queue's `freshness_watchdog`
// (`private`); OSS gets a no-op stub. Runtime-gated on an Enterprise
// license like the audit export above. Safe on concurrent servers — the
// watchdog claims per-script state rows atomically before pushing.
let pipeline_freshness_watchdog_f = async {
if server_mode
&& !*DISABLE_FRESHNESS_WATCHDOG
&& iteration.is_some()
&& iteration.as_ref().unwrap().should_run(2)
{
if let Some(db) = conn.as_sql() {
if matches!(
windmill_common::ee_oss::get_license_plan().await,
windmill_common::ee_oss::LicensePlan::Enterprise
) {
windmill_queue::freshness_watchdog::tick(db).await;
}
}
}
};
join!(
expired_items_f,
zombie_jobs_f,
@@ -3021,6 +3051,7 @@ pub async fn monitor_db(
manage_audit_partitions_f,
export_audit_logs_to_object_store_f,
cleanup_scheduled_job_deletions_f,
pipeline_freshness_watchdog_f,
);
}
backend/tests/freshness_watchdog.rs
+294
View File
@@ -0,0 +1,294 @@
//! End-to-end tests for the pipeline freshness watchdog (Enterprise).
//!
//! `windmill_queue::freshness_watchdog::tick` is called directly against
//! seeded `script` / `v2_job(_completed)` rows — no worker or API server is
//! needed, since the watchdog's job ends at the push (the pushed job sitting
//! in `v2_job_queue` is itself part of the assertions). Covers: staleness on
//! never-ran and aged-out members, the fresh short-circuit + state reset,
//! the in-flight suppression, the backoff claim, and the skip rules
//! (partitioned, malformed window, non-pipeline scripts).
#![cfg(feature = "private")]
use sqlx::{Pool, Postgres};
use windmill_queue::freshness_watchdog::tick;
use windmill_test_utils::initialize_tracing;
const WS: &str = "test-workspace";
const PATH: &str = "u/test-user/freshness_producer";
/// Seed a deployed pipeline-member script. Mirrors the deploy path's output:
/// `auto_kind = 'pipeline'`, empty (non-NULL) lock so run-by-path resolution
/// treats it as deployed, hash derived from path+content for uniqueness.
async fn seed_pipeline_script(
db: &Pool<Postgres>,
path: &str,
content: &str,
) -> anyhow::Result<()> {
let mut h = 0i64;
for b in path.bytes().chain(content.bytes()) {
h = h.wrapping_mul(31).wrapping_add(b as i64);
}
sqlx::query(
r#"INSERT INTO script (workspace_id, hash, path, summary, description, content,
created_by, language, tag, lock, auto_kind)
VALUES ($1, $2, $3, '', '', $4, 'test-user', 'bash'::script_lang, 'bash', '', 'pipeline')
ON CONFLICT DO NOTHING"#,
)
.bind(WS)
.bind(h)
.bind(path)
.bind(content)
.execute(db)
.await?;
// Process-global deployed-script caches are keyed by (workspace, path) /
// (workspace, hash) and would leak between #[sqlx::test] isolated DBs
// that reuse both — resolve everything from this test's own DB.
windmill_common::DEPLOYED_SCRIPT_CACHE_DISABLED
.store(true, std::sync::atomic::Ordering::Relaxed);
Ok(())
}
/// Seed a completed root run of `path` that finished `age_s` seconds ago.
async fn seed_completed_run(
db: &Pool<Postgres>,
path: &str,
age_s: i64,
success: bool,
) -> anyhow::Result<()> {
let id = uuid::Uuid::new_v4();
sqlx::query(
r#"INSERT INTO v2_job (id, workspace_id, runnable_path, kind, created_at,
created_by, permissioned_as, permissioned_as_email, tag)
VALUES ($1, $2, $3, 'script'::job_kind,
now() - ($4::bigint::text || ' seconds')::interval,
'test-user', 'u/test-user', 'test@windmill.dev', 'bash')"#,
)
.bind(id)
.bind(WS)
.bind(path)
.bind(age_s)
.execute(db)
.await?;
sqlx::query(
r#"INSERT INTO v2_job_completed (id, workspace_id, duration_ms, status, started_at, completed_at)
VALUES ($1, $2, 0, CASE WHEN $3 THEN 'success'::job_status ELSE 'failure'::job_status END,
now() - ($4::bigint::text || ' seconds')::interval,
now() - ($4::bigint::text || ' seconds')::interval)"#,
)
.bind(id)
.bind(WS)
.bind(success)
.bind(age_s)
.execute(db)
.await?;
Ok(())
}
/// Jobs the watchdog pushed: (path, created_by, args) rows attributed to
/// `trigger_kind = 'freshness'`.
async fn fetch_pushed(
db: &Pool<Postgres>,
) -> anyhow::Result<Vec<(String, String, Option<serde_json::Value>)>> {
let rows = sqlx::query!(
r#"SELECT runnable_path AS "runnable_path!", created_by AS "created_by!",
args AS "args: sqlx::types::Json<serde_json::Value>"
FROM v2_job
WHERE workspace_id = $1 AND trigger_kind = 'freshness'
ORDER BY created_at"#,
WS,
)
.fetch_all(db)
.await?;
Ok(rows
.into_iter()
.map(|r| (r.runnable_path, r.created_by, r.args.map(|a| a.0)))
.collect())
}
async fn state_row(db: &Pool<Postgres>, path: &str) -> anyhow::Result<Option<i32>> {
Ok(sqlx::query_scalar!(
"SELECT attempts FROM pipeline_freshness_state WHERE workspace_id = $1 AND script_path = $2",
WS,
path,
)
.fetch_optional(db)
.await?)
}
#[sqlx::test(fixtures("base"))]
async fn never_ran_member_is_pushed_once(db: Pool<Postgres>) -> anyhow::Result<()> {
initialize_tracing().await;
seed_pipeline_script(&db, PATH, "# pipeline\n# freshness 30s\necho hi\n").await?;
tick(&db).await;
let pushed = fetch_pushed(&db).await?;
assert_eq!(pushed.len(), 1, "one watchdog push expected");
let (path, created_by, args) = &pushed[0];
assert_eq!(path, PATH);
assert_eq!(created_by, &format!("freshness-{PATH}"));
let args = args.as_ref().expect("args recorded");
assert_eq!(
args.get("_wmill_skip_asset_dispatch"),
Some(&serde_json::json!(true)),
"watchdog runs must not re-fire the cascade"
);
assert_eq!(
args.pointer("/trigger/kind"),
Some(&serde_json::json!("freshness"))
);
assert_eq!(state_row(&db, PATH).await?, Some(1), "claim row recorded");
// Second tick: the pushed job is queued-and-due, so the in-flight guard
// suppresses a duplicate regardless of backoff.
tick(&db).await;
assert_eq!(
fetch_pushed(&db).await?.len(),
1,
"no duplicate while queued"
);
// Simulate the queued job vanishing without a completion: the backoff
// claim (next_attempt_at in the future) now carries the suppression.
sqlx::query!("DELETE FROM v2_job_queue WHERE workspace_id = $1", WS)
.execute(&db)
.await?;
tick(&db).await;
assert_eq!(fetch_pushed(&db).await?.len(), 1, "backoff holds the retry");
// Force the backoff window open: the watchdog retries and escalates.
sqlx::query!(
"UPDATE pipeline_freshness_state SET next_attempt_at = now() - interval '1 second'
WHERE workspace_id = $1 AND script_path = $2",
WS,
PATH,
)
.execute(&db)
.await?;
tick(&db).await;
assert_eq!(fetch_pushed(&db).await?.len(), 2, "due retry pushed");
assert_eq!(state_row(&db, PATH).await?, Some(2), "attempts escalated");
Ok(())
}
#[sqlx::test(fixtures("base"))]
async fn fresh_member_is_skipped_and_state_reset(db: Pool<Postgres>) -> anyhow::Result<()> {
initialize_tracing().await;
seed_pipeline_script(&db, PATH, "# pipeline\n# freshness 1h\necho hi\n").await?;
seed_completed_run(&db, PATH, 10, true).await?;
// Leftover backoff row from an earlier staleness episode.
sqlx::query!(
"INSERT INTO pipeline_freshness_state (workspace_id, script_path, attempts) VALUES ($1, $2, 3)",
WS,
PATH,
)
.execute(&db)
.await?;
tick(&db).await;
assert!(
fetch_pushed(&db).await?.is_empty(),
"fresh member not pushed"
);
assert_eq!(state_row(&db, PATH).await?, None, "backoff reset on fresh");
Ok(())
}
#[sqlx::test(fixtures("base"))]
async fn aged_out_member_is_pushed(db: Pool<Postgres>) -> anyhow::Result<()> {
initialize_tracing().await;
seed_pipeline_script(&db, PATH, "# pipeline\n# freshness 1h\necho hi\n").await?;
// Old success outside the window + a recent failure: still stale.
seed_completed_run(&db, PATH, 7200, true).await?;
seed_completed_run(&db, PATH, 60, false).await?;
tick(&db).await;
assert_eq!(fetch_pushed(&db).await?.len(), 1, "aged-out member pushed");
Ok(())
}
#[sqlx::test(fixtures("base"))]
async fn partitioned_malformed_and_plain_members_are_skipped(
db: Pool<Postgres>,
) -> anyhow::Result<()> {
initialize_tracing().await;
// Partitioned: freshness means partition-gap detection, out of scope.
seed_pipeline_script(
&db,
"u/test-user/partitioned",
"# pipeline\n# partitioned daily\n# freshness 1h\necho hi\n",
)
.await?;
// Malformed window: fails safe to unwatched.
seed_pipeline_script(
&db,
"u/test-user/malformed",
"# pipeline\n# freshness soonish\necho hi\n",
)
.await?;
// Freshness only in prose (parser must reject; ILIKE prefilter passes).
seed_pipeline_script(
&db,
"u/test-user/prose",
"# pipeline\n# ensure freshness of data below\necho hi\n",
)
.await?;
tick(&db).await;
assert!(fetch_pushed(&db).await?.is_empty(), "no member is watched");
let rows = sqlx::query_scalar!(
r#"SELECT COUNT(*) AS "count!" FROM pipeline_freshness_state WHERE workspace_id = $1"#,
WS,
)
.fetch_one(&db)
.await?;
assert_eq!(rows, 0, "no state rows for unwatched members");
Ok(())
}
#[sqlx::test(fixtures("base"))]
async fn archived_workspace_is_not_resurrected(db: Pool<Postgres>) -> anyhow::Result<()> {
initialize_tracing().await;
// Workspace archival stops all execution but keeps script rows for
// unarchival — the watchdog must not keep pushing runs there.
seed_pipeline_script(&db, PATH, "# pipeline\n# freshness 30s\necho hi\n").await?;
sqlx::query!("UPDATE workspace SET deleted = true WHERE id = $1", WS)
.execute(&db)
.await?;
tick(&db).await;
assert!(
fetch_pushed(&db).await?.is_empty(),
"no pushes into an archived workspace"
);
assert_eq!(
state_row(&db, PATH).await?,
None,
"no state bookkeeping either"
);
Ok(())
}
#[sqlx::test(fixtures("base"))]
async fn state_of_unwatched_member_is_cleaned_up(db: Pool<Postgres>) -> anyhow::Result<()> {
initialize_tracing().await;
// A stale-bookkeeping row whose script no longer declares freshness
// (e.g. annotation removed and redeployed) must not survive the sweep.
sqlx::query!(
"INSERT INTO pipeline_freshness_state (workspace_id, script_path) VALUES ($1, $2)",
WS,
"u/test-user/gone",
)
.execute(&db)
.await?;
tick(&db).await;
assert_eq!(state_row(&db, "u/test-user/gone").await?, None);
Ok(())
}
backend/windmill-ai/src/ai_bedrock.rs
+174 -10
View File
@@ -857,22 +857,91 @@ pub fn bedrock_stream_event_is_block_stop(event: &ConverseStreamOutput) -> bool
matches!(event, ConverseStreamOutput::ContentBlockStop(_))
}
/// Convert accumulated streaming tool calls to OpenAI format
pub fn streaming_tool_calls_to_openai(tool_calls: Vec<StreamingToolCall>) -> Vec<OpenAIToolCall> {
/// Convert accumulated streaming tool calls to OpenAI format.
///
/// When thinking is enabled, `reasoning` carries the turn's Claude reasoning
/// block; it is attached to the first tool call so the next request replays it
/// before `toolUse` (required by Claude when thinking is enabled — see
/// [`convert_message`]). When `None` (thinking off), tool calls carry no extra
/// content, byte-identical to the pre-feature output.
pub fn streaming_tool_calls_to_openai(
tool_calls: Vec<StreamingToolCall>,
reasoning: Option<crate::ai_types::BedrockExtraContent>,
) -> Vec<OpenAIToolCall> {
let mut reasoning = reasoning;
tool_calls
.into_iter()
.map(|tc| OpenAIToolCall {
id: tc.id,
function: OpenAIFunction { name: tc.name, arguments: tc.arguments },
r#type: FUNCTION_TYPE.to_string(),
// Worker agent requests never enable thinking, so there is no
// reasoning block to round-trip here (the chat proxy path does —
// see providers/bedrock.rs).
extra_content: None,
.map(|tc| {
// `take` so only the first tool call carries the reasoning block.
let extra_content = reasoning.take().map(|block| crate::ai_types::ExtraContent {
bedrock: Some(block),
..Default::default()
});
OpenAIToolCall {
id: tc.id,
function: OpenAIFunction { name: tc.name, arguments: tc.arguments },
r#type: FUNCTION_TYPE.to_string(),
extra_content,
}
})
.collect()
}
/// Fold a Bedrock `ReasoningContent` stream delta into an accumulating reasoning
/// block (text + signature, or redacted bytes). Returns the readable text delta
/// when the event carried one, so the caller can stream it as reasoning content.
/// Mirrors the proxy path's accumulation in `providers/bedrock.rs`.
pub fn bedrock_stream_event_to_reasoning_delta(
event: &ConverseStreamOutput,
reasoning: &mut Option<crate::ai_types::BedrockExtraContent>,
) -> Option<String> {
let ConverseStreamOutput::ContentBlockDelta(delta_event) = event else {
return None;
};
let aws_sdk_bedrockruntime::types::ContentBlockDelta::ReasoningContent(rc) =
delta_event.delta()?
else {
return None;
};
let entry = reasoning.get_or_insert_with(Default::default);
match rc {
aws_sdk_bedrockruntime::types::ReasoningContentBlockDelta::Text(text) => {
entry
.reasoning_text
.get_or_insert_with(String::new)
.push_str(text);
Some(text.clone())
}
aws_sdk_bedrockruntime::types::ReasoningContentBlockDelta::Signature(signature) => {
entry
.signature
.get_or_insert_with(String::new)
.push_str(signature);
None
}
aws_sdk_bedrockruntime::types::ReasoningContentBlockDelta::RedactedContent(blob) => {
// Base64 of concatenated fragments != concatenated base64 fragments,
// so accumulate raw bytes and re-encode.
let mut bytes = entry
.redacted_content
.as_deref()
.and_then(|existing| {
base64::Engine::decode(&base64::engine::general_purpose::STANDARD, existing)
.ok()
})
.unwrap_or_default();
bytes.extend_from_slice(blob.as_ref());
entry.redacted_content = Some(base64::Engine::encode(
&base64::engine::general_purpose::STANDARD,
bytes,
));
None
}
_ => None,
}
}
// ============================================================================
// Tool Configuration Builder
// ============================================================================
@@ -1028,6 +1097,101 @@ mod tests {
assert!(matches!(&content[1], ContentBlock::ToolUse(_)));
}
#[test]
fn streaming_tool_calls_to_openai_attaches_reasoning_to_first_call_only() {
let calls = vec![
StreamingToolCall {
id: "call_1".to_string(),
name: "a".to_string(),
arguments: "{}".to_string(),
},
StreamingToolCall {
id: "call_2".to_string(),
name: "b".to_string(),
arguments: "{}".to_string(),
},
];
let reasoning = crate::ai_types::BedrockExtraContent {
reasoning_text: Some("thinking".to_string()),
signature: Some("sig".to_string()),
redacted_content: None,
};
let openai = streaming_tool_calls_to_openai(calls, Some(reasoning));
let with_reasoning = openai
.iter()
.filter(|tc| {
tc.extra_content
.as_ref()
.and_then(|e| e.bedrock.as_ref())
.is_some()
})
.count();
assert_eq!(
with_reasoning, 1,
"exactly one tool call carries the reasoning block"
);
let block = openai[0]
.extra_content
.as_ref()
.and_then(|e| e.bedrock.as_ref())
.expect("first tool call carries reasoning");
assert_eq!(block.reasoning_text.as_deref(), Some("thinking"));
assert_eq!(block.signature.as_deref(), Some("sig"));
}
#[test]
fn streaming_tool_calls_to_openai_without_reasoning_has_no_extra_content() {
let calls = vec![StreamingToolCall {
id: "call_1".to_string(),
name: "a".to_string(),
arguments: "{}".to_string(),
}];
let openai = streaming_tool_calls_to_openai(calls, None);
assert!(openai[0].extra_content.is_none());
}
#[test]
fn bedrock_reasoning_delta_accumulates_text_and_signature() {
use aws_sdk_bedrockruntime::types::{
ContentBlockDelta, ContentBlockDeltaEvent, ConverseStreamOutput,
ReasoningContentBlockDelta,
};
let mut reasoning = None;
let text_event = ConverseStreamOutput::ContentBlockDelta(
ContentBlockDeltaEvent::builder()
.content_block_index(0)
.delta(ContentBlockDelta::ReasoningContent(
ReasoningContentBlockDelta::Text("let me ".to_string()),
))
.build()
.unwrap(),
);
assert_eq!(
bedrock_stream_event_to_reasoning_delta(&text_event, &mut reasoning).as_deref(),
Some("let me ")
);
let sig_event = ConverseStreamOutput::ContentBlockDelta(
ContentBlockDeltaEvent::builder()
.content_block_index(0)
.delta(ContentBlockDelta::ReasoningContent(
ReasoningContentBlockDelta::Signature("sig".to_string()),
))
.build()
.unwrap(),
);
assert!(bedrock_stream_event_to_reasoning_delta(&sig_event, &mut reasoning).is_none());
let block = reasoning.expect("reasoning accumulated");
assert_eq!(block.reasoning_text.as_deref(), Some("let me "));
assert_eq!(block.signature.as_deref(), Some("sig"));
}
#[test]
fn openai_messages_to_bedrock_skips_cache_points_when_disabled() {
let messages = vec![
backend/windmill-ai/src/ai_google.rs
+3 -5
View File
@@ -312,7 +312,7 @@ impl GeminiToolCallEvent {
pub fn to_extra_content(&self) -> Option<ExtraContent> {
self.thought_signature.as_ref().map(|sig| ExtraContent {
google: Some(GoogleExtraContent { thought_signature: Some(sig.clone()) }),
bedrock: None,
..Default::default()
})
}
}
@@ -1001,10 +1001,8 @@ mod tests {
#[test]
fn gemini_streaming_without_usage_emits_no_usage_chunk() {
let parsed = GeminiParsedEvent {
text: Some("the answer".to_string()),
..Default::default()
};
let parsed =
GeminiParsedEvent { text: Some("the answer".to_string()), ..Default::default() };
let mut tool_call_index = 0;
let chunks = gemini_event_to_openai_sse_chunks(
backend/windmill-ai/src/ai_types.rs
+18
View File
@@ -118,6 +118,22 @@ pub struct BedrockExtraContent {
pub redacted_content: Option<String>,
}
/// Native-Anthropic reasoning block emitted in the same assistant turn as a
/// tool call. Like Bedrock, Anthropic requires the thinking block (text +
/// unmodified signature, or redacted bytes) to precede `tool_use` on replay when
/// thinking is enabled, so it is round-tripped through the OpenAI-shaped tool call.
#[derive(Deserialize, Serialize, Clone, Debug, Default)]
pub struct AnthropicExtraContent {
#[serde(skip_serializing_if = "Option::is_none")]
pub thinking: Option<String>,
#[serde(skip_serializing_if = "Option::is_none")]
pub signature: Option<String>,
/// Base64 `data` of a redacted (encrypted) thinking block, when the provider
/// returned one instead of readable text.
#[serde(skip_serializing_if = "Option::is_none")]
pub redacted_thinking: Option<String>,
}
/// Extra content for provider-specific metadata (e.g., Google thought signatures)
#[derive(Deserialize, Serialize, Clone, Debug, Default)]
pub struct ExtraContent {
@@ -125,6 +141,8 @@ pub struct ExtraContent {
pub google: Option<GoogleExtraContent>,
#[serde(skip_serializing_if = "Option::is_none")]
pub bedrock: Option<BedrockExtraContent>,
#[serde(skip_serializing_if = "Option::is_none")]
pub anthropic: Option<AnthropicExtraContent>,
}
#[derive(Deserialize, Serialize, Clone, Debug)]
backend/windmill-ai/src/providers/anthropic.rs
+204 -2
View File
@@ -92,6 +92,10 @@ pub enum AnthropicRequestContent {
#[serde(skip_serializing_if = "Option::is_none")]
cache_control: Option<CacheControl>,
},
#[serde(rename = "thinking")]
Thinking { thinking: String, signature: String },
#[serde(rename = "redacted_thinking")]
RedactedThinking { data: String },
#[serde(rename = "image")]
Image { source: AnthropicBase64Source },
#[serde(rename = "document")]
@@ -131,6 +135,26 @@ pub struct AnthropicMessage {
pub content: Vec<AnthropicRequestContent>,
}
/// Adaptive thinking config for Anthropic native API. `summarized` display
/// matches the chat proxy path (renders a summarized thinking stream).
#[derive(Serialize, Debug)]
pub struct AnthropicThinking {
pub r#type: &'static str,
pub display: &'static str,
}
impl AnthropicThinking {
fn adaptive() -> Self {
Self { r#type: "adaptive", display: "summarized" }
}
}
/// Carries the reasoning effort token alongside adaptive thinking.
#[derive(Serialize, Debug)]
pub struct AnthropicOutputConfig {
pub effort: String,
}
/// Anthropic-specific request structure for standard API
#[derive(Serialize)]
pub struct AnthropicRequest<'a> {
@@ -145,6 +169,10 @@ pub struct AnthropicRequest<'a> {
#[serde(skip_serializing_if = "Option::is_none")]
pub temperature: Option<f32>,
#[serde(skip_serializing_if = "Option::is_none")]
pub thinking: Option<AnthropicThinking>,
#[serde(skip_serializing_if = "Option::is_none")]
pub output_config: Option<AnthropicOutputConfig>,
#[serde(skip_serializing_if = "Option::is_none")]
pub max_tokens: Option<u32>,
pub stream: bool,
}
@@ -166,6 +194,10 @@ pub struct AnthropicVertexRequest {
#[serde(skip_serializing_if = "Option::is_none")]
pub temperature: Option<f32>,
#[serde(skip_serializing_if = "Option::is_none")]
pub thinking: Option<AnthropicThinking>,
#[serde(skip_serializing_if = "Option::is_none")]
pub output_config: Option<AnthropicOutputConfig>,
#[serde(skip_serializing_if = "Option::is_none")]
pub max_tokens: Option<u32>,
pub stream: bool,
}
@@ -189,6 +221,25 @@ fn convert_messages_to_anthropic(messages: &[OpenAIMessage]) -> Vec<AnthropicMes
"assistant" => {
let mut content: Vec<AnthropicRequestContent> = Vec::new();
// Replay the turn's thinking block first: when thinking is enabled,
// Claude requires the thinking block (with its unmodified signature)
// to precede tool_use in the assistant turn it was emitted in. It is
// round-tripped on the tool call's extra_content (see AnthropicExtraContent).
if let Some(reasoning) = msg
.tool_calls
.as_ref()
.and_then(|tcs| {
tcs.iter().find_map(|tc| {
tc.extra_content
.as_ref()
.and_then(|ec| ec.anthropic.as_ref())
})
})
.and_then(anthropic_reasoning_block_from_extra)
{
content.push(reasoning);
}
// Add text content if present
if let Some(ref c) = msg.content {
let text = extract_text_content(c);
@@ -246,6 +297,25 @@ fn convert_messages_to_anthropic(messages: &[OpenAIMessage]) -> Vec<AnthropicMes
result
}
/// Rebuild an Anthropic thinking content block from the round-tripped
/// [`AnthropicExtraContent`](crate::ai_types::AnthropicExtraContent). Returns
/// None when there is no replayable block (thinking text without a signature is
/// rejected by Anthropic, so it is dropped rather than sent).
fn anthropic_reasoning_block_from_extra(
extra: &crate::ai_types::AnthropicExtraContent,
) -> Option<AnthropicRequestContent> {
if let Some(data) = extra.redacted_thinking.as_deref() {
return Some(AnthropicRequestContent::RedactedThinking { data: data.to_string() });
}
match (extra.thinking.as_deref(), extra.signature.as_deref()) {
(Some(thinking), Some(signature)) => Some(AnthropicRequestContent::Thinking {
thinking: thinking.to_string(),
signature: signature.to_string(),
}),
_ => None,
}
}
/// Convert OpenAI content to Anthropic content blocks
fn convert_content_to_anthropic(content: &Option<OpenAIContent>) -> Vec<AnthropicRequestContent> {
let Some(content) = content else {
@@ -591,6 +661,17 @@ impl AnthropicQueryBuilder {
}
}
// Adaptive thinking rejects sampling params, so drop temperature when
// reasoning is on (Anthropic returns a hard 400 otherwise).
let (thinking, output_config, temperature) = match args.reasoning_effort {
Some(effort) => (
Some(AnthropicThinking::adaptive()),
Some(AnthropicOutputConfig { effort: effort.to_string() }),
None,
),
None => (None, None, args.temperature),
};
// Build request based on platform
if self.is_vertex() {
// For Vertex AI: no model field, anthropic_version in body
@@ -600,7 +681,9 @@ impl AnthropicQueryBuilder {
messages: anthropic_messages,
tools: tools_option,
tool_choice,
temperature: args.temperature,
temperature,
thinking,
output_config,
max_tokens,
stream: true,
};
@@ -614,7 +697,9 @@ impl AnthropicQueryBuilder {
messages: anthropic_messages,
tools: tools_option,
tool_choice,
temperature: args.temperature,
temperature,
thinking,
output_config,
max_tokens,
stream: true,
};
@@ -843,6 +928,123 @@ mod tests {
assert!(has_header(&request.headers, "anthropic-beta", "some-beta"));
}
#[test]
fn convert_messages_replays_thinking_block_before_tool_use() {
let tool_call = crate::ai_types::OpenAIToolCall {
id: "call_1".to_string(),
function: crate::ai_types::OpenAIFunction {
name: "lookup".to_string(),
arguments: "{}".to_string(),
},
r#type: "function".to_string(),
extra_content: Some(crate::ai_types::ExtraContent {
anthropic: Some(crate::ai_types::AnthropicExtraContent {
thinking: Some("let me think".to_string()),
signature: Some("sig-abc".to_string()),
redacted_thinking: None,
}),
..Default::default()
}),
};
let assistant = OpenAIMessage {
role: "assistant".to_string(),
tool_calls: Some(vec![tool_call]),
..Default::default()
};
let messages = convert_messages_to_anthropic(&[assistant]);
let content = &messages[0].content;
match &content[0] {
AnthropicRequestContent::Thinking { thinking, signature } => {
assert_eq!(thinking, "let me think");
assert_eq!(signature, "sig-abc");
}
other => panic!("expected thinking block first, got {:?}", other),
}
assert!(matches!(
&content[1],
AnthropicRequestContent::ToolUse { .. }
));
}
#[test]
fn convert_messages_drops_thinking_without_signature() {
let tool_call = crate::ai_types::OpenAIToolCall {
id: "call_1".to_string(),
function: crate::ai_types::OpenAIFunction {
name: "lookup".to_string(),
arguments: "{}".to_string(),
},
r#type: "function".to_string(),
extra_content: Some(crate::ai_types::ExtraContent {
anthropic: Some(crate::ai_types::AnthropicExtraContent {
thinking: Some("unsigned".to_string()),
signature: None,
redacted_thinking: None,
}),
..Default::default()
}),
};
let assistant = OpenAIMessage {
role: "assistant".to_string(),
tool_calls: Some(vec![tool_call]),
..Default::default()
};
let messages = convert_messages_to_anthropic(&[assistant]);
// An unsigned thinking block can't be replayed, so only tool_use remains.
assert!(matches!(
messages[0].content[0],
AnthropicRequestContent::ToolUse { .. }
));
}
#[test]
fn anthropic_request_serializes_adaptive_thinking_without_temperature() {
let request = AnthropicRequest {
model: "claude-opus-4-8",
system: None,
messages: vec![],
tools: None,
tool_choice: None,
temperature: None,
thinking: Some(AnthropicThinking::adaptive()),
output_config: Some(AnthropicOutputConfig { effort: "high".to_string() }),
max_tokens: Some(64000),
stream: true,
};
let body: serde_json::Value =
serde_json::from_str(&serde_json::to_string(&request).unwrap()).unwrap();
assert_eq!(body["thinking"]["type"], "adaptive");
assert_eq!(body["thinking"]["display"], "summarized");
assert_eq!(body["output_config"]["effort"], "high");
// Sampling params are rejected alongside adaptive thinking.
assert!(body.get("temperature").is_none());
}
#[test]
fn anthropic_request_omits_thinking_when_reasoning_off() {
let request = AnthropicRequest {
model: "claude-opus-4-8",
system: None,
messages: vec![],
tools: None,
tool_choice: None,
temperature: Some(0.5),
thinking: None,
output_config: None,
max_tokens: Some(64000),
stream: true,
};
let body: serde_json::Value =
serde_json::from_str(&serde_json::to_string(&request).unwrap()).unwrap();
assert!(body.get("thinking").is_none());
assert!(body.get("output_config").is_none());
assert_eq!(body["temperature"], 0.5);
}
#[test]
fn rejects_vertex_proxy_request_without_model() {
let credentials = credentials(AIPlatform::GoogleVertexAi);
backend/windmill-ai/src/providers/bedrock.rs
+43 -53
View File
@@ -9,10 +9,10 @@
use crate::{
ai_bedrock::{
bedrock_model_supports_prompt_caching, bedrock_stream_event_is_block_stop,
bedrock_stream_event_to_text, bedrock_stream_event_to_tool_delta,
bedrock_stream_event_to_tool_delta_with_block_index, bedrock_stream_event_to_tool_start,
bedrock_stream_event_to_tool_start_with_block_index, build_tool_config,
create_inference_config, format_bedrock_error, json_to_document,
bedrock_stream_event_to_reasoning_delta, bedrock_stream_event_to_text,
bedrock_stream_event_to_tool_delta, bedrock_stream_event_to_tool_delta_with_block_index,
bedrock_stream_event_to_tool_start, bedrock_stream_event_to_tool_start_with_block_index,
build_tool_config, create_inference_config, format_bedrock_error, json_to_document,
openai_messages_to_bedrock, streaming_tool_calls_to_openai, BearerTokenProvider,
BedrockClient, StreamingToolCall,
},
@@ -641,57 +641,13 @@ fn bedrock_sse_chunks_for_event(
/// Fold a `ReasoningContent` stream delta into the state's pending reasoning
/// block. Returns the text delta (for a `reasoning_content` SSE chunk) when the
/// event carried readable reasoning text.
/// event carried readable reasoning text. Shares the worker path's folding so
/// the two never drift.
fn accumulate_reasoning_delta(
event: &aws_sdk_bedrockruntime::types::ConverseStreamOutput,
state: &mut BedrockSseStreamState,
) -> Option<String> {
let aws_sdk_bedrockruntime::types::ConverseStreamOutput::ContentBlockDelta(delta_event) = event
else {
return None;
};
let aws_sdk_bedrockruntime::types::ContentBlockDelta::ReasoningContent(reasoning) =
delta_event.delta()?
else {
return None;
};
let entry = state.reasoning.get_or_insert_with(Default::default);
match reasoning {
aws_sdk_bedrockruntime::types::ReasoningContentBlockDelta::Text(text) => {
entry
.reasoning_text
.get_or_insert_with(String::new)
.push_str(text);
Some(text.clone())
}
aws_sdk_bedrockruntime::types::ReasoningContentBlockDelta::Signature(signature) => {
entry
.signature
.get_or_insert_with(String::new)
.push_str(signature);
None
}
aws_sdk_bedrockruntime::types::ReasoningContentBlockDelta::RedactedContent(blob) => {
// Base64 of concatenated fragments != concatenated base64 fragments,
// so accumulate raw bytes and re-encode.
let mut bytes = entry
.redacted_content
.as_deref()
.and_then(|existing| {
base64::Engine::decode(&base64::engine::general_purpose::STANDARD, existing)
.ok()
})
.unwrap_or_default();
bytes.extend_from_slice(blob.as_ref());
entry.redacted_content = Some(base64::Engine::encode(
&base64::engine::general_purpose::STANDARD,
bytes,
));
None
}
_ => None,
}
bedrock_stream_event_to_reasoning_delta(event, &mut state.reasoning)
}
async fn handle_bedrock_sdk_non_streaming(
@@ -942,6 +898,7 @@ impl BedrockQueryBuilder {
tools: Option<&[ToolDef]>,
model: &str,
temperature: Option<f32>,
reasoning_effort: Option<&str>,
max_tokens: Option<u32>,
api_key: &str,
region: &str,
@@ -977,6 +934,9 @@ impl BedrockQueryBuilder {
let (bedrock_messages, system_prompts) =
openai_messages_to_bedrock(&prepared_messages, enable_prompt_caching)?;
// Adaptive thinking rejects sampling params; drop temperature when reasoning is on.
let temperature = reasoning_effort.is_none().then_some(temperature).flatten();
// Build inference configuration using shared helper
let inference_config = create_inference_config(temperature, max_tokens.map(|t| t as i32));
@@ -994,6 +954,7 @@ impl BedrockQueryBuilder {
system_prompts,
inference_config,
tool_config,
reasoning_effort,
stream_event_sink,
)
.await
@@ -1008,6 +969,7 @@ impl BedrockQueryBuilder {
system_prompts: Vec<aws_sdk_bedrockruntime::types::SystemContentBlock>,
inference_config: Option<aws_sdk_bedrockruntime::types::InferenceConfiguration>,
tool_config: Option<aws_sdk_bedrockruntime::types::ToolConfiguration>,
reasoning_effort: Option<&str>,
stream_event_sink: Option<Box<dyn StreamEventSink>>,
) -> Result<ParsedResponse, Error> {
tracing::debug!(
@@ -1035,6 +997,11 @@ impl BedrockQueryBuilder {
request_builder = request_builder.set_tool_config(Some(config));
}
if let Some(effort) = reasoning_effort {
request_builder =
request_builder.additional_model_request_fields(bedrock_thinking_fields(effort));
}
let mut stream = request_builder
.send()
.await
@@ -1053,11 +1020,32 @@ impl BedrockQueryBuilder {
let mut accumulated_tool_calls: HashMap<String, StreamingToolCall> = HashMap::new();
let mut current_tool_use_id: Option<String> = None;
let mut usage: Option<TokenUsage> = None;
// Claude reasoning block for the turn (only populated when thinking is on),
// attached to the first tool call for replay before toolUse.
let mut reasoning: Option<crate::ai_types::BedrockExtraContent> = None;
// Process stream events using shared parsing functions
loop {
match stream.recv().await {
Ok(Some(event)) => {
// Fold reasoning deltas into the turn's reasoning block (for
// replay before toolUse, required by Claude when thinking is
// on) and stream the readable summary as a thinking affordance.
if let Some(reasoning_delta) =
bedrock_stream_event_to_reasoning_delta(&event, &mut reasoning)
{
if let Some(processor) = stream_event_sink.as_ref() {
processor
.send(
StreamingEvent::ReasoningTokenDelta {
content: reasoning_delta,
},
&mut events_str,
)
.await?;
}
}
// Handle tool use start using shared parser
if let Some(tool_call) = bedrock_stream_event_to_tool_start(&event) {
current_tool_use_id = Some(tool_call.id.clone());
@@ -1143,8 +1131,10 @@ impl BedrockQueryBuilder {
Some(accumulated_text)
};
let tool_calls =
streaming_tool_calls_to_openai(accumulated_tool_calls.into_values().collect());
let tool_calls = streaming_tool_calls_to_openai(
accumulated_tool_calls.into_values().collect(),
reasoning,
);
Ok(ParsedResponse::Text {
content,
backend/windmill-ai/src/providers/google_ai.rs
+7 -2
View File
@@ -153,13 +153,18 @@ impl GoogleAIQueryBuilder {
(None, None)
};
// Map the effort token onto Gemini's native thinking controls; without
// one, leave provider defaults untouched.
let thinking_config = args
.reasoning_effort
.map(|effort| gemini_thinking_config(args.model, effort));
build_gemini_generation_config(
args.temperature,
args.max_tokens,
response_mime_type,
response_schema,
// Worker AI-agent requests carry no reasoning knob; keep provider defaults.
None,
thinking_config,
)
}
}
backend/windmill-ai/src/providers/openai.rs
+15
View File
@@ -192,6 +192,15 @@ pub struct ResponsesApiTextFormat {
pub format: ResponsesApiTextFormatConfig,
}
/// Reasoning config for the Responses API (`reasoning: { effort }`).
/// The summary is intentionally not requested, mirroring the copilot chat: OpenAI
/// gates reasoning summaries behind organization verification, so asking for one
/// would fail the request for unverified orgs.
#[derive(Serialize)]
pub struct ResponsesApiReasoning {
pub effort: String,
}
#[derive(Serialize)]
pub struct ResponsesApiRequest<'a> {
pub model: &'a str,
@@ -204,6 +213,8 @@ pub struct ResponsesApiRequest<'a> {
#[serde(skip_serializing_if = "Option::is_none")]
pub temperature: Option<f32>,
#[serde(skip_serializing_if = "Option::is_none")]
pub reasoning: Option<ResponsesApiReasoning>,
#[serde(skip_serializing_if = "Option::is_none")]
pub max_output_tokens: Option<u32>,
#[serde(skip_serializing_if = "Option::is_none")]
pub text: Option<ResponsesApiTextFormat>,
@@ -409,6 +420,9 @@ impl OpenAIQueryBuilder {
tools,
stream: Some(true),
temperature: args.temperature,
reasoning: args
.reasoning_effort
.map(|effort| ResponsesApiReasoning { effort: effort.to_string() }),
max_output_tokens: args.max_tokens,
text,
};
@@ -464,6 +478,7 @@ impl OpenAIQueryBuilder {
tools,
stream: None, // Image generation doesn't use streaming
temperature: args.temperature,
reasoning: None, // Image generation models don't take a reasoning effort
max_output_tokens: args.max_tokens,
text: None, // No structured output for image generation
};
backend/windmill-ai/src/providers/other.rs
+88 -2
View File
@@ -35,6 +35,12 @@ pub struct OpenAICompletionRequest<'a> {
#[serde(skip_serializing_if = "Option::is_none")]
pub temperature: Option<f32>,
#[serde(skip_serializing_if = "Option::is_none")]
pub reasoning_effort: Option<&'a str>,
/// DeepSeek disables reasoning via a `thinking` object rather than an effort
/// token (`reasoning_effort: "none"` is rejected by their API).
#[serde(skip_serializing_if = "Option::is_none")]
pub thinking: Option<serde_json::Value>,
#[serde(skip_serializing_if = "Option::is_none")]
pub max_completion_tokens: Option<u32>,
#[serde(skip_serializing_if = "Option::is_none")]
pub response_format: Option<ResponseFormat>,
@@ -51,6 +57,32 @@ pub struct BuiltRequests {
pub without_usage: String,
}
/// Provider-specific reasoning translation for OpenAI-compatible providers,
/// mirroring the chat proxy's `applyReasoningToConfig`. Returns the effort token
/// to send, an optional `thinking` object, and the temperature to keep:
/// - DeepSeek disables reasoning via `thinking: {type: disabled}` (the effort
/// token `"none"` is rejected by their API).
/// - Mistral rejects sampling params when reasoning is on (greedy sampling
/// requires `top_p == 1`), so temperature is dropped.
fn provider_reasoning_fields<'a>(
provider: &AIProvider,
reasoning_effort: Option<&'a str>,
temperature: Option<f32>,
) -> (Option<&'a str>, Option<serde_json::Value>, Option<f32>) {
let (effort, thinking) = match provider {
AIProvider::DeepSeek if reasoning_effort == Some("none") => {
(None, Some(serde_json::json!({ "type": "disabled" })))
}
_ => (reasoning_effort, None),
};
let temperature = if *provider == AIProvider::Mistral && reasoning_effort.is_some() {
None
} else {
temperature
};
(effort, thinking, temperature)
}
/// Query builder for providers using the OpenAI-compatible completion endpoint
/// (Mistral, DeepSeek, Groq, TogetherAI, CustomAI, etc.)
pub struct OtherQueryBuilder {
@@ -109,12 +141,17 @@ impl OtherQueryBuilder {
None
};
let (reasoning_effort, thinking, temperature) =
provider_reasoning_fields(&self.provider_kind, args.reasoning_effort, args.temperature);
// Build request with stream_options for usage tracking
let request_with_usage = OpenAICompletionRequest {
model: args.model,
messages: &prepared_messages,
tools: args.tools,
temperature: args.temperature,
temperature,
reasoning_effort,
thinking: thinking.clone(),
max_completion_tokens: args.max_tokens,
response_format: response_format.clone(),
tool_choice: tool_choice.clone(),
@@ -127,7 +164,9 @@ impl OtherQueryBuilder {
model: args.model,
messages: &prepared_messages,
tools: args.tools,
temperature: args.temperature,
temperature,
reasoning_effort,
thinking,
max_completion_tokens: args.max_tokens,
response_format,
tool_choice,
@@ -268,3 +307,50 @@ impl QueryBuilder for OtherQueryBuilder {
}
}
}
#[cfg(test)]
mod tests {
use super::*;
#[test]
fn deepseek_off_translates_to_thinking_disabled() {
let (effort, thinking, temperature) =
provider_reasoning_fields(&AIProvider::DeepSeek, Some("none"), Some(0.5));
// "none" is rejected as an effort token, so it becomes a thinking disable.
assert_eq!(effort, None);
assert_eq!(thinking, Some(serde_json::json!({ "type": "disabled" })));
assert_eq!(temperature, Some(0.5));
}
#[test]
fn deepseek_level_passes_effort_through() {
let (effort, thinking, _) =
provider_reasoning_fields(&AIProvider::DeepSeek, Some("high"), None);
assert_eq!(effort, Some("high"));
assert!(thinking.is_none());
}
#[test]
fn mistral_drops_temperature_when_reasoning_on() {
let (effort, thinking, temperature) =
provider_reasoning_fields(&AIProvider::Mistral, Some("high"), Some(0.7));
assert_eq!(effort, Some("high"));
assert!(thinking.is_none());
assert_eq!(temperature, None);
}
#[test]
fn mistral_keeps_temperature_when_reasoning_off() {
let (_, _, temperature) = provider_reasoning_fields(&AIProvider::Mistral, None, Some(0.7));
assert_eq!(temperature, Some(0.7));
}
#[test]
fn other_providers_pass_reasoning_and_temperature_through() {
let (effort, thinking, temperature) =
provider_reasoning_fields(&AIProvider::OpenRouter, Some("high"), Some(0.3));
assert_eq!(effort, Some("high"));
assert!(thinking.is_none());
assert_eq!(temperature, Some(0.3));
}
}
backend/windmill-ai/src/query_builder.rs
+3
View File
@@ -12,6 +12,9 @@ pub struct BuildRequestArgs<'a> {
pub tools: Option<&'a [ToolDef]>,
pub model: &'a str,
pub temperature: Option<f32>,
/// Provider-native reasoning effort token (e.g. `low`, `high`, `none`).
/// Each `build_request` maps it onto the provider's thinking config.
pub reasoning_effort: Option<&'a str>,
pub max_tokens: Option<u32>,
pub output_schema: Option<&'a OpenAPISchema>,
pub output_type: &'a OutputType,
backend/windmill-ai/src/sse.rs
+184 -3
View File
@@ -9,7 +9,9 @@ use windmill_common::{error::Error, utils::rd_string};
use crate::{
ai_google::{parse_gemini_sse_event, GeminiUsageMetadata},
ai_types::UrlCitation,
ai_types::{ExtraContent, GoogleExtraContent, OpenAIFunction, OpenAIToolCall},
ai_types::{
AnthropicExtraContent, ExtraContent, GoogleExtraContent, OpenAIFunction, OpenAIToolCall,
},
query_builder::StreamEventSink,
types::StreamingEvent,
};
@@ -30,6 +32,10 @@ pub struct OpenAIChoiceDeltaToolCall {
#[derive(Deserialize)]
pub struct OpenAIChoiceDelta {
pub content: Option<String>,
/// Reasoning summary streamed by providers that expose it (e.g. DeepSeek's
/// `reasoning_content`). Rendered as a "thinking" affordance.
#[serde(default)]
pub reasoning_content: Option<String>,
pub tool_calls: Option<Vec<OpenAIChoiceDeltaToolCall>>,
}
@@ -142,6 +148,13 @@ impl SSEParser for OpenAISSEParser {
if let Some(mut choices) = event.choices.filter(|s| !s.is_empty()) {
if let Some(delta) = choices.remove(0).delta {
if let Some(reasoning) = delta.reasoning_content.filter(|s| !s.is_empty()) {
let event = StreamingEvent::ReasoningTokenDelta { content: reasoning };
self.stream_event_processor
.send(event, &mut self.events_str)
.await?;
}
if let Some(content) = delta.content.filter(|s| !s.is_empty()) {
self.accumulated_content.push_str(&content);
let event = StreamingEvent::TokenDelta { content };
@@ -216,6 +229,16 @@ pub enum AnthropicContentBlockStart {
id: String,
name: String,
},
#[serde(rename = "thinking")]
Thinking {
#[serde(default)]
thinking: String,
},
#[serde(rename = "redacted_thinking")]
RedactedThinking {
#[serde(default)]
data: String,
},
#[serde(other)]
Unknown,
}
@@ -240,6 +263,10 @@ pub enum AnthropicDelta {
InputJsonDelta { partial_json: String },
#[serde(rename = "citations_delta")]
CitationsDelta { citation: AnthropicCitationDelta },
#[serde(rename = "thinking_delta")]
ThinkingDelta { thinking: String },
#[serde(rename = "signature_delta")]
SignatureDelta { signature: String },
#[serde(other)]
Unknown,
}
@@ -292,6 +319,7 @@ pub enum AnthropicSSEEvent {
#[allow(dead_code)]
enum ContentBlockState {
Text,
Thinking,
ToolUse { id: String, name: String },
Unknown,
}
@@ -310,6 +338,11 @@ pub struct AnthropicSSEParser {
pub used_websearch: bool,
/// Token usage from message_delta event
pub usage: Option<AnthropicUsage>,
/// Claude thinking block accumulated from `thinking`/`signature` deltas
/// (or a redacted block). Attached to the first tool call of the turn so it
/// can be replayed before `tool_use` (required by Claude when thinking is on).
pending_reasoning: Option<AnthropicExtraContent>,
reasoning_attached: bool,
}
impl AnthropicSSEParser {
@@ -323,6 +356,8 @@ impl AnthropicSSEParser {
annotations: Vec::new(),
used_websearch: false,
usage: None,
pending_reasoning: None,
reasoning_attached: false,
}
}
}
@@ -363,6 +398,17 @@ impl SSEParser for AnthropicSSEParser {
self.stream_event_processor
.send(event, &mut self.events_str)
.await?;
// Attach the turn's thinking block to the first tool call so it
// can be replayed before tool_use on the next request.
let extra_content = if self.reasoning_attached {
None
} else {
self.reasoning_attached = true;
self.pending_reasoning.take().map(|anthropic| ExtraContent {
anthropic: Some(anthropic),
..Default::default()
})
};
// Initialize tool call accumulator
self.accumulated_tool_calls.insert(
index as i64,
@@ -370,10 +416,35 @@ impl SSEParser for AnthropicSSEParser {
id,
function: OpenAIFunction { name, arguments: String::new() },
r#type: "function".to_string(),
extra_content: None,
extra_content,
},
);
}
AnthropicContentBlockStart::Thinking { thinking } => {
self.content_blocks
.insert(index, ContentBlockState::Thinking);
let entry = self.pending_reasoning.get_or_insert_with(Default::default);
if !thinking.is_empty() {
entry
.thinking
.get_or_insert_with(String::new)
.push_str(&thinking);
self.stream_event_processor
.send(
StreamingEvent::ReasoningTokenDelta { content: thinking },
&mut self.events_str,
)
.await?;
}
}
AnthropicContentBlockStart::RedactedThinking { data } => {
// Redacted blocks arrive whole (no deltas).
self.content_blocks
.insert(index, ContentBlockState::Unknown);
self.pending_reasoning
.get_or_insert_with(Default::default)
.redacted_thinking = Some(data);
}
AnthropicContentBlockStart::ServerToolUse { name, .. } => {
// Detect websearch tool usage
if name == "web_search" {
@@ -417,6 +488,34 @@ impl SSEParser for AnthropicSSEParser {
title: citation.title,
});
}
AnthropicDelta::ThinkingDelta { thinking } => {
if let Some(ContentBlockState::Thinking) =
self.content_blocks.get(&index)
{
self.pending_reasoning
.get_or_insert_with(Default::default)
.thinking
.get_or_insert_with(String::new)
.push_str(&thinking);
self.stream_event_processor
.send(
StreamingEvent::ReasoningTokenDelta { content: thinking },
&mut self.events_str,
)
.await?;
}
}
AnthropicDelta::SignatureDelta { signature } => {
if let Some(ContentBlockState::Thinking) =
self.content_blocks.get(&index)
{
self.pending_reasoning
.get_or_insert_with(Default::default)
.signature
.get_or_insert_with(String::new)
.push_str(&signature);
}
}
AnthropicDelta::Unknown => {}
}
}
@@ -495,6 +594,15 @@ impl SSEParser for GeminiSSEParser {
return Ok(());
};
if let Some(reasoning) = parsed.reasoning.filter(|s| !s.is_empty()) {
self.stream_event_processor
.send(
StreamingEvent::ReasoningTokenDelta { content: reasoning },
&mut self.events_str,
)
.await?;
}
if let Some(text) = parsed.text {
self.accumulated_content.push_str(&text);
self.stream_event_processor
@@ -522,7 +630,7 @@ impl SSEParser for GeminiSSEParser {
let extra_content = tool_call.thought_signature.map(|sig| ExtraContent {
google: Some(GoogleExtraContent { thought_signature: Some(sig) }),
bedrock: None,
..Default::default()
});
self.accumulated_tool_calls.insert(
@@ -805,3 +913,76 @@ impl SSEParser for OpenAIResponsesSSEParser {
Ok(())
}
}
#[cfg(test)]
mod tests {
use super::*;
#[test]
fn reasoning_token_delta_serializes_with_snake_case_tag() {
let event = StreamingEvent::ReasoningTokenDelta { content: "hmm".to_string() };
let json: serde_json::Value =
serde_json::from_str(&serde_json::to_string(&event).unwrap()).unwrap();
assert_eq!(json["type"], "reasoning_token_delta");
assert_eq!(json["content"], "hmm");
}
#[test]
fn openai_delta_parses_reasoning_content() {
// DeepSeek and similar stream reasoning under `reasoning_content`.
let delta: OpenAIChoiceDelta =
serde_json::from_str(r#"{"reasoning_content":"let me think"}"#).unwrap();
assert_eq!(delta.reasoning_content.as_deref(), Some("let me think"));
}
#[test]
fn parses_anthropic_thinking_events() {
let start: AnthropicSSEEvent = serde_json::from_str(
r#"{"type":"content_block_start","index":0,"content_block":{"type":"thinking","thinking":"x"}}"#,
)
.unwrap();
assert!(matches!(
start,
AnthropicSSEEvent::ContentBlockStart {
content_block: AnthropicContentBlockStart::Thinking { .. },
..
}
));
let thinking_delta: AnthropicSSEEvent = serde_json::from_str(
r#"{"type":"content_block_delta","index":0,"delta":{"type":"thinking_delta","thinking":"more"}}"#,
)
.unwrap();
assert!(matches!(
thinking_delta,
AnthropicSSEEvent::ContentBlockDelta {
delta: AnthropicDelta::ThinkingDelta { .. },
..
}
));
let signature_delta: AnthropicSSEEvent = serde_json::from_str(
r#"{"type":"content_block_delta","index":0,"delta":{"type":"signature_delta","signature":"sig"}}"#,
)
.unwrap();
assert!(matches!(
signature_delta,
AnthropicSSEEvent::ContentBlockDelta {
delta: AnthropicDelta::SignatureDelta { .. },
..
}
));
let redacted: AnthropicSSEEvent = serde_json::from_str(
r#"{"type":"content_block_start","index":0,"content_block":{"type":"redacted_thinking","data":"abc"}}"#,
)
.unwrap();
assert!(matches!(
redacted,
AnthropicSSEEvent::ContentBlockStart {
content_block: AnthropicContentBlockStart::RedactedThinking { .. },
..
}
));
}
}
backend/windmill-ai/src/types.rs
+14
View File
@@ -203,6 +203,10 @@ pub struct ProviderWithResource {
pub kind: AIProvider,
pub resource: ProviderResource,
pub model: String,
/// Provider-native reasoning effort token (e.g. `low`, `high`, `none`).
/// Belongs to the model selection, so it rides on the provider config.
#[serde(default)]
pub reasoning_effort: Option<String>,
}
impl ProviderWithResource {
@@ -214,6 +218,12 @@ impl ProviderWithResource {
&self.model
}
/// The reasoning effort to thread to the provider, treating an empty string
/// (e.g. a cleared flow input) as unset.
pub fn get_reasoning_effort(&self) -> Option<&str> {
self.reasoning_effort.as_deref().filter(|s| !s.is_empty())
}
pub async fn get_base_url(&self, db: &DB) -> Result<String, Error> {
self.kind
.get_base_url(self.resource.base_url.clone(), db)
@@ -365,6 +375,10 @@ pub struct AIAgentResult<'a> {
pub enum StreamingEvent {
/// Individual token from the AI response
TokenDelta { content: String },
/// Individual token of the model's reasoning / thinking summary. Emitted
/// before the answer when reasoning is enabled; renderable as a "thinking"
/// affordance (thinking tokens bill regardless of whether they are shown).
ReasoningTokenDelta { content: String },
/// Tool call has started
ToolCall { call_id: String, function_name: String },
/// Tool call arguments are complete
backend/windmill-api-assets/src/lib.rs
+308 -1
View File
@@ -614,6 +614,21 @@ struct GraphQuery {
struct GraphAssetNode {
kind: AssetKind,
path: String,
// Fork workspaces only: 'fork' when the fork has materialized this ducklake asset itself,
// 'deferred' when reads fall back to the parent workspace's current table (defer view).
// Absent outside forks, for non-ducklake assets, and for assets never materialized
// anywhere. Lockstep with TS `AssetGraphAssetNode.fork_materialization`.
#[serde(skip_serializing_if = "Option::is_none")]
fork_materialization: Option<String>,
// The base dimension this asset is the SCD2 `<dim>_current` companion view
// of — set only on a `ducklake://…/<dim>_current` node whose producer
// declares `// materialize … history` on `<dim>`. The producer edge already
// links it to that script (both writes are registered at deploy); this lets
// the canvas render it as a derived "current view" of the base rather than an
// unrelated table. Absent for every other asset. Lockstep with TS
// `AssetGraphAssetNode.derived_from`.
#[serde(skip_serializing_if = "Option::is_none")]
derived_from: Option<String>,
}
#[derive(Serialize, Debug)]
@@ -632,6 +647,14 @@ struct GraphRunnableNode {
partition_kind: Option<String>,
#[serde(skip_serializing_if = "Option::is_none", default)]
freshness: Option<String>,
// Completion time of the most recently started successful run of this
// pipeline member. The canvas checks it against the `// freshness` window
// to color the badge fresh/stale. The badge itself is passive; on EE the
// freshness watchdog (windmill-queue) separately re-runs stale
// unpartitioned producers. Absent when no successful run is visible to
// the caller (job RLS applies).
#[serde(skip_serializing_if = "Option::is_none", default)]
last_success_at: Option<chrono::DateTime<chrono::Utc>>,
#[serde(skip_serializing_if = "Option::is_none", default)]
tag: Option<String>,
#[serde(skip_serializing_if = "Option::is_none", default)]
@@ -656,6 +679,13 @@ struct GraphRunnableNode {
// `merge` / any partitioned write INSERTs into a fixed-schema table.
#[serde(skip_serializing_if = "Option::is_none", default)]
materialize_strategy: Option<String>,
// `on_schema_change=ignore` on the managed materialize — the producer's
// opt-out from downstream schema-contract warnings. Threaded to the editor
// so its client-side contract mirror suppresses the same warnings the
// server check does. Only serialized when set to `ignore` (default `warn`
// is absent). Lockstep with TS `AssetGraphRunnableNode.materialize_on_schema_change`.
#[serde(skip_serializing_if = "Option::is_none", default)]
materialize_on_schema_change: Option<String>,
// Macros this script provides to the workspace registry (deployed
// `// macros` library). Drives the library node state + details-pane
// signature list. Lockstep with TS `AssetGraphRunnableNode.macros`.
@@ -776,6 +806,25 @@ enum TriggerEdge {
},
}
// Ordering-only "must-run-after" edge: `runnable_path`'s data test reads
// `asset` (a `// data_test relationships` ref, or a custom test whose body
// reads a known pipeline asset), so the asset's in-pipeline producer must
// materialize before `runnable_path` runs. NOT a data-consumption edge — the
// tested script doesn't ingest the asset's rows, it only needs the table to
// exist at test time. Rendered dashed (like macro edges) and fed into the
// cascade topo-sort so a cold cascade orders the referenced dimension first.
// Only emitted when the referenced asset has a producer in the graph; an
// external table (no producer) adds no edge — the runtime error stands.
#[derive(Serialize, Debug)]
struct TestEdge {
producer_kind: AssetUsageKind,
producer_path: String,
runnable_kind: AssetUsageKind,
runnable_path: String,
asset_kind: AssetKind,
asset_path: String,
}
#[derive(Serialize, Debug)]
struct AssetGraphResponse {
assets: Vec<GraphAssetNode>,
@@ -784,12 +833,15 @@ struct AssetGraphResponse {
triggers: Vec<TriggerEdge>,
#[serde(skip_serializing_if = "Vec::is_empty", default)]
macro_edges: Vec<MacroEdge>,
#[serde(skip_serializing_if = "Vec::is_empty", default)]
test_edges: Vec<TestEdge>,
}
async fn asset_graph(
authed: ApiAuthed,
Path(w_id): Path<String>,
Extension(user_db): Extension<UserDB>,
Extension(db): Extension<windmill_common::DB>,
Query(q): Query<GraphQuery>,
) -> JsonResult<AssetGraphResponse> {
let mut tx = user_db.begin(&authed).await?;
@@ -920,6 +972,46 @@ async fn asset_graph(
.fetch_all(&mut *tx)
.await?;
// Newest successful completed run per pipeline member, for the passive
// freshness status on the canvas. Correlated per-path lookup walks
// ix_job_root_job_index_by_path_2 newest-first until the first success,
// so cost is bounded by the member count, not run history. "Newest" is
// by created_at (the index order), not completed_at: with overlapping
// runs of one path this can pick an earlier completion, erring toward
// stale — never toward false-fresh. Inside the user tx so job-visibility
// RLS applies — a caller who can't see the runs gets no timestamp rather
// than leaked completion times.
let member_paths: Vec<String> = pipeline_member_paths
.iter()
.map(|r| r.path.clone())
.collect();
let last_success_rows = sqlx::query!(
r#"
SELECT p.path AS "path!",
(SELECT c.completed_at
FROM v2_job j
JOIN v2_job_completed c ON c.id = j.id
WHERE j.workspace_id = $1
AND j.runnable_path = p.path
AND j.parent_job IS NULL
-- No 'singlestepflow': flows may share a script's path, and
-- a same-path flow run must not read as the script being
-- fresh (false-fresh). Script retries land as native
-- 'script' jobs; only the rare flow-wrapper fallback is
-- missed, which errs stale. Kept in lockstep with the
-- freshness watchdog's queries (freshness_watchdog_ee).
AND j.kind IN ('script', 'preview')
AND c.status = 'success'
ORDER BY j.created_at DESC
LIMIT 1) AS last_success_at
FROM unnest($2::text[]) AS p(path)
"#,
&w_id,
&member_paths,
)
.fetch_all(&mut *tx)
.await?;
// Existing scripts / flows in the workspace. Used to filter out
// orphan trigger rows whose `script_path` no longer resolves — those
// would otherwise be added to `runnable_set` below and surface as
@@ -1010,6 +1102,11 @@ async fn asset_graph(
(r.path.clone(), lineage)
})
.collect();
let last_success_by_path: std::collections::HashMap<String, chrono::DateTime<chrono::Utc>> =
last_success_rows
.into_iter()
.filter_map(|r| r.last_success_at.map(|t| (r.path, t)))
.collect();
let pipeline_member_script_paths: std::collections::HashSet<String> =
pipeline_member_paths.into_iter().map(|r| r.path).collect();
let existing_script_paths: std::collections::HashSet<String> =
@@ -1190,9 +1287,204 @@ async fn asset_graph(
runnable_set.insert((AssetUsageKind::Script, e.consumer_path.clone()));
}
// Data-test ordering edges. A `// data_test relationships <col> -> <asset>`
// (and, best-effort, a custom `// data_test <script>` whose body reads a
// pipeline asset) needs the referenced asset materialized before the tested
// script runs — but that dependency is otherwise absent from the execution
// DAG, so a cold cascade can run the tested script first and hard-fail
// ("Catalog Error: Table … does not exist"). Resolve the referenced asset's
// producer(s) from the write edges built above and add an ordering-only
// edge producer → testing_script. No producer (external table) ⇒ no edge.
let mut producers_by_asset: std::collections::HashMap<
(AssetKind, String),
Vec<(AssetUsageKind, String)>,
> = Default::default();
for e in &edges {
if matches!(e.access_type.as_deref(), Some("w") | Some("rw")) {
producers_by_asset
.entry((e.asset_kind, e.asset_path.clone()))
.or_default()
.push((e.runnable_kind, e.runnable_path.clone()));
}
}
// Assets a runnable reads, keyed by (usage_kind, path) — used to order a
// custom-test *script*'s producers before whichever member declares
// `// data_test <path>`. Keyed on the kind too because a flow can share a
// script's path; `// data_test <path>` resolves a deployed script body, so
// the lookup below pins `Script` and never pulls a same-path flow's reads.
let mut reads_by_runnable: std::collections::HashMap<
(AssetUsageKind, String),
Vec<(AssetKind, String)>,
> = Default::default();
for e in &edges {
if matches!(e.access_type.as_deref(), None | Some("r") | Some("rw")) {
reads_by_runnable
.entry((e.runnable_kind, e.runnable_path.clone()))
.or_default()
.push((e.asset_kind, e.asset_path.clone()));
}
}
let mut test_edges: Vec<TestEdge> = Vec::new();
// Dedup on (producer, tested_script, asset) so several tests referencing the
// same asset, or a repeated producer, yield one edge.
let mut seen_test_edges: std::collections::HashSet<(String, String, AssetKind, String)> =
Default::default();
for (member_path, ann) in &annotations_by_path {
for dt in &ann.data_tests {
use windmill_common::assets::DataTest;
let referenced: Vec<(AssetKind, String)> = match dt {
DataTest::Relationships { to_kind, to_path, .. } => {
vec![(
windmill_common::assets::asset_kind_from_parser(*to_kind),
to_path.clone(),
)]
}
// Best-effort: the custom test *script*'s parsed reads. Reading
// the member's OWN output resolves to the member as producer and
// is dropped by the self-edge guard below.
DataTest::Custom { path } => reads_by_runnable
.get(&(AssetUsageKind::Script, path.clone()))
.cloned()
.unwrap_or_default(),
_ => vec![],
};
for (asset_kind, asset_path) in referenced {
let Some(producers) = producers_by_asset.get(&(asset_kind, asset_path.clone()))
else {
continue;
};
for (producer_kind, producer_path) in producers {
// Skip a self-edge: the tested script produces the asset it
// tests (a materialize + relationships/custom on its own output).
if *producer_kind == AssetUsageKind::Script && producer_path == member_path {
continue;
}
if seen_test_edges.insert((
producer_path.clone(),
member_path.clone(),
asset_kind,
asset_path.clone(),
)) {
test_edges.push(TestEdge {
producer_kind: *producer_kind,
producer_path: producer_path.clone(),
runnable_kind: AssetUsageKind::Script,
runnable_path: member_path.clone(),
asset_kind,
asset_path: asset_path.clone(),
});
}
}
}
}
}
for e in &test_edges {
runnable_set.insert((e.producer_kind, e.producer_path.clone()));
runnable_set.insert((e.runnable_kind, e.runnable_path.clone()));
}
// Fork data-environment state per ducklake asset. The parent's rows are read on the plain
// pool: fork membership does not imply parent membership, and defer already exposes the
// parent's data to fork jobs — surfacing its materialization status is strictly less.
let fork_materialization_by_path: std::collections::HashMap<String, &'static str> = {
// The WHOLE ancestor chain, matching defer discovery: a grandchild fork whose direct
// parent only defers a table still reads it (from the grandparent), so it must show
// as deferred, not unmarked.
let ancestors = windmill_common::workspaces::fork_ancestor_chain(&db, &w_id).await?;
match ancestors {
// An orphaned `wm-fork-*` (parent deleted, chain empty) is still isolated — its
// own materializations must keep their 'fork' chips (nothing can be 'deferred').
a if a.is_empty() && !w_id.starts_with(windmill_common::workspaces::WM_FORK_PREFIX) => {
std::collections::HashMap::new()
}
ancestors => {
// Lakes the fork chose to SHARE at creation have no fork namespace — their
// assets live in the parent's tables for real, so no chip applies.
let shared_lakes: std::collections::HashSet<String> = sqlx::query_scalar!(
"SELECT ducklake->'ducklakes' FROM workspace_settings WHERE workspace_id = $1",
&w_id,
)
.fetch_optional(&db)
.await?
.flatten()
.and_then(|v| v.as_object().cloned())
.map(|lakes| {
lakes
.into_iter()
.filter(|(_, cfg)| {
cfg.get("fork_behavior").and_then(|b| b.as_str()) == Some("shared")
})
.map(|(name, _)| name)
.collect()
})
.unwrap_or_default();
sqlx::query!(
r#"
-- Fork rows also count when a snapshot ever committed (a failed run
-- preserves it): the physical table exists, so reads hit the FORK's
-- data — showing 'deferred' would misstate what a query returns.
-- Ancestor rows still require a clean materialization.
SELECT DISTINCT asset_path AS "asset_path!", workspace_id AS "workspace_id!"
FROM materialized_partition
WHERE (workspace_id = $1 OR workspace_id = ANY($2))
AND asset_kind = 'ducklake'
AND (status = 'materialized'
OR (workspace_id = $1 AND snapshot_id IS NOT NULL))
"#,
&w_id,
&ancestors,
)
.fetch_all(&db)
.await?
.into_iter()
.filter(|r| {
!shared_lakes.contains(r.asset_path.split('/').next().unwrap_or_default())
})
.fold(std::collections::HashMap::new(), |mut m, r| {
// A fork row wins over an inherited 'deferred' from any ancestor's row.
if r.workspace_id == w_id {
m.insert(r.asset_path, "fork");
} else {
m.entry(r.asset_path).or_insert("deferred");
}
m
})
}
}
};
// (kind, `<dim>_current` path) → base `<dim>` path, for every managed scd2
// producer in the graph. Reads of the companion view resolve to a node
// whose producer edge already exists (the deploy path registers both writes)
// — this map lets that node advertise which base dimension it derives from.
let scd2_current_base: std::collections::HashMap<(AssetKind, String), String> =
annotations_by_path
.values()
.filter_map(|a| a.materialize.as_ref())
.filter_map(|m| {
m.scd2_current_target().map(|(k, current)| {
(
(windmill_common::assets::asset_kind_from_parser(k), current),
m.target_path.clone(),
)
})
})
.collect();
let mut assets: Vec<GraphAssetNode> = asset_set
.into_iter()
.map(|(kind, path)| GraphAssetNode { kind, path })
.map(|(kind, path)| GraphAssetNode {
fork_materialization: (kind == AssetKind::Ducklake)
.then(|| {
fork_materialization_by_path
.get(&path)
.map(|s| s.to_string())
})
.flatten(),
derived_from: scd2_current_base.get(&(kind, path.clone())).cloned(),
kind,
path,
})
.collect();
assets.sort_by(|a, b| a.path.cmp(&b.path));
@@ -1215,6 +1507,10 @@ async fn asset_graph(
freshness: ann
.and_then(|a| a.freshness.as_ref())
.map(|f| f.duration.clone()),
last_success_at: (usage_kind == AssetUsageKind::Script)
.then(|| last_success_by_path.get(&path))
.flatten()
.copied(),
tag: ann.and_then(|a| a.tag.clone()),
retry: ann.and_then(|a| a.retry.clone()),
data_tests: ann.map(|a| a.data_tests.clone()).unwrap_or_default(),
@@ -1232,8 +1528,12 @@ async fn asset_graph(
}
}),
materialize_strategy: ann.and_then(|a| a.materialize.as_ref()).and_then(|m| {
// Precedence mirrors the runtime strategy derivation:
// scd2 (`history`) > append > merge (`key=`) > replace.
if m.manual {
None
} else if m.scd2 {
Some("scd2".to_string())
} else if m.append {
Some("append".to_string())
} else if m.unique_key.is_some() {
@@ -1242,6 +1542,12 @@ async fn asset_graph(
Some("replace".to_string())
}
}),
materialize_on_schema_change: ann
.and_then(|a| a.materialize.as_ref())
.filter(|m| {
m.on_schema_change == windmill_common::assets::OnSchemaChange::Ignore
})
.map(|_| "ignore".to_string()),
macros: (usage_kind == AssetUsageKind::Script)
.then(|| macros_by_provider.get(&path))
.flatten()
@@ -1260,6 +1566,7 @@ async fn asset_graph(
edges,
triggers,
macro_edges,
test_edges,
}))
}
backend/windmill-api-schedule/src/lib.rs
+37 -2
View File
@@ -182,6 +182,19 @@ fn to_json_raw_opt(
value.map(|v| sqlx::types::Json(to_raw_value(&v)))
}
/// Managed ducklake-maintenance schedules live under a reserved path prefix;
/// their lifecycle is owned by the workspace ducklake settings, so the
/// schedule API refuses to create/edit/delete/toggle them.
fn reject_reserved_schedule_path(path: &str) -> Result<()> {
if path.starts_with(windmill_common::workspaces::DUCKLAKE_MAINTENANCE_PATH_PREFIX) {
return Err(Error::BadRequest(format!(
"Schedules under {} are managed by the workspace ducklake settings",
windmill_common::workspaces::DUCKLAKE_MAINTENANCE_PATH_PREFIX
)));
}
Ok(())
}
/// Validate that a dynamic skip handler (script or flow) exists
async fn validate_dynamic_skip<'c>(
tx: &mut Transaction<'c, Postgres>,
@@ -219,6 +232,7 @@ async fn create_schedule(
Json(ns): Json<NewSchedule>,
) -> Result<String> {
check_scopes(&authed, || format!("schedules:write:{}", ns.path))?;
reject_reserved_schedule_path(&ns.path)?;
let authed = maybe_refresh_folders(&ns.path, &w_id, authed, &db).await;
@@ -468,6 +482,7 @@ async fn edit_schedule(
) -> Result<String> {
let path = path.to_path();
check_scopes(&authed, || format!("schedules:write:{}", path))?;
reject_reserved_schedule_path(path)?;
let authed = maybe_refresh_folders(&path, &w_id, authed, &db).await;
let mut tx = user_db.begin(&authed).await?;
@@ -750,6 +765,15 @@ async fn list_schedule(
)
.order_by("edited_at", true)
.and_where("workspace_id = ?".bind(&w_id))
// managed ducklake-maintenance schedules are edited from the
// workspace ducklake settings, not the schedules UI/CLI.
// starts_with, not LIKE: the prefix contains `_` which LIKE treats as
// a wildcard, and a user folder like `ducklake-maintenance` must not
// be swept up.
.and_where(
"NOT starts_with(path, ?)"
.bind(&windmill_common::workspaces::DUCKLAKE_MAINTENANCE_PATH_PREFIX),
)
.offset(offset)
.limit(per_page)
.clone();
@@ -923,12 +947,13 @@ async fn list_schedule_with_jobs(
ORDER BY completed_at DESC
LIMIT 20
) AS jobs) t
WHERE workspace_id = $1
WHERE workspace_id = $1 AND NOT starts_with(schedule.path, $4)
ORDER BY edited_at DESC
LIMIT $2 OFFSET $3",
w_id,
per_page as i64,
offset as i64
offset as i64,
windmill_common::workspaces::DUCKLAKE_MAINTENANCE_PATH_PREFIX
)
.fetch_all(&mut *tx)
.await?;
@@ -1014,6 +1039,7 @@ pub async fn set_enabled(
let mut tx = user_db.begin(&authed).await?;
let path = path.to_path();
check_scopes(&authed, || format!("schedules:write:{}", path))?;
reject_reserved_schedule_path(path)?;
// Block enabling a schedule in a fork when the parent has the same path
// (regardless of parent's enabled flag), unless force=true. Two enabled
@@ -1190,6 +1216,7 @@ async fn delete_schedule(
) -> Result<String> {
let path = path.to_path();
check_scopes(&authed, || format!("schedules:write:{}", path))?;
reject_reserved_schedule_path(path)?;
let mut tx = user_db.begin(&authed).await?;
clear_schedule(&mut tx, path, &w_id).await?;
@@ -1399,6 +1426,14 @@ async fn set_default_error_handler(
}
}
for updated_schedule_path in updated_schedules {
// managed ducklake-maintenance rows get the handler update (their
// failures should reach workspace handlers) but must not be
// pushed into git-sync as deployed schedules
if updated_schedule_path
.starts_with(windmill_common::workspaces::DUCKLAKE_MAINTENANCE_PATH_PREFIX)
{
continue;
}
handle_deployment_metadata(
&authed.email,
&authed.username,
backend/windmill-api-scripts/src/scripts.rs
+86 -23
View File
@@ -141,6 +141,8 @@ pub fn workspaced_service() -> Router {
// CI test results
.route("/ci_test_results/{kind}/{*path}", get(get_ci_test_results))
.route("/ci_test_results_batch", post(get_ci_test_results_batch))
// Save-time schema-contract check (pipelines gap #2b)
.route("/check_schema_contracts", post(check_schema_contracts))
}
#[derive(Serialize, FromRow)]
@@ -1252,16 +1254,6 @@ async fn create_script_internal<'c>(
// membership; parsed writes tell us what is produced (we don't record
// them in auto_kind itself).
let pipeline_annotations = parse_pipeline_annotations(&ns.content);
// `// freshness` is parsed but enforcement is a not-yet-implemented
// enterprise feature (skeleton in windmill_common::pipeline_advanced).
// Surface a clear TODO at deploy rather than silently accepting an
// annotation that does nothing.
if pipeline_annotations.freshness.is_some() {
tracing::warn!(
"{}",
windmill_common::pipeline_advanced::freshness_enforcement_todo()
);
}
// `// materialize` materializes a `ducklake://<name>/<table>` target from a
// DuckDB script. These two constraints hold for *both* modes: a non-DuckLake
// target would otherwise deploy, register a producer in the asset graph, then
@@ -1315,6 +1307,38 @@ async fn create_script_internal<'c>(
ns.path
);
}
// SCD2 (`key=<col> history` / `scd2`) option-combination checks the
// runtime cannot honor: reject at deploy with a clear message rather
// than letting the script deploy and fail on its first run. Mirrors the
// executor's safety-net `MaterializeSpec::validate` (duckdb_executor).
if let Err(e) = m.validate(pipeline_annotations.partition.is_some()) {
return Err(Error::BadRequest(e));
}
// `manual` materialize never captures a schema (no wrap codegen), so
// there is no contract for `on_schema_change` to mute downstream.
if m.manual && m.on_schema_change == windmill_parser::asset_parser::OnSchemaChange::Ignore {
tracing::warn!(
"script {}: `on_schema_change=ignore` on a `manual` materialize is inert — manual mode captures no schema, so consumers have no contract to check",
ns.path
);
}
}
// A `// data_test` line whose right-hand side doesn't parse into a known
// check is dropped silently by the parser (fail-safe). For a data-quality
// assertion that is a footgun — a typo disables the test with no signal —
// so warn at deploy. Warning, not rejection: it matches the other pipeline
// deploy checks above and can't false-positive a deploy on an edge the
// coarse count and the parser disagree on.
let malformed_data_tests =
windmill_parser::asset_parser::count_malformed_data_tests(&ns.content);
if malformed_data_tests > 0 {
tracing::warn!(
"script {}: {} `// data_test` line(s) are malformed and were dropped — the intended \
check(s) will not run. Fix the syntax (e.g. `data_test accepted_values <col> = a,b,c`, \
`data_test relationships <col> -> <asset>.<refcol>`).",
ns.path,
malformed_data_tests
);
}
// `// macros` — this script is a workspace macro library: its body is
// CREATE [OR REPLACE] MACRO statements plus plain setup, registered into
@@ -1422,21 +1446,17 @@ async fn create_script_internal<'c>(
// body's `SELECT` doesn't express the write (the runtime generates it), so
// server-side inference wouldn't otherwise link it.
let effective_assets = if let Some(m) = pipeline_annotations.materialize.as_ref() {
let kind = windmill_common::assets::asset_kind_from_parser(m.target_kind);
let mut a = effective_assets.unwrap_or_default();
// Produced assets: the managed table, plus — for managed scd2 — the
// `<dim>_current` companion view the runtime (re)creates each run.
// Registering the view as a write asset lets `// on
// ducklake://…/<dim>_current` subscribers be dispatched by the cascade
// (which fans out from these deploy-time asset rows); without it a
// subscriber on the view would silently never fire. Gated on `!manual`:
// manual mode owns its own DDL and short-circuits before the scd2 codegen
// (no view is created), so registering it there would be a false edge.
let mut targets = vec![m.target_path.clone()];
if m.scd2 && !m.manual {
targets.push(format!("{}_current", m.target_path));
}
for path in targets {
// `<dim>_current` companion view the runtime (re)creates each run
// (`MaterializeSpec::write_targets`). Registering the view as a write
// asset lets `// on ducklake://…/<dim>_current` subscribers be
// dispatched by the cascade (which fans out from these deploy-time asset
// rows) and connects a consumer that reads only the view back to this
// producer; without it a subscriber on the view would silently never
// fire and the view would be an orphan node in the lineage graph.
for (target_kind, path) in m.write_targets() {
let kind = windmill_common::assets::asset_kind_from_parser(target_kind);
if !a.iter().any(|x| x.kind == kind && x.path == path) {
a.push(windmill_common::assets::AssetWithAltAccessType {
path,
@@ -3976,3 +3996,46 @@ async fn get_ci_test_results_batch(
Ok(Json(result_map))
}
#[derive(Deserialize)]
struct CheckSchemaContractsRequest {
language: ScriptLang,
content: String,
}
#[derive(Serialize)]
struct CheckSchemaContractsResponse {
warnings: Vec<windmill_common::schema_contracts::ContractWarning>,
}
// Save-time schema-contract check (pipelines gap #2b): validate the given
// script content's asset references (body column reads, `// column` lineage,
// `// data_test relationships`) against the latest captured producer schemas
// and return WARNINGS — never errors, and deploy never blocks on this. The
// frontend calls it right after a successful deploy (post-commit, so a
// self-produced target resolves to the fresh content) and the editor mirrors
// the same diff client-side; this endpoint is the authoritative check. Parsing
// uses the same server path as deploy (`effective_script_assets` +
// `parse_pipeline_annotations`) so the verdict matches what deployed.
async fn check_schema_contracts(
authed: ApiAuthed,
Extension(user_db): Extension<UserDB>,
Path(w_id): Path<String>,
Json(req): Json<CheckSchemaContractsRequest>,
) -> JsonResult<CheckSchemaContractsResponse> {
let assets = crate::asset_inference::effective_script_assets(&req.language, &req.content, None)
.unwrap_or_default();
let ann = parse_pipeline_annotations(&req.content);
let mut tx = user_db.begin(&authed).await?;
let warnings = windmill_common::schema_contracts::check_schema_contracts(
&mut tx,
&w_id,
&assets,
&ann.column_lineage,
&ann.data_tests,
ann.materialize.as_ref(),
)
.await?;
tx.commit().await?;
Ok(Json(CheckSchemaContractsResponse { warnings }))
}
backend/windmill-api-workspaces/Cargo.toml
+7 -2
View File
@@ -11,10 +11,14 @@ path = "src/lib.rs"
[features]
default = []
enterprise = ["windmill-common/enterprise"]
private = ["windmill-common/private"]
private = [
"windmill-common/private",
"windmill-queue/private",
"windmill-audit/private",
]
cloud = ["windmill-common/cloud"]
no_auth = ["windmill-api-auth/no_auth"]
parquet = ["windmill-object-store/parquet"]
parquet = ["windmill-object-store/parquet", "dep:futures"]
[dependencies]
windmill-common = { workspace = true, default-features = false }
@@ -31,6 +35,7 @@ windmill-store.workspace = true
axum.workspace = true
chrono.workspace = true
futures = { workspace = true, optional = true }
hex.workspace = true
magic-crypt.workspace = true
http.workspace = true
backend/windmill-api-workspaces/src/workspaces.rs
+170 -15
View File
@@ -184,6 +184,10 @@ pub fn workspaced_service() -> Router {
"/drop_forked_datatable_databases",
post(crate::workspaces_extra::drop_forked_datatable_databases),
)
.route(
"/drop_forked_ducklake_namespaces",
post(crate::workspaces_extra::drop_forked_ducklake_namespaces),
)
.route(
"/get_datatable_full_schema",
post(get_datatable_full_schema),
@@ -456,6 +460,11 @@ struct CreateWorkspaceFork {
/// forked workspace's datatable config to point to the new database.
#[serde(default)]
forked_datatables: Vec<ForkedDatatableInfo>,
/// Lakes the user explicitly chose to SHARE with the parent (the fork then reads and
/// writes the parent's lake directly). Every lake not listed gets the default isolated
/// fork namespace + read-defer.
#[serde(default)]
shared_ducklakes: Vec<String>,
/// Create the fork as a persistent dev workspace: the id is not required to carry the
/// `wm-fork-` prefix, and at most one dev workspace may exist per parent.
#[serde(default)]
@@ -2309,6 +2318,17 @@ async fn edit_ducklake_config(
require_admin(is_admin, &username)?;
let is_superadmin = require_super_admin(&db, &email).await.is_ok();
// Lake names end up interpolated in `ATTACH 'ducklake://<name>'`,
// generated maintenance SQL and the reserved maintenance schedule path
// (CHECK-constrained to [\w-]+ segments).
for name in new_config.settings.ducklakes.keys() {
if !windmill_common::workspaces::is_valid_ducklake_name(name) {
return Err(Error::BadRequest(format!(
"Invalid ducklake name '{name}': only letters, digits, '_' and '-' are allowed"
)));
}
}
let mut tx = db.begin().await?;
let args_for_audit = format!("{:?}", new_config.settings);
@@ -2323,21 +2343,22 @@ async fn edit_ducklake_config(
)
.await?;
let old_ducklakes = sqlx::query_scalar!(
r#"
SELECT ws.ducklake->'ducklakes' AS ducklake_name
FROM workspace_settings ws
WHERE ws.workspace_id = $1
"#,
&w_id
)
.fetch_one(&mut *tx)
.await?
.unwrap_or(serde_json::Value::Null);
let old_ducklakes: HashMap<String, Ducklake> =
serde_json::from_value(old_ducklakes).unwrap_or_default();
// Check that non-superadmins are not abusing Instance databases
if !is_superadmin {
let old_ducklakes = sqlx::query_scalar!(
r#"
SELECT ws.ducklake->'ducklakes' AS ducklake_name
FROM workspace_settings ws
WHERE ws.workspace_id = $1
"#,
&w_id
)
.fetch_one(&mut *tx)
.await?
.unwrap_or(serde_json::Value::Null);
let old_ducklakes: HashMap<String, Ducklake> =
serde_json::from_value(old_ducklakes).unwrap_or_default();
for (name, dl) in new_config.settings.ducklakes.iter() {
if dl.catalog.resource_type == DucklakeCatalogResourceType::Instance {
let old_dl = old_ducklakes.get(name);
@@ -2355,7 +2376,7 @@ async fn edit_ducklake_config(
}
}
let config: serde_json::Value = serde_json::to_value(new_config.settings)
let config: serde_json::Value = serde_json::to_value(&new_config.settings)
.map_err(|err| Error::internal_err(err.to_string()))?;
sqlx::query!(
@@ -2366,6 +2387,19 @@ async fn edit_ducklake_config(
.execute(&mut *tx)
.await?;
// Same tx as the settings update: a failed schedule sync/push must fail
// the whole save — nothing reconciles a half-applied state later.
let tx = windmill_queue::ducklake_maintenance::sync_ducklake_maintenance_schedules(
&db,
tx,
&w_id,
&new_config.settings.ducklakes,
&old_ducklakes,
&username,
&email,
)
.await?;
tx.commit().await?;
Ok(format!("Edit ducklake config for workspace {}", &w_id))
@@ -4101,6 +4135,7 @@ async fn clone_workspace_data(
// Clone CI test references
clone_ci_test_references(tx, source_workspace_id, target_workspace_id).await?;
clone_macro_registry(tx, source_workspace_id, target_workspace_id).await?;
clone_asset_usages_and_triggers(tx, source_workspace_id, target_workspace_id).await?;
// Clone flows with new versions
clone_flows(tx, source_workspace_id, target_workspace_id).await?;
@@ -4142,6 +4177,9 @@ async fn clone_triggers_and_schedules(
source_workspace_id: &str,
target_workspace_id: &str,
) -> Result<()> {
// Managed ducklake-maintenance schedules are excluded: the fork starts
// with no ducklake config, so cloned rows could never resolve a lake and
// the schedule API refuses mutations under the reserved prefix.
sqlx::query!(
r#"INSERT INTO schedule (
workspace_id, path, edited_by, edited_at, schedule, enabled, script_path,
@@ -4158,9 +4196,10 @@ async fn clone_triggers_and_schedules(
on_recovery_times, on_recovery_extra_args, ws_error_handler_muted, retry,
summary, no_flow_overlap, tag, paused_until, on_success, on_success_extra_args,
cron_version, description, dynamic_skip, permissioned_as, labels
FROM schedule WHERE workspace_id = $2"#,
FROM schedule WHERE workspace_id = $2 AND NOT starts_with(path, $3)"#,
target_workspace_id,
source_workspace_id,
windmill_common::workspaces::DUCKLAKE_MAINTENANCE_PATH_PREFIX,
)
.execute(&mut **tx)
.await?;
@@ -4693,6 +4732,39 @@ async fn clone_macro_registry(
Ok(())
}
// Asset usage rows and `// on` subscriber triggers are deploy-derived like
// ci_test_reference / the macro registry: without cloning them the fork's
// pipeline graph has no asset nodes or lineage edges, and — worse — the asset
// dispatch cascade never fires in the fork (it reads `script_trigger`), so
// materializing an upstream node can't trigger its consumers until every
// script is manually redeployed. `job`-kind usage rows (runtime-detected,
// ephemeral) are skipped like the graph does.
async fn clone_asset_usages_and_triggers(
tx: &mut Transaction<'_, Postgres>,
source_workspace_id: &str,
target_workspace_id: &str,
) -> Result<()> {
sqlx::query!(
"INSERT INTO asset (workspace_id, path, kind, usage_access_type, usage_path, usage_kind, columns)
SELECT $2, path, kind, usage_access_type, usage_path, usage_kind, columns
FROM asset WHERE workspace_id = $1 AND usage_kind IN ('script', 'flow')",
source_workspace_id,
target_workspace_id,
)
.execute(&mut **tx)
.await?;
sqlx::query!(
"INSERT INTO script_trigger (workspace_id, runnable_kind, runnable_path, trigger_kind, trigger_ref, join_all, debounce_s, retry_count, retry_delay_s)
SELECT $2, runnable_kind, runnable_path, trigger_kind, trigger_ref, join_all, debounce_s, retry_count, retry_delay_s
FROM script_trigger WHERE workspace_id = $1",
source_workspace_id,
target_workspace_id,
)
.execute(&mut **tx)
.await?;
Ok(())
}
async fn clone_flows(
tx: &mut Transaction<'_, Postgres>,
source_workspace_id: &str,
@@ -5414,6 +5486,42 @@ async fn create_workspace_fork(
// conflict instead of a misleading "maximum number of workspaces" error.
check_fork_w_id_conflict(&db, &nw.id).await?;
purge_stale_fork_diff_state(&db, &nw.id).await?;
// A previously deleted fork with this id may have left ducklake namespaces behind if its
// physical cleanup failed after the delete committed (registry rows are the durable
// ledger — no FK, they outlive the workspace). Retry that cleanup now, and refuse to
// proceed while any metadata schema still can't be dropped: creating the fork anyway
// would silently reattach the deterministic namespace and its stale tables. Data-file
// leftovers alone don't block — once the schema is gone they are inert (a deleted fork's
// `$res:` storage resource is gone forever, so they may never resolve again), and the
// surviving registry row has the next successful same-prefix cleanup sweep them.
// `$res:` fallback = the workspace being forked: the deleted fork's resource clones came
// from a parent, so the new parent is the natural donor for the retry's credentials.
let leftover_issues = crate::workspaces_extra::drop_forked_ducklake_namespaces_impl(
&db,
&nw.id,
Some(&parent_workspace_id),
)
.await?;
let blocking: Vec<&str> = leftover_issues
.iter()
.filter(|i| i.blocking)
.map(|i| i.msg.as_str())
.collect();
if !blocking.is_empty() {
return Err(Error::BadRequest(format!(
"a previously deleted workspace with id '{}' left ducklake namespaces that could \
not be cleaned up: {}; retry once the catalog is reachable again",
nw.id,
blocking.join("; ")
)));
}
for i in &leftover_issues {
tracing::warn!(
"creating fork {}: leftover ducklake cleanup: {}",
nw.id,
i.msg
);
}
#[cfg(not(feature = "enterprise"))]
_check_nb_of_workspaces(&db).await?;
@@ -5511,6 +5619,42 @@ async fn create_workspace_fork(
apply_forked_datatable(&db, &mut tx, &parent_workspace_id, &forked_id, fdt).await?;
}
// The settings clone copies the source's ducklake config verbatim — including a parent
// fork's own `fork_behavior` stamps. Sharing is a per-fork-creation choice, never
// inherited: reset any cloned stamps first, then apply this fork's requested list.
sqlx::query!(
r#"UPDATE workspace_settings
SET ducklake = jsonb_set(ducklake, '{ducklakes}', (
SELECT COALESCE(jsonb_object_agg(key, value - 'fork_behavior'), '{}'::jsonb)
FROM jsonb_each(ducklake->'ducklakes')
))
WHERE workspace_id = $1 AND jsonb_typeof(ducklake->'ducklakes') = 'object'"#,
&forked_id,
)
.execute(&mut *tx)
.await?;
// Stamp the per-lake ducklake fork choice into the fork's own settings. Only the `shared`
// opt-out needs stamping — absent `fork_behavior` already means isolated (the default),
// so unlisted lakes and API callers that omit the field stay safe.
for lake in &nw.shared_ducklakes {
let stamped = sqlx::query_scalar!(
r#"UPDATE workspace_settings
SET ducklake = jsonb_set(ducklake, ARRAY['ducklakes', $2, 'fork_behavior'], '"shared"')
WHERE workspace_id = $1 AND ducklake->'ducklakes' ? $2
RETURNING 1 AS "one!""#,
&forked_id,
lake,
)
.fetch_optional(&mut *tx)
.await?;
if stamped.is_none() {
return Err(Error::BadRequest(format!(
"cannot mark ducklake `{lake}` as shared: no such lake in the workspace settings"
)));
}
}
// Lock the parent ("prod") so edits are funneled through this dev workspace.
let locked_prod = nw.is_dev_workspace && (nw.lock_prod_deploy || nw.lock_prod_forking);
if locked_prod {
@@ -5535,6 +5679,10 @@ async fn create_workspace_fork(
.await?;
tx.commit().await?;
// A pre-creation lookup could have cached an EMPTY ancestor chain for this id, which
// would bypass ducklake fork isolation for the TTL.
windmill_common::workspaces::invalidate_fork_ancestor_chain_cache(&forked_id);
if locked_prod {
windmill_common::workspaces::invalidate_protection_rules_cache(&parent_workspace_id);
}
@@ -5708,6 +5856,13 @@ async fn attach_dev_workspace(
// The dev workspace's parent just changed (none -> prod); drop its cached fork->parent mapping
// so per-workspace job tags route to the prod family immediately rather than after the TTL.
windmill_queue::tags::invalidate_fork_parent_cache(&dev_w_id);
// Drop the cached ancestor chains too — the workspace existed BEFORE the attach, so a
// cached empty chain reads as "not a fork" and its ducklake jobs would write the shared
// lake until the TTL. Descendants' chains also gained the new root.
windmill_common::workspaces::invalidate_fork_ancestor_chain_cache(&dev_w_id);
for id in windmill_common::workspaces::list_fork_descendants(&db, &dev_w_id).await? {
windmill_common::workspaces::invalidate_fork_ancestor_chain_cache(&id);
}
// Same reparent invalidates the billing-workspace mapping so its usage meters to prod at once. The
// candidate can bring its own fork subtree, whose descendants had resolved their (now-stale) root
// to the candidate's old family; invalidate them too so they meter to prod without waiting out the
backend/windmill-api-workspaces/src/workspaces_extra.rs
+580 -22
View File
@@ -749,6 +749,16 @@ pub(crate) async fn change_workspace_id(
// (fork-of-fork) self-heal via the 60s billing-cache TTL.
for child in &reparented_children {
windmill_queue::tags::invalidate_fork_parent_cache(child);
windmill_common::workspaces::invalidate_fork_ancestor_chain_cache(child);
// Grandchildren's cached chains contain the old (renamed-away) ancestor id; a stale
// chain drops all defer ancestors in the ducklake resolver, so sweep the subtree
// rather than letting it wait out the TTL.
for id in windmill_common::workspaces::list_fork_descendants(&db, child)
.await
.unwrap_or_default()
{
windmill_common::workspaces::invalidate_fork_ancestor_chain_cache(&id);
}
#[cfg(feature = "cloud")]
windmill_common::workspaces::invalidate_billing_workspace_cache(child);
}
@@ -829,28 +839,19 @@ pub(crate) async fn delete_workspace(
// Deleting an attached dev workspace removes the parent prod's dev_workspace_lock (below), so it
// must be a prod-admin action, not just the dev's own owner (dev ownership can diverge from
// prod's) — mirrors detach_dev_workspace, which is prod-admin gated.
if let Some(prod) = sqlx::query_scalar!(
"SELECT parent_workspace_id FROM workspace WHERE id = $1 AND is_dev_workspace",
&w_id
)
.fetch_optional(&mut *tx)
.await?
.flatten()
{
let is_prod_admin = sqlx::query_scalar!(
"SELECT is_admin FROM usr WHERE workspace_id = $1 AND email = $2",
&prod,
&authed.email
)
.fetch_optional(&mut *tx)
.await?
.unwrap_or(false);
if !is_prod_admin && !is_super_admin_email(&db, &authed.email).await? {
return Err(Error::PermissionDenied(format!(
"Deleting dev workspace '{w_id}' requires being an admin of its parent prod workspace '{prod}' (or a superadmin)"
)));
}
}
require_prod_admin_for_dev_workspace(&db, &authed, &w_id).await?;
// Snapshot the fork's ducklake namespaces + RESOLVE their connection material NOW — the
// registry rows and the fork's `$res:` resources both CASCADE with the workspace row —
// but the destructive cleanup itself runs only after the commit below: a delete that
// fails mid-way must never leave a live workspace with its fork data destroyed and no
// registry row to retry from. Read-only: nothing is dropped here.
let fork_ducklake_cleanups = prepare_fork_ducklake_cleanups(&db, &w_id, None)
.await
.unwrap_or_else(|e| {
tracing::warn!("deleting workspace {w_id}: preparing ducklake cleanup: {e:#}");
vec![]
});
sqlx::query!("DELETE FROM ai_agent_memory WHERE workspace_id = $1", &w_id)
.execute(&mut *tx)
@@ -1086,6 +1087,17 @@ pub(crate) async fn delete_workspace(
.await?;
tx.commit().await?;
// Physical ducklake-namespace cleanup, post-commit, from the pre-read snapshot: fork
// namespaces are deterministic from (id, lake), so an orphan would silently REATTACH to a
// recreated identical fork id. Runs inline so every delete path is covered (CLI,
// force-delete dialog, direct API — not just the sidebar flow, which still calls the
// endpoint first for per-lake error toasts; the rerun is an idempotent no-op). Best
// effort: failures are logged — the workspace row is already gone, and broken storage
// credentials must not have made it undeletable.
for e in cleanup_fork_ducklake_namespaces(&db, &w_id, fork_ducklake_cleanups).await {
tracing::warn!("deleted workspace {w_id}: ducklake namespace cleanup: {}", e.msg);
}
if let Some(parent) = dev_lock_parent {
windmill_common::workspaces::invalidate_protection_rules_cache(&parent);
}
@@ -1099,12 +1111,26 @@ pub(crate) async fn delete_workspace(
// billing TTL.
for id in std::iter::once(&w_id).chain(orphaned_children.iter()) {
windmill_queue::tags::invalidate_fork_parent_cache(id);
windmill_common::workspaces::invalidate_fork_ancestor_chain_cache(id);
#[cfg(feature = "cloud")]
{
windmill_common::workspaces::invalidate_billing_workspace_cache(id);
windmill_common::workspaces::invalidate_team_plan_cache(id);
}
}
// Deeper descendants' cached ancestor CHAINS still contain the deleted workspace; unlike
// the sibling caches (which self-heal harmlessly via TTL), a stale chain makes the
// ducklake resolver drop all defer ancestors (all-or-nothing on broken links) — a visible
// defer/chips outage for up to the TTL. Anchor at the orphaned children: the deleted row
// is gone, but their subtrees are intact.
for child in orphaned_children.iter() {
for id in windmill_common::workspaces::list_fork_descendants(&db, child)
.await
.unwrap_or_default()
{
windmill_common::workspaces::invalidate_fork_ancestor_chain_cache(&id);
}
}
Ok(format!("Deleted workspace {}", &w_id))
}
@@ -1266,6 +1292,538 @@ pub async fn drop_forked_datatable_databases(
Ok(Json(errors))
}
/// Drop this fork workspace's ducklake namespaces: the `wm_fork_*` metadata schema in each
/// lake's catalog database, plus (best effort) the fork's `__wm_forks/<wid>/…` data files in
/// the workspace storage. Driven by the `fork_ducklake_namespace` registry written at first
/// fork attach, so it works even after settings drift. Returns errors per lake that failed;
/// the registry row is only deleted once both cleanups succeeded, so a retry resumes.
/// Same permission as delete_workspace: fork owner or super admin. `delete_workspace` also
/// runs this inline (the UI calls this endpoint first for error visibility; the inline run
/// covers every other delete path — CLI, force-delete dialogs, direct API — whose row delete
/// would otherwise CASCADE the registry away while orphaning the physical namespaces, which a
/// recreated identical fork id would then silently reattach).
pub async fn drop_forked_ducklake_namespaces(
authed: ApiAuthed,
Extension(db): Extension<DB>,
Path(w_id): Path<String>,
) -> Result<Json<Vec<String>>> {
let is_fork = workspace_is_fork(&db, &w_id).await?;
let mut tx = db.begin().await?;
if !(is_fork && is_workspace_owner(&authed, &w_id, &mut tx).await?)
&& !is_super_admin_email(&db, &authed.email).await?
{
return Err(Error::PermissionDenied(
"Dropping forked ducklake namespaces requires being the fork's owner or a superadmin"
.to_string(),
));
}
tx.commit().await?;
require_prod_admin_for_dev_workspace(&db, &authed, &w_id).await?;
Ok(Json(
drop_forked_ducklake_namespaces_impl(&db, &w_id, None)
.await?
.into_iter()
.map(|i| i.msg)
.collect(),
))
}
/// The cleanup itself, shared by the endpoint and the post-commit `delete_workspace` run. No
/// authorization of its own — callers gate.
pub(crate) async fn drop_forked_ducklake_namespaces_impl(
db: &DB,
w_id: &str,
res_fallback_w_id: Option<&str>,
) -> Result<Vec<ForkDucklakeCleanupIssue>> {
let prepared = prepare_fork_ducklake_cleanups(db, w_id, res_fallback_w_id).await?;
Ok(cleanup_fork_ducklake_namespaces(db, w_id, prepared).await)
}
struct ForkDucklakeNamespaceRow {
ducklake_name: String,
metadata_schema: String,
catalog: String,
storage: String,
storage_ref: String,
schema_dropped: bool,
data_path: String,
}
/// A namespace row plus its RESOLVED connection material. `delete_workspace` prepares these
/// BEFORE its transaction commits — the registry rows AND the fork's `$res:` resources both
/// disappear with the workspace row — and runs the destructive cleanup only AFTER the commit:
/// a delete that fails mid-way must never leave a live workspace with its fork data
/// destroyed, and a committed delete must still be able to reach catalogs/storages whose
/// credentials lived in the (now gone) fork resources. Per-row resolution failures are
/// carried as strings so they surface with the other cleanup errors.
pub(crate) struct PreparedForkDucklakeCleanup {
ns: ForkDucklakeNamespaceRow,
catalog_pg: std::result::Result<windmill_common::PgDatabase, String>,
store: std::result::Result<(windmill_types::s3::LargeFileStorage, serde_json::Value), String>,
}
/// One row per (lake, catalog, storage, data path) ever attached — settings drift adds rows,
/// so every location the fork wrote gets cleaned, not just the first. Read-only: resolves
/// credentials but destroys nothing.
/// `res_fallback_w_id`: second workspace to resolve `$res:` catalog/storage paths against
/// when the row's own workspace no longer has them — the retry path runs AFTER the fork and
/// its resources were deleted. Fork resources are clones of a parent's, so the workspace
/// being forked again is the natural donor. None on live-workspace paths.
pub(crate) async fn prepare_fork_ducklake_cleanups(
db: &DB,
w_id: &str,
res_fallback_w_id: Option<&str>,
) -> Result<Vec<PreparedForkDucklakeCleanup>> {
let rows = sqlx::query_as!(
ForkDucklakeNamespaceRow,
r#"SELECT ducklake_name AS "ducklake_name!", metadata_schema AS "metadata_schema!",
catalog AS "catalog!", storage AS "storage!",
storage_ref AS "storage_ref!", data_path AS "data_path!",
schema_dropped AS "schema_dropped!"
FROM fork_ducklake_namespace WHERE workspace_id = $1"#,
w_id
)
.fetch_all(db)
.await?;
let mut prepared = Vec::with_capacity(rows.len());
for ns in rows {
let catalog_pg = if ns.schema_dropped {
// Schema phase already done — the retry needs no catalog connection at all
// (its credentials may be unresolvable for good with the fork's resources gone).
Err("unused: metadata schema already dropped".to_string())
} else {
resolve_fork_catalog_pg(db, w_id, res_fallback_w_id, &ns.ducklake_name, &ns.catalog)
.await
.map_err(|e| e.to_string())
};
let storage = Some(ns.storage.as_str()).filter(|s| !s.is_empty());
let store = resolve_fork_storage(db, w_id, res_fallback_w_id, storage, &ns.storage_ref)
.await
.map_err(|e| e.to_string());
prepared.push(PreparedForkDucklakeCleanup { ns, catalog_pg, store });
}
Ok(prepared)
}
/// Drop the prepared namespaces' metadata schemas + data files, deleting each registry row
/// only after both succeed (a no-op when the row already cascaded away with the workspace).
/// Returns per-namespace error strings; never fails as a whole.
/// One failed step of a fork ducklake cleanup. `blocking` = the metadata schema (or its
/// guards) failed, so the namespace is still attachable and a same-id fork must NOT be
/// created. Non-blocking = the schema is gone and only data files (or the registry-row
/// delete) failed — inert storage leftovers, tracked by the surviving row and swept by the
/// next successful cleanup of the same prefix.
pub(crate) struct ForkDucklakeCleanupIssue {
pub(crate) blocking: bool,
pub(crate) msg: String,
}
pub(crate) async fn cleanup_fork_ducklake_namespaces(
db: &DB,
w_id: &str,
prepared: Vec<PreparedForkDucklakeCleanup>,
) -> Vec<ForkDucklakeCleanupIssue> {
// The registration once-cache must not outlive the rows it mirrors: a same-id fork
// recreated within the TTL would otherwise skip re-registration, and ITS eventual
// deletion would find no rows — orphaning the deterministic namespace for the next
// same-id fork to silently reattach.
windmill_common::workspaces::invalidate_fork_ducklake_registration_cache(w_id);
let mut errors: Vec<ForkDucklakeCleanupIssue> = Vec::new();
for PreparedForkDucklakeCleanup { ns, catalog_pg, store } in prepared {
// Hard guards mirroring the forked-datatable drop: never touch a schema outside the
// fork prefix, never delete outside the fork data dir — even if a registry row was
// somehow tampered with.
if !ns
.metadata_schema
.starts_with(windmill_common::workspaces::FORK_DUCKLAKE_SCHEMA_PREFIX)
{
errors.push(ForkDucklakeCleanupIssue {
blocking: true,
msg: format!(
"Refusing to drop schema '{}' for ducklake://{}: name does not start with '{}'",
ns.metadata_schema,
ns.ducklake_name,
windmill_common::workspaces::FORK_DUCKLAKE_SCHEMA_PREFIX
),
});
continue;
}
// The fork's directory segment, NOT the raw workspace id: ids are only
// git-branch-safe and may contain `/`, which raw would let one fork's prefix nest
// inside a sibling's (`wm-fork-a/b` under `wm-fork-a`) and be swept by its cleanup.
let expected_prefix = format!(
"{}/{}/",
windmill_common::workspaces::FORK_DUCKLAKE_DATA_DIR,
windmill_common::workspaces::fork_data_dir_segment(w_id)
);
if !format!("{}/", ns.data_path.trim_end_matches('/')).starts_with(&expected_prefix) {
errors.push(ForkDucklakeCleanupIssue {
blocking: true,
msg: format!(
"Refusing to delete data path '{}' for ducklake://{}: not under '{}'",
ns.data_path, ns.ducklake_name, expected_prefix
),
});
continue;
}
let drop_res = if ns.schema_dropped {
// Recorded as already dropped by a prior partial cleanup; skipping means no
// catalog credentials are needed. Registration resets the flag when a live fork
// re-attaches (recreating the schema).
Ok(())
} else {
match catalog_pg {
Ok(pg) => drop_fork_ducklake_metadata_schema(db, pg, &ns.metadata_schema).await,
Err(e) => Err(Error::internal_err(e)),
}
};
if let Err(e) = drop_res {
errors.push(ForkDucklakeCleanupIssue {
blocking: true,
msg: format!(
"Could not drop metadata schema '{}' for ducklake://{}: {e}",
ns.metadata_schema, ns.ducklake_name
),
});
continue;
}
let delete_res = match store {
Ok((lfs, resource_value)) => {
delete_fork_ducklake_data(lfs, resource_value, &ns.data_path).await
}
Err(e) => Err(Error::internal_err(e)),
};
if let Err(e) = delete_res {
// Record the completed schema phase so retries never need catalog credentials
// again (best effort — a failed update just means the next retry re-drops an
// absent schema, which requires the catalog to be reachable).
sqlx::query!(
"UPDATE fork_ducklake_namespace SET schema_dropped = true
WHERE workspace_id = $1 AND ducklake_name = $2 AND catalog = $3
AND storage = $4 AND storage_ref = $5 AND data_path = $6",
w_id,
&ns.ducklake_name,
&ns.catalog,
&ns.storage,
&ns.storage_ref,
&ns.data_path,
)
.execute(db)
.await
.ok();
errors.push(ForkDucklakeCleanupIssue {
blocking: false,
msg: format!(
"Dropped metadata schema but could not delete data files under '{}' for ducklake://{}: {e}",
ns.data_path, ns.ducklake_name
),
});
continue;
}
sqlx::query!(
"DELETE FROM fork_ducklake_namespace
WHERE workspace_id = $1 AND ducklake_name = $2 AND catalog = $3
AND storage = $4 AND storage_ref = $5 AND data_path = $6",
w_id,
&ns.ducklake_name,
&ns.catalog,
&ns.storage,
&ns.storage_ref,
&ns.data_path,
)
.execute(db)
.await
.map_err(|e| {
errors.push(ForkDucklakeCleanupIssue {
blocking: false,
msg: format!(
"Cleaned ducklake://{} but could not delete its registry row: {e}",
ns.ducklake_name
),
})
})
.ok();
}
errors
}
/// Drop the fork's metadata schema in the catalog database recorded by the registry row —
/// NOT whatever the fork's settings point at by now: a drifted catalog resource must not make
/// cleanup drop a schema in the wrong database while orphaning the real one. The pg schema
/// holds only DuckLake metadata tables (auto-created at first fork attach), so a plain
/// `DROP SCHEMA … CASCADE` on the catalog connection removes the whole fork namespace.
/// Resolve the catalog connection recorded in the registry row (read-only): instance
/// identities rebuild instance creds; resource identities resolve their `$res:` in the fork's
/// workspace — which is why this must run BEFORE the workspace (and its resources) are
/// deleted. Mysql never registers (rejected at resolution).
async fn resolve_fork_catalog_pg(
db: &DB,
w_id: &str,
res_fallback_w_id: Option<&str>,
ducklake_name: &str,
catalog: &str,
) -> Result<windmill_common::PgDatabase> {
let (resource_type, resource_path) = catalog.split_once(':').ok_or_else(|| {
Error::internal_err(format!(
"ducklake://{ducklake_name}: malformed registry catalog identity `{catalog}`"
))
})?;
let catalog_resource = if resource_type == "instance" {
let mut pg_creds = windmill_common::PgDatabase::parse_uri(
&windmill_common::get_database_url().await?.as_str().await,
)?;
pg_creds.dbname = resource_path.to_string();
pg_creds.user = Some("custom_instance_user".to_string());
pg_creds.password =
Some(windmill_common::utils::get_custom_pg_instance_password(db).await?);
serde_json::to_value(&pg_creds)
.map_err(|e| Error::internal_err(format!("serializing pg creds: {e}")))?
} else {
resolve_res_with_fallback(db, w_id, res_fallback_w_id, resource_path).await?
};
serde_json::from_value(catalog_resource).map_err(|e| {
Error::internal_err(format!(
"ducklake://{ducklake_name}: catalog resource is not a postgres database: {e}"
))
})
}
async fn drop_fork_ducklake_metadata_schema(
db: &DB,
pg: windmill_common::PgDatabase,
metadata_schema: &str,
) -> Result<()> {
let (client, connection) = pg.connect(Some(db)).await?;
let join_handle = tokio::spawn(async move { connection.await });
let res = client
.execute(
&format!(
"DROP SCHEMA IF EXISTS \"{}\" CASCADE",
metadata_schema.replace('"', "\"\"")
),
&[],
)
.await;
drop(client);
let _ = join_handle.await;
res.map_err(|e| Error::internal_err(format!("{e:#}")))?;
Ok(())
}
/// Resolve the storage identified by the registry's `storage_ref` (read-only) — the storage
/// that was active when the fork's data was WRITTEN, not whatever the logical storage name
/// points at by deletion time (a repointed storage must not orphan the original fork data,
/// nor get a colliding prefix deleted). `storage_ref` = '' falls back to resolving the
/// logical name against current settings (registration couldn't identify the storage — best
/// effort). Resolves the `$res:` in the fork's workspace, which is why this must run BEFORE
/// the workspace is deleted.
async fn resolve_fork_storage(
db: &DB,
w_id: &str,
res_fallback_w_id: Option<&str>,
storage: Option<&str>,
storage_ref: &str,
) -> Result<(windmill_types::s3::LargeFileStorage, serde_json::Value)> {
use windmill_types::s3::{
AzureBlobStorage, FilesystemStorage, GoogleCloudStorage, LargeFileStorage, S3Storage,
};
let lfs: LargeFileStorage = if let Some((typ, path)) = storage_ref.split_once(':') {
// Rebuild the LFS entry from the registered identity; only the variant (which
// resource parser applies) and the path matter to `lfs_to_object_store_resource`.
let s3 = |p: &str| S3Storage {
s3_resource_path: p.to_string(),
public_resource: None,
advanced_permissions: None,
};
match typ {
"S3Storage" => LargeFileStorage::S3Storage(s3(path)),
"S3AwsOidc" => LargeFileStorage::S3AwsOidc(s3(path)),
"AzureBlobStorage" => LargeFileStorage::AzureBlobStorage(AzureBlobStorage {
azure_blob_resource_path: path.to_string(),
public_resource: None,
advanced_permissions: None,
}),
"AzureWorkloadIdentity" => LargeFileStorage::AzureWorkloadIdentity(AzureBlobStorage {
azure_blob_resource_path: path.to_string(),
public_resource: None,
advanced_permissions: None,
}),
"GoogleCloudStorage" => LargeFileStorage::GoogleCloudStorage(GoogleCloudStorage {
gcs_resource_path: path.to_string(),
public_resource: None,
advanced_permissions: None,
}),
"FilesystemStorage" => LargeFileStorage::FilesystemStorage(FilesystemStorage {
root_path: path.to_string(),
public_resource: None,
advanced_permissions: None,
}),
other => {
return Err(Error::internal_err(format!(
"unknown registered storage type `{other}`"
)))
}
}
} else {
let lfs_json = sqlx::query_scalar!(
"SELECT large_file_storage FROM workspace_settings WHERE workspace_id = $1",
w_id
)
.fetch_optional(db)
.await?
.flatten()
.ok_or_else(|| Error::BadRequest("workspace has no storage configured".to_string()))?;
// Named storages live under `secondary_storage`; `None`/`_default_` is the primary.
match storage.filter(|s| *s != "_default_") {
None => serde_json::from_value(lfs_json.clone())
.map_err(|e| Error::internal_err(format!("parsing large_file_storage: {e}")))?,
Some(name) => serde_json::from_value(
lfs_json
.get("secondary_storage")
.and_then(|s| s.get(name))
.cloned()
.ok_or_else(|| {
Error::BadRequest(format!("workspace has no storage named {name}"))
})?,
)
.map_err(|e| Error::internal_err(format!("parsing storage {name}: {e}")))?,
}
};
// Filesystem storage stores a direct path (`lfs_to_object_store_resource` ignores the
// resource value); everything else references a resource whose stored path may or may not
// carry the `$res:` prefix — same normalization as `get_workspace_s3_resource_from_lfs`.
let resource_value = if matches!(lfs, LargeFileStorage::FilesystemStorage(_)) {
serde_json::Value::Null
} else {
let path = lfs.get_s3_resource_path();
let path = path.strip_prefix("$res:").unwrap_or(path);
resolve_res_with_fallback(db, w_id, res_fallback_w_id, path).await?
};
Ok((lfs, resource_value))
}
/// Resolve a `$res:` path in `w_id`, falling back to the same path in `res_fallback_w_id`
/// when the first lookup fails — retry-path cleanups run after the fork workspace (and its
/// cloned resource rows) were deleted, and the fork's resources were clones of a parent's.
async fn resolve_res_with_fallback(
db: &DB,
w_id: &str,
res_fallback_w_id: Option<&str>,
resource_path: &str,
) -> Result<serde_json::Value> {
let res = windmill_common::workspaces::transform_json_value_unchecked(
&serde_json::Value::String(format!("$res:{resource_path}")),
w_id,
db,
)
.await;
match (res, res_fallback_w_id) {
(Ok(v), _) => Ok(v),
(Err(e), None) => Err(e),
(Err(_), Some(fb)) => {
windmill_common::workspaces::transform_json_value_unchecked(
&serde_json::Value::String(format!("$res:{resource_path}")),
fb,
db,
)
.await
}
}
}
/// Delete every object under the fork's data prefix in the pre-resolved storage. Requires the
/// `parquet` (object store) feature; without it the metadata schema is still dropped and the
/// unreachable data files are left for manual cleanup.
#[cfg(feature = "parquet")]
async fn delete_fork_ducklake_data(
lfs: windmill_types::s3::LargeFileStorage,
resource_value: serde_json::Value,
data_path: &str,
) -> Result<()> {
use futures::{StreamExt, TryStreamExt};
let store = windmill_object_store::build_object_store_client(
&windmill_object_store::lfs_to_object_store_resource(&lfs, resource_value)?,
)
.await?;
let prefix = windmill_object_store::object_store_reexports::Path::from(
data_path.trim_matches('/').to_string(),
);
let locations: Vec<_> = store
.list(Some(&prefix))
.map_ok(|m| m.location)
.try_collect()
.await
.map_err(windmill_object_store::object_store_error_to_error)?;
// The object_store crate evaluates list prefixes on a path-SEGMENT basis (`a/b` does not
// match `a/bc/…`), so sibling fork segments sharing a string prefix are already excluded.
// Filter anyway — deletion must not depend on a listing implementation detail.
let boundary = format!("{}/", prefix.as_ref());
let locations: Vec<_> = locations
.into_iter()
.filter(|l| l.as_ref().starts_with(&boundary))
.collect();
// 1000-object chunks: S3 DeleteObjects caps a batch at 1000 keys.
for chunk in locations.chunks(1000) {
store
.delete_stream(futures::stream::iter(chunk.iter().cloned().map(Ok)).boxed())
.try_collect::<Vec<_>>()
.await
.map_err(windmill_object_store::object_store_error_to_error)?;
}
Ok(())
}
#[cfg(not(feature = "parquet"))]
async fn delete_fork_ducklake_data(
_lfs: windmill_types::s3::LargeFileStorage,
_resource_value: serde_json::Value,
_data_path: &str,
) -> Result<()> {
Err(Error::internal_err(
"object storage support (parquet feature) is not compiled in".to_string(),
))
}
/// Destroying an ATTACHED dev workspace (or its data environments) must be a prod-admin
/// action, not just the dev's own owner (dev ownership can diverge from prod's) — mirrors
/// detach_dev_workspace. Shared by `delete_workspace` and `drop_forked_ducklake_namespaces`
/// so the two gates cannot drift: the sidebar calls the drop endpoint BEFORE deleteWorkspace,
/// and a weaker gate on the drop would let a non-prod-admin dev owner destroy the live dev's
/// materializations and then have the deletion itself rejected. No-op for non-dev workspaces.
async fn require_prod_admin_for_dev_workspace(
db: &DB,
authed: &ApiAuthed,
w_id: &str,
) -> Result<()> {
if let Some(prod) = sqlx::query_scalar!(
"SELECT parent_workspace_id FROM workspace WHERE id = $1 AND is_dev_workspace",
w_id
)
.fetch_optional(db)
.await?
.flatten()
{
let is_prod_admin = sqlx::query_scalar!(
"SELECT is_admin FROM usr WHERE workspace_id = $1 AND email = $2",
&prod,
&authed.email
)
.fetch_optional(db)
.await?
.unwrap_or(false);
if !is_prod_admin && !is_super_admin_email(db, &authed.email).await? {
return Err(Error::PermissionDenied(format!(
"Destroying dev workspace '{w_id}' or its data requires being an admin of its parent prod workspace '{prod}' (or a superadmin)"
)));
}
}
Ok(())
}
async fn is_workspace_owner(
authed: &ApiAuthed,
w_id: &str,
backend/windmill-api/Cargo.toml
+1 -1
View File
@@ -10,7 +10,7 @@ path = "src/lib.rs"
[features]
default = []
private = ["windmill-audit/private", "windmill-common/private", "windmill-api-auth/private", "windmill-store/private", "windmill-api-users/private", "windmill-api-workspaces/private", "windmill-api-groups/private", "windmill-api-configs/private", "windmill-api-settings/private", "windmill-api-assets/private", "windmill-api-agent-workers?/private", "windmill-trigger-kafka?/private", "windmill-trigger-postgres?/private", "windmill-trigger-mqtt?/private", "windmill-trigger-websocket?/private", "windmill-trigger-nats?/private", "windmill-trigger-sqs?/private", "windmill-trigger-gcp?/private", "windmill-trigger-azure?/private", "windmill-trigger-email?/private", "windmill-git-sync/private", "windmill-autoscaling?/private"]
private = ["windmill-audit/private", "windmill-common/private", "windmill-api-auth/private", "windmill-store/private", "windmill-api-users/private", "windmill-api-workspaces/private", "windmill-api-groups/private", "windmill-api-configs/private", "windmill-api-settings/private", "windmill-api-assets/private", "windmill-api-agent-workers?/private", "windmill-trigger-kafka?/private", "windmill-trigger-postgres?/private", "windmill-trigger-mqtt?/private", "windmill-trigger-websocket?/private", "windmill-trigger-nats?/private", "windmill-trigger-sqs?/private", "windmill-trigger-gcp?/private", "windmill-trigger-azure?/private", "windmill-trigger-email?/private", "windmill-git-sync/private", "windmill-autoscaling?/private", "windmill-object-store/private"]
enterprise = ["windmill-queue/enterprise", "windmill-audit/enterprise", "windmill-git-sync/enterprise", "windmill-common/enterprise", "windmill-worker?/enterprise", "windmill-api-auth/enterprise", "windmill-store/enterprise", "windmill-api-jobs/enterprise", "windmill-api-scripts/enterprise", "windmill-api-flows/enterprise", "windmill-api-users/enterprise", "windmill-api-workspaces/enterprise", "windmill-api-groups/enterprise", "windmill-api-configs/enterprise", "windmill-api-settings/enterprise", "windmill-api-schedule/enterprise", "windmill-api-agent-workers?/enterprise", "windmill-trigger/enterprise", "windmill-trigger-kafka?/enterprise", "windmill-trigger-postgres?/enterprise", "windmill-trigger-mqtt?/enterprise", "windmill-trigger-websocket?/enterprise", "windmill-trigger-email?/enterprise", "windmill-trigger-nats?/enterprise", "windmill-trigger-sqs?/enterprise", "windmill-trigger-gcp?/enterprise", "windmill-trigger-azure?/enterprise", "windmill-trigger-http?/enterprise", "windmill-native-triggers?/enterprise", "dep:windmill-autoscaling", "windmill-autoscaling/enterprise", "license"]
stripe = []
run_inline = ["dep:windmill-worker", "windmill-api-configs/run_inline"]
backend/windmill-api/openapi.yaml
+165 -1
View File
@@ -1,7 +1,7 @@
openapi: "3.0.3"
info:
version: 1.747.0
version: 1.749.0
title: Windmill API
contact:
@@ -4983,6 +4983,24 @@ paths:
items:
type: string
/w/{workspace}/workspaces/drop_forked_ducklake_namespaces:
post:
summary: drop this fork workspace's ducklake namespaces (catalog metadata schemas + data files)
operationId: dropForkedDucklakeNamespaces
tags:
- workspace
parameters:
- $ref: "#/components/parameters/WorkspaceId"
responses:
"200":
description: list of errors (empty if all succeeded)
content:
application/json:
schema:
type: array
items:
type: string
/w/{workspace}/workspaces/import_pg_database:
post:
summary: import a PostgreSQL database from source to target via pg_dump
@@ -9504,6 +9522,50 @@ paths:
items:
$ref: "#/components/schemas/CiTestResult"
/w/{workspace}/scripts/check_schema_contracts:
post:
summary: check a script's asset references against captured producer schemas
description: |
Save-time schema-contract check for data pipelines: validates the given
script content's asset references (body column reads, `// column` lineage,
`// data_test relationships`) against the latest captured producer schemas
and returns warnings. Warnings never block a save/deploy; an asset whose
producer declares `on_schema_change=ignore` is suppressed to a single
informational entry.
operationId: checkSchemaContracts
tags:
- script
parameters:
- $ref: "#/components/parameters/WorkspaceId"
requestBody:
required: true
content:
application/json:
schema:
type: object
required:
- language
- content
properties:
language:
$ref: "#/components/schemas/ScriptLang"
content:
type: string
responses:
"200":
description: contract warnings (empty when all references match)
content:
application/json:
schema:
type: object
required:
- warnings
properties:
warnings:
type: array
items:
$ref: "#/components/schemas/ContractWarning"
/w/{workspace}/scripts/raw_temp/store:
post:
summary: store raw script content temporarily for CLI lock generation
@@ -21737,6 +21799,10 @@ paths:
$ref: "#/components/schemas/AssetKind"
path:
type: string
fork_materialization:
type: string
enum: [fork, deferred]
description: Fork workspaces only — 'fork' when this ducklake asset was materialized in the fork itself, 'deferred' when reads fall back to the parent workspace's current table via a defer view. Omitted otherwise.
runnables:
type: array
items:
@@ -21846,6 +21912,38 @@ paths:
type: string
via_use:
type: boolean
test_edges:
type: array
description: >-
Ordering-only "must-run-after" edges — a `// data_test relationships`
(or custom test reading a pipeline asset) requires the referenced
asset's producer to run before the tested script. Not a data-consumption
edge; fed into the cascade topo-sort so cold runs order correctly.
Omitted when empty.
items:
type: object
required:
[
producer_kind,
producer_path,
runnable_kind,
runnable_path,
asset_kind,
asset_path,
]
properties:
producer_kind:
$ref: "#/components/schemas/AssetUsageKind"
producer_path:
type: string
runnable_kind:
$ref: "#/components/schemas/AssetUsageKind"
runnable_path:
type: string
asset_kind:
$ref: "#/components/schemas/AssetKind"
asset_path:
type: string
/w/{workspace}/assets/macros:
get:
@@ -25909,6 +26007,7 @@ components:
- google
- github
- asset
- freshness
TriggerMode:
description: job trigger mode
@@ -28450,6 +28549,11 @@ components:
new_dbname:
type: string
description: "New database name for the fork"
shared_ducklakes:
type: array
items:
type: string
description: "Lake names the fork SHARES with the parent (reads and writes the parent's lake directly). Every lake not listed gets the default isolated fork namespace with read-defer to the parent."
is_dev_workspace:
type: boolean
description: "Create the fork as a persistent dev workspace (id not required to carry the wm-fork- prefix; at most one per parent)"
@@ -29189,6 +29293,33 @@ components:
- path
extra_args:
type: string
fork_behavior:
type: string
enum: [isolated, shared]
description:
Fork workspaces only - how this lake behaves in the fork, stamped at fork
creation. Absent = isolated (fork-scoped namespace + read-defer to parent).
maintenance:
type: object
description:
Scheduled maintenance (enterprise) - snapshot expiry, adjacent-file
compaction and orphaned-file cleanup, run as a managed per-lake schedule
required: [enabled]
properties:
enabled:
type: boolean
schedule:
type: string
description: cron cadence (v2, UTC); defaults to daily at 03h with a per-lake minute offset
retention_days:
type: integer
description: snapshot retention window in days (default 7); time-travel older than this stops working
compaction:
type: boolean
description: merge adjacent small parquet files (default true)
orphan_cleanup:
type: boolean
description: delete orphaned files older than max(retention, 1 day) (default true)
DataTableSettings:
type: object
@@ -30260,6 +30391,39 @@ components:
kind:
$ref: "#/components/schemas/AssetKind"
required: [path, kind]
ContractWarning:
description: |
One save-time schema-contract warning: a consumer reference that does
not match the referenced asset's latest captured schema.
`schema_version`/`captured_at` identify the capture the check ran
against (as-of the producer's last run, not its latest save).
type: object
required: [kind, asset_path, message]
properties:
kind:
type: string
enum:
- missing_column
- missing_lineage_source
- missing_relationship_column
- relationship_type_mismatch
- suppressed
asset_path:
type: string
column:
type: string
expected_type:
type: string
found_type:
type: string
schema_version:
type: integer
format: int64
captured_at:
type: string
format: date-time
message:
type: string
Volume:
type: object
required:
backend/windmill-api/src/workspaces_export.rs
+6 -1
View File
@@ -919,11 +919,16 @@ pub(crate) async fn tarball_workspace(
}
if include_schedules.unwrap_or(false) {
// Managed ducklake-maintenance schedules are excluded: they are
// derived from the workspace ducklake settings (and admins bypass the
// RLS that hides them), so exporting them would drag unsyncable rows
// into git.
let schedules = sqlx::query_as::<_, Schedule>(
"SELECT workspace_id, path, edited_by, edited_at, schedule, timezone, enabled, script_path, is_flow, args, extra_perms, email, permissioned_as, error, on_failure, on_failure_times, on_failure_exact, on_failure_extra_args, on_recovery, on_recovery_times, on_recovery_extra_args, on_success, on_success_extra_args, ws_error_handler_muted, retry, no_flow_overlap, summary, description, tag, paused_until, cron_version, dynamic_skip, labels FROM schedule
WHERE workspace_id = $1",
WHERE workspace_id = $1 AND NOT starts_with(path, $2)",
)
.bind(&w_id)
.bind(windmill_common::workspaces::DUCKLAKE_MAINTENANCE_PATH_PREFIX)
.fetch_all(&mut *tx)
.await?;
backend/windmill-common/src/assets.rs
+44 -1
View File
@@ -6,7 +6,7 @@ use crate::{error, scripts::ScriptHash};
pub use windmill_parser::asset_parser::{
merge_column_lineage, parse_pipeline_annotations, ColumnLineage, ColumnRef, DataTest,
PartitionKind, PipelineAnnotations, RetrySpec, TriggerSpec, PARTITION_TOKEN,
OnSchemaChange, PartitionKind, PipelineAnnotations, RetrySpec, TriggerSpec, PARTITION_TOKEN,
};
pub use windmill_types::assets::*;
@@ -352,6 +352,10 @@ mod debounce_duration_tests {
assert_eq!(parse_duration_secs("5m"), Some(300));
assert_eq!(parse_duration_secs("2h"), Some(7200));
assert_eq!(parse_duration_secs(" 1d "), Some(86400));
// Explicit plus sign comes free with i64 parsing; the TS mirror
// (parseDurationSecs) matches it — keep the two in lockstep.
assert_eq!(parse_duration_secs("+5m"), Some(300));
assert_eq!(parse_duration_secs("+45"), Some(45));
}
#[test]
@@ -365,6 +369,45 @@ mod debounce_duration_tests {
}
}
#[cfg(test)]
mod trigger_ref_roundtrip_tests {
use super::{parse_asset_trigger_ref, trigger_spec_to_row, AssetKind, ScriptTriggerKind};
use windmill_parser::asset_parser::{parse_asset_syntax, AssetKind as PAssetKind, TriggerSpec};
// `trigger_spec_to_row` rebuilds a stored ref as `s3://<path>`, and
// `parse_asset_trigger_ref` parses it back. The two must be inverse for
// every S3 URI form, or a consumer's `// on` trigger lands on a different
// graph node than the producer's inferred write. Because `parse_asset_syntax`
// strips ALL leading slashes, a canonical path never starts with `/`, so the
// naive `prefix + path` rebuild round-trips — including the `S3Object(s3="/x")`
// quad-slash case that previously desynced (path `/x` rebuilt to `s3:///x`,
// which re-parsed to `x`).
fn roundtrip(uri: &str) -> String {
let (pkind, path) = parse_asset_syntax(uri, false).expect("parse uri");
assert_eq!(pkind, PAssetKind::S3Object);
let spec = TriggerSpec::Asset { asset_kind: pkind, path: path.to_string(), debounce: None };
let (kind, stored) = trigger_spec_to_row(&spec).expect("to row");
assert_eq!(kind, ScriptTriggerKind::Asset);
let (rkind, rpath) = parse_asset_trigger_ref(&stored).expect("parse ref");
assert_eq!(rkind, AssetKind::S3Object);
// The producer path and the round-tripped consumer path must match.
assert_eq!(
rpath, path,
"round-trip diverged for {uri} (stored {stored})"
);
rpath
}
#[test]
fn s3_trigger_ref_roundtrips_for_every_uri_form() {
assert_eq!(roundtrip("s3:///exports/x"), "exports/x"); // SDK default storage
assert_eq!(roundtrip("s3://exports/x"), "exports/x"); // DuckDB / bare
assert_eq!(roundtrip("s3://mybucket/exports/x"), "mybucket/exports/x"); // explicit
assert_eq!(roundtrip("s3:////x"), "x"); // S3Object(s3="/x") quad-slash
assert_eq!(roundtrip("s3:///y=2024/f.parquet"), "y=2024/f.parquet"); // Hive
}
}
// Inverse of trigger_spec_to_row for the Asset variant: parses a stored
// trigger_ref (e.g. `s3://foo`, `$res:bar`) back into the (kind, path) pair
// used as a graph node id. Returns None for refs that don't match any known
backend/windmill-common/src/lib.rs
+1
View File
@@ -65,6 +65,7 @@ pub mod materialization;
pub mod min_version;
pub mod notify_events;
pub mod runtime_assets;
pub mod schema_contracts;
pub mod workspace_dependencies;
#[cfg(feature = "private")]
backend/windmill-common/src/materialization.rs
+97 -1
View File
@@ -94,7 +94,11 @@ pub async fn record_materialization<'e>(
VALUES ($1, $2, $3, $4, $5, $6, $7, $8, now(), $9)
ON CONFLICT (workspace_id, asset_kind, asset_path, partition)
DO UPDATE SET status = EXCLUDED.status,
snapshot_id = EXCLUDED.snapshot_id,
-- A failed run records no snapshot, but must not erase the last
-- committed one: a physical table from an earlier commit (or from a
-- committed write whose data tests then failed) still exists, and
-- fork defer/graph state keys on that evidence.
snapshot_id = COALESCE(EXCLUDED.snapshot_id, materialized_partition.snapshot_id),
row_count = EXCLUDED.row_count,
job_id = EXCLUDED.job_id,
materialized_at = now(),
@@ -255,6 +259,98 @@ pub async fn record_asset_schema(
Ok(true)
}
/// One table a fork workspace should read from an ancestor through a defer view.
#[derive(Debug, Clone, Serialize, Deserialize)]
pub struct ForkDeferTable {
/// Lake-internal table name: the `asset_path` minus its `<lake>/` prefix (may itself be
/// `schema.table`).
pub table: String,
/// The owning ancestor's latest captured schema carries the SCD2 marker column
/// (`is_current`), so that lake also holds a managed `<table>_current` companion view
/// that consumers read — defer it alongside the table.
#[serde(default)]
pub with_current_view: bool,
/// Index into `DucklakeForkDefer.ancestors` (nearest-first) of the NEAREST ancestor that
/// materialized this table — the defer view must target that ancestor's namespace. In a
/// `fork → parent → root` chain where only root materialized a table, the parent has no
/// physical copy (it defers too), so a view over the parent would not bind. Defaults to 0
/// (the direct parent) for wire compatibility with agents that predate the field.
#[serde(default)]
pub ancestor_idx: u32,
}
/// Tables of lake `lake_name` materialized somewhere in the fork's ancestor chain
/// (nearest-first) but not (yet) in the fork — the fork's read-defer set, each mapped to the
/// nearest ancestor that owns a physical copy. Only `Materialized` rows count on every side: a
/// deferred table must physically exist in the targeted ancestor (defer views bind at CREATE
/// and would otherwise fail the whole job), and any successful fork materialization makes the
/// fork's own table authoritative.
///
/// **Authorization:** performs no access control; trusted server-side callers only. It reads
/// ancestor workspaces' rows on behalf of a fork — acceptable because defer itself exposes
/// the ancestors' table contents to fork members.
pub async fn list_fork_defer_tables<'e>(
executor: impl PgExecutor<'e>,
ancestor_workspace_ids: &[String],
fork_workspace_id: &str,
lake_name: &str,
) -> Result<Vec<ForkDeferTable>> {
let rows = sqlx::query!(
r#"
WITH ancestor AS (
SELECT wid, ord FROM unnest($1::text[]) WITH ORDINALITY AS a(wid, ord)
), anc_mat AS (
-- Per table, the nearest ancestor (lowest ord) that materialized it.
SELECT DISTINCT ON (mp.asset_path) mp.asset_path, a.wid, a.ord
FROM materialized_partition mp
JOIN ancestor a ON a.wid = mp.workspace_id
WHERE mp.asset_kind = 'ducklake' AND mp.status = 'materialized'
AND split_part(mp.asset_path, '/', 1) = $3 AND mp.asset_path LIKE '%/%'
ORDER BY mp.asset_path, a.ord
), fork_mat AS (
-- Fork-OWNED assets: anything whose physical table exists in the fork
-- namespace, not just clean materializations. A committed write whose data
-- tests failed afterwards records status='failed' WITH a snapshot — its table
-- is real, and a defer view emitted over it would silently yield to it
-- (CREATE VIEW IF NOT EXISTS) while claiming the read defers to the parent.
SELECT DISTINCT asset_path FROM materialized_partition
WHERE workspace_id = $2 AND asset_kind = 'ducklake'
AND (status = 'materialized' OR snapshot_id IS NOT NULL)
), latest_schema AS (
SELECT DISTINCT ON (workspace_id, asset_path) workspace_id, asset_path, columns
FROM materialized_asset_schema
WHERE workspace_id = ANY($1) AND asset_kind = 'ducklake'
ORDER BY workspace_id, asset_path, version DESC
)
SELECT am.asset_path AS "asset_path!",
am.ord AS "ord!",
COALESCE(EXISTS (
SELECT 1 FROM jsonb_array_elements(ls.columns) e
WHERE e->>'name' = 'is_current'
), false) AS "has_current!"
FROM anc_mat am
LEFT JOIN latest_schema ls
ON ls.asset_path = am.asset_path AND ls.workspace_id = am.wid
WHERE am.asset_path NOT IN (SELECT asset_path FROM fork_mat)
ORDER BY am.asset_path
"#,
ancestor_workspace_ids,
fork_workspace_id,
lake_name,
)
.fetch_all(executor)
.await?;
Ok(rows
.into_iter()
.map(|r| ForkDeferTable {
table: r.asset_path[lake_name.len() + 1..].to_string(),
with_current_view: r.has_current,
// WITH ORDINALITY is 1-based; ancestors vec is 0-based.
ancestor_idx: (r.ord - 1).max(0) as u32,
})
.collect())
}
/// All captured schema versions for one asset, newest version first.
///
/// **Authorization:** performs no access control (mirrors

Some files were not shown because too many files have changed in this diff Show More