diff --git a/AGENTS.md b/AGENTS.md index ac00741c49..83b68f7d36 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -24,7 +24,8 @@ Open-source platform for internal tools, workflows, API integrations, background ## Dev Environment - **Backend**: `cargo run` from `backend/` (API at http://localhost:8000) -- **DuckDB local jobs**: before running DuckDB scripts locally, build the FFI shared library with `cd backend/windmill-duckdb-ffi-internal && ./build_dev.sh`. Re-run it after clean builds or when `backend/target/debug/libwindmill_duckdb_ffi_internal.*` is missing. +- **DuckDB local jobs**: before running DuckDB scripts locally, build the FFI shared library with `cd backend/windmill-duckdb-ffi-internal && ./build_dev.sh`. Re-run it after clean builds or when `backend/target/debug/libwindmill_duckdb_ffi_internal.*` is missing. The bundled DuckDB compile (~2min) is cached in a per-user dir shared across worktrees, so a fresh worktree reuses it and the build is near-instant. +- **Data pipelines (DuckLake) from source**: a plain `cargo run` (even `--features quickjs`) advertises a `duckdb` worker tag but **cannot** execute DuckDB scripts and has **no** working S3 proxy (DuckLake writes 404). Build CE DuckLake with `cargo run --features quickjs,duckdb,parquet,private` (add `,python` for Python scripts, `,enterprise,license` for EE) **and** build the FFI (bullet above). See `backend/CLAUDE.md` → "Running data pipelines (DuckLake) from source" for the exact feature sets and the two feature-gate gotchas. - **Frontend**: `REMOTE=http://localhost:8000 npm run dev` from `frontend/` (port 3000+) - **DB**: `psql postgres://postgres:changeme@localhost:5432/windmill` - **Login**: `admin@windmill.dev` / `changeme` diff --git a/CHANGELOG.md b/CHANGELOG.md index b5342e3024..365084fbfb 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,5 +1,49 @@ # Changelog +## [1.749.0](https://github.com/windmill-labs/windmill/compare/v1.748.0...v1.749.0) (2026-07-05) + + +### Features + +* **pipelines:** mid-DAG selective execution (dbt `model+`) for pipeline runs ([#9945](https://github.com/windmill-labs/windmill/issues/9945)) ([2d3a773](https://github.com/windmill-labs/windmill/commit/2d3a77344104a587548f23f1b614ceffd52a5778)) +* **pipelines:** partition run-arg picker + first-run setup signpost ([#9943](https://github.com/windmill-labs/windmill/issues/9943)) ([475b072](https://github.com/windmill-labs/windmill/commit/475b072987b33d50111f5251a5f69f4245f930ae)) +* **pipelines:** self-teaching custom data_test errors + scaffold ([#9937](https://github.com/windmill-labs/windmill/issues/9937)) ([0ad174f](https://github.com/windmill-labs/windmill/commit/0ad174fa490e17eeb26280b5bdfd62956dfed9ff)) + + +### Bug Fixes + +* **cli:** macro-library parity in --local pipeline graph + read-only run --dry-run ([#9942](https://github.com/windmill-labs/windmill/issues/9942)) ([e3f4303](https://github.com/windmill-labs/windmill/commit/e3f43033cafcdb5df253aeb55ce93e599b2584d2)) +* **datatable:** self-teaching error for unresolved datatable:// references ([#9941](https://github.com/windmill-labs/windmill/issues/9941)) ([55451db](https://github.com/windmill-labs/windmill/commit/55451db009e3060c21948ece2c97e102a3c9b171)) +* **object-storage:** remove 20-file bucket-browser listing cap in CE ([#9935](https://github.com/windmill-labs/windmill/issues/9935)) ([22452ce](https://github.com/windmill-labs/windmill/commit/22452ce54034a9bea8f7d48946818fd148b938c0)) +* **pipelines:** dedup guard for keyed merge + deploy-time SCD2 validation ([#9936](https://github.com/windmill-labs/windmill/issues/9936)) ([52ce805](https://github.com/windmill-labs/windmill/commit/52ce805f619747af4f998cde7819a164c754205a)) +* **pipelines:** link SCD2 <dim>_current view to its producer across all graph surfaces ([#9933](https://github.com/windmill-labs/windmill/issues/9933)) ([574d3ac](https://github.com/windmill-labs/windmill/commit/574d3ac9ff5015b5d3f53040c9d4dfbfd161a076)) +* **pipelines:** order data_test relationships refs before the tested script in a cascade ([#9934](https://github.com/windmill-labs/windmill/issues/9934)) ([46be39d](https://github.com/windmill-labs/windmill/commit/46be39dfb7fbfb2b70e61819d6065b45810c41c9)) +* **pipelines:** pipeline-level run control, tables label, data-test rollback + fork badges ([#9944](https://github.com/windmill-labs/windmill/issues/9944)) ([6ae8dd3](https://github.com/windmill-labs/windmill/commit/6ae8dd37b1de930ab17344cebf7c28385c6cfdba)) +* rebuild windows bun loader main.ts filter from forward-slash cdir ([#9946](https://github.com/windmill-labs/windmill/issues/9946)) ([a582e04](https://github.com/windmill-labs/windmill/commit/a582e04bf40cf685f88bceaf88e3d24bde3d420a)) + +## [1.748.0](https://github.com/windmill-labs/windmill/compare/v1.747.0...v1.748.0) (2026-07-05) + + +### Features + +* **ai-agent:** support reasoning effort in AI agent workflow steps ([#9886](https://github.com/windmill-labs/windmill/issues/9886)) ([a368d49](https://github.com/windmill-labs/windmill/commit/a368d49bd8786a2dca6771f2051f1d44d1b2363d)) +* **ducklake:** scheduled lake maintenance (expiry, compaction, orphan cleanup) ([#9916](https://github.com/windmill-labs/windmill/issues/9916)) ([3352150](https://github.com/windmill-labs/windmill/commit/33521505dbc34f22b575d21fda1cc76d698a8840)) +* **pipelines:** asset freshness — fresh/stale badge (CE) + watchdog (EE) ([#9909](https://github.com/windmill-labs/windmill/issues/9909)) ([5d7fb6d](https://github.com/windmill-labs/windmill/commit/5d7fb6deca3e02e89d77e5d3856483beb8b8bfeb)) +* **pipelines:** capture violating-row samples for data tests ([#9919](https://github.com/windmill-labs/windmill/issues/9919)) ([d4b4374](https://github.com/windmill-labs/windmill/commit/d4b4374de8f8a7875b050c16d1236fcd0355812b)) +* **pipelines:** fork data environments for ducklake materialization (dev data) ([#9915](https://github.com/windmill-labs/windmill/issues/9915)) ([39eb9de](https://github.com/windmill-labs/windmill/commit/39eb9de1bce400109c130a081807e40e995ae068)) +* **pipelines:** on_schema_change write guardrails + data_test deploy validation ([#9930](https://github.com/windmill-labs/windmill/issues/9930)) ([377c02e](https://github.com/windmill-labs/windmill/commit/377c02ec47389e64b7ef5cbbae0de05df648266d)) +* **pipelines:** record upstream snapshot ids on cascade-dispatched jobs ([#9910](https://github.com/windmill-labs/windmill/issues/9910)) ([af36498](https://github.com/windmill-labs/windmill/commit/af36498432e643108308e1c03b5d986d0f0f8888)) +* **pipelines:** schema contracts — save-time consumer checks vs captured schemas ([#9917](https://github.com/windmill-labs/windmill/issues/9917)) ([42e11c6](https://github.com/windmill-labs/windmill/commit/42e11c6570b62ffaa86598438fa8ddf462c4035f)) +* **pipeline:** write-audit-publish for materialization data tests ([#9911](https://github.com/windmill-labs/windmill/issues/9911)) ([dce247c](https://github.com/windmill-labs/windmill/commit/dce247c6d2678a2c95bd728027e17ae3965638e2)) +* **sdk:** enforce s3:// URIs for string S3 params + ingestion (EL) docs ([#9912](https://github.com/windmill-labs/windmill/issues/9912)) ([5ad2de9](https://github.com/windmill-labs/windmill/commit/5ad2de91a26b312bf27124ceca16ef331621bde8)) + + +### Bug Fixes + +* **cli:** pipeline + workspace UX batch (init/bind stub, run errors, macro libs, lock-job report, upgrade errors) ([#9929](https://github.com/windmill-labs/windmill/issues/9929)) ([28a6b08](https://github.com/windmill-labs/windmill/commit/28a6b086c842105298f236baa0a61868f71a5eb1)) +* **cli:** publish all windmill-parser-wasm-* deps so local pipeline graph keeps write edges ([#9926](https://github.com/windmill-labs/windmill/issues/9926)) ([744a759](https://github.com/windmill-labs/windmill/commit/744a7597edaf3ca9a7fd2b21a34fb33457913a64)) +* **pipelines:** activity-axis label clarity + select failed node on cascade failure ([#9931](https://github.com/windmill-labs/windmill/issues/9931)) ([5769b60](https://github.com/windmill-labs/windmill/commit/5769b6036cf14b0cb424c5b3d9d878c600a5652e)) + ## [1.747.0](https://github.com/windmill-labs/windmill/compare/v1.746.0...v1.747.0) (2026-07-03) diff --git a/backend/.sqlx/query-089d7bc7acdbb97cf477159e111bc7e9ee85289ff5c52af43166928337c257e7.json b/backend/.sqlx/query-089d7bc7acdbb97cf477159e111bc7e9ee85289ff5c52af43166928337c257e7.json index a779aa0e95..6efb66005d 100644 --- a/backend/.sqlx/query-089d7bc7acdbb97cf477159e111bc7e9ee85289ff5c52af43166928337c257e7.json +++ b/backend/.sqlx/query-089d7bc7acdbb97cf477159e111bc7e9ee85289ff5c52af43166928337c257e7.json @@ -35,7 +35,8 @@ "ci_test", "github", "azure", - "asset" + "asset", + "freshness" ] } } diff --git a/backend/.sqlx/query-0e22295e40bfc173d85f0a962d491f3afd87eeea5e4e8d935b1b86eb02f061e7.json b/backend/.sqlx/query-0e22295e40bfc173d85f0a962d491f3afd87eeea5e4e8d935b1b86eb02f061e7.json new file mode 100644 index 0000000000..906c710fdf --- /dev/null +++ b/backend/.sqlx/query-0e22295e40bfc173d85f0a962d491f3afd87eeea5e4e8d935b1b86eb02f061e7.json @@ -0,0 +1,15 @@ +{ + "db_name": "PostgreSQL", + "query": "DELETE FROM pipeline_freshness_state WHERE workspace_id = $1 AND script_path = $2", + "describe": { + "columns": [], + "parameters": { + "Left": [ + "Text", + "Text" + ] + }, + "nullable": [] + }, + "hash": "0e22295e40bfc173d85f0a962d491f3afd87eeea5e4e8d935b1b86eb02f061e7" +} diff --git a/backend/.sqlx/query-0f141ca6a58901dee1ddcf25694d76a4e5702c0324a27ef3a5740ef5d98d8946.json b/backend/.sqlx/query-0f141ca6a58901dee1ddcf25694d76a4e5702c0324a27ef3a5740ef5d98d8946.json new file mode 100644 index 0000000000..0dbcf04c6e --- /dev/null +++ b/backend/.sqlx/query-0f141ca6a58901dee1ddcf25694d76a4e5702c0324a27ef3a5740ef5d98d8946.json @@ -0,0 +1,31 @@ +{ + "db_name": "PostgreSQL", + "query": "SELECT\n schedule.path, t.jobs FROM schedule,\n LATERAL(SELECT ARRAY(\n SELECT json_build_object('id', id, 'success', status = 'success', 'duration_ms', duration_ms)\n FROM v2_job_completed c JOIN v2_job j USING (id)\n WHERE trigger_kind = 'schedule'\n AND trigger = schedule.path\n AND c.workspace_id = $1\n AND j.workspace_id = $1\n AND parent_job IS NULL AND runnable_path = schedule.script_path\n AND status <> 'skipped'\n ORDER BY completed_at DESC\n LIMIT 20\n ) AS jobs) t\n WHERE workspace_id = $1 AND NOT starts_with(schedule.path, $4)\n ORDER BY edited_at DESC\n LIMIT $2 OFFSET $3", + "describe": { + "columns": [ + { + "ordinal": 0, + "name": "path", + "type_info": "Varchar" + }, + { + "ordinal": 1, + "name": "jobs", + "type_info": "JsonArray" + } + ], + "parameters": { + "Left": [ + "Text", + "Int8", + "Int8", + "Text" + ] + }, + "nullable": [ + false, + null + ] + }, + "hash": "0f141ca6a58901dee1ddcf25694d76a4e5702c0324a27ef3a5740ef5d98d8946" +} diff --git a/backend/.sqlx/query-142ae6939440654da21ebb06acf838563f28e832e37e5c165329441d8471acdb.json b/backend/.sqlx/query-142ae6939440654da21ebb06acf838563f28e832e37e5c165329441d8471acdb.json new file mode 100644 index 0000000000..c19df789d4 --- /dev/null +++ b/backend/.sqlx/query-142ae6939440654da21ebb06acf838563f28e832e37e5c165329441d8471acdb.json @@ -0,0 +1,16 @@ +{ + "db_name": "PostgreSQL", + "query": "INSERT INTO schedule (\n workspace_id, path, edited_by, edited_at, schedule, enabled, script_path,\n args, extra_perms, is_flow, email, error, timezone, on_failure,\n on_recovery, on_failure_times, on_failure_exact, on_failure_extra_args,\n on_recovery_times, on_recovery_extra_args, ws_error_handler_muted, retry,\n summary, no_flow_overlap, tag, paused_until, on_success, on_success_extra_args,\n cron_version, description, dynamic_skip, permissioned_as, labels\n )\n SELECT\n $1, path, edited_by, edited_at, schedule, FALSE, script_path,\n args, extra_perms, is_flow, email, error, timezone, on_failure,\n on_recovery, on_failure_times, on_failure_exact, on_failure_extra_args,\n on_recovery_times, on_recovery_extra_args, ws_error_handler_muted, retry,\n summary, no_flow_overlap, tag, paused_until, on_success, on_success_extra_args,\n cron_version, description, dynamic_skip, permissioned_as, labels\n FROM schedule WHERE workspace_id = $2 AND path NOT LIKE $3", + "describe": { + "columns": [], + "parameters": { + "Left": [ + "Varchar", + "Text", + "Text" + ] + }, + "nullable": [] + }, + "hash": "142ae6939440654da21ebb06acf838563f28e832e37e5c165329441d8471acdb" +} diff --git a/backend/.sqlx/query-212553c83e4dcdc6d045eb2fe2dadbb2860ce52d37a56b2861de1215260ecff8.json b/backend/.sqlx/query-212553c83e4dcdc6d045eb2fe2dadbb2860ce52d37a56b2861de1215260ecff8.json index f9bc97d29d..704883d4f1 100644 --- a/backend/.sqlx/query-212553c83e4dcdc6d045eb2fe2dadbb2860ce52d37a56b2861de1215260ecff8.json +++ b/backend/.sqlx/query-212553c83e4dcdc6d045eb2fe2dadbb2860ce52d37a56b2861de1215260ecff8.json @@ -39,7 +39,8 @@ "ci_test", "github", "azure", - "asset" + "asset", + "freshness" ] } } @@ -77,7 +78,8 @@ "ci_test", "github", "azure", - "asset" + "asset", + "freshness" ] } } diff --git a/backend/.sqlx/query-256a302afef987857016f2fa636d15474599734bc46ede6930650e2cba5d8a46.json b/backend/.sqlx/query-256a302afef987857016f2fa636d15474599734bc46ede6930650e2cba5d8a46.json new file mode 100644 index 0000000000..5f74d57491 --- /dev/null +++ b/backend/.sqlx/query-256a302afef987857016f2fa636d15474599734bc46ede6930650e2cba5d8a46.json @@ -0,0 +1,15 @@ +{ + "db_name": "PostgreSQL", + "query": "INSERT INTO script_trigger (workspace_id, runnable_kind, runnable_path, trigger_kind, trigger_ref, join_all, debounce_s, retry_count, retry_delay_s)\n SELECT $2, runnable_kind, runnable_path, trigger_kind, trigger_ref, join_all, debounce_s, retry_count, retry_delay_s\n FROM script_trigger WHERE workspace_id = $1", + "describe": { + "columns": [], + "parameters": { + "Left": [ + "Text", + "Varchar" + ] + }, + "nullable": [] + }, + "hash": "256a302afef987857016f2fa636d15474599734bc46ede6930650e2cba5d8a46" +} diff --git a/backend/.sqlx/query-28ec31481bdaae8b0512d21d074da5d913caa8d935b33ad8f21336dedefd1c23.json b/backend/.sqlx/query-28ec31481bdaae8b0512d21d074da5d913caa8d935b33ad8f21336dedefd1c23.json new file mode 100644 index 0000000000..a1e3edbbe2 --- /dev/null +++ b/backend/.sqlx/query-28ec31481bdaae8b0512d21d074da5d913caa8d935b33ad8f21336dedefd1c23.json @@ -0,0 +1,15 @@ +{ + "db_name": "PostgreSQL", + "query": "DELETE FROM schedule WHERE workspace_id = $1 AND path = ANY($2)", + "describe": { + "columns": [], + "parameters": { + "Left": [ + "Text", + "TextArray" + ] + }, + "nullable": [] + }, + "hash": "28ec31481bdaae8b0512d21d074da5d913caa8d935b33ad8f21336dedefd1c23" +} diff --git a/backend/.sqlx/query-333a703a53cc760842b047ad420efef25885a9bca64bb2ba4bb173bc19dcbdba.json b/backend/.sqlx/query-333a703a53cc760842b047ad420efef25885a9bca64bb2ba4bb173bc19dcbdba.json new file mode 100644 index 0000000000..dfbc2b75e6 --- /dev/null +++ b/backend/.sqlx/query-333a703a53cc760842b047ad420efef25885a9bca64bb2ba4bb173bc19dcbdba.json @@ -0,0 +1,23 @@ +{ + "db_name": "PostgreSQL", + "query": "SELECT path FROM schedule WHERE workspace_id = $1 AND path LIKE $2", + "describe": { + "columns": [ + { + "ordinal": 0, + "name": "path", + "type_info": "Varchar" + } + ], + "parameters": { + "Left": [ + "Text", + "Text" + ] + }, + "nullable": [ + false + ] + }, + "hash": "333a703a53cc760842b047ad420efef25885a9bca64bb2ba4bb173bc19dcbdba" +} diff --git a/backend/.sqlx/query-36da136cf9b9554702e3b3ec364368a6862e35350ae43383fb2335e2c82a9419.json b/backend/.sqlx/query-36da136cf9b9554702e3b3ec364368a6862e35350ae43383fb2335e2c82a9419.json new file mode 100644 index 0000000000..d05abd4f3a --- /dev/null +++ b/backend/.sqlx/query-36da136cf9b9554702e3b3ec364368a6862e35350ae43383fb2335e2c82a9419.json @@ -0,0 +1,29 @@ +{ + "db_name": "PostgreSQL", + "query": "SELECT DISTINCT path AS \"asset_path!\", usage_path AS \"producer_path!\"\n FROM asset\n WHERE workspace_id = $1 AND kind = 'ducklake' AND path = ANY($2)\n AND usage_kind = 'script' AND usage_access_type IN ('w', 'rw')", + "describe": { + "columns": [ + { + "ordinal": 0, + "name": "asset_path!", + "type_info": "Varchar" + }, + { + "ordinal": 1, + "name": "producer_path!", + "type_info": "Varchar" + } + ], + "parameters": { + "Left": [ + "Text", + "TextArray" + ] + }, + "nullable": [ + false, + false + ] + }, + "hash": "36da136cf9b9554702e3b3ec364368a6862e35350ae43383fb2335e2c82a9419" +} diff --git a/backend/.sqlx/query-3c84781704b84b8a927ecce5a3fcb3adcf0175d0a71013f2497397c1c8ccc619.json b/backend/.sqlx/query-3c84781704b84b8a927ecce5a3fcb3adcf0175d0a71013f2497397c1c8ccc619.json index 3568d1723e..f5ee767768 100644 --- a/backend/.sqlx/query-3c84781704b84b8a927ecce5a3fcb3adcf0175d0a71013f2497397c1c8ccc619.json +++ b/backend/.sqlx/query-3c84781704b84b8a927ecce5a3fcb3adcf0175d0a71013f2497397c1c8ccc619.json @@ -128,7 +128,8 @@ "ci_test", "github", "azure", - "asset" + "asset", + "freshness" ] } } diff --git a/backend/.sqlx/query-3ea82f16050318a0b3b87c12beb66b49254036ca93f7a6415ceb4e316ed64eed.json b/backend/.sqlx/query-3ea82f16050318a0b3b87c12beb66b49254036ca93f7a6415ceb4e316ed64eed.json new file mode 100644 index 0000000000..60dbee76ab --- /dev/null +++ b/backend/.sqlx/query-3ea82f16050318a0b3b87c12beb66b49254036ca93f7a6415ceb4e316ed64eed.json @@ -0,0 +1,29 @@ +{ + "db_name": "PostgreSQL", + "query": "\n SELECT p.path AS \"path!\",\n (SELECT c.completed_at\n FROM v2_job j\n JOIN v2_job_completed c ON c.id = j.id\n WHERE j.workspace_id = $1\n AND j.runnable_path = p.path\n AND j.parent_job IS NULL\n -- No 'singlestepflow': flows may share a script's path, and\n -- a same-path flow run must not read as the script being\n -- fresh (false-fresh). Script retries land as native\n -- 'script' jobs; only the rare flow-wrapper fallback is\n -- missed, which errs stale. Kept in lockstep with the\n -- freshness watchdog's queries (freshness_watchdog_ee).\n AND j.kind IN ('script', 'preview')\n AND c.status = 'success'\n ORDER BY j.created_at DESC\n LIMIT 1) AS last_success_at\n FROM unnest($2::text[]) AS p(path)\n ", + "describe": { + "columns": [ + { + "ordinal": 0, + "name": "path!", + "type_info": "Text" + }, + { + "ordinal": 1, + "name": "last_success_at", + "type_info": "Timestamptz" + } + ], + "parameters": { + "Left": [ + "Text", + "TextArray" + ] + }, + "nullable": [ + null, + null + ] + }, + "hash": "3ea82f16050318a0b3b87c12beb66b49254036ca93f7a6415ceb4e316ed64eed" +} diff --git a/backend/.sqlx/query-3eeecc4bb26163cf6c2c19bc7c2950a1563bd516e730aaa9d0f1501ea8cea7f9.json b/backend/.sqlx/query-3eeecc4bb26163cf6c2c19bc7c2950a1563bd516e730aaa9d0f1501ea8cea7f9.json new file mode 100644 index 0000000000..4599073796 --- /dev/null +++ b/backend/.sqlx/query-3eeecc4bb26163cf6c2c19bc7c2950a1563bd516e730aaa9d0f1501ea8cea7f9.json @@ -0,0 +1,58 @@ +{ + "db_name": "PostgreSQL", + "query": "SELECT ducklake_name AS \"ducklake_name!\", metadata_schema AS \"metadata_schema!\",\n catalog AS \"catalog!\", storage AS \"storage!\",\n storage_ref AS \"storage_ref!\", data_path AS \"data_path!\",\n schema_dropped AS \"schema_dropped!\"\n FROM fork_ducklake_namespace WHERE workspace_id = $1", + "describe": { + "columns": [ + { + "ordinal": 0, + "name": "ducklake_name!", + "type_info": "Varchar" + }, + { + "ordinal": 1, + "name": "metadata_schema!", + "type_info": "Varchar" + }, + { + "ordinal": 2, + "name": "catalog!", + "type_info": "Text" + }, + { + "ordinal": 3, + "name": "storage!", + "type_info": "Text" + }, + { + "ordinal": 4, + "name": "storage_ref!", + "type_info": "Text" + }, + { + "ordinal": 5, + "name": "data_path!", + "type_info": "Text" + }, + { + "ordinal": 6, + "name": "schema_dropped!", + "type_info": "Bool" + } + ], + "parameters": { + "Left": [ + "Text" + ] + }, + "nullable": [ + false, + false, + false, + false, + false, + false, + false + ] + }, + "hash": "3eeecc4bb26163cf6c2c19bc7c2950a1563bd516e730aaa9d0f1501ea8cea7f9" +} diff --git a/backend/.sqlx/query-431a5eb2eb5a0b57e096cc78f68eb5a0665c90c4926f1afb689d6cd16bd5b722.json b/backend/.sqlx/query-431a5eb2eb5a0b57e096cc78f68eb5a0665c90c4926f1afb689d6cd16bd5b722.json new file mode 100644 index 0000000000..b554ef8b55 --- /dev/null +++ b/backend/.sqlx/query-431a5eb2eb5a0b57e096cc78f68eb5a0665c90c4926f1afb689d6cd16bd5b722.json @@ -0,0 +1,29 @@ +{ + "db_name": "PostgreSQL", + "query": "SELECT DISTINCT ON (path) path, content\n FROM script\n WHERE workspace_id = $1 AND path = ANY($2)\n AND archived = false AND deleted = false\n ORDER BY path, created_at DESC", + "describe": { + "columns": [ + { + "ordinal": 0, + "name": "path", + "type_info": "Varchar" + }, + { + "ordinal": 1, + "name": "content", + "type_info": "Text" + } + ], + "parameters": { + "Left": [ + "Text", + "TextArray" + ] + }, + "nullable": [ + false, + false + ] + }, + "hash": "431a5eb2eb5a0b57e096cc78f68eb5a0665c90c4926f1afb689d6cd16bd5b722" +} diff --git a/backend/.sqlx/query-45a539e61a3c35098fb6d59c3d5de8b6fcebe83002c74647666e3a6f5b8490dd.json b/backend/.sqlx/query-45a539e61a3c35098fb6d59c3d5de8b6fcebe83002c74647666e3a6f5b8490dd.json new file mode 100644 index 0000000000..5b3e70acbf --- /dev/null +++ b/backend/.sqlx/query-45a539e61a3c35098fb6d59c3d5de8b6fcebe83002c74647666e3a6f5b8490dd.json @@ -0,0 +1,23 @@ +{ + "db_name": "PostgreSQL", + "query": "\n SELECT EXISTS(\n SELECT 1 FROM v2_job_queue q\n JOIN v2_job j ON j.id = q.id\n WHERE q.workspace_id = $1\n AND j.runnable_path = $2\n AND j.parent_job IS NULL\n AND j.kind IN ('script', 'preview')\n AND (q.running = true OR q.scheduled_for <= now())\n ) AS \"in_flight!\"\n ", + "describe": { + "columns": [ + { + "ordinal": 0, + "name": "in_flight!", + "type_info": "Bool" + } + ], + "parameters": { + "Left": [ + "Text", + "Text" + ] + }, + "nullable": [ + null + ] + }, + "hash": "45a539e61a3c35098fb6d59c3d5de8b6fcebe83002c74647666e3a6f5b8490dd" +} diff --git a/backend/.sqlx/query-45c73bd8742601753ad947c88b65ff3ae182ff7c0631d4fa44b67b11a133498f.json b/backend/.sqlx/query-45c73bd8742601753ad947c88b65ff3ae182ff7c0631d4fa44b67b11a133498f.json new file mode 100644 index 0000000000..6dc5fdb9a6 --- /dev/null +++ b/backend/.sqlx/query-45c73bd8742601753ad947c88b65ff3ae182ff7c0631d4fa44b67b11a133498f.json @@ -0,0 +1,17 @@ +{ + "db_name": "PostgreSQL", + "query": "\n UPDATE pipeline_freshness_state\n SET attempts = attempts + 1,\n last_push_at = now(),\n next_attempt_at = now()\n + (LEAST($3::bigint, $4::bigint * (1::bigint << LEAST(attempts + 1, 20)))::text\n || ' seconds')::interval\n WHERE workspace_id = $1 AND script_path = $2 AND next_attempt_at <= now()\n ", + "describe": { + "columns": [], + "parameters": { + "Left": [ + "Text", + "Text", + "Int8", + "Int8" + ] + }, + "nullable": [] + }, + "hash": "45c73bd8742601753ad947c88b65ff3ae182ff7c0631d4fa44b67b11a133498f" +} diff --git a/backend/.sqlx/query-4a43d4df6c5b2e8dda4308dcb88c23caf312ec377dd91e5307f00d3fb8ec325d.json b/backend/.sqlx/query-4a43d4df6c5b2e8dda4308dcb88c23caf312ec377dd91e5307f00d3fb8ec325d.json index 7d950f6d8f..7474818bf3 100644 --- a/backend/.sqlx/query-4a43d4df6c5b2e8dda4308dcb88c23caf312ec377dd91e5307f00d3fb8ec325d.json +++ b/backend/.sqlx/query-4a43d4df6c5b2e8dda4308dcb88c23caf312ec377dd91e5307f00d3fb8ec325d.json @@ -80,7 +80,8 @@ "ci_test", "github", "azure", - "asset" + "asset", + "freshness" ] } } diff --git a/backend/.sqlx/query-5323858a017814206179d35fdb708c547742851975163550ec4f3823057e91fe.json b/backend/.sqlx/query-5323858a017814206179d35fdb708c547742851975163550ec4f3823057e91fe.json new file mode 100644 index 0000000000..a8facb9d1b --- /dev/null +++ b/backend/.sqlx/query-5323858a017814206179d35fdb708c547742851975163550ec4f3823057e91fe.json @@ -0,0 +1,20 @@ +{ + "db_name": "PostgreSQL", + "query": "INSERT INTO schedule (\n workspace_id, path, schedule, timezone, edited_by, script_path,\n is_flow, enabled, email, permissioned_as, summary, tag, cron_version\n ) VALUES ($1, $2, $3, 'Etc/UTC', $4, $2, false, true, $5, $6, $7, 'duckdb', 'v2')\n ON CONFLICT (workspace_id, path) DO UPDATE SET\n schedule = EXCLUDED.schedule,\n timezone = EXCLUDED.timezone,\n edited_by = EXCLUDED.edited_by,\n edited_at = now(),\n script_path = EXCLUDED.script_path,\n is_flow = false,\n enabled = true,\n email = EXCLUDED.email,\n permissioned_as = EXCLUDED.permissioned_as,\n summary = EXCLUDED.summary,\n tag = EXCLUDED.tag,\n cron_version = EXCLUDED.cron_version,\n error = NULL", + "describe": { + "columns": [], + "parameters": { + "Left": [ + "Varchar", + "Varchar", + "Varchar", + "Varchar", + "Varchar", + "Varchar", + "Varchar" + ] + }, + "nullable": [] + }, + "hash": "5323858a017814206179d35fdb708c547742851975163550ec4f3823057e91fe" +} diff --git a/backend/.sqlx/query-53a1ec83aedcec2cdf6495637464e6219434de4d2f9ae5e484ed8ce5da426109.json b/backend/.sqlx/query-53a1ec83aedcec2cdf6495637464e6219434de4d2f9ae5e484ed8ce5da426109.json new file mode 100644 index 0000000000..8bce508e75 --- /dev/null +++ b/backend/.sqlx/query-53a1ec83aedcec2cdf6495637464e6219434de4d2f9ae5e484ed8ce5da426109.json @@ -0,0 +1,16 @@ +{ + "db_name": "PostgreSQL", + "query": "DELETE FROM schedule WHERE workspace_id = $1 AND path = ANY($2) AND path != ALL($3)", + "describe": { + "columns": [], + "parameters": { + "Left": [ + "Text", + "TextArray", + "TextArray" + ] + }, + "nullable": [] + }, + "hash": "53a1ec83aedcec2cdf6495637464e6219434de4d2f9ae5e484ed8ce5da426109" +} diff --git a/backend/.sqlx/query-572b80f7b16fd5d1a05f8fe9bbb07667fc1b5e802dc2465ffd13178a042e4f89.json b/backend/.sqlx/query-572b80f7b16fd5d1a05f8fe9bbb07667fc1b5e802dc2465ffd13178a042e4f89.json new file mode 100644 index 0000000000..580666f07d --- /dev/null +++ b/backend/.sqlx/query-572b80f7b16fd5d1a05f8fe9bbb07667fc1b5e802dc2465ffd13178a042e4f89.json @@ -0,0 +1,24 @@ +{ + "db_name": "PostgreSQL", + "query": "\n SELECT EXISTS(\n SELECT 1 FROM v2_job j\n JOIN v2_job_completed c ON c.id = j.id\n WHERE j.workspace_id = $1\n AND j.runnable_path = $2\n AND j.parent_job IS NULL\n AND j.kind IN ('script', 'preview')\n AND c.status = 'success'\n AND c.completed_at > now() - ($3::bigint::text || ' seconds')::interval\n ) AS \"fresh!\"\n ", + "describe": { + "columns": [ + { + "ordinal": 0, + "name": "fresh!", + "type_info": "Bool" + } + ], + "parameters": { + "Left": [ + "Text", + "Text", + "Int8" + ] + }, + "nullable": [ + null + ] + }, + "hash": "572b80f7b16fd5d1a05f8fe9bbb07667fc1b5e802dc2465ffd13178a042e4f89" +} diff --git a/backend/.sqlx/query-5e0569092c559a1b19f636fae68f6d740a1720f7746c01f9ea693c70271ff55c.json b/backend/.sqlx/query-5e0569092c559a1b19f636fae68f6d740a1720f7746c01f9ea693c70271ff55c.json new file mode 100644 index 0000000000..18e0936d3a --- /dev/null +++ b/backend/.sqlx/query-5e0569092c559a1b19f636fae68f6d740a1720f7746c01f9ea693c70271ff55c.json @@ -0,0 +1,17 @@ +{ + "db_name": "PostgreSQL", + "query": "\n INSERT INTO pipeline_freshness_state\n (workspace_id, script_path, attempts, last_push_at, next_attempt_at)\n VALUES ($1, $2, 1, now(),\n now() + (LEAST($3::bigint, $4::bigint * 2)::text || ' seconds')::interval)\n ON CONFLICT (workspace_id, script_path) DO NOTHING\n ", + "describe": { + "columns": [], + "parameters": { + "Left": [ + "Varchar", + "Varchar", + "Int8", + "Int8" + ] + }, + "nullable": [] + }, + "hash": "5e0569092c559a1b19f636fae68f6d740a1720f7746c01f9ea693c70271ff55c" +} diff --git a/backend/.sqlx/query-63b0f39e3ea2b8d01d4c055258b6fc3dc4e9e1f237ba7c86cbdd70140bc55541.json b/backend/.sqlx/query-63b0f39e3ea2b8d01d4c055258b6fc3dc4e9e1f237ba7c86cbdd70140bc55541.json new file mode 100644 index 0000000000..1bcb7e6e53 --- /dev/null +++ b/backend/.sqlx/query-63b0f39e3ea2b8d01d4c055258b6fc3dc4e9e1f237ba7c86cbdd70140bc55541.json @@ -0,0 +1,36 @@ +{ + "db_name": "PostgreSQL", + "query": "\n WITH ancestor AS (\n SELECT wid, ord FROM unnest($1::text[]) WITH ORDINALITY AS a(wid, ord)\n ), anc_mat AS (\n -- Per table, the nearest ancestor (lowest ord) that materialized it.\n SELECT DISTINCT ON (mp.asset_path) mp.asset_path, a.wid, a.ord\n FROM materialized_partition mp\n JOIN ancestor a ON a.wid = mp.workspace_id\n WHERE mp.asset_kind = 'ducklake' AND mp.status = 'materialized'\n AND split_part(mp.asset_path, '/', 1) = $3 AND mp.asset_path LIKE '%/%'\n ORDER BY mp.asset_path, a.ord\n ), fork_mat AS (\n -- Fork-OWNED assets: anything whose physical table exists in the fork\n -- namespace, not just clean materializations. A committed write whose data\n -- tests failed afterwards records status='failed' WITH a snapshot — its table\n -- is real, and a defer view emitted over it would silently yield to it\n -- (CREATE VIEW IF NOT EXISTS) while claiming the read defers to the parent.\n SELECT DISTINCT asset_path FROM materialized_partition\n WHERE workspace_id = $2 AND asset_kind = 'ducklake'\n AND (status = 'materialized' OR snapshot_id IS NOT NULL)\n ), latest_schema AS (\n SELECT DISTINCT ON (workspace_id, asset_path) workspace_id, asset_path, columns\n FROM materialized_asset_schema\n WHERE workspace_id = ANY($1) AND asset_kind = 'ducklake'\n ORDER BY workspace_id, asset_path, version DESC\n )\n SELECT am.asset_path AS \"asset_path!\",\n am.ord AS \"ord!\",\n COALESCE(EXISTS (\n SELECT 1 FROM jsonb_array_elements(ls.columns) e\n WHERE e->>'name' = 'is_current'\n ), false) AS \"has_current!\"\n FROM anc_mat am\n LEFT JOIN latest_schema ls\n ON ls.asset_path = am.asset_path AND ls.workspace_id = am.wid\n WHERE am.asset_path NOT IN (SELECT asset_path FROM fork_mat)\n ORDER BY am.asset_path\n ", + "describe": { + "columns": [ + { + "ordinal": 0, + "name": "asset_path!", + "type_info": "Varchar" + }, + { + "ordinal": 1, + "name": "ord!", + "type_info": "Int8" + }, + { + "ordinal": 2, + "name": "has_current!", + "type_info": "Bool" + } + ], + "parameters": { + "Left": [ + "TextArray", + "Text", + "Text" + ] + }, + "nullable": [ + false, + null, + null + ] + }, + "hash": "63b0f39e3ea2b8d01d4c055258b6fc3dc4e9e1f237ba7c86cbdd70140bc55541" +} diff --git a/backend/.sqlx/query-67e25a7c19ea0ffaf7ea5303fcd04af5a7eb488c76f783e690af0c2153b1d6a8.json b/backend/.sqlx/query-67e25a7c19ea0ffaf7ea5303fcd04af5a7eb488c76f783e690af0c2153b1d6a8.json index 08ebe6bba5..fcc16e9a7c 100644 --- a/backend/.sqlx/query-67e25a7c19ea0ffaf7ea5303fcd04af5a7eb488c76f783e690af0c2153b1d6a8.json +++ b/backend/.sqlx/query-67e25a7c19ea0ffaf7ea5303fcd04af5a7eb488c76f783e690af0c2153b1d6a8.json @@ -161,7 +161,8 @@ "ci_test", "github", "azure", - "asset" + "asset", + "freshness" ] } } diff --git a/backend/.sqlx/query-6ca5cbe50df89e6eead890758e34a6ae5091c37ec901975c3d564fd817a405b7.json b/backend/.sqlx/query-6ca5cbe50df89e6eead890758e34a6ae5091c37ec901975c3d564fd817a405b7.json new file mode 100644 index 0000000000..9a0bbf46ef --- /dev/null +++ b/backend/.sqlx/query-6ca5cbe50df89e6eead890758e34a6ae5091c37ec901975c3d564fd817a405b7.json @@ -0,0 +1,23 @@ +{ + "db_name": "PostgreSQL", + "query": "UPDATE workspace_settings\n SET ducklake = jsonb_set(ducklake, ARRAY['ducklakes', $2, 'fork_behavior'], '\"shared\"')\n WHERE workspace_id = $1 AND ducklake->'ducklakes' ? $2\n RETURNING 1 AS \"one!\"", + "describe": { + "columns": [ + { + "ordinal": 0, + "name": "one!", + "type_info": "Int4" + } + ], + "parameters": { + "Left": [ + "Text", + "Text" + ] + }, + "nullable": [ + null + ] + }, + "hash": "6ca5cbe50df89e6eead890758e34a6ae5091c37ec901975c3d564fd817a405b7" +} diff --git a/backend/.sqlx/query-6d81570849980f316b08487a03f6017386cf2767c953e0d152e7983a00218c6f.json b/backend/.sqlx/query-6d81570849980f316b08487a03f6017386cf2767c953e0d152e7983a00218c6f.json new file mode 100644 index 0000000000..4e8c7acee2 --- /dev/null +++ b/backend/.sqlx/query-6d81570849980f316b08487a03f6017386cf2767c953e0d152e7983a00218c6f.json @@ -0,0 +1,15 @@ +{ + "db_name": "PostgreSQL", + "query": "UPDATE pipeline_freshness_state SET next_attempt_at = now() - interval '1 second'\n WHERE workspace_id = $1 AND script_path = $2", + "describe": { + "columns": [], + "parameters": { + "Left": [ + "Text", + "Text" + ] + }, + "nullable": [] + }, + "hash": "6d81570849980f316b08487a03f6017386cf2767c953e0d152e7983a00218c6f" +} diff --git a/backend/.sqlx/query-76af9b978fccef61f50a5eab335d4e427342fb78c66980984aefa7b8c7592ecb.json b/backend/.sqlx/query-76af9b978fccef61f50a5eab335d4e427342fb78c66980984aefa7b8c7592ecb.json new file mode 100644 index 0000000000..b6b0f06be5 --- /dev/null +++ b/backend/.sqlx/query-76af9b978fccef61f50a5eab335d4e427342fb78c66980984aefa7b8c7592ecb.json @@ -0,0 +1,22 @@ +{ + "db_name": "PostgreSQL", + "query": "SELECT ducklake->'ducklakes' FROM workspace_settings WHERE workspace_id = $1", + "describe": { + "columns": [ + { + "ordinal": 0, + "name": "?column?", + "type_info": "Jsonb" + } + ], + "parameters": { + "Left": [ + "Text" + ] + }, + "nullable": [ + null + ] + }, + "hash": "76af9b978fccef61f50a5eab335d4e427342fb78c66980984aefa7b8c7592ecb" +} diff --git a/backend/.sqlx/query-7cd0b27f4b4a20b05ac876861b934128c3a6ade024d6662379857905a7447a2a.json b/backend/.sqlx/query-7cd0b27f4b4a20b05ac876861b934128c3a6ade024d6662379857905a7447a2a.json new file mode 100644 index 0000000000..5c8ab6fb5b --- /dev/null +++ b/backend/.sqlx/query-7cd0b27f4b4a20b05ac876861b934128c3a6ade024d6662379857905a7447a2a.json @@ -0,0 +1,32 @@ +{ + "db_name": "PostgreSQL", + "query": "\n SELECT DISTINCT ON (workspace_id, path)\n workspace_id AS \"workspace_id!\", path AS \"path!\", content AS \"content!\"\n FROM script\n WHERE auto_kind = 'pipeline'\n AND archived = false\n AND deleted = false\n AND content ILIKE '%freshness%'\n -- Workspace archival stops all execution but leaves script rows\n -- intact for unarchival; without this the watchdog would keep\n -- resurrecting runs in a workspace the admin shut down.\n AND EXISTS (SELECT 1 FROM workspace w\n WHERE w.id = script.workspace_id AND w.deleted = false)\n ORDER BY workspace_id, path, created_at DESC\n ", + "describe": { + "columns": [ + { + "ordinal": 0, + "name": "workspace_id!", + "type_info": "Varchar" + }, + { + "ordinal": 1, + "name": "path!", + "type_info": "Varchar" + }, + { + "ordinal": 2, + "name": "content!", + "type_info": "Text" + } + ], + "parameters": { + "Left": [] + }, + "nullable": [ + false, + false, + false + ] + }, + "hash": "7cd0b27f4b4a20b05ac876861b934128c3a6ade024d6662379857905a7447a2a" +} diff --git a/backend/.sqlx/query-81d1c56c369c5328d947f1ad21da995a8a555e4a9c2d1387dd8cee9c81efd40b.json b/backend/.sqlx/query-81d1c56c369c5328d947f1ad21da995a8a555e4a9c2d1387dd8cee9c81efd40b.json new file mode 100644 index 0000000000..3f6a4a8136 --- /dev/null +++ b/backend/.sqlx/query-81d1c56c369c5328d947f1ad21da995a8a555e4a9c2d1387dd8cee9c81efd40b.json @@ -0,0 +1,15 @@ +{ + "db_name": "PostgreSQL", + "query": "\n DELETE FROM pipeline_freshness_state s\n WHERE NOT EXISTS (\n SELECT 1 FROM unnest($1::text[], $2::text[]) AS w(workspace_id, script_path)\n WHERE w.workspace_id = s.workspace_id AND w.script_path = s.script_path\n )\n ", + "describe": { + "columns": [], + "parameters": { + "Left": [ + "TextArray", + "TextArray" + ] + }, + "nullable": [] + }, + "hash": "81d1c56c369c5328d947f1ad21da995a8a555e4a9c2d1387dd8cee9c81efd40b" +} diff --git a/backend/.sqlx/query-5e50ba0ae27b09a3ea1530c223e5039aa631bb5b0993ad09bc9a1381f6715f19.json b/backend/.sqlx/query-853230da671371fd8bbeeb4c87f8c20a9e310092ae787efe4b0657399c799424.json similarity index 63% rename from backend/.sqlx/query-5e50ba0ae27b09a3ea1530c223e5039aa631bb5b0993ad09bc9a1381f6715f19.json rename to backend/.sqlx/query-853230da671371fd8bbeeb4c87f8c20a9e310092ae787efe4b0657399c799424.json index 17d372249b..5aacf1a295 100644 --- a/backend/.sqlx/query-5e50ba0ae27b09a3ea1530c223e5039aa631bb5b0993ad09bc9a1381f6715f19.json +++ b/backend/.sqlx/query-853230da671371fd8bbeeb4c87f8c20a9e310092ae787efe4b0657399c799424.json @@ -1,6 +1,6 @@ { "db_name": "PostgreSQL", - "query": "INSERT INTO materialized_partition\n (workspace_id, asset_kind, asset_path, partition, status,\n snapshot_id, row_count, job_id, materialized_at, error)\n VALUES ($1, $2, $3, $4, $5, $6, $7, $8, now(), $9)\n ON CONFLICT (workspace_id, asset_kind, asset_path, partition)\n DO UPDATE SET status = EXCLUDED.status,\n snapshot_id = EXCLUDED.snapshot_id,\n row_count = EXCLUDED.row_count,\n job_id = EXCLUDED.job_id,\n materialized_at = now(),\n error = EXCLUDED.error", + "query": "INSERT INTO materialized_partition\n (workspace_id, asset_kind, asset_path, partition, status,\n snapshot_id, row_count, job_id, materialized_at, error)\n VALUES ($1, $2, $3, $4, $5, $6, $7, $8, now(), $9)\n ON CONFLICT (workspace_id, asset_kind, asset_path, partition)\n DO UPDATE SET status = EXCLUDED.status,\n -- A failed run records no snapshot, but must not erase the last\n -- committed one: a physical table from an earlier commit (or from a\n -- committed write whose data tests then failed) still exists, and\n -- fork defer/graph state keys on that evidence.\n snapshot_id = COALESCE(EXCLUDED.snapshot_id, materialized_partition.snapshot_id),\n row_count = EXCLUDED.row_count,\n job_id = EXCLUDED.job_id,\n materialized_at = now(),\n error = EXCLUDED.error", "describe": { "columns": [], "parameters": { @@ -43,5 +43,5 @@ }, "nullable": [] }, - "hash": "5e50ba0ae27b09a3ea1530c223e5039aa631bb5b0993ad09bc9a1381f6715f19" + "hash": "853230da671371fd8bbeeb4c87f8c20a9e310092ae787efe4b0657399c799424" } diff --git a/backend/.sqlx/query-8756a8100d6fd6ecb64ef73d6f2c8453d53c973dce61cc8ace7d278773c7506f.json b/backend/.sqlx/query-8756a8100d6fd6ecb64ef73d6f2c8453d53c973dce61cc8ace7d278773c7506f.json new file mode 100644 index 0000000000..5741a1aa5b --- /dev/null +++ b/backend/.sqlx/query-8756a8100d6fd6ecb64ef73d6f2c8453d53c973dce61cc8ace7d278773c7506f.json @@ -0,0 +1,31 @@ +{ + "db_name": "PostgreSQL", + "query": "SELECT\n schedule.path, t.jobs FROM schedule,\n LATERAL(SELECT ARRAY(\n SELECT json_build_object('id', id, 'success', status = 'success', 'duration_ms', duration_ms)\n FROM v2_job_completed c JOIN v2_job j USING (id)\n WHERE trigger_kind = 'schedule'\n AND trigger = schedule.path\n AND c.workspace_id = $1\n AND j.workspace_id = $1\n AND parent_job IS NULL AND runnable_path = schedule.script_path\n AND status <> 'skipped'\n ORDER BY completed_at DESC\n LIMIT 20\n ) AS jobs) t\n WHERE workspace_id = $1 AND schedule.path NOT LIKE $4\n ORDER BY edited_at DESC\n LIMIT $2 OFFSET $3", + "describe": { + "columns": [ + { + "ordinal": 0, + "name": "path", + "type_info": "Varchar" + }, + { + "ordinal": 1, + "name": "jobs", + "type_info": "JsonArray" + } + ], + "parameters": { + "Left": [ + "Text", + "Int8", + "Int8", + "Text" + ] + }, + "nullable": [ + false, + null + ] + }, + "hash": "8756a8100d6fd6ecb64ef73d6f2c8453d53c973dce61cc8ace7d278773c7506f" +} diff --git a/backend/.sqlx/query-8837a126698d924eb9014cfee0e6afc85a8350706fe694935b25ef46a847409e.json b/backend/.sqlx/query-8837a126698d924eb9014cfee0e6afc85a8350706fe694935b25ef46a847409e.json new file mode 100644 index 0000000000..829083e0ec --- /dev/null +++ b/backend/.sqlx/query-8837a126698d924eb9014cfee0e6afc85a8350706fe694935b25ef46a847409e.json @@ -0,0 +1,22 @@ +{ + "db_name": "PostgreSQL", + "query": "\n SELECT ws.ducklake->'ducklakes' AS ducklake_name\n FROM workspace_settings ws\n WHERE ws.workspace_id = $1\n ", + "describe": { + "columns": [ + { + "ordinal": 0, + "name": "ducklake_name", + "type_info": "Jsonb" + } + ], + "parameters": { + "Left": [ + "Text" + ] + }, + "nullable": [ + null + ] + }, + "hash": "8837a126698d924eb9014cfee0e6afc85a8350706fe694935b25ef46a847409e" +} diff --git a/backend/.sqlx/query-8a1e2e59c05b0f67e32c517a0c46e1a9307335b0e7a27613cd690e99812a99f3.json b/backend/.sqlx/query-8a1e2e59c05b0f67e32c517a0c46e1a9307335b0e7a27613cd690e99812a99f3.json new file mode 100644 index 0000000000..9fc8cf1336 --- /dev/null +++ b/backend/.sqlx/query-8a1e2e59c05b0f67e32c517a0c46e1a9307335b0e7a27613cd690e99812a99f3.json @@ -0,0 +1,23 @@ +{ + "db_name": "PostgreSQL", + "query": "SELECT path FROM schedule WHERE workspace_id = $1 AND starts_with(path, $2)", + "describe": { + "columns": [ + { + "ordinal": 0, + "name": "path", + "type_info": "Varchar" + } + ], + "parameters": { + "Left": [ + "Text", + "Text" + ] + }, + "nullable": [ + false + ] + }, + "hash": "8a1e2e59c05b0f67e32c517a0c46e1a9307335b0e7a27613cd690e99812a99f3" +} diff --git a/backend/.sqlx/query-8dec884a4c3b2d105afe22d1d7a01fcab837629fd3c1a09cf82b21641a890444.json b/backend/.sqlx/query-8dec884a4c3b2d105afe22d1d7a01fcab837629fd3c1a09cf82b21641a890444.json new file mode 100644 index 0000000000..efd12114f3 --- /dev/null +++ b/backend/.sqlx/query-8dec884a4c3b2d105afe22d1d7a01fcab837629fd3c1a09cf82b21641a890444.json @@ -0,0 +1,29 @@ +{ + "db_name": "PostgreSQL", + "query": "\n -- Fork rows also count when a snapshot ever committed (a failed run\n -- preserves it): the physical table exists, so reads hit the FORK's\n -- data — showing 'deferred' would misstate what a query returns.\n -- Ancestor rows still require a clean materialization.\n SELECT DISTINCT asset_path AS \"asset_path!\", workspace_id AS \"workspace_id!\"\n FROM materialized_partition\n WHERE (workspace_id = $1 OR workspace_id = ANY($2))\n AND asset_kind = 'ducklake'\n AND (status = 'materialized'\n OR (workspace_id = $1 AND snapshot_id IS NOT NULL))\n ", + "describe": { + "columns": [ + { + "ordinal": 0, + "name": "asset_path!", + "type_info": "Varchar" + }, + { + "ordinal": 1, + "name": "workspace_id!", + "type_info": "Varchar" + } + ], + "parameters": { + "Left": [ + "Text", + "TextArray" + ] + }, + "nullable": [ + false, + false + ] + }, + "hash": "8dec884a4c3b2d105afe22d1d7a01fcab837629fd3c1a09cf82b21641a890444" +} diff --git a/backend/.sqlx/query-9ecb404e46a4eac55f977f05a3afbafe5dc3cdecc17a3d5a7476b160c1b6e7e1.json b/backend/.sqlx/query-9ecb404e46a4eac55f977f05a3afbafe5dc3cdecc17a3d5a7476b160c1b6e7e1.json index ad9e57801e..ab01730c02 100644 --- a/backend/.sqlx/query-9ecb404e46a4eac55f977f05a3afbafe5dc3cdecc17a3d5a7476b160c1b6e7e1.json +++ b/backend/.sqlx/query-9ecb404e46a4eac55f977f05a3afbafe5dc3cdecc17a3d5a7476b160c1b6e7e1.json @@ -35,7 +35,8 @@ "ci_test", "github", "azure", - "asset" + "asset", + "freshness" ] } } diff --git a/backend/.sqlx/query-a0faa5b042bac00bcd09b925bcf548e2783c2de7e92ba0ff7e6f4637419fcf7d.json b/backend/.sqlx/query-a0faa5b042bac00bcd09b925bcf548e2783c2de7e92ba0ff7e6f4637419fcf7d.json new file mode 100644 index 0000000000..e3e8e3f2ad --- /dev/null +++ b/backend/.sqlx/query-a0faa5b042bac00bcd09b925bcf548e2783c2de7e92ba0ff7e6f4637419fcf7d.json @@ -0,0 +1,16 @@ +{ + "db_name": "PostgreSQL", + "query": "DELETE FROM schedule WHERE workspace_id = $1 AND path LIKE $2 AND path != ALL($3)", + "describe": { + "columns": [], + "parameters": { + "Left": [ + "Text", + "Text", + "TextArray" + ] + }, + "nullable": [] + }, + "hash": "a0faa5b042bac00bcd09b925bcf548e2783c2de7e92ba0ff7e6f4637419fcf7d" +} diff --git a/backend/.sqlx/query-a4b6371d33206010b2f3ffd2b09e33244fe8ab9a803248fc23f334034d24aad4.json b/backend/.sqlx/query-a4b6371d33206010b2f3ffd2b09e33244fe8ab9a803248fc23f334034d24aad4.json index 9a21f228ea..405904604a 100644 --- a/backend/.sqlx/query-a4b6371d33206010b2f3ffd2b09e33244fe8ab9a803248fc23f334034d24aad4.json +++ b/backend/.sqlx/query-a4b6371d33206010b2f3ffd2b09e33244fe8ab9a803248fc23f334034d24aad4.json @@ -191,7 +191,8 @@ "ci_test", "github", "azure", - "asset" + "asset", + "freshness" ] } } diff --git a/backend/.sqlx/query-a7f0ed0081e98a9861ea4b081b8a14576509147e51a86a95a2cc786bee3199d9.json b/backend/.sqlx/query-a7f0ed0081e98a9861ea4b081b8a14576509147e51a86a95a2cc786bee3199d9.json new file mode 100644 index 0000000000..21e0f4613b --- /dev/null +++ b/backend/.sqlx/query-a7f0ed0081e98a9861ea4b081b8a14576509147e51a86a95a2cc786bee3199d9.json @@ -0,0 +1,16 @@ +{ + "db_name": "PostgreSQL", + "query": "DELETE FROM schedule WHERE workspace_id = $1 AND starts_with(path, $2) AND path != ALL($3)", + "describe": { + "columns": [], + "parameters": { + "Left": [ + "Text", + "Text", + "TextArray" + ] + }, + "nullable": [] + }, + "hash": "a7f0ed0081e98a9861ea4b081b8a14576509147e51a86a95a2cc786bee3199d9" +} diff --git a/backend/.sqlx/query-b10e21ec9edf2d22ff956fdcd085bd601b7dcabdf70e85543455cb4fc3986d74.json b/backend/.sqlx/query-b10e21ec9edf2d22ff956fdcd085bd601b7dcabdf70e85543455cb4fc3986d74.json new file mode 100644 index 0000000000..d5b10b13b9 --- /dev/null +++ b/backend/.sqlx/query-b10e21ec9edf2d22ff956fdcd085bd601b7dcabdf70e85543455cb4fc3986d74.json @@ -0,0 +1,23 @@ +{ + "db_name": "PostgreSQL", + "query": "SELECT attempts FROM pipeline_freshness_state WHERE workspace_id = $1 AND script_path = $2", + "describe": { + "columns": [ + { + "ordinal": 0, + "name": "attempts", + "type_info": "Int4" + } + ], + "parameters": { + "Left": [ + "Text", + "Text" + ] + }, + "nullable": [ + false + ] + }, + "hash": "b10e21ec9edf2d22ff956fdcd085bd601b7dcabdf70e85543455cb4fc3986d74" +} diff --git a/backend/.sqlx/query-b3771b690c5966272b1f42c9965bb6a8f961c119516e4c33dc928cd3b4f4edbc.json b/backend/.sqlx/query-b3771b690c5966272b1f42c9965bb6a8f961c119516e4c33dc928cd3b4f4edbc.json index 3efa843923..092d15e592 100644 --- a/backend/.sqlx/query-b3771b690c5966272b1f42c9965bb6a8f961c119516e4c33dc928cd3b4f4edbc.json +++ b/backend/.sqlx/query-b3771b690c5966272b1f42c9965bb6a8f961c119516e4c33dc928cd3b4f4edbc.json @@ -166,7 +166,8 @@ "ci_test", "github", "azure", - "asset" + "asset", + "freshness" ] } } diff --git a/backend/.sqlx/query-bc6f566bc958923045dc3e1ff1639a26f649c960a4ee8d0a584884a236129d43.json b/backend/.sqlx/query-bc6f566bc958923045dc3e1ff1639a26f649c960a4ee8d0a584884a236129d43.json new file mode 100644 index 0000000000..ce77275faf --- /dev/null +++ b/backend/.sqlx/query-bc6f566bc958923045dc3e1ff1639a26f649c960a4ee8d0a584884a236129d43.json @@ -0,0 +1,15 @@ +{ + "db_name": "PostgreSQL", + "query": "INSERT INTO pipeline_freshness_state (workspace_id, script_path, attempts) VALUES ($1, $2, 3)", + "describe": { + "columns": [], + "parameters": { + "Left": [ + "Varchar", + "Varchar" + ] + }, + "nullable": [] + }, + "hash": "bc6f566bc958923045dc3e1ff1639a26f649c960a4ee8d0a584884a236129d43" +} diff --git a/backend/.sqlx/query-bcfa34cf80abea05f0c24883b9e77429c51e6166c414bcc5ce2e97fac25bcd77.json b/backend/.sqlx/query-bcfa34cf80abea05f0c24883b9e77429c51e6166c414bcc5ce2e97fac25bcd77.json index f6ff25a4bf..419ab26383 100644 --- a/backend/.sqlx/query-bcfa34cf80abea05f0c24883b9e77429c51e6166c414bcc5ce2e97fac25bcd77.json +++ b/backend/.sqlx/query-bcfa34cf80abea05f0c24883b9e77429c51e6166c414bcc5ce2e97fac25bcd77.json @@ -80,7 +80,8 @@ "ci_test", "github", "azure", - "asset" + "asset", + "freshness" ] } } diff --git a/backend/.sqlx/query-be6d2c92a62b7b284651c45af809746147aa9b8d0a81642a7b7cb4738a0cad66.json b/backend/.sqlx/query-be6d2c92a62b7b284651c45af809746147aa9b8d0a81642a7b7cb4738a0cad66.json index 8dc66064dc..b9d33a6b5f 100644 --- a/backend/.sqlx/query-be6d2c92a62b7b284651c45af809746147aa9b8d0a81642a7b7cb4738a0cad66.json +++ b/backend/.sqlx/query-be6d2c92a62b7b284651c45af809746147aa9b8d0a81642a7b7cb4738a0cad66.json @@ -111,7 +111,8 @@ "ci_test", "github", "azure", - "asset" + "asset", + "freshness" ] } } diff --git a/backend/.sqlx/query-c6c186eb3a108e1f695bf5730c23ef4ab1ccfdc4ed33904e83e5130ac0bf5896.json b/backend/.sqlx/query-c6c186eb3a108e1f695bf5730c23ef4ab1ccfdc4ed33904e83e5130ac0bf5896.json new file mode 100644 index 0000000000..7cfc1f109f --- /dev/null +++ b/backend/.sqlx/query-c6c186eb3a108e1f695bf5730c23ef4ab1ccfdc4ed33904e83e5130ac0bf5896.json @@ -0,0 +1,41 @@ +{ + "db_name": "PostgreSQL", + "query": "SELECT DISTINCT ON (asset_path)\n asset_path, version, columns AS \"columns: Json>\", captured_at\n FROM materialized_asset_schema\n WHERE workspace_id = $1 AND asset_kind = 'ducklake' AND asset_path = ANY($2)\n ORDER BY asset_path, version DESC", + "describe": { + "columns": [ + { + "ordinal": 0, + "name": "asset_path", + "type_info": "Varchar" + }, + { + "ordinal": 1, + "name": "version", + "type_info": "Int8" + }, + { + "ordinal": 2, + "name": "columns: Json>", + "type_info": "Jsonb" + }, + { + "ordinal": 3, + "name": "captured_at", + "type_info": "Timestamptz" + } + ], + "parameters": { + "Left": [ + "Text", + "TextArray" + ] + }, + "nullable": [ + false, + false, + false, + false + ] + }, + "hash": "c6c186eb3a108e1f695bf5730c23ef4ab1ccfdc4ed33904e83e5130ac0bf5896" +} diff --git a/backend/.sqlx/query-c832ebc7ff0be446bf36628f6e06f59d0fbe8df62e0bd0be636d7cf70dcf95f0.json b/backend/.sqlx/query-c832ebc7ff0be446bf36628f6e06f59d0fbe8df62e0bd0be636d7cf70dcf95f0.json new file mode 100644 index 0000000000..d316c16fff --- /dev/null +++ b/backend/.sqlx/query-c832ebc7ff0be446bf36628f6e06f59d0fbe8df62e0bd0be636d7cf70dcf95f0.json @@ -0,0 +1,16 @@ +{ + "db_name": "PostgreSQL", + "query": "INSERT INTO schedule (\n workspace_id, path, edited_by, edited_at, schedule, enabled, script_path,\n args, extra_perms, is_flow, email, error, timezone, on_failure,\n on_recovery, on_failure_times, on_failure_exact, on_failure_extra_args,\n on_recovery_times, on_recovery_extra_args, ws_error_handler_muted, retry,\n summary, no_flow_overlap, tag, paused_until, on_success, on_success_extra_args,\n cron_version, description, dynamic_skip, permissioned_as, labels\n )\n SELECT\n $1, path, edited_by, edited_at, schedule, FALSE, script_path,\n args, extra_perms, is_flow, email, error, timezone, on_failure,\n on_recovery, on_failure_times, on_failure_exact, on_failure_extra_args,\n on_recovery_times, on_recovery_extra_args, ws_error_handler_muted, retry,\n summary, no_flow_overlap, tag, paused_until, on_success, on_success_extra_args,\n cron_version, description, dynamic_skip, permissioned_as, labels\n FROM schedule WHERE workspace_id = $2 AND NOT starts_with(path, $3)", + "describe": { + "columns": [], + "parameters": { + "Left": [ + "Varchar", + "Text", + "Text" + ] + }, + "nullable": [] + }, + "hash": "c832ebc7ff0be446bf36628f6e06f59d0fbe8df62e0bd0be636d7cf70dcf95f0" +} diff --git a/backend/.sqlx/query-cbbc6a894b6421d2cd49e59b7ed650fb861aa19a009d6dfe9937304057f2b91f.json b/backend/.sqlx/query-cbbc6a894b6421d2cd49e59b7ed650fb861aa19a009d6dfe9937304057f2b91f.json new file mode 100644 index 0000000000..0fac858572 --- /dev/null +++ b/backend/.sqlx/query-cbbc6a894b6421d2cd49e59b7ed650fb861aa19a009d6dfe9937304057f2b91f.json @@ -0,0 +1,22 @@ +{ + "db_name": "PostgreSQL", + "query": "\n WITH RECURSIVE chain AS (\n SELECT id, parent_workspace_id, 0 AS depth\n FROM workspace WHERE id = $1\n UNION ALL\n SELECT w.id, w.parent_workspace_id, chain.depth + 1\n FROM workspace w\n JOIN chain ON w.id = chain.parent_workspace_id\n WHERE chain.depth < 20\n )\n SELECT id AS \"id!\" FROM chain WHERE depth > 0 ORDER BY depth\n ", + "describe": { + "columns": [ + { + "ordinal": 0, + "name": "id!", + "type_info": "Varchar" + } + ], + "parameters": { + "Left": [ + "Text" + ] + }, + "nullable": [ + null + ] + }, + "hash": "cbbc6a894b6421d2cd49e59b7ed650fb861aa19a009d6dfe9937304057f2b91f" +} diff --git a/backend/.sqlx/query-cd492743a27a158c9f5ff1fdb50a48ebfed85ad8800e293945eadc874d7e42f3.json b/backend/.sqlx/query-cd492743a27a158c9f5ff1fdb50a48ebfed85ad8800e293945eadc874d7e42f3.json new file mode 100644 index 0000000000..83f7394250 --- /dev/null +++ b/backend/.sqlx/query-cd492743a27a158c9f5ff1fdb50a48ebfed85ad8800e293945eadc874d7e42f3.json @@ -0,0 +1,19 @@ +{ + "db_name": "PostgreSQL", + "query": "UPDATE fork_ducklake_namespace SET schema_dropped = true\n WHERE workspace_id = $1 AND ducklake_name = $2 AND catalog = $3\n AND storage = $4 AND storage_ref = $5 AND data_path = $6", + "describe": { + "columns": [], + "parameters": { + "Left": [ + "Text", + "Text", + "Text", + "Text", + "Text", + "Text" + ] + }, + "nullable": [] + }, + "hash": "cd492743a27a158c9f5ff1fdb50a48ebfed85ad8800e293945eadc874d7e42f3" +} diff --git a/backend/.sqlx/query-d1142126245cdaf5d01dc67ab2488958b335f6a24d1e93641eab6aaac86301f7.json b/backend/.sqlx/query-d1142126245cdaf5d01dc67ab2488958b335f6a24d1e93641eab6aaac86301f7.json new file mode 100644 index 0000000000..f705dbc337 --- /dev/null +++ b/backend/.sqlx/query-d1142126245cdaf5d01dc67ab2488958b335f6a24d1e93641eab6aaac86301f7.json @@ -0,0 +1,14 @@ +{ + "db_name": "PostgreSQL", + "query": "UPDATE workspace_settings\n SET ducklake = jsonb_set(ducklake, '{ducklakes}', (\n SELECT COALESCE(jsonb_object_agg(key, value - 'fork_behavior'), '{}'::jsonb)\n FROM jsonb_each(ducklake->'ducklakes')\n ))\n WHERE workspace_id = $1 AND jsonb_typeof(ducklake->'ducklakes') = 'object'", + "describe": { + "columns": [], + "parameters": { + "Left": [ + "Text" + ] + }, + "nullable": [] + }, + "hash": "d1142126245cdaf5d01dc67ab2488958b335f6a24d1e93641eab6aaac86301f7" +} diff --git a/backend/.sqlx/query-d1a1f64cb6b14c561c53c44910cad8bc7a2d226a9f22418782bcc66e46d0f706.json b/backend/.sqlx/query-d1a1f64cb6b14c561c53c44910cad8bc7a2d226a9f22418782bcc66e46d0f706.json new file mode 100644 index 0000000000..cbf96dad52 --- /dev/null +++ b/backend/.sqlx/query-d1a1f64cb6b14c561c53c44910cad8bc7a2d226a9f22418782bcc66e46d0f706.json @@ -0,0 +1,15 @@ +{ + "db_name": "PostgreSQL", + "query": "INSERT INTO pipeline_freshness_state (workspace_id, script_path) VALUES ($1, $2)", + "describe": { + "columns": [], + "parameters": { + "Left": [ + "Varchar", + "Varchar" + ] + }, + "nullable": [] + }, + "hash": "d1a1f64cb6b14c561c53c44910cad8bc7a2d226a9f22418782bcc66e46d0f706" +} diff --git a/backend/.sqlx/query-d41ea93fd58381b89e151c965eae1ea2fe96a1b94f5a92953fb1c1642d15c016.json b/backend/.sqlx/query-d41ea93fd58381b89e151c965eae1ea2fe96a1b94f5a92953fb1c1642d15c016.json index d5365ffe94..d97c02d26b 100644 --- a/backend/.sqlx/query-d41ea93fd58381b89e151c965eae1ea2fe96a1b94f5a92953fb1c1642d15c016.json +++ b/backend/.sqlx/query-d41ea93fd58381b89e151c965eae1ea2fe96a1b94f5a92953fb1c1642d15c016.json @@ -111,7 +111,8 @@ "ci_test", "github", "azure", - "asset" + "asset", + "freshness" ] } } diff --git a/backend/.sqlx/query-d4211392e174a0e8f89c7fcebdf120e5b0f629f9f04e08a2982df33ff23ac7a9.json b/backend/.sqlx/query-d4211392e174a0e8f89c7fcebdf120e5b0f629f9f04e08a2982df33ff23ac7a9.json index 33a5534b42..052d83fcd9 100644 --- a/backend/.sqlx/query-d4211392e174a0e8f89c7fcebdf120e5b0f629f9f04e08a2982df33ff23ac7a9.json +++ b/backend/.sqlx/query-d4211392e174a0e8f89c7fcebdf120e5b0f629f9f04e08a2982df33ff23ac7a9.json @@ -251,7 +251,8 @@ "ci_test", "github", "azure", - "asset" + "asset", + "freshness" ] } } diff --git a/backend/.sqlx/query-caf4dc1046769f410d1277cdd3747763e0d674e8a1d9f3512629cb5a9e5a3439.json b/backend/.sqlx/query-dc60d43814e2b5220db648677dbb3656e27f1c962c2f0438e0cc06530661fbf8.json similarity index 53% rename from backend/.sqlx/query-caf4dc1046769f410d1277cdd3747763e0d674e8a1d9f3512629cb5a9e5a3439.json rename to backend/.sqlx/query-dc60d43814e2b5220db648677dbb3656e27f1c962c2f0438e0cc06530661fbf8.json index f2398bf143..0d36677337 100644 --- a/backend/.sqlx/query-caf4dc1046769f410d1277cdd3747763e0d674e8a1d9f3512629cb5a9e5a3439.json +++ b/backend/.sqlx/query-dc60d43814e2b5220db648677dbb3656e27f1c962c2f0438e0cc06530661fbf8.json @@ -1,17 +1,16 @@ { "db_name": "PostgreSQL", - "query": "\n SELECT ws.datatable->'datatables'->$2 AS config\n FROM workspace_settings ws\n WHERE ws.workspace_id = $1\n ", + "query": "\n SELECT ws.datatable->'datatables' AS datatables\n FROM workspace_settings ws\n WHERE ws.workspace_id = $1\n ", "describe": { "columns": [ { "ordinal": 0, - "name": "config", + "name": "datatables", "type_info": "Jsonb" } ], "parameters": { "Left": [ - "Text", "Text" ] }, @@ -19,5 +18,5 @@ null ] }, - "hash": "caf4dc1046769f410d1277cdd3747763e0d674e8a1d9f3512629cb5a9e5a3439" + "hash": "dc60d43814e2b5220db648677dbb3656e27f1c962c2f0438e0cc06530661fbf8" } diff --git a/backend/.sqlx/query-e27cc16414c181a055bf0666eb0cb16a356cb6be5c48f95b5ba200e7e6ab0efd.json b/backend/.sqlx/query-e27cc16414c181a055bf0666eb0cb16a356cb6be5c48f95b5ba200e7e6ab0efd.json new file mode 100644 index 0000000000..37bd25eb98 --- /dev/null +++ b/backend/.sqlx/query-e27cc16414c181a055bf0666eb0cb16a356cb6be5c48f95b5ba200e7e6ab0efd.json @@ -0,0 +1,20 @@ +{ + "db_name": "PostgreSQL", + "query": "INSERT INTO fork_ducklake_namespace\n (workspace_id, ducklake_name, metadata_schema, catalog, storage, storage_ref, data_path)\n VALUES ($1, $2, $3, $4, $5, $6, $7)\n ON CONFLICT (workspace_id, ducklake_name, catalog, storage, storage_ref, data_path)\n DO UPDATE SET schema_dropped = false", + "describe": { + "columns": [], + "parameters": { + "Left": [ + "Varchar", + "Varchar", + "Varchar", + "Text", + "Text", + "Text", + "Text" + ] + }, + "nullable": [] + }, + "hash": "e27cc16414c181a055bf0666eb0cb16a356cb6be5c48f95b5ba200e7e6ab0efd" +} diff --git a/backend/.sqlx/query-e4d71278fb80126a7a9da73f1889352d4d1e3cb3a8a08f1c9c03055a1cab1235.json b/backend/.sqlx/query-e4d71278fb80126a7a9da73f1889352d4d1e3cb3a8a08f1c9c03055a1cab1235.json index 470c651020..a35373a959 100644 --- a/backend/.sqlx/query-e4d71278fb80126a7a9da73f1889352d4d1e3cb3a8a08f1c9c03055a1cab1235.json +++ b/backend/.sqlx/query-e4d71278fb80126a7a9da73f1889352d4d1e3cb3a8a08f1c9c03055a1cab1235.json @@ -191,7 +191,8 @@ "ci_test", "github", "azure", - "asset" + "asset", + "freshness" ] } } diff --git a/backend/.sqlx/query-e50bce4bba01c90171973862bcfa088b5c0c1ad601146ff1c57d4a3ddc30556a.json b/backend/.sqlx/query-e50bce4bba01c90171973862bcfa088b5c0c1ad601146ff1c57d4a3ddc30556a.json new file mode 100644 index 0000000000..8c80c14ea4 --- /dev/null +++ b/backend/.sqlx/query-e50bce4bba01c90171973862bcfa088b5c0c1ad601146ff1c57d4a3ddc30556a.json @@ -0,0 +1,34 @@ +{ + "db_name": "PostgreSQL", + "query": "SELECT runnable_path AS \"runnable_path!\", created_by AS \"created_by!\",\n args AS \"args: sqlx::types::Json\"\n FROM v2_job\n WHERE workspace_id = $1 AND trigger_kind = 'freshness'\n ORDER BY created_at", + "describe": { + "columns": [ + { + "ordinal": 0, + "name": "runnable_path!", + "type_info": "Varchar" + }, + { + "ordinal": 1, + "name": "created_by!", + "type_info": "Varchar" + }, + { + "ordinal": 2, + "name": "args: sqlx::types::Json", + "type_info": "Jsonb" + } + ], + "parameters": { + "Left": [ + "Text" + ] + }, + "nullable": [ + true, + false, + true + ] + }, + "hash": "e50bce4bba01c90171973862bcfa088b5c0c1ad601146ff1c57d4a3ddc30556a" +} diff --git a/backend/.sqlx/query-f26cde14890cc754e04341264c4dba01f2f3a02bb36a8d7c20e1990dc81e0472.json b/backend/.sqlx/query-f26cde14890cc754e04341264c4dba01f2f3a02bb36a8d7c20e1990dc81e0472.json new file mode 100644 index 0000000000..8257722c5c --- /dev/null +++ b/backend/.sqlx/query-f26cde14890cc754e04341264c4dba01f2f3a02bb36a8d7c20e1990dc81e0472.json @@ -0,0 +1,22 @@ +{ + "db_name": "PostgreSQL", + "query": "SELECT COUNT(*) AS \"count!\" FROM pipeline_freshness_state WHERE workspace_id = $1", + "describe": { + "columns": [ + { + "ordinal": 0, + "name": "count!", + "type_info": "Int8" + } + ], + "parameters": { + "Left": [ + "Text" + ] + }, + "nullable": [ + null + ] + }, + "hash": "f26cde14890cc754e04341264c4dba01f2f3a02bb36a8d7c20e1990dc81e0472" +} diff --git a/backend/.sqlx/query-f3fef6964a211872b01c984b19192da8ee231c772f88dd9145f01d7cf3b8f8c1.json b/backend/.sqlx/query-f3fef6964a211872b01c984b19192da8ee231c772f88dd9145f01d7cf3b8f8c1.json new file mode 100644 index 0000000000..7049fafe37 --- /dev/null +++ b/backend/.sqlx/query-f3fef6964a211872b01c984b19192da8ee231c772f88dd9145f01d7cf3b8f8c1.json @@ -0,0 +1,15 @@ +{ + "db_name": "PostgreSQL", + "query": "INSERT INTO asset (workspace_id, path, kind, usage_access_type, usage_path, usage_kind, columns)\n SELECT $2, path, kind, usage_access_type, usage_path, usage_kind, columns\n FROM asset WHERE workspace_id = $1 AND usage_kind IN ('script', 'flow')", + "describe": { + "columns": [], + "parameters": { + "Left": [ + "Text", + "Varchar" + ] + }, + "nullable": [] + }, + "hash": "f3fef6964a211872b01c984b19192da8ee231c772f88dd9145f01d7cf3b8f8c1" +} diff --git a/backend/.sqlx/query-fbd1b02b9bbf86e17d6ee0da3e9b134780e55a521a5cee8eb657b67bbd4b2588.json b/backend/.sqlx/query-fbd1b02b9bbf86e17d6ee0da3e9b134780e55a521a5cee8eb657b67bbd4b2588.json new file mode 100644 index 0000000000..ad17a60330 --- /dev/null +++ b/backend/.sqlx/query-fbd1b02b9bbf86e17d6ee0da3e9b134780e55a521a5cee8eb657b67bbd4b2588.json @@ -0,0 +1,19 @@ +{ + "db_name": "PostgreSQL", + "query": "DELETE FROM fork_ducklake_namespace\n WHERE workspace_id = $1 AND ducklake_name = $2 AND catalog = $3\n AND storage = $4 AND storage_ref = $5 AND data_path = $6", + "describe": { + "columns": [], + "parameters": { + "Left": [ + "Text", + "Text", + "Text", + "Text", + "Text", + "Text" + ] + }, + "nullable": [] + }, + "hash": "fbd1b02b9bbf86e17d6ee0da3e9b134780e55a521a5cee8eb657b67bbd4b2588" +} diff --git a/backend/CLAUDE.md b/backend/CLAUDE.md index ff5d9b2f22..24cf2cb837 100644 --- a/backend/CLAUDE.md +++ b/backend/CLAUDE.md @@ -11,6 +11,45 @@ cd backend/windmill-duckdb-ffi-internal && ./build_dev.sh ``` Re-run after clean builds or when `target/debug/libwindmill_duckdb_ffi_internal.*` is missing. + The bundled DuckDB compile (~2min) is cached in a per-user dir shared across + worktrees (keyed by the crate's `Cargo.lock`), so a fresh worktree reuses it and + the build is near-instant — you don't pay the full compile per worktree. Editing + the FFI crate's own source falls back to an isolated per-worktree `./target`. +- **Running data pipelines (DuckLake) from source**: see the section below — a plain build + advertises the `duckdb` tag but cannot execute DuckDB scripts and has no working S3 proxy. + +## Running data pipelines (DuckLake) from source + +DuckLake pipelines need **both** the right cargo features **and** the prebuilt DuckDB FFI. A +plain `cargo run` (or `cargo run --features quickjs`) does **not** suffice, and the failure modes +are silent-ish, so agents lose time. Verify feature names against `backend/Cargo.toml` `[features]`. + +**Feature sets** (run from `backend/`): + +| Goal | Command | +|---|---| +| CE DuckLake (DuckDB scripts + S3 proxy) | `cargo run --features quickjs,duckdb,parquet,private` | +| + Python scripts | add `,python` | +| EE features (WAP, partitioning, forks, …) | add `,enterprise,license` | + +`enterprise` already pulls in `license`, but list both when you want the license-gated paths. +`quickjs` is for JS eval, not DuckLake per se — keep it if your baseline build had it. + +**Before running any DuckDB script**, build the FFI (see the bullet above): +`cd backend/windmill-duckdb-ffi-internal && ./build_dev.sh`. + +**Two gotchas that a wrong feature set produces:** + +1. **`duckdb` tag advertised, feature missing.** The `duckdb` worker tag is in the *unconditional* + default tag list (`windmill-common/src/worker.rs`, `DEFAULT_TAGS`), so a worker advertises it even + without the `duckdb` feature. Jobs then dispatch but fail at execution with + `"Duck DB requires the duckdb feature to be enabled"` (`windmill-worker/src/worker.rs`). Fix: + compile with `--features duckdb`. +2. **DuckLake writes 404 (no S3 proxy).** The workspace S3 proxy (`/w/{ws}/s3_proxy/*`) that + DuckLake uses for reads/writes only mounts the real service under + `#[cfg(all(feature = "private", feature = "parquet"))]` (`windmill-api/src/s3_proxy_oss.rs`); + otherwise it's an empty router and every proxied request 404s. Fix: compile with **both** + `private` and `parquet`. ## Cloud vs self-hosted gating diff --git a/backend/Cargo.lock b/backend/Cargo.lock index 36858ddfda..88eaca4bbd 100644 --- a/backend/Cargo.lock +++ b/backend/Cargo.lock @@ -2057,9 +2057,9 @@ dependencies = [ [[package]] name = "cc" -version = "1.2.65" +version = "1.2.66" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "e228eec9be7c17ccb640b59b36a5cd805ea2a564a4c5e162c2f659fea30d3b96" +checksum = "f5d6cac793997bd970000024b2934968efe83b382de4fdcf4fcb46b6ee4ad996" dependencies = [ "find-msvc-tools", "jobserver", @@ -5072,9 +5072,11 @@ source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "300e883d756b2e4ec94e02791f39b04b522276138852cfc41d9fb7e904106099" dependencies = [ "cfg-if", + "js-sys", "libc", "r-efi 6.0.0", "rand_core 0.10.1", + "wasm-bindgen", ] [[package]] @@ -7563,9 +7565,9 @@ dependencies = [ [[package]] name = "num-bigint" -version = "0.4.7" +version = "0.4.8" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "c863e9ab5e7bf9c99ba75e1050f1e4d624ae87ed3532d6238ffbdc7b585dbbe6" +checksum = "c89e69e7e0f03bea5ef08013795c25018e101932225a656383bd384495ecc367" dependencies = [ "num-integer", "num-traits", @@ -8361,9 +8363,9 @@ checksum = "9b4f627cb1b25917193a259e49bdad08f671f8d9708acfd5fe0a8c1455d87220" [[package]] name = "pest" -version = "2.8.6" +version = "2.8.7" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "e0848c601009d37dfa3430c4666e147e49cdcf1b92ecd3e63657d8a5f19da662" +checksum = "47627dd7305c6a2d6c8c6bcd24c5a4c17dbbf425f4f9c5313e724b38fc9782e9" dependencies = [ "memchr", "ucd-trie", @@ -8371,9 +8373,9 @@ dependencies = [ [[package]] name = "pest_derive" -version = "2.8.6" +version = "2.8.7" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "11f486f1ea21e6c10ed15d5a7c77165d0ee443402f0780849d1768e7d9d6fe77" +checksum = "4b4254325ecad416ab689e27ba51da03ba01a9632bc6e108f5fe7c3c4ad29d58" dependencies = [ "pest", "pest_generator", @@ -8381,9 +8383,9 @@ dependencies = [ [[package]] name = "pest_generator" -version = "2.8.6" +version = "2.8.7" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "8040c4647b13b210a963c1ed407c1ff4fdfa01c31d6d2a098218702e6664f94f" +checksum = "6c4c0e91ead7a8f7acecbca6f003fc2e8282b1dbe2dd9c9d2f16aba42995e0a7" dependencies = [ "pest", "pest_meta", @@ -8394,12 +8396,11 @@ dependencies = [ [[package]] name = "pest_meta" -version = "2.8.6" +version = "2.8.7" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "89815c69d36021a140146f26659a81d6c2afa33d216d736dd4be5381a7362220" +checksum = "f9744bc48116fee06334924bb5f2bad41eed5e89bd26e29b0b799f9a3f82c210" dependencies = [ "pest", - "sha2 0.10.9", ] [[package]] @@ -8995,15 +8996,16 @@ dependencies = [ [[package]] name = "quinn-proto" -version = "0.11.15" +version = "0.11.16" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "4fcb935c5bec503c2f0e306bdd3e58bb9029dcb14fa8d9ac76e3a5256ac0763e" +checksum = "2f4bfc015262b9df63c8845072ce59068853ff5872180c2ce2f13038b970e560" dependencies = [ "aws-lc-rs", "bytes", - "getrandom 0.3.4", + "getrandom 0.4.3", "lru-slab", - "rand 0.9.0", + "rand 0.10.2", + "rand_pcg", "ring 0.17.14", "rustc-hash 2.1.3", "rustls 0.23.35", @@ -9017,16 +9019,16 @@ dependencies = [ [[package]] name = "quinn-udp" -version = "0.5.14" +version = "0.5.15" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "addec6a0dcad8a8d96a771f815f0eaf55f9d1805756410b39f5fa81332574cbd" +checksum = "35a133f956daabe89a61a685c2649f13d82d5aa4bd5d12d1277e1072a21c0694" dependencies = [ "cfg_aliases", "libc", "once_cell", "socket2 0.6.4", "tracing", - "windows-sys 0.60.2", + "windows-sys 0.61.2", ] [[package]] @@ -9184,6 +9186,15 @@ dependencies = [ "rand_core 0.5.1", ] +[[package]] +name = "rand_pcg" +version = "0.10.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "caa0f4137e1c0a72f4c651489402276c8e8e1cf081f3b0ba156d2cbeef09e86a" +dependencies = [ + "rand_core 0.10.1", +] + [[package]] name = "raw-cpuid" version = "11.6.0" @@ -13322,9 +13333,9 @@ checksum = "7fcfc827f90e53a02eaef5e535ee14266c1d569214c6aa70133a624d8a3164ba" [[package]] name = "utf8-width" -version = "0.1.8" +version = "0.1.9" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "1292c0d970b54115d14f2492fe0170adf21d68a1de108eebc51c1df4f346a091" +checksum = "159a7cadce548703edd50d24069bc294c5415ecab0a480e0cd1ca06d112dc94a" [[package]] name = "utf8_iter" @@ -13735,7 +13746,7 @@ dependencies = [ [[package]] name = "windmill" -version = "1.747.0" +version = "1.749.0" dependencies = [ "anyhow", "async-nats", @@ -13817,7 +13828,7 @@ dependencies = [ [[package]] name = "windmill-ai" -version = "1.747.0" +version = "1.749.0" dependencies = [ "async-stream", "async-trait", @@ -13850,7 +13861,7 @@ dependencies = [ [[package]] name = "windmill-alerting" -version = "1.747.0" +version = "1.749.0" dependencies = [ "axum 0.8.9", "chrono", @@ -13863,7 +13874,7 @@ dependencies = [ [[package]] name = "windmill-api" -version = "1.747.0" +version = "1.749.0" dependencies = [ "anyhow", "argon2", @@ -14001,7 +14012,7 @@ dependencies = [ [[package]] name = "windmill-api-agent-workers" -version = "1.747.0" +version = "1.749.0" dependencies = [ "axum 0.8.9", "chrono", @@ -14024,7 +14035,7 @@ dependencies = [ [[package]] name = "windmill-api-assets" -version = "1.747.0" +version = "1.749.0" dependencies = [ "axum 0.8.9", "chrono", @@ -14039,7 +14050,7 @@ dependencies = [ [[package]] name = "windmill-api-auth" -version = "1.747.0" +version = "1.749.0" dependencies = [ "anyhow", "axum 0.8.9", @@ -14065,7 +14076,7 @@ dependencies = [ [[package]] name = "windmill-api-client" -version = "1.747.0" +version = "1.749.0" dependencies = [ "reqwest 0.12.28", "serde", @@ -14075,7 +14086,7 @@ dependencies = [ [[package]] name = "windmill-api-configs" -version = "1.747.0" +version = "1.749.0" dependencies = [ "axum 0.8.9", "chrono", @@ -14092,7 +14103,7 @@ dependencies = [ [[package]] name = "windmill-api-debug" -version = "1.747.0" +version = "1.749.0" dependencies = [ "axum 0.8.9", "base64 0.22.1", @@ -14114,7 +14125,7 @@ dependencies = [ [[package]] name = "windmill-api-embeddings" -version = "1.747.0" +version = "1.749.0" dependencies = [ "anyhow", "axum 0.8.9", @@ -14137,7 +14148,7 @@ dependencies = [ [[package]] name = "windmill-api-flow-conversations" -version = "1.747.0" +version = "1.749.0" dependencies = [ "axum 0.8.9", "chrono", @@ -14153,7 +14164,7 @@ dependencies = [ [[package]] name = "windmill-api-flows" -version = "1.747.0" +version = "1.749.0" dependencies = [ "axum 0.8.9", "chrono", @@ -14174,7 +14185,7 @@ dependencies = [ [[package]] name = "windmill-api-groups" -version = "1.747.0" +version = "1.749.0" dependencies = [ "axum 0.8.9", "chrono", @@ -14195,7 +14206,7 @@ dependencies = [ [[package]] name = "windmill-api-inputs" -version = "1.747.0" +version = "1.749.0" dependencies = [ "axum 0.8.9", "chrono", @@ -14209,7 +14220,7 @@ dependencies = [ [[package]] name = "windmill-api-integration-tests" -version = "1.747.0" +version = "1.749.0" dependencies = [ "anyhow", "async-nats", @@ -14244,7 +14255,7 @@ dependencies = [ [[package]] name = "windmill-api-jobs" -version = "1.747.0" +version = "1.749.0" dependencies = [ "anyhow", "axum 0.8.9", @@ -14269,7 +14280,7 @@ dependencies = [ [[package]] name = "windmill-api-npm-proxy" -version = "1.747.0" +version = "1.749.0" dependencies = [ "axum 0.8.9", "flate2", @@ -14287,7 +14298,7 @@ dependencies = [ [[package]] name = "windmill-api-openapi" -version = "1.747.0" +version = "1.749.0" dependencies = [ "anyhow", "axum 0.8.9", @@ -14309,7 +14320,7 @@ dependencies = [ [[package]] name = "windmill-api-schedule" -version = "1.747.0" +version = "1.749.0" dependencies = [ "axum 0.8.9", "chrono", @@ -14329,7 +14340,7 @@ dependencies = [ [[package]] name = "windmill-api-scripts" -version = "1.747.0" +version = "1.749.0" dependencies = [ "axum 0.8.9", "chrono", @@ -14366,7 +14377,7 @@ dependencies = [ [[package]] name = "windmill-api-settings" -version = "1.747.0" +version = "1.749.0" dependencies = [ "anyhow", "axum 0.8.9", @@ -14394,7 +14405,7 @@ dependencies = [ [[package]] name = "windmill-api-sse" -version = "1.747.0" +version = "1.749.0" dependencies = [ "lazy_static", "serde", @@ -14406,7 +14417,7 @@ dependencies = [ [[package]] name = "windmill-api-users" -version = "1.747.0" +version = "1.749.0" dependencies = [ "argon2", "axum 0.8.9", @@ -14431,7 +14442,7 @@ dependencies = [ [[package]] name = "windmill-api-workers" -version = "1.747.0" +version = "1.749.0" dependencies = [ "axum 0.8.9", "chrono", @@ -14445,10 +14456,11 @@ dependencies = [ [[package]] name = "windmill-api-workspaces" -version = "1.747.0" +version = "1.749.0" dependencies = [ "axum 0.8.9", "chrono", + "futures", "hex", "http 1.4.2", "hyper 1.10.1", @@ -14479,7 +14491,7 @@ dependencies = [ [[package]] name = "windmill-audit" -version = "1.747.0" +version = "1.749.0" dependencies = [ "chrono", "lazy_static", @@ -14493,7 +14505,7 @@ dependencies = [ [[package]] name = "windmill-autoscaling" -version = "1.747.0" +version = "1.749.0" dependencies = [ "anyhow", "axum 0.8.9", @@ -14512,7 +14524,7 @@ dependencies = [ [[package]] name = "windmill-common" -version = "1.747.0" +version = "1.749.0" dependencies = [ "aes-gcm", "aho-corasick", @@ -14614,7 +14626,7 @@ dependencies = [ [[package]] name = "windmill-dep-map" -version = "1.747.0" +version = "1.749.0" dependencies = [ "chrono", "itertools 0.14.0", @@ -14633,7 +14645,7 @@ dependencies = [ [[package]] name = "windmill-git-sync" -version = "1.747.0" +version = "1.749.0" dependencies = [ "regex", "serde", @@ -14648,7 +14660,7 @@ dependencies = [ [[package]] name = "windmill-indexer" -version = "1.747.0" +version = "1.749.0" dependencies = [ "anyhow", "astral-tokio-tar", @@ -14672,7 +14684,7 @@ dependencies = [ [[package]] name = "windmill-jseval" -version = "1.747.0" +version = "1.749.0" dependencies = [ "anyhow", "futures", @@ -14689,7 +14701,7 @@ dependencies = [ [[package]] name = "windmill-macros" -version = "1.747.0" +version = "1.749.0" dependencies = [ "itertools 0.14.0", "lazy_static", @@ -14705,7 +14717,7 @@ dependencies = [ [[package]] name = "windmill-mcp" -version = "1.747.0" +version = "1.749.0" dependencies = [ "anyhow", "async-trait", @@ -14726,7 +14738,7 @@ dependencies = [ [[package]] name = "windmill-native-triggers" -version = "1.747.0" +version = "1.749.0" dependencies = [ "anyhow", "async-trait", @@ -14757,7 +14769,7 @@ dependencies = [ [[package]] name = "windmill-oauth" -version = "1.747.0" +version = "1.749.0" dependencies = [ "anyhow", "arc-swap", @@ -14782,7 +14794,7 @@ dependencies = [ [[package]] name = "windmill-object-store" -version = "1.747.0" +version = "1.749.0" dependencies = [ "anyhow", "async-stream", @@ -14816,7 +14828,7 @@ dependencies = [ [[package]] name = "windmill-operator" -version = "1.747.0" +version = "1.749.0" dependencies = [ "anyhow", "futures", @@ -14834,7 +14846,7 @@ dependencies = [ [[package]] name = "windmill-parser" -version = "1.747.0" +version = "1.749.0" dependencies = [ "convert_case 0.6.0", "serde", @@ -14843,7 +14855,7 @@ dependencies = [ [[package]] name = "windmill-parser-bash" -version = "1.747.0" +version = "1.749.0" dependencies = [ "anyhow", "lazy_static", @@ -14855,7 +14867,7 @@ dependencies = [ [[package]] name = "windmill-parser-csharp" -version = "1.747.0" +version = "1.749.0" dependencies = [ "anyhow", "serde_json", @@ -14867,7 +14879,7 @@ dependencies = [ [[package]] name = "windmill-parser-go" -version = "1.747.0" +version = "1.749.0" dependencies = [ "anyhow", "gosyn", @@ -14879,7 +14891,7 @@ dependencies = [ [[package]] name = "windmill-parser-graphql" -version = "1.747.0" +version = "1.749.0" dependencies = [ "anyhow", "lazy_static", @@ -14891,7 +14903,7 @@ dependencies = [ [[package]] name = "windmill-parser-java" -version = "1.747.0" +version = "1.749.0" dependencies = [ "anyhow", "serde_json", @@ -14903,7 +14915,7 @@ dependencies = [ [[package]] name = "windmill-parser-nu" -version = "1.747.0" +version = "1.749.0" dependencies = [ "anyhow", "nu-parser", @@ -14914,7 +14926,7 @@ dependencies = [ [[package]] name = "windmill-parser-php" -version = "1.747.0" +version = "1.749.0" dependencies = [ "anyhow", "itertools 0.14.0", @@ -14925,7 +14937,7 @@ dependencies = [ [[package]] name = "windmill-parser-py" -version = "1.747.0" +version = "1.749.0" dependencies = [ "anyhow", "itertools 0.14.0", @@ -14937,7 +14949,7 @@ dependencies = [ [[package]] name = "windmill-parser-py-asset" -version = "1.747.0" +version = "1.749.0" dependencies = [ "anyhow", "rustpython-ast", @@ -14948,7 +14960,7 @@ dependencies = [ [[package]] name = "windmill-parser-py-imports" -version = "1.747.0" +version = "1.749.0" dependencies = [ "anyhow", "async-recursion", @@ -14970,7 +14982,7 @@ dependencies = [ [[package]] name = "windmill-parser-r" -version = "1.747.0" +version = "1.749.0" dependencies = [ "anyhow", "serde_json", @@ -14982,7 +14994,7 @@ dependencies = [ [[package]] name = "windmill-parser-ruby" -version = "1.747.0" +version = "1.749.0" dependencies = [ "anyhow", "lazy_static", @@ -14996,7 +15008,7 @@ dependencies = [ [[package]] name = "windmill-parser-rust" -version = "1.747.0" +version = "1.749.0" dependencies = [ "anyhow", "convert_case 0.6.0", @@ -15013,7 +15025,7 @@ dependencies = [ [[package]] name = "windmill-parser-sql" -version = "1.747.0" +version = "1.749.0" dependencies = [ "anyhow", "lazy_static", @@ -15026,7 +15038,7 @@ dependencies = [ [[package]] name = "windmill-parser-sql-asset" -version = "1.747.0" +version = "1.749.0" dependencies = [ "anyhow", "serde", @@ -15038,7 +15050,7 @@ dependencies = [ [[package]] name = "windmill-parser-ts" -version = "1.747.0" +version = "1.749.0" dependencies = [ "anyhow", "lazy_static", @@ -15056,7 +15068,7 @@ dependencies = [ [[package]] name = "windmill-parser-ts-asset" -version = "1.747.0" +version = "1.749.0" dependencies = [ "anyhow", "serde-wasm-bindgen", @@ -15072,7 +15084,7 @@ dependencies = [ [[package]] name = "windmill-parser-wac" -version = "1.747.0" +version = "1.749.0" dependencies = [ "anyhow", "rustpython-ast", @@ -15088,7 +15100,7 @@ dependencies = [ [[package]] name = "windmill-parser-yaml" -version = "1.747.0" +version = "1.749.0" dependencies = [ "anyhow", "serde", @@ -15099,7 +15111,7 @@ dependencies = [ [[package]] name = "windmill-queue" -version = "1.747.0" +version = "1.749.0" dependencies = [ "anyhow", "async-recursion", @@ -15138,7 +15150,7 @@ dependencies = [ [[package]] name = "windmill-runtime-nativets" -version = "1.747.0" +version = "1.749.0" dependencies = [ "anyhow", "const_format", @@ -15177,7 +15189,7 @@ dependencies = [ [[package]] name = "windmill-sql-datatype-parser-wasm" -version = "1.747.0" +version = "1.749.0" dependencies = [ "getrandom 0.3.4", "wasm-bindgen", @@ -15188,7 +15200,7 @@ dependencies = [ [[package]] name = "windmill-store" -version = "1.747.0" +version = "1.749.0" dependencies = [ "anyhow", "async-recursion", @@ -15222,7 +15234,7 @@ dependencies = [ [[package]] name = "windmill-test-utils" -version = "1.747.0" +version = "1.749.0" dependencies = [ "anyhow", "async-trait", @@ -15246,7 +15258,7 @@ dependencies = [ [[package]] name = "windmill-trigger" -version = "1.747.0" +version = "1.749.0" dependencies = [ "anyhow", "async-trait", @@ -15279,7 +15291,7 @@ dependencies = [ [[package]] name = "windmill-trigger-azure" -version = "1.747.0" +version = "1.749.0" dependencies = [ "anyhow", "async-trait", @@ -15312,7 +15324,7 @@ dependencies = [ [[package]] name = "windmill-trigger-email" -version = "1.747.0" +version = "1.749.0" dependencies = [ "anyhow", "async-trait", @@ -15332,7 +15344,7 @@ dependencies = [ [[package]] name = "windmill-trigger-gcp" -version = "1.747.0" +version = "1.749.0" dependencies = [ "anyhow", "async-trait", @@ -15366,7 +15378,7 @@ dependencies = [ [[package]] name = "windmill-trigger-http" -version = "1.747.0" +version = "1.749.0" dependencies = [ "anyhow", "async-trait", @@ -15402,7 +15414,7 @@ dependencies = [ [[package]] name = "windmill-trigger-kafka" -version = "1.747.0" +version = "1.749.0" dependencies = [ "anyhow", "async-trait", @@ -15425,7 +15437,7 @@ dependencies = [ [[package]] name = "windmill-trigger-mqtt" -version = "1.747.0" +version = "1.749.0" dependencies = [ "anyhow", "async-trait", @@ -15449,7 +15461,7 @@ dependencies = [ [[package]] name = "windmill-trigger-nats" -version = "1.747.0" +version = "1.749.0" dependencies = [ "anyhow", "async-nats", @@ -15473,7 +15485,7 @@ dependencies = [ [[package]] name = "windmill-trigger-postgres" -version = "1.747.0" +version = "1.749.0" dependencies = [ "anyhow", "async-trait", @@ -15508,7 +15520,7 @@ dependencies = [ [[package]] name = "windmill-trigger-sqs" -version = "1.747.0" +version = "1.749.0" dependencies = [ "anyhow", "async-trait", @@ -15536,7 +15548,7 @@ dependencies = [ [[package]] name = "windmill-trigger-websocket" -version = "1.747.0" +version = "1.749.0" dependencies = [ "anyhow", "async-trait", @@ -15561,7 +15573,7 @@ dependencies = [ [[package]] name = "windmill-types" -version = "1.747.0" +version = "1.749.0" dependencies = [ "anyhow", "bitflags 2.13.0", @@ -15580,7 +15592,7 @@ dependencies = [ [[package]] name = "windmill-worker" -version = "1.747.0" +version = "1.749.0" dependencies = [ "anyhow", "async-once-cell", @@ -15690,7 +15702,7 @@ dependencies = [ [[package]] name = "windmill-worker-volumes" -version = "1.747.0" +version = "1.749.0" dependencies = [ "bytes", "futures", diff --git a/backend/Cargo.toml b/backend/Cargo.toml index a91518dc38..48e48da15d 100644 --- a/backend/Cargo.toml +++ b/backend/Cargo.toml @@ -1,6 +1,6 @@ [package] name = "windmill" -version = "1.747.0" +version = "1.749.0" authors.workspace = true edition.workspace = true @@ -87,7 +87,7 @@ members = [ exclude = ["./windmill-duckdb-ffi-internal", "./parsers/windmill-parser-wasm"] [workspace.package] -version = "1.747.0" +version = "1.749.0" authors = ["Ruben Fiszel "] edition = "2021" diff --git a/backend/ee-repo-ref.txt b/backend/ee-repo-ref.txt index f0cdaad407..71a3acdda6 100644 --- a/backend/ee-repo-ref.txt +++ b/backend/ee-repo-ref.txt @@ -1 +1 @@ -d36555eb8891becdb66f2135684387367450ed1d \ No newline at end of file +6c287041cd7edd4a77a4bc07ad0e156cec32cce4 diff --git a/backend/migrations/20260703165613_pipeline_freshness_watchdog.down.sql b/backend/migrations/20260703165613_pipeline_freshness_watchdog.down.sql new file mode 100644 index 0000000000..aed2f9d32a --- /dev/null +++ b/backend/migrations/20260703165613_pipeline_freshness_watchdog.down.sql @@ -0,0 +1,5 @@ +-- Postgres has no ALTER TYPE ... DROP VALUE for enums. The 'freshness' value +-- stays even on rollback, consistent with prior job_trigger_kind additions +-- (see 20260510174213_asset_trigger_dispatch). +DROP INDEX IF EXISTS idx_script_pipeline_freshness_scan; +DROP TABLE IF EXISTS pipeline_freshness_state; diff --git a/backend/migrations/20260703165613_pipeline_freshness_watchdog.up.sql b/backend/migrations/20260703165613_pipeline_freshness_watchdog.up.sql new file mode 100644 index 0000000000..acf9545c96 --- /dev/null +++ b/backend/migrations/20260703165613_pipeline_freshness_watchdog.up.sql @@ -0,0 +1,35 @@ +-- Attribution for runs pushed by the pipeline freshness watchdog (the EE +-- background loop that re-runs a `// freshness`-annotated producer whose +-- output aged past its window). +ALTER TYPE job_trigger_kind ADD VALUE IF NOT EXISTS 'freshness'; + +-- Per-(workspace, script) watchdog state: exponential-backoff bookkeeping so +-- a persistently failing producer isn't re-pushed on every scan tick, and an +-- atomic claim so concurrent servers can't double-push in the same tick +-- (claim = the UPDATE/INSERT that advances next_attempt_at; only the winner +-- pushes). Rows exist only while a script is stale — observing it fresh (or +-- its annotation gone) deletes the row, resetting the backoff. +CREATE TABLE pipeline_freshness_state ( + workspace_id VARCHAR(50) NOT NULL REFERENCES workspace (id) ON DELETE CASCADE, + script_path VARCHAR(510) NOT NULL, + attempts INTEGER NOT NULL DEFAULT 1, + last_push_at TIMESTAMPTZ NOT NULL DEFAULT now(), + next_attempt_at TIMESTAMPTZ NOT NULL DEFAULT now(), + PRIMARY KEY (workspace_id, script_path) +); + +-- Written only by the server monitor loop on the raw (non-RLS) pool, but +-- granted like every other app table so a future user-transaction reader +-- doesn't hit the recurring missing-GRANT class of bug. +GRANT ALL ON pipeline_freshness_state TO windmill_user; +GRANT ALL ON pipeline_freshness_state TO windmill_admin; + +-- The watchdog's ~60s candidate scan (latest deployed pipeline members) +-- filters on this exact predicate and orders by (workspace_id, path, +-- created_at DESC); without a matching partial index it seq-scans the whole +-- script-version heap on every tick, on instances that mostly have zero +-- pipeline scripts. (idx_script_pipeline_path is text_pattern_ops for +-- prefix LIKE — it can't serve this ordering.) +CREATE INDEX idx_script_pipeline_freshness_scan + ON script (workspace_id, path, created_at DESC) + WHERE auto_kind = 'pipeline' AND archived = false AND deleted = false; diff --git a/backend/migrations/20260703170745_fork_ducklake_namespace.down.sql b/backend/migrations/20260703170745_fork_ducklake_namespace.down.sql new file mode 100644 index 0000000000..fd722bf414 --- /dev/null +++ b/backend/migrations/20260703170745_fork_ducklake_namespace.down.sql @@ -0,0 +1 @@ +DROP TABLE IF EXISTS fork_ducklake_namespace; diff --git a/backend/migrations/20260703170745_fork_ducklake_namespace.up.sql b/backend/migrations/20260703170745_fork_ducklake_namespace.up.sql new file mode 100644 index 0000000000..7762304199 --- /dev/null +++ b/backend/migrations/20260703170745_fork_ducklake_namespace.up.sql @@ -0,0 +1,51 @@ +-- Registry of ducklake namespaces provisioned for fork/dev workspaces. One row per +-- (fork workspace, lake name): records the exact catalog metadata schema and data +-- sub-path the fork's jobs attach to, so fork deletion can drop the pg schema and +-- delete the S3 prefix deterministically (the row is written on first resolution, +-- before any physical state exists). +-- +-- Deliberately NO foreign key to workspace(id): rows are the durable cleanup ledger and +-- must OUTLIVE the workspace row when physical cleanup fails after the delete commits +-- (unreachable catalog, storage outage) — a CASCADE would erase the only record of the +-- orphaned namespace, letting a recreated same-id fork silently reattach stale tables. +-- Rows are deleted explicitly after each successful cleanup; leftover rows for a reused +-- id are retried at fork creation, which refuses to proceed while they cannot be cleaned. +CREATE TABLE fork_ducklake_namespace ( + workspace_id VARCHAR(50) NOT NULL, + ducklake_name VARCHAR(255) NOT NULL, + metadata_schema VARCHAR(63) NOT NULL, + -- Canonical identity of the catalog database the metadata schema lives in + -- (`:`, e.g. `instance:wm_ducklake` or + -- `postgres:u/admin/pg`). Cleanup connects to THIS catalog, not whatever the fork's + -- settings point at by then — a drifted catalog resource must not make cleanup drop a + -- schema in the wrong database and orphan the real one. + catalog TEXT NOT NULL, + -- Named workspace storage holding the fork's data files; '' = the default storage + -- (part of the PK, which cannot hold NULL). + storage TEXT NOT NULL DEFAULT '', + -- The storage's RESOLVED identity at registration time (`:`, e.g. `s3:u/admin/minio` or `filesystem:/data/lfs`; '' = unknown). Cleanup + -- deletes the fork prefix from THIS storage, not whatever the logical name points at by + -- then — repointing a storage after attach must not orphan the original fork data (or + -- delete a colliding prefix from the new one). + storage_ref TEXT NOT NULL DEFAULT '', + -- The fork namespace's data path within that storage (a bucket-root + -- `__wm_forks//…` prefix). + data_path TEXT NOT NULL, + created_at TIMESTAMPTZ NOT NULL DEFAULT now(), + -- Cleanup phase state: true once the metadata schema has been dropped but data files (or + -- the row delete) still failed. Later retries then skip the schema phase entirely — they + -- need NO catalog credentials, which may be gone for good with the deleted fork's + -- resources. Reset to false whenever a live fork re-registers the row (re-attaching + -- recreates the schema). + schema_dropped BOOLEAN NOT NULL DEFAULT false, + -- One row per physical location EVER attached: if the fork's lake settings drift + -- (catalog/storage/path change), later attaches add rows rather than replace them, so + -- cleanup covers every location the fork wrote, not just the first. + PRIMARY KEY (workspace_id, ducklake_name, catalog, storage, storage_ref, data_path) +); + +-- Resolution runs under user_db transactions (SET LOCAL ROLE) in API contexts, so the +-- windmill roles need explicit grants (default privileges don't apply to app-created tables). +GRANT ALL ON fork_ducklake_namespace TO windmill_user; +GRANT ALL ON fork_ducklake_namespace TO windmill_admin; diff --git a/backend/parsers/windmill-parser-py-asset/src/lib.rs b/backend/parsers/windmill-parser-py-asset/src/lib.rs index 2282025092..8d2e87a53e 100644 --- a/backend/parsers/windmill-parser-py-asset/src/lib.rs +++ b/backend/parsers/windmill-parser-py-asset/src/lib.rs @@ -319,12 +319,24 @@ fn dict_str_value(dict: &rustpython_ast::ExprDict, name: &str) -> Option /// `write_s3_file` to a canonical asset path, mirroring `windmill-parser-ts-asset`: /// `S3Object(s3="", storage=""?)` — or the equivalent dict literal — /// maps to the URI `s3:///` (empty bucket for default storage, i.e. -/// `s3:///`), and a bare `"s3://bucket/key"` string is passed through. +/// `s3:///`), and a `"s3://bucket/key"` URI string is passed through. +/// String args mirror the runtime `parse_s3_object` contract exactly: only a +/// `s3:///` URI with a non-empty key is valid — any other +/// string (bare key, `s3://x`, empty key) raises at run time, so recording an +/// edge for it would be a phantom node for a call that can only error. /// The resulting URI is fed through `parse_asset_syntax` so the stored path /// matches the TS object form and the `# on s3:///…` trigger form exactly. fn s3_object_arg_path(expr: &Expr) -> Option { let uri = match expr { - Expr::Constant(ExprConstant { value: Constant::Str(s), .. }) => s.clone(), + Expr::Constant(ExprConstant { value: Constant::Str(s), .. }) => { + match s + .strip_prefix("s3://") + .and_then(|rest| rest.split_once('/')) + { + Some((_, key)) if !key.is_empty() => s.clone(), + _ => return None, + } + } Expr::Call(call) => { // `S3Object(...)` imported directly or as `wmill.S3Object(...)` let func_name = call @@ -371,13 +383,47 @@ def main(): s.map_err(|e| e.to_string()), Ok(vec![ParseAssetsResult { kind: AssetKind::S3Object, - path: "/test.csv".to_string(), + path: "test.csv".to_string(), access_type: Some(R), columns: None, },]) ); } + #[test] + fn test_py_asset_parser_bare_string_no_asset() { + // A plain (non-`s3://`) string is rejected by the runtime + // `parse_s3_object`, so the parser must not record a phantom asset + // for a call that can only error. + let input = r#" +import wmill +def main(): + wmill.write_s3_file("pipelines/etl/out.jsonl", b"") +"#; + let s = parse_assets(input).map(|o| o.assets); + assert_eq!(s.map_err(|e| e.to_string()), Ok(vec![])); + } + + #[test] + fn test_py_asset_parser_invalid_uri_no_write_edge() { + // `s3://x` (no key part) and `s3://bucket/` (empty key) are rejected + // by the runtime `parse_s3_object` — same rule for the SDK-arg path: + // no R/W edge. The generic URI-literal scan may still record them as + // ambiguous (`access_type: None`) assets, like any `s3://…` string + // constant anywhere in a script. + let input = r#" +import wmill +def main(): + wmill.write_s3_file("s3://broken", b"") + wmill.write_s3_file("s3://bucket/", b"") +"#; + let assets = parse_assets(input).expect("parse").assets; + assert!( + assets.iter().all(|a| a.access_type.is_none()), + "invalid URIs must not produce R/W edges: {assets:?}" + ); + } + #[test] fn test_py_asset_parser_write_s3_object_constructor() { // The SDK signature is `write_s3_file(s3object: S3Object | str, ...)` and @@ -395,7 +441,31 @@ def main(): s.map_err(|e| e.to_string()), Ok(vec![ParseAssetsResult { kind: AssetKind::S3Object, - path: "/analytics/x.csv".to_string(), + path: "analytics/x.csv".to_string(), + access_type: Some(W), + columns: None, + },]) + ); + } + + #[test] + fn test_py_write_key_matches_duckdb_read_key() { + // Cross-language lineage: this write records `exports/x`, the same path a + // DuckDB `read_csv('s3://exports/x')` resolves to (see + // windmill-parser-sql-asset `test_duckdb_read_key_matches_sdk_write_key`), + // so the producer and consumer connect in the pipeline graph. + let input = r#" +import wmill +from wmill import S3Object +def main(): + wmill.write_s3_file(S3Object(s3="exports/x"), b"content") +"#; + let s = parse_assets(input).map(|o| o.assets); + assert_eq!( + s.map_err(|e| e.to_string()), + Ok(vec![ParseAssetsResult { + kind: AssetKind::S3Object, + path: "exports/x".to_string(), access_type: Some(W), columns: None, },]) @@ -438,7 +508,7 @@ def main(): s.map_err(|e| e.to_string()), Ok(vec![ParseAssetsResult { kind: AssetKind::S3Object, - path: "/dir/in.csv".to_string(), + path: "dir/in.csv".to_string(), access_type: Some(R), columns: None, },]) @@ -461,14 +531,14 @@ def main(): Ok(vec![ ParseAssetsResult { kind: AssetKind::S3Object, - path: "/out.json".to_string(), - access_type: Some(W), + path: "mybucket/dir/in.csv".to_string(), + access_type: Some(R), columns: None, }, ParseAssetsResult { kind: AssetKind::S3Object, - path: "mybucket/dir/in.csv".to_string(), - access_type: Some(R), + path: "out.json".to_string(), + access_type: Some(W), columns: None, }, ]) @@ -494,25 +564,25 @@ def main(): Ok(vec![ ParseAssetsResult { kind: AssetKind::S3Object, - path: "/pipelines/km_real/enriched.json".to_string(), + path: "pipelines/km_real/enriched.json".to_string(), access_type: Some(W), columns: None, }, ParseAssetsResult { kind: AssetKind::S3Object, - path: "/pipelines/km_real/raw_events.json".to_string(), + path: "pipelines/km_real/raw_events.json".to_string(), access_type: Some(W), columns: None, }, ParseAssetsResult { kind: AssetKind::S3Object, - path: "/pipelines/km_real/report.json".to_string(), + path: "pipelines/km_real/report.json".to_string(), access_type: Some(W), columns: None, }, ParseAssetsResult { kind: AssetKind::S3Object, - path: "/pipelines/km_real/summary.json".to_string(), + path: "pipelines/km_real/summary.json".to_string(), access_type: Some(W), columns: None, }, diff --git a/backend/parsers/windmill-parser-sql-asset/src/asset_parser.rs b/backend/parsers/windmill-parser-sql-asset/src/asset_parser.rs index c4f93efdb7..d2ab5b51a7 100644 --- a/backend/parsers/windmill-parser-sql-asset/src/asset_parser.rs +++ b/backend/parsers/windmill-parser-sql-asset/src/asset_parser.rs @@ -1145,13 +1145,13 @@ mod tests { Ok(vec![ ParseAssetsResult { kind: AssetKind::S3Object, - path: "/a.parquet".to_string(), + path: "a.parquet".to_string(), access_type: Some(R), columns: None }, ParseAssetsResult { kind: AssetKind::S3Object, - path: "/c.parquet".to_string(), + path: "c.parquet".to_string(), access_type: Some(W), columns: None }, @@ -1165,6 +1165,30 @@ mod tests { ); } + #[test] + fn test_duckdb_read_key_matches_sdk_write_key() { + // Cross-language lineage: a TS `writeS3File({ s3: "exports/x" })` or + // Python `write_s3_file(S3Object(s3="exports/x"))` records the asset path + // `exports/x` (default storage). A DuckDB reader of the same object must + // resolve to the identical path so the graph connects the producer and + // consumer — both the bare `s3://exports/x` and the triple-slash + // `s3:///exports/x` default-storage form must yield `exports/x`. + for uri in ["s3://exports/x", "s3:///exports/x"] { + let input = format!("SELECT * FROM read_csv('{uri}');"); + let assets = parse_assets(&input).expect("parse").assets; + assert_eq!( + assets, + vec![ParseAssetsResult { + kind: AssetKind::S3Object, + path: "exports/x".to_string(), + access_type: Some(R), + columns: None + }], + "DuckDB read of {uri} must resolve to the SDK write key" + ); + } + } + #[test] fn test_copy_target_echoed_in_select_stays_write_only() { // The trailing summary SELECT merely mentions the COPY target — it @@ -1178,7 +1202,7 @@ mod tests { s.map_err(|e| e.to_string()), Ok(vec![ParseAssetsResult { kind: AssetKind::S3Object, - path: "/out.csv".to_string(), + path: "out.csv".to_string(), access_type: Some(W), columns: None }]) @@ -1195,7 +1219,7 @@ mod tests { s.map_err(|e| e.to_string()), Ok(vec![ParseAssetsResult { kind: AssetKind::S3Object, - path: "/referenced.csv".to_string(), + path: "referenced.csv".to_string(), access_type: Some(R), columns: None }]) @@ -1215,7 +1239,7 @@ mod tests { s.map_err(|e| e.to_string()), Ok(vec![ParseAssetsResult { kind: AssetKind::S3Object, - path: "/data.csv".to_string(), + path: "data.csv".to_string(), access_type: Some(RW), columns: None }]) @@ -1235,7 +1259,7 @@ mod tests { s.map_err(|e| e.to_string()), Ok(vec![ParseAssetsResult { kind: AssetKind::S3Object, - path: "/data.parquet".to_string(), + path: "data.parquet".to_string(), access_type: Some(RW), columns: None }]) @@ -1253,13 +1277,13 @@ mod tests { Ok(vec![ ParseAssetsResult { kind: AssetKind::S3Object, - path: "/a.parquet".to_string(), + path: "a.parquet".to_string(), access_type: Some(R), columns: None }, ParseAssetsResult { kind: AssetKind::S3Object, - path: "/b.parquet".to_string(), + path: "b.parquet".to_string(), access_type: Some(R), columns: None } @@ -1279,7 +1303,7 @@ mod tests { s.map_err(|e| e.to_string()), Ok(vec![ParseAssetsResult { kind: AssetKind::S3Object, - path: "/data.parquet".to_string(), + path: "data.parquet".to_string(), access_type: Some(RW), columns: None }]) @@ -1961,7 +1985,7 @@ mod tests { assert_eq!(result.len(), 1); assert_eq!(result[0].kind, AssetKind::S3Object); - assert_eq!(result[0].path, "/example_file.parquet"); + assert_eq!(result[0].path, "example_file.parquet"); assert_eq!(result[0].access_type, Some(R)); let columns = result[0].columns.as_ref().expect("Should have columns"); @@ -1992,7 +2016,7 @@ mod tests { assert_eq!(result.len(), 1); assert_eq!(result[0].kind, AssetKind::S3Object); - assert_eq!(result[0].path, "/example_file.parquet"); + assert_eq!(result[0].path, "example_file.parquet"); assert!(result[0].columns.is_none()); } @@ -2005,7 +2029,7 @@ mod tests { assert_eq!(result.len(), 1); assert_eq!(result[0].kind, AssetKind::S3Object); - assert_eq!(result[0].path, "/example_file.parquet"); + assert_eq!(result[0].path, "example_file.parquet"); let columns = result[0].columns.as_ref().expect("Should have columns"); assert_eq!(columns.get("col1"), Some(&R)); @@ -2024,7 +2048,7 @@ mod tests { assert_eq!(result.len(), 2); assert!(result.iter().any(|a| { - a.path == "/file1.parquet" + a.path == "file1.parquet" && a.columns.as_ref().map_or(false, |c| c.contains_key("col1")) })); assert!(result.iter().any(|a| { @@ -2057,7 +2081,7 @@ mod tests { assert_eq!(result.len(), 1); assert_eq!(result[0].kind, AssetKind::S3Object); - assert_eq!(result[0].path, "/test.parquet"); + assert_eq!(result[0].path, "test.parquet"); assert_eq!(result[0].access_type, Some(R)); let columns = result[0].columns.as_ref().expect("Should have columns"); diff --git a/backend/parsers/windmill-parser-ts-asset/src/lib.rs b/backend/parsers/windmill-parser-ts-asset/src/lib.rs index b7fe0fea5c..f273db5cbe 100644 --- a/backend/parsers/windmill-parser-ts-asset/src/lib.rs +++ b/backend/parsers/windmill-parser-ts-asset/src/lib.rs @@ -341,12 +341,25 @@ fn object_str_prop(obj: &ObjectLit, name: &str) -> Option { /// `writeS3File` to a canonical asset path, mirroring the runtime /// `parseS3Object`: an object `{ s3: "", storage?: "" }` maps to /// the URI `s3:///` (empty bucket for default storage, i.e. -/// `s3:///`), and a bare `"s3://bucket/key"` string is passed through. +/// `s3:///`), and a `"s3://bucket/key"` URI string is passed through. +/// String args mirror the runtime `parseS3Object` contract exactly: only a +/// `s3:///` URI with a non-empty key is valid — any other +/// string (bare key, `s3://x`, empty key) throws at run time, so recording an +/// edge for it would be a phantom node for a call that can only error. /// The resulting URI is fed through `parse_asset_syntax` so the stored path /// matches the `// on s3:///…` trigger form exactly. fn s3_object_arg_path(arg: &Expr) -> Option { let uri = match arg { - Expr::Lit(Lit::Str(s)) => s.value.to_string(), + Expr::Lit(Lit::Str(s)) => { + let v = s.value.to_string(); + match v + .strip_prefix("s3://") + .and_then(|rest| rest.split_once('/')) + { + Some((_, key)) if !key.is_empty() => v, + _ => return None, + } + } Expr::Object(obj) => { let key = object_str_prop(obj, "s3")?; let storage = object_str_prop(obj, "storage").unwrap_or_default(); @@ -420,7 +433,7 @@ mod tests { s.map(|r| r.assets).map_err(|e| e.to_string()), Ok(vec![ParseAssetsResult { kind: AssetKind::S3Object, - path: "/test.csv".to_string(), + path: "test.csv".to_string(), access_type: Some(R), columns: None, },]) @@ -448,7 +461,31 @@ mod tests { s.map(|r| r.assets).map_err(|e| e.to_string()), Ok(vec![ParseAssetsResult { kind: AssetKind::S3Object, - path: "/pipelines/km_real/raw_events.json".to_string(), + path: "pipelines/km_real/raw_events.json".to_string(), + access_type: Some(W), + columns: None, + },]) + ); + } + + #[test] + fn test_ts_write_key_matches_duckdb_read_key() { + // Cross-language lineage: this write records `exports/x`, the same path a + // DuckDB `read_csv('s3://exports/x')` resolves to (see + // windmill-parser-sql-asset `test_duckdb_read_key_matches_sdk_write_key`), + // so the producer and consumer connect in the pipeline graph. + let input = r#" + import * as wmill from "windmill-client" + export async function main() { + await wmill.writeS3File({ s3: "exports/x" }, "[]") + } + "#; + let s = parse_assets(input); + assert_eq!( + s.map(|r| r.assets).map_err(|e| e.to_string()), + Ok(vec![ParseAssetsResult { + kind: AssetKind::S3Object, + path: "exports/x".to_string(), access_type: Some(W), columns: None, },]) @@ -477,6 +514,42 @@ mod tests { ); } + #[test] + fn test_ts_asset_parser_bare_string_no_asset() { + // A plain (non-`s3://`) string is rejected by the runtime + // `parseS3Object`, so the parser must not record a phantom asset for + // a call that can only error. + let input = r#" + import * as wmill from "windmill-client" + export async function main() { + await wmill.writeS3File("pipelines/etl/out.jsonl", "[]") + } + "#; + let s = parse_assets(input); + assert_eq!(s.map(|r| r.assets).map_err(|e| e.to_string()), Ok(vec![])); + } + + #[test] + fn test_ts_asset_parser_invalid_uri_no_write_edge() { + // `s3://x` (no key part) and `s3://bucket/` (empty key) are rejected + // by the runtime `parseS3Object` — same rule for the SDK-arg path: + // no R/W edge. The generic URI-literal scan may still record them as + // ambiguous (`access_type: None`) assets, like any `s3://…` string + // constant anywhere in a script. + let input = r#" + import * as wmill from "windmill-client" + export async function main() { + await wmill.writeS3File("s3://broken", "[]") + await wmill.writeS3File("s3://bucket/", "[]") + } + "#; + let assets = parse_assets(input).expect("parse").assets; + assert!( + assets.iter().all(|a| a.access_type.is_none()), + "invalid URIs must not produce R/W edges: {assets:?}" + ); + } + #[test] fn test_ts_asset_parser_multiple_s3_object_writes() { // Mirrors the f/km/r_seed shape: several direct object-form writes in @@ -497,25 +570,25 @@ mod tests { Ok(vec![ ParseAssetsResult { kind: AssetKind::S3Object, - path: "/pipelines/km_real/enriched.json".to_string(), + path: "pipelines/km_real/enriched.json".to_string(), access_type: Some(W), columns: None, }, ParseAssetsResult { kind: AssetKind::S3Object, - path: "/pipelines/km_real/raw_events.json".to_string(), + path: "pipelines/km_real/raw_events.json".to_string(), access_type: Some(W), columns: None, }, ParseAssetsResult { kind: AssetKind::S3Object, - path: "/pipelines/km_real/report.json".to_string(), + path: "pipelines/km_real/report.json".to_string(), access_type: Some(W), columns: None, }, ParseAssetsResult { kind: AssetKind::S3Object, - path: "/pipelines/km_real/summary.json".to_string(), + path: "pipelines/km_real/summary.json".to_string(), access_type: Some(W), columns: None, }, @@ -536,7 +609,7 @@ mod tests { s.map(|r| r.assets).map_err(|e| e.to_string()), Ok(vec![ParseAssetsResult { kind: AssetKind::S3Object, - path: "/out.json".to_string(), + path: "out.json".to_string(), access_type: Some(W), columns: None, },]) diff --git a/backend/parsers/windmill-parser-wasm/Cargo.lock b/backend/parsers/windmill-parser-wasm/Cargo.lock index 815d62644f..6d36c2f8f2 100644 --- a/backend/parsers/windmill-parser-wasm/Cargo.lock +++ b/backend/parsers/windmill-parser-wasm/Cargo.lock @@ -6191,7 +6191,7 @@ checksum = "712e227841d057c1ee1cd2fb22fa7e5a5461ae8e48fa2ca79ec42cfc1931183f" [[package]] name = "windmill-common" -version = "1.747.0" +version = "1.749.0" dependencies = [ "aho-corasick", "anyhow", @@ -6272,7 +6272,7 @@ dependencies = [ [[package]] name = "windmill-macros" -version = "1.747.0" +version = "1.749.0" dependencies = [ "proc-macro2", "quote", @@ -6284,7 +6284,7 @@ dependencies = [ [[package]] name = "windmill-parser" -version = "1.747.0" +version = "1.749.0" dependencies = [ "convert_case", "serde", @@ -6293,7 +6293,7 @@ dependencies = [ [[package]] name = "windmill-parser-bash" -version = "1.747.0" +version = "1.749.0" dependencies = [ "anyhow", "lazy_static", @@ -6305,7 +6305,7 @@ dependencies = [ [[package]] name = "windmill-parser-csharp" -version = "1.747.0" +version = "1.749.0" dependencies = [ "anyhow", "serde_json", @@ -6317,7 +6317,7 @@ dependencies = [ [[package]] name = "windmill-parser-go" -version = "1.747.0" +version = "1.749.0" dependencies = [ "anyhow", "gosyn", @@ -6329,7 +6329,7 @@ dependencies = [ [[package]] name = "windmill-parser-graphql" -version = "1.747.0" +version = "1.749.0" dependencies = [ "anyhow", "lazy_static", @@ -6341,7 +6341,7 @@ dependencies = [ [[package]] name = "windmill-parser-java" -version = "1.747.0" +version = "1.749.0" dependencies = [ "anyhow", "serde_json", @@ -6353,7 +6353,7 @@ dependencies = [ [[package]] name = "windmill-parser-nu" -version = "1.747.0" +version = "1.749.0" dependencies = [ "anyhow", "nu-parser", @@ -6364,7 +6364,7 @@ dependencies = [ [[package]] name = "windmill-parser-php" -version = "1.747.0" +version = "1.749.0" dependencies = [ "anyhow", "itertools 0.14.0", @@ -6375,7 +6375,7 @@ dependencies = [ [[package]] name = "windmill-parser-py" -version = "1.747.0" +version = "1.749.0" dependencies = [ "anyhow", "itertools 0.14.0", @@ -6387,7 +6387,7 @@ dependencies = [ [[package]] name = "windmill-parser-py-asset" -version = "1.747.0" +version = "1.749.0" dependencies = [ "anyhow", "rustpython-ast", @@ -6398,7 +6398,7 @@ dependencies = [ [[package]] name = "windmill-parser-py-imports" -version = "1.747.0" +version = "1.749.0" dependencies = [ "anyhow", "async-recursion", @@ -6420,7 +6420,7 @@ dependencies = [ [[package]] name = "windmill-parser-r" -version = "1.747.0" +version = "1.749.0" dependencies = [ "anyhow", "serde_json", @@ -6432,7 +6432,7 @@ dependencies = [ [[package]] name = "windmill-parser-ruby" -version = "1.747.0" +version = "1.749.0" dependencies = [ "anyhow", "lazy_static", @@ -6446,7 +6446,7 @@ dependencies = [ [[package]] name = "windmill-parser-rust" -version = "1.747.0" +version = "1.749.0" dependencies = [ "anyhow", "convert_case", @@ -6463,7 +6463,7 @@ dependencies = [ [[package]] name = "windmill-parser-sql" -version = "1.747.0" +version = "1.749.0" dependencies = [ "anyhow", "lazy_static", @@ -6476,7 +6476,7 @@ dependencies = [ [[package]] name = "windmill-parser-sql-asset" -version = "1.747.0" +version = "1.749.0" dependencies = [ "anyhow", "serde", @@ -6488,7 +6488,7 @@ dependencies = [ [[package]] name = "windmill-parser-ts" -version = "1.747.0" +version = "1.749.0" dependencies = [ "anyhow", "lazy_static", @@ -6506,7 +6506,7 @@ dependencies = [ [[package]] name = "windmill-parser-ts-asset" -version = "1.747.0" +version = "1.749.0" dependencies = [ "anyhow", "serde-wasm-bindgen", @@ -6522,7 +6522,7 @@ dependencies = [ [[package]] name = "windmill-parser-wac" -version = "1.747.0" +version = "1.749.0" dependencies = [ "anyhow", "rustpython-ast", @@ -6538,7 +6538,7 @@ dependencies = [ [[package]] name = "windmill-parser-wasm" -version = "1.747.0" +version = "1.749.0" dependencies = [ "anyhow", "getrandom 0.2.17", @@ -6570,7 +6570,7 @@ dependencies = [ [[package]] name = "windmill-parser-yaml" -version = "1.747.0" +version = "1.749.0" dependencies = [ "anyhow", "serde", @@ -6581,7 +6581,7 @@ dependencies = [ [[package]] name = "windmill-types" -version = "1.747.0" +version = "1.749.0" dependencies = [ "anyhow", "bitflags", diff --git a/backend/parsers/windmill-parser-wasm/Cargo.toml b/backend/parsers/windmill-parser-wasm/Cargo.toml index 9111ca8ccb..b8b9263a91 100644 --- a/backend/parsers/windmill-parser-wasm/Cargo.toml +++ b/backend/parsers/windmill-parser-wasm/Cargo.toml @@ -12,7 +12,7 @@ resolver = "2" members = ["."] [workspace.package] -version = "1.747.0" +version = "1.749.0" edition = "2021" authors = ["Ruben Fiszel "] diff --git a/backend/parsers/windmill-parser/src/asset_parser.rs b/backend/parsers/windmill-parser/src/asset_parser.rs index c22359f6b0..94b43c1d7f 100644 --- a/backend/parsers/windmill-parser/src/asset_parser.rs +++ b/backend/parsers/windmill-parser/src/asset_parser.rs @@ -77,10 +77,12 @@ pub struct ParseAssetsOutput { #[serde(skip_serializing_if = "Option::is_none", default)] pub partition: Option, // `// freshness ` — SLA stating outputs must be at most - // `duration` old. Active backstop: when no other trigger has fired the - // script within the window, a watchdog re-runs it. Distinct from - // schedule (which is producer cadence); freshness is consumer SLA and - // applies regardless of which trigger last fired. + // `duration` old. Drives passive monitoring in CE (the asset graph + // colors the node's badge fresh/stale against its last successful run) + // and the enterprise watchdog (windmill-queue `freshness_watchdog`), + // which re-runs a stale unpartitioned producer. Distinct from schedule + // (which is producer cadence); freshness is consumer SLA and applies + // regardless of which trigger last fired. #[serde(skip_serializing_if = "Option::is_none", default)] pub freshness: Option, // `// trigger all` → AND join barrier; default (`any`) = OR (current @@ -250,6 +252,8 @@ pub struct RetrySpec { // history (`valid_from`/`valid_to`/`is_current`). The leading keyword `scd2` is a // recognized alias for `history`. `deletes=close` (scd2 only) also closes a key // that disappears from the snapshot; default leaves absent keys current. +// `on_schema_change=ignore` suppresses downstream schema-contract warnings for +// the produced asset (save-time metadata only; default `warn`). #[derive(Serialize, Debug, PartialEq, Clone)] pub struct MaterializeSpec { pub target_kind: AssetKind, @@ -274,6 +278,117 @@ pub struct MaterializeSpec { // `hard_deletes=close`). Default (false) leaves absent keys current. #[serde(skip_serializing_if = "std::ops::Not::not", default)] pub close_deleted: bool, + // `on_schema_change=warn|ignore|fail|sync` governs two orthogonal things: + // • Save-time contract warnings (gap #2b): consumers referencing columns + // the captured schema no longer has warn by default; only `ignore` + // suppresses those warnings (`warn`/`fail`/`sync` all keep them). + // • Run-time write guardrails for the persist-and-mutate strategies + // (partitioned replace, merge, append), where the table schema is fixed + // at first CREATE and the write is positional — a renamed/added/removed + // SELECT column silently lands in the wrong column. `warn` logs the + // drift and proceeds positionally; `fail` aborts before mutating; `sync` + // ALTERs the table to match and writes by name. Whole-table replace and + // scd2 are unaffected. See `sql_materialize.rs`. + #[serde(skip_serializing_if = "OnSchemaChange::is_warn", default)] + pub on_schema_change: OnSchemaChange, +} + +impl MaterializeSpec { + /// The `_current` SCD2 companion view this managed materialize also + /// produces, or `None` when it isn't a managed scd2 target. Managed scd2 + /// creates the base table *and* a `_current` "latest row per key" view + /// each run (see `sql_materialize.rs`); `manual` mode owns its own DDL and + /// short-circuits before that codegen, so it produces no companion. + pub fn scd2_current_target(&self) -> Option<(AssetKind, String)> { + (self.scd2 && !self.manual) + .then(|| (self.target_kind, format!("{}_current", self.target_path))) + } + + /// Every asset this managed materialize produces: the base table, plus — for + /// managed scd2 — the `_current` companion view. The producer's + /// trailing `SELECT` doesn't express these writes (the runtime generates the + /// DDL), so this is the single source of truth every graph surface (deploy + /// asset rows, the CLI `--local` graph, and the frontend live graph) uses to + /// link reads of the base *and* the `_current` view back to this producer. + pub fn write_targets(&self) -> Vec<(AssetKind, String)> { + let mut targets = vec![(self.target_kind, self.target_path.clone())]; + targets.extend(self.scd2_current_target()); + targets + } +} + +// dbt's `on_schema_change`, covering both the save-time contract check and the +// run-time write guardrail for the positional persist-and-mutate strategies: +// • `warn` (default): surface consumer contract warnings; at write time, log +// the drift loudly and proceed with the positional write against the fixed +// table schema. +// • `ignore`: suppress consumer contract warnings; at write time, no guard +// (the pre-guardrail behaviour). +// • `fail`: keep contract warnings; at write time, abort the run before +// mutating when the SELECT's column *set* diverges from the table's. +// • `sync`: keep contract warnings; at write time, ALTER the table to match +// the SELECT (add/drop columns) and INSERT BY NAME. +// `warn`/`fail` drift detection is name-set based (added/removed columns), which +// is what the positional persist-and-mutate INSERT can misalign on. It does NOT +// flag a pure *reorder* of same-named columns: `SELECT b, a` into a `(a, b)` +// table has an identical column set, so `fail` does not abort and the positional +// INSERT swaps the values. Reorder-safety is exactly what `sync` provides +// (INSERT BY NAME maps by name), so a SELECT whose column order is not pinned to +// the table's should use `sync`, not `fail`. (An ordered-list comparison would +// close this, but a false positive there would abort a correctly-aligned write, +// so the guard stays on the set difference.) +// `fail`/`sync` only affect partitioned replace, merge and append; whole-table +// replace already rebuilds each run, and scd2 has no positional write — for an +// scd2 target `sync` degrades to `warn` (no write-time effect; deploy-time +// rejection is out of scope here). +#[derive(Serialize, Debug, PartialEq, Eq, Clone, Copy, Default)] +#[serde(rename_all = "lowercase")] +pub enum OnSchemaChange { + #[default] + Warn, + Ignore, + Fail, + Sync, +} + +impl OnSchemaChange { + pub fn is_warn(&self) -> bool { + matches!(self, OnSchemaChange::Warn) + } +} + +impl MaterializeSpec { + /// Deploy-time validation of the option combination against the script's + /// partitioning, returning a human-facing error for combinations the + /// runtime cannot honor. Called at save (`create_script_internal`) so a + /// misconfigured script is rejected up front, and again in the DuckDB + /// executor as a safety net for preview/test runs that never deploy. Both + /// checks are SCD2-specific and inert for `manual` mode (which owns its DDL + /// and ignores the reconciliation strategy). `partitioned` is whether the + /// script declares `// partitioned`. + pub fn validate(&self, partitioned: bool) -> Result<(), String> { + if self.manual || !self.scd2 { + return Ok(()); + } + // SCD2 needs a natural key to identify an entity across versions. + if self.unique_key.as_deref().map_or(true, str::is_empty) { + return Err( + "materialize scd2: requires a natural key — add `key=` (e.g. \ + `// materialize ducklake:/// key=id history`)" + .to_string(), + ); + } + // SCD2's diff/close/open shape has no partition-scoped form in v1. + if partitioned { + return Err( + "materialize scd2: `// partitioned` is not supported with scd2 in v1 — remove \ + `// partitioned`, or drop `history`/`scd2` to materialize the partition without \ + history" + .to_string(), + ); + } + Ok(()) + } } // `// data_test …` — a data-quality assertion run against the @@ -522,6 +637,24 @@ pub fn parse_asset_syntax(s: &str, enable_default_syntax: bool) -> Option<(Asset for (prefix, kind) in ASSET_KINDS.iter() { if s.starts_with(prefix) { let path = &s[prefix.len()..]; + // Canonicalize S3 keys to a single asset identity. The SDK object + // form (`{ s3: "key" }` / `S3Object(s3="key")`, default storage) + // resolves to `s3:///key`, whose path is `/key`, while DuckDB + // `s3://key` and `// on s3://key` yield the bare `key`. Strip every + // leading slash so the triple-slash default-storage form and the + // `s3://storage/key` form share one path — otherwise a TS/Python + // writer and a DuckDB reader of the same object become disconnected + // nodes in the pipeline graph. Stripping ALL leading slashes (not + // just one) keeps the identity stable through URI reconstruction: + // `trigger_spec_to_row` rebuilds `s3://`, so a canonical path + // must never itself start with `/` or the rebuilt ref would parse + // back to a different key. Only leading slashes are touched, so + // Hive-partition keys (`s3://b/y=2024/f.parquet`) are untouched. + let path = if matches!(kind, AssetKind::S3Object) { + path.trim_start_matches('/') + } else { + path + }; return Some((*kind, path)); } } @@ -637,7 +770,7 @@ fn parse_kv_opts(s: &str) -> BTreeMap { // - `on ` → asset / native trigger edge (including // the marker-only `on schedule` form) // - `partitioned [opts]` → partition declaration -// - `freshness ` → SLA / active backstop +// - `freshness ` → SLA window (badge + EE watchdog) // - `tag ` → worker-tag override (annotation wins // over UI-set value at deploy) // - `retry []` → cascade-only retry policy @@ -839,6 +972,39 @@ pub fn parse_pipeline_annotations(code: &str) -> PipelineAnnotations { out } +// Count `// data_test <…>` lines in the leading comment header whose right-hand +// side fails to parse into a check. `parse_pipeline_annotations` drops these +// fail-safe (a malformed line just yields no check), which is the wrong default +// for a data-quality assertion: a typo silently disables the test. The deploy +// path uses this count to warn. Same leading-block boundary as the parser (stop +// at the first non-comment line) so a body comment can't be miscounted, and the +// same `parse_data_test_spec` grammar so "malformed" means exactly what the +// parser rejects — no second grammar to drift. +pub fn count_malformed_data_tests(code: &str) -> usize { + let mut malformed = 0; + for raw_line in code.lines() { + let line = raw_line.trim_start(); + if line.is_empty() { + continue; + } + let rest = if let Some(r) = line.strip_prefix("//") { + r + } else if let Some(r) = line.strip_prefix("--") { + r + } else if let Some(r) = line.strip_prefix('#') { + r + } else { + break; + }; + if let Some(after_kw) = consume_keyword(rest.trim_start(), "data_test") { + if parse_data_test_spec(after_kw.trim()).is_none() { + malformed += 1; + } + } + } + malformed +} + // Parse a `// retry []` right-hand side. `` is a // non-negative decimal; `` is an optional raw duration string left // for `parse_duration_secs` to validate at deploy. A bare zero count (or @@ -915,6 +1081,15 @@ fn parse_materialize_spec(s: &str) -> Option { // `deletes=close` (scd2 only) opts into hard-delete-close; any other value // (or absence) keeps the soft-delete default. let close_deleted = opts.get("deletes").map(|v| v == "close").unwrap_or(false); + // `on_schema_change=ignore|fail|sync`; any other value (or absence) keeps + // the `warn` default, fail-safe like `deletes=` above (a typo must never + // silently disable the guardrail). + let on_schema_change = match opts.get("on_schema_change").map(String::as_str) { + Some("ignore") => OnSchemaChange::Ignore, + Some("fail") => OnSchemaChange::Fail, + Some("sync") => OnSchemaChange::Sync, + _ => OnSchemaChange::Warn, + }; Some(MaterializeSpec { target_kind, target_path: path.to_string(), @@ -924,6 +1099,7 @@ fn parse_materialize_spec(s: &str) -> Option { scd2, track, close_deleted, + on_schema_change, }) } @@ -1123,6 +1299,83 @@ fn parse_trigger_spec(s: &str) -> Option { mod pipeline_annotation_tests { use super::*; + #[test] + fn s3_key_normalization_unifies_uri_forms() { + // A TS/Python SDK write of `{ s3: "exports/x" }` (default storage) + // resolves to the URI `s3:///exports/x`, while a DuckDB read of + // `s3://exports/x` and the `// on s3://exports/x` trigger form yield the + // bare `exports/x`. All three must canonicalize to one asset key so + // the writer and reader connect in the pipeline graph. + let sdk_write = parse_asset_syntax("s3:///exports/x", false); + let duckdb_read = parse_asset_syntax("s3://exports/x", false); + assert_eq!(sdk_write, Some((AssetKind::S3Object, "exports/x"))); + assert_eq!(duckdb_read, Some((AssetKind::S3Object, "exports/x"))); + assert_eq!(sdk_write, duckdb_read); + + // The `// on` trigger annotation goes through the same function. + assert_eq!( + parse_asset_syntax("s3:///exports/x", true), + parse_asset_syntax("s3://exports/x", true) + ); + + // Explicit-storage form is unaffected (no leading slash to strip). + assert_eq!( + parse_asset_syntax("s3://mybucket/exports/x", false), + Some((AssetKind::S3Object, "mybucket/exports/x")) + ); + + // Hive-partition keys and nested paths under default storage are + // preserved verbatim (only leading slashes are stripped). + assert_eq!( + parse_asset_syntax("s3:///t/year=2024/month=01/f.parquet", false), + Some((AssetKind::S3Object, "t/year=2024/month=01/f.parquet")) + ); + + // Every leading slash is stripped so a canonical S3 path never starts + // with `/`. `S3Object(s3="/x")` resolves to the quad-slash URI + // `s3:////x`; the identity must be the bare `x` (not `/x`) so the ref + // that `trigger_spec_to_row` rebuilds round-trips back to it. + assert_eq!( + parse_asset_syntax("s3:////x", false), + Some((AssetKind::S3Object, "x")) + ); + assert_eq!( + parse_asset_syntax("s3://///deep///", false), + Some((AssetKind::S3Object, "deep///")) + ); + + // Non-S3 kinds keep their leading slash (their paths are workspace- + // relative and the slash is significant). + assert_eq!( + parse_asset_syntax("res://f/foo", false), + Some((AssetKind::Resource, "f/foo")) + ); + assert_eq!( + parse_asset_syntax("ducklake://analytics/orders", false), + Some((AssetKind::Ducklake, "analytics/orders")) + ); + } + + #[test] + fn s3_explicit_storage_aliases_default_storage_nested_key() { + // Accepted tradeoff of one canonical key: the explicit-storage form + // `s3://storage/key` and the default-storage nested-key form + // `s3:///storage/key` collapse to the same node `storage/key`, even + // though they name different objects. This is a best-effort lineage + // graph that does not split the first segment as a storage name; the + // collision only happens when a storage config is named to match a + // default-storage prefix. Pinned so the aliasing is intentional, not a + // latent surprise. + assert_eq!( + parse_asset_syntax("s3://mybucket/x", false), + parse_asset_syntax("s3:///mybucket/x", false) + ); + assert_eq!( + parse_asset_syntax("s3://mybucket/x", false), + Some((AssetKind::S3Object, "mybucket/x")) + ); + } + #[test] fn bare_pipeline_marker() { let out = parse_pipeline_annotations("// pipeline\nconsole.log('hi')"); @@ -1606,6 +1859,62 @@ mod pipeline_annotation_tests { assert!(!m.close_deleted); } + #[test] + fn materialize_scd2_write_targets_include_current_view() { + // A managed scd2 materialize produces the base table AND the + // `_current` companion view, so the producer must be recorded as + // writing both (reads of the view otherwise resolve to an orphan asset). + let out = parse_pipeline_annotations( + "// materialize ducklake://main/dim_customers key=id history", + ); + let m = out.materialize.expect("materialize"); + assert_eq!( + m.write_targets(), + vec![ + (AssetKind::Ducklake, "main/dim_customers".to_string()), + ( + AssetKind::Ducklake, + "main/dim_customers_current".to_string() + ), + ] + ); + assert_eq!( + m.scd2_current_target(), + Some(( + AssetKind::Ducklake, + "main/dim_customers_current".to_string() + )) + ); + } + + #[test] + fn materialize_non_scd2_write_targets_are_base_only() { + // A plain merge (no `history`) creates no companion view — only the base. + let out = parse_pipeline_annotations("// materialize ducklake://main/dim_customers key=id"); + let m = out.materialize.expect("materialize"); + assert_eq!( + m.write_targets(), + vec![(AssetKind::Ducklake, "main/dim_customers".to_string())] + ); + assert_eq!(m.scd2_current_target(), None); + } + + #[test] + fn materialize_manual_scd2_has_no_companion_view() { + // `manual` mode owns its own DDL and never creates the `_current` view, + // so registering it would be a false producer edge. + let out = parse_pipeline_annotations( + "// materialize manual ducklake://main/dim_customers key=id history", + ); + let m = out.materialize.expect("materialize"); + assert!(m.manual && m.scd2); + assert_eq!(m.scd2_current_target(), None); + assert_eq!( + m.write_targets(), + vec![(AssetKind::Ducklake, "main/dim_customers".to_string())] + ); + } + #[test] fn materialize_scd2_deletes_close_opt() { let out = parse_pipeline_annotations( @@ -1621,6 +1930,93 @@ mod pipeline_annotation_tests { assert!(!out.materialize.expect("materialize").close_deleted); } + #[test] + fn materialize_validate_scd2_requires_key() { + // scd2 without `key=` is rejected at deploy (was a run-time error). + let m = parse_pipeline_annotations("// materialize ducklake://a/dim history") + .materialize + .expect("materialize"); + assert!(m.scd2 && m.unique_key.is_none()); + let err = m.validate(false).expect_err("scd2 without key must fail"); + assert!(err.contains("requires a natural key")); + // with a key it validates + let m = parse_pipeline_annotations("// materialize ducklake://a/dim key=id history") + .materialize + .expect("materialize"); + assert!(m.validate(false).is_ok()); + } + + #[test] + fn materialize_validate_scd2_rejects_partitioned() { + // scd2 + `// partitioned` has no v1 form — rejected at deploy. + let m = parse_pipeline_annotations("// materialize ducklake://a/dim key=id history") + .materialize + .expect("materialize"); + let err = m.validate(true).expect_err("scd2 + partitioned must fail"); + assert!(err.contains("`// partitioned` is not supported with scd2")); + // unpartitioned scd2 is fine + assert!(m.validate(false).is_ok()); + } + + #[test] + fn materialize_validate_non_scd2_and_manual_are_inert() { + // Non-scd2 strategies are unconstrained by these checks, partitioned or not. + let m = parse_pipeline_annotations("// materialize ducklake://a/orders key=id") + .materialize + .expect("materialize"); + assert!(m.validate(true).is_ok()); + assert!(m.validate(false).is_ok()); + // `manual` owns its DDL and ignores the strategy — never rejected here, + // even with a partitioned scd2-looking combo. + let m = parse_pipeline_annotations("// materialize manual ducklake://a/dim history") + .materialize + .expect("materialize"); + assert!(m.manual && m.scd2); + assert!(m.validate(true).is_ok()); + } + + #[test] + fn materialize_on_schema_change_opt() { + let out = parse_pipeline_annotations( + "// materialize ducklake://a/orders on_schema_change=ignore", + ); + let m = out.materialize.expect("materialize"); + assert_eq!(m.on_schema_change, OnSchemaChange::Ignore); + // default is warn + let out = parse_pipeline_annotations("// materialize ducklake://a/orders"); + assert_eq!( + out.materialize.expect("materialize").on_schema_change, + OnSchemaChange::Warn + ); + // fail + sync parse to their own variants + let out = + parse_pipeline_annotations("// materialize ducklake://a/orders on_schema_change=fail"); + assert_eq!( + out.materialize.expect("materialize").on_schema_change, + OnSchemaChange::Fail + ); + let out = + parse_pipeline_annotations("// materialize ducklake://a/orders on_schema_change=sync"); + assert_eq!( + out.materialize.expect("materialize").on_schema_change, + OnSchemaChange::Sync + ); + // unknown/junk value keeps the warn default (fail-safe, like `deletes=`) + let out = + parse_pipeline_annotations("// materialize ducklake://a/orders on_schema_change=bogus"); + assert_eq!( + out.materialize.expect("materialize").on_schema_change, + OnSchemaChange::Warn + ); + // composes with other opts + let out = parse_pipeline_annotations( + "// materialize ducklake://a/dim key=id history on_schema_change=ignore", + ); + let m = out.materialize.expect("materialize"); + assert!(m.scd2); + assert_eq!(m.on_schema_change, OnSchemaChange::Ignore); + } + #[test] fn materialize_key_without_history_is_plain_merge() { let out = parse_pipeline_annotations("// materialize ducklake://a/dim key=id"); @@ -1950,4 +2346,23 @@ mod pipeline_annotation_tests { )); assert!(out.column_lineage.is_empty()); } + + #[test] + fn count_malformed_data_tests_counts_only_broken_header_lines() { + let n = count_malformed_data_tests(concat!( + "-- data_test not_null id\n", // valid + "-- data_test unique id\n", // valid + "-- data_test accepted_values status paid\n", // malformed: missing `=` + "-- data_test relationships cust ducklake\n", // malformed: no `->` + "-- data_test\n", // malformed: bare keyword + "-- data_test f/tests/custom\n", // valid: custom script path + "SELECT 1 -- data_test not_a_test\n", // body line: not counted + )); + assert_eq!(n, 3); + // A clean header has zero. + assert_eq!( + count_malformed_data_tests("-- data_test unique id\nSELECT 1"), + 0 + ); + } } diff --git a/backend/parsers/windmill-parser/src/sql_materialize.rs b/backend/parsers/windmill-parser/src/sql_materialize.rs index e40cd0f057..1b24a5bccd 100644 --- a/backend/parsers/windmill-parser/src/sql_materialize.rs +++ b/backend/parsers/windmill-parser/src/sql_materialize.rs @@ -390,9 +390,39 @@ pub struct MaterializeCodegen<'a> { /// and the partition column / `SET PARTITIONED BY` are omitted. pub partitioned: bool, pub strategy: MaterializeStrategy, + /// Write-time guardrail for a drifted SELECT vs the fixed table schema. + /// Only the persist-and-mutate strategies (partitioned replace, merge, + /// append) act on it: `Fail` emits an in-txn guard that raises on drift, + /// `Sync` writes BY NAME and expects the executor to inject `ALTER TABLE` + /// DDL at the [`SYNC_ALTER_SENTINEL`] slot, `Warn`/`Ignore` write + /// positionally (drift surfaced by the summary in `Warn`, silent in + /// `Ignore`). See [`MaterializeCodegen::is_persist_and_mutate`]. + pub on_schema_change: OnSchemaChange, } +/// The exact statement the `sync` codegen emits right after `BEGIN +/// TRANSACTION;` as the injection slot for `ALTER TABLE … ADD/DROP COLUMN` +/// DDL. The executor computes the drift with a pre-pass probe and replaces this +/// literal in the assembled query text (with the DDL, or removes it when there +/// is no drift). Classified `Write` so the EE write-audit-publish reassembly +/// keeps it inside the transaction with the mutations; a plain no-op SELECT so +/// that if it is somehow left un-replaced the run still succeeds unchanged. +pub const SYNC_ALTER_SENTINEL: &str = "SELECT '__wm_sync_alter_sentinel__' AS _wm_sync;"; + impl<'a> MaterializeCodegen<'a> { + /// Whether this (strategy, partitioned) uses the positional persist-and- + /// mutate write whose table schema is fixed at first CREATE — the only case + /// the `on_schema_change` write-time guardrail applies to. Whole-table + /// replace (`CREATE OR REPLACE`) and scd2 self-heal / are name-mapped, so + /// they are excluded. + pub fn is_persist_and_mutate(&self) -> bool { + match self.strategy { + MaterializeStrategy::Scd2 { .. } => false, + MaterializeStrategy::Replace => self.partitioned, + MaterializeStrategy::Append | MaterializeStrategy::Merge { .. } => true, + } + } + /// The ordered statements that perform the materialization, to be run after /// the setup blocks and inside the caller's execution. The first-run /// bootstrap is idempotent (`IF NOT EXISTS`), so this is safe to run every @@ -440,22 +470,39 @@ impl<'a> MaterializeCodegen<'a> { "CREATE TABLE IF NOT EXISTS {t} AS SELECT * FROM ({sel}) WHERE false;" )); } - out.push("BEGIN TRANSACTION;".to_string()); + // Write-time schema guardrail, emitted right after BEGIN so it runs + // before any mutation (a failing guard aborts before touching data; the + // sync ALTERs run before the INSERT so the sets match). Reached only on + // the persist-and-mutate path here (whole-table replace and scd2 return + // above), so no extra strategy gate is needed. + match self.on_schema_change { + OnSchemaChange::Fail => out.push(schema_drift_guard_sql(sel, t, pcol)), + OnSchemaChange::Sync => out.push(SYNC_ALTER_SENTINEL.to_string()), + OnSchemaChange::Warn | OnSchemaChange::Ignore => {} + } // The rows to write, with the partition column appended when partitioned. let source = if self.partitioned { format!("SELECT *, {pval} AS {pcol} FROM ({sel})") } else { format!("SELECT * FROM ({sel})") }; + // `sync` maps columns by name (positional would cross-wire: an ALTERed + // ADD COLUMN appends at the end, so a positional INSERT of the SELECT + // would fill it from the wrong source column). + let by_name = if self.on_schema_change == OnSchemaChange::Sync { + " BY NAME" + } else { + "" + }; match &self.strategy { MaterializeStrategy::Replace => { // Only reached when partitioned (whole-table replace returned above). out.push(format!("DELETE FROM {t} WHERE {pcol} = {pval};")); - out.push(format!("INSERT INTO {t} {source};")); + out.push(format!("INSERT INTO {t}{by_name} {source};")); } MaterializeStrategy::Append => { - out.push(format!("INSERT INTO {t} {source};")); + out.push(format!("INSERT INTO {t}{by_name} {source};")); } MaterializeStrategy::Merge { unique_key } => { // Upsert within the slice via delete-by-key + insert (dbt's @@ -467,6 +514,11 @@ impl<'a> MaterializeCodegen<'a> { // same write shape as `replace`, which is reliable. The DELETE is // scoped to the current partition when partitioned so it stays // slice-local (a key present in another partition is untouched). + // + // Guard first: the DELETE+INSERT does not dedup the source, so + // two incoming rows sharing a key would both persist under it. + // Raise instead of silently double-writing (see the helper). + out.push(duplicate_source_key_guard_sql(sel, unique_key)); let scope = if self.partitioned { format!("{pcol} = {pval} AND ") } else { @@ -475,7 +527,7 @@ impl<'a> MaterializeCodegen<'a> { out.push(format!( "DELETE FROM {t} WHERE {scope}{unique_key} IN (SELECT {unique_key} FROM ({sel}));" )); - out.push(format!("INSERT INTO {t} {source};")); + out.push(format!("INSERT INTO {t}{by_name} {source};")); } // Handled by the early return above (scd2 has no partitioned form). MaterializeStrategy::Scd2 { .. } => unreachable!("scd2 handled before this match"), @@ -578,7 +630,7 @@ impl<'a> MaterializeCodegen<'a> { ]; // Hard-delete-close (`deletes=close`): the keys that vanished from the // snapshot — present-and-current in the table, absent from the SELECT. - // Captured before the transaction (like `changed`) and disjoint from it (a + // Captured before the close (like `changed`) and disjoint from it (a // key is either in the snapshot or not), so the two closes never overlap. if close_deleted { out.push(format!( @@ -633,6 +685,112 @@ impl<'a> MaterializeCodegen<'a> { } } +// --------------------------------------------------------------------------- +// on_schema_change drift detection (write-time guardrail) +// --------------------------------------------------------------------------- +// +// Drift is computed entirely in SQL against the live DuckDB session: the +// SELECT's output columns come from `DESCRIBE`, the table's from `DESCRIBE` of +// the target (the managed `_wm_partition` column is excluded so it is compared +// as the producer's logical output, matching schema capture). `added` = SELECT +// columns absent from the table, `removed` = table columns absent from the +// SELECT. When the table was just created this run (first materialize) the two +// DESCRIBEs agree, so both lists are empty and no guard fires. + +/// A `DESCRIBE`-derived set of column names of `rel_sql` (any SELECT-able +/// relation, already parenthesized/qualified by the caller), optionally +/// dropping the managed partition column. +fn describe_col_names(rel_sql: &str, exclude_col: Option<&str>) -> String { + let filter = match exclude_col { + Some(c) => format!(" WHERE column_name <> {}", quote_lit(c)), + None => String::new(), + }; + format!("SELECT column_name FROM (DESCRIBE SELECT * FROM {rel_sql}){filter}") +} + +/// Scalar subqueries `(added, removed)` — the list of column names in the +/// SELECT but not the table, and vice versa. Each is a `list(...)` over an +/// `EXCEPT`; empty ⇒ `list()` yields an empty list (`len` 0). The partition +/// column is excluded on the table side only. +/// +/// Set difference, not ordered: this catches the add/remove/rename that a +/// positional INSERT misaligns on, but by design NOT a pure reorder of +/// same-named columns (identical sets ⇒ empty added/removed). See the +/// `OnSchemaChange` doc in asset_parser.rs — reorder-safety is `sync`'s job +/// (INSERT BY NAME); the set difference is deliberately kept over an ordered +/// comparison so the `fail` guard cannot false-positive on a correctly-aligned +/// write. +fn drift_lists(sel_sql: &str, target_qualified: &str, partition_col: &str) -> (String, String) { + let sel = describe_col_names(&format!("({sel_sql})"), None); + let tbl = describe_col_names(target_qualified, Some(partition_col)); + let added = format!("(SELECT list(column_name) FROM (({sel}) EXCEPT ({tbl})))"); + let removed = format!("(SELECT list(column_name) FROM (({tbl}) EXCEPT ({sel})))"); + (added, removed) +} + +/// The `fail`-mode guard: a single statement that raises via DuckDB `error(...)` +/// when the SELECT's columns diverge from the table's, naming the added/removed +/// columns and the target. Both `CASE` branches are cast to VARCHAR so the +/// planner cannot constant-fold the `error(...)` away, and the condition depends +/// on the runtime drift subqueries so it is never folded to a constant. +fn schema_drift_guard_sql(sel_sql: &str, target_qualified: &str, partition_col: &str) -> String { + let (added, removed) = drift_lists(sel_sql, target_qualified, partition_col); + // `target_qualified` is safe in table-reference position (quoted identifiers) + // but here it lands inside a SQL string literal, so single-quotes must be + // doubled (same as `quote_lit`). + let tq = target_qualified.replace('\'', "''"); + format!( + "SELECT CASE WHEN coalesce(len(_wm_added), 0) + coalesce(len(_wm_removed), 0) > 0 \ + THEN CAST(error('managed materialize: on_schema_change=fail blocked a schema-drifted \ + write to {tq} — added column(s): [' || coalesce(array_to_string(_wm_added, ', '), '') || \ + '], removed column(s): [' || coalesce(array_to_string(_wm_removed, ', '), '') || \ + ']. The table schema is fixed at first create; set on_schema_change=sync to auto-migrate, \ + or align the SELECT with the table.') AS VARCHAR) ELSE 'ok' END \ + FROM (SELECT {added} AS _wm_added, {removed} AS _wm_removed);", + tq = tq, + ) +} + +/// In-transaction guard for the keyed `merge` strategy: raises via DuckDB +/// `error(...)` when the source SELECT holds more than one row for the same +/// non-NULL `unique_key`. A keyed merge is delete-by-key + insert-all (it does +/// NOT deduplicate the source), so duplicate source keys would land every +/// duplicate row under one key — the exact silent double-write this guards +/// against. Erroring keeps the semantics explicit: the author must deduplicate +/// in the SELECT (or use `append`). NULL keys are excluded to match the delete's +/// `key IN (...)` scope, which never matches NULL. `unique_key` is embedded raw +/// in identifier position (matching the merge's own DELETE/IN) and doubled-quote +/// escaped where it lands inside the error string literal. +fn duplicate_source_key_guard_sql(sel_sql: &str, unique_key: &str) -> String { + let key_lit = unique_key.replace('\'', "''"); + format!( + "SELECT CASE WHEN _wm_dup_keys > 0 THEN CAST(error('managed materialize: keyed merge on \ + `{key_lit}` blocked — the source has ' || _wm_dup_keys || ' key value(s) with more than \ + one row. A keyed merge keeps one row per key and does not deduplicate; deduplicate in the \ + SELECT (e.g. QUALIFY row_number() OVER (PARTITION BY {key_lit} ORDER BY …) = 1) or use \ + `append`.') AS VARCHAR) ELSE 'ok' END FROM (SELECT count(*) AS _wm_dup_keys FROM (SELECT \ + {unique_key} FROM ({sel_sql}) WHERE {unique_key} IS NOT NULL GROUP BY {unique_key} HAVING \ + count(*) > 1));" + ) +} + +/// The `warn`-mode summary column: a `schema_drift` struct `{added, removed}` +/// when the SELECT drifted from the table, else NULL. Appended to the +/// materialize summary row so the executor can log it and fold it into the job +/// result without an extra round-trip. +fn schema_drift_summary_field( + sel_sql: &str, + target_qualified: &str, + partition_col: &str, +) -> String { + let (added, removed) = drift_lists(sel_sql, target_qualified, partition_col); + format!( + "(SELECT CASE WHEN coalesce(len(_wm_added), 0) + coalesce(len(_wm_removed), 0) > 0 \ + THEN {{'added': _wm_added, 'removed': _wm_removed}} END \ + FROM (SELECT {added} AS _wm_added, {removed} AS _wm_removed)) AS schema_drift" + ) +} + /// The read that captures the DuckLake snapshot id produced by the write, for /// the given attach alias (e.g. `_wm_target`). The worker runs this last and /// records the result into `materialized_partition`. @@ -646,6 +804,44 @@ pub fn snapshot_capture_sql(alias: &str) -> String { /// from the target ducklake's config and passes it in as `target_attach`. pub const TARGET_ALIAS: &str = "_wm_target"; +/// Structural role of one statement in a [`MaterializePlan`]. The public build +/// executes the plan verbatim, so the kinds are pure metadata there; they exist +/// so a downstream assembler (`pipeline_advanced::finalize_materialize_query`) +/// can reason about the plan without parsing SQL. +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +pub enum MaterializeStmtKind { + /// Pre-write statement: user setup, the target ATTACH, referenced-asset + /// ATTACHes. + Setup, + /// Write work against the target: bootstrap DDL, SCD2 temp-table captures, + /// the mutation itself, the `_current` view. + Write, + /// The `BEGIN TRANSACTION;` marker emitted by the strategy codegen. + TxnBegin, + /// The `COMMIT;` marker emitted by the strategy codegen. + TxnCommit, + /// The trailing one-row summary read (asset / rows / snapshot_id / + /// data_tests breakdown). + Summary, +} + +/// One planned statement: its structural role and the SQL text. +#[derive(Debug, Clone, PartialEq, Eq)] +pub struct MaterializeStmt { + pub kind: MaterializeStmtKind, + pub sql: String, +} + +/// The full ordered materialization plan [`build_wrap_blocks`] produces: +/// statements in execution order plus the compiled data-test checks (also +/// embedded in the summary statement's breakdown). Assembled into the final +/// statement list by `pipeline_advanced::finalize_materialize_query`. +#[derive(Debug, Clone, PartialEq, Eq)] +pub struct MaterializePlan { + pub stmts: Vec, + pub checks: Vec, +} + /// Assemble the full ordered statement list the DuckDB executor runs for a /// managed `// materialize` script. This is the single entry point the worker /// calls; it composes the already-tested pieces (classifier split → target @@ -659,6 +855,14 @@ pub const TARGET_ALIAS: &str = "_wm_target"; /// the full `/
` for the result summary. The trailing statement is /// a one-row summary read (asset / rows / snapshot_id) that is both the job's /// result (a useful preview) and what the worker records. +/// +/// Returns a [`MaterializePlan`] — the statements plus their structural role +/// and the compiled data-test checks — rather than raw SQL: the executor hands +/// the plan to `windmill_common::pipeline_advanced::finalize_materialize_query` +/// (which this crate cannot depend on), whose public-build implementation +/// assembles the statements verbatim. Everything this function produces runs +/// as-is on the public build; the plan's structure is metadata about it, not a +/// second mode. pub fn build_wrap_blocks( plan: &WrapPlan, target_attach: &str, @@ -668,18 +872,11 @@ pub fn build_wrap_blocks( partition_value_sql: &str, partitioned: bool, strategy: MaterializeStrategy, + on_schema_change: OnSchemaChange, tests: &[DataTestResolved], -) -> Result, String> { +) -> Result { let target_qualified = format!("{TARGET_ALIAS}.{target_table}"); let scd2 = matches!(strategy, MaterializeStrategy::Scd2 { .. }); - let cg = MaterializeCodegen { - target_qualified: &target_qualified, - select_sql: &plan.output, - partition_col, - partition_value_sql, - partitioned, - strategy, - }; let ctx = DataTestCtx { target_qualified: &target_qualified, asset_path, @@ -689,27 +886,60 @@ pub fn build_wrap_blocks( scd2, }; let test_sql = build_data_test_checks(tests, &ctx)?; - let mut blocks: Vec = Vec::new(); + let cg = MaterializeCodegen { + target_qualified: &target_qualified, + select_sql: &plan.output, + partition_col, + partition_value_sql, + partitioned, + strategy, + on_schema_change, + }; + // `warn` folds the post-write drift into the summary row (executor logs it + + // returns it) — only for the positional persist-and-mutate path, and only in + // `warn`: `fail`/`sync` guard the write itself, `ignore` is silent. + let drift_summary_select = + if on_schema_change == OnSchemaChange::Warn && cg.is_persist_and_mutate() { + Some(plan.output.as_str()) + } else { + None + }; + let mut stmts: Vec = Vec::new(); + let setup = |sql: String| MaterializeStmt { kind: MaterializeStmtKind::Setup, sql }; // Setup blocks come from the splitter with their `;` stripped — re-terminate // each so that when the executor re-joins and re-splits the assembled query, // adjacent statements (e.g. the user ATTACH and the synthetic target ATTACH) // don't merge into one malformed statement. - blocks.extend(plan.setup.iter().map(|s| terminate(s))); - blocks.push(target_attach.to_string()); + stmts.extend(plan.setup.iter().map(|s| setup(terminate(s)))); + stmts.push(setup(target_attach.to_string())); // Referenced-asset ATTACHes (relationships tests) — read-only, before the // write and the summary that probes them. - blocks.extend(test_sql.attaches); - blocks.extend(cg.statements()); + stmts.extend(test_sql.attaches.into_iter().map(setup)); + // Classify the codegen statements by matching the exact transaction-marker + // literals this module emits (`BEGIN TRANSACTION;` / `COMMIT;`); everything + // else the codegen produces is write work. + stmts.extend(cg.statements().into_iter().map(|sql| { + let kind = match sql.as_str() { + "BEGIN TRANSACTION;" => MaterializeStmtKind::TxnBegin, + "COMMIT;" => MaterializeStmtKind::TxnCommit, + _ => MaterializeStmtKind::Write, + }; + MaterializeStmt { kind, sql } + })); // The summary read carries the per-test breakdown (when any tests apply). - blocks.push(materialize_result_sql( - &target_qualified, - asset_path, - partition_col, - partition_value_sql, - partitioned, - &test_sql.checks, - )); - Ok(blocks) + stmts.push(MaterializeStmt { + kind: MaterializeStmtKind::Summary, + sql: materialize_result_sql( + &target_qualified, + asset_path, + partition_col, + partition_value_sql, + partitioned, + &test_sql.checks, + drift_summary_select, + ), + }); + Ok(MaterializePlan { stmts, checks: test_sql.checks }) } /// The trailing one-row summary the materialize run returns: the asset it @@ -723,6 +953,10 @@ pub fn materialize_result_sql( partition_value_sql: &str, partitioned: bool, checks: &[DataTestCheck], + // `on_schema_change=warn` on a persist-and-mutate strategy: the SELECT to + // diff against the (post-write) table for the `schema_drift` summary column. + // `None` ⇒ no drift column (every other mode / strategy). + drift_summary_select: Option<&str>, ) -> String { let (count_expr, partition_sel) = if partitioned { // Row count is the slice this run wrote (the partition); `partition` @@ -769,25 +1003,47 @@ pub fn materialize_result_sql( FROM (SELECT column_name, column_type, row_number() OVER () AS _wm_ord \ FROM (DESCRIBE SELECT * FROM {target_qualified}){partition_filter})) AS output_schema" ); + // `on_schema_change=warn`: fold the drift `{added, removed}` (or NULL) into + // the same row so the executor logs it + returns it with no extra probe. + let drift_col = match drift_summary_select { + Some(sel) => format!( + ", {}", + schema_drift_summary_field(sel, target_qualified, partition_col) + ), + None => String::new(), + }; let base_cols = format!( "'ducklake://{asset_path}' AS materialized, \ {partition_sel}{count_expr} AS rows, \ (SELECT max(snapshot_id) FROM ducklake_snapshots('{TARGET_ALIAS}')) AS snapshot_id, \ - {schema_capture}" + {schema_capture}{drift_col}" ); if checks.is_empty() { return format!("SELECT {base_cols};"); } - // Per-test breakdown. Each check's violating-count is computed once as a CTE - // column (`c0`, `c1`, …); the `data_tests` list-of-struct then references - // those columns — DuckDB rejects scalar subqueries *inside* a struct/list - // literal, hence the CTE. Names are single-quote-escaped. The result row - // carries the whole breakdown so the worker runs every test (no - // abort-on-first) and decides pass/fail itself. - let cte_cols = checks + // Per-test breakdown. Each check's one-row probe `(v, s)` becomes a CTE + // (`_wm_t0`, `_wm_t1`, …); `_wm_tr` cross-joins them (all one-row, so the + // join stays one row) and the `data_tests` list-of-struct references the + // flattened columns — DuckDB rejects scalar subqueries *inside* a + // struct/list literal, hence the CTE lift. Names are single-quote-escaped. + // The result row carries the whole breakdown so the worker runs every + // test (no abort-on-first) and decides pass/fail itself. + let probe_ctes = checks .iter() .enumerate() - .map(|(i, c)| format!("{} AS c{i}", c.violating)) + .map(|(i, c)| format!("_wm_t{i} AS ({})", c.probe)) + .collect::>() + .join(", "); + let tr_cols = checks + .iter() + .enumerate() + .map(|(i, _)| format!("_wm_t{i}.v AS c{i}, _wm_t{i}.s AS s{i}")) + .collect::>() + .join(", "); + let tr_from = checks + .iter() + .enumerate() + .map(|(i, _)| format!("_wm_t{i}")) .collect::>() .join(", "); let list_items = checks @@ -795,12 +1051,12 @@ pub fn materialize_result_sql( .enumerate() .map(|(i, c)| { let name = c.name.replace('\'', "''"); - format!("{{'test': '{name}', 'violating': c{i}}}") + format!("{{'test': '{name}', 'violating': c{i}, 'sample': s{i}}}") }) .collect::>() .join(", "); format!( - "WITH _wm_tr AS (SELECT {cte_cols}) \ + "WITH {probe_ctes}, _wm_tr AS (SELECT {tr_cols} FROM {tr_from}) \ SELECT {base_cols}, [{list_items}] AS data_tests FROM _wm_tr;" ) } @@ -821,20 +1077,21 @@ fn terminate(stmt: &str) -> String { // // A data test is the FIRST extensible annotation: the parser yields a // `DataTest` from a known vocabulary, and this module turns each into a -// *check* — a `(name, violating-row-count query)` pair — that runs against the -// freshly-materialized target after the write commits. The materialize summary -// query embeds every check's count in one `data_tests` column, so all tests -// run in a single pass (no abort-on-first) and the worker, not the SQL, -// decides pass/fail and reports the full per-test breakdown. +// *check* — a `(name, probe)` pair whose probe counts and samples the +// violating rows — that runs against the freshly-materialized target after +// the write commits. The materialize summary query embeds every check's +// outcome in one `data_tests` column, so all tests run in a single pass (no +// abort-on-first) and the worker, not the SQL, decides pass/fail and reports +// the full per-test breakdown. // -// The pattern is deliberately open: a verifier is just `(name, count query)`. -// Built-ins differ only in their count query; the `Custom` escape hatch -// supplies its own (a user SELECT returning the violating rows). A sibling -// annotation family (column-lineage) can emit its own checks through the same -// `push_check` shape rather than bolting on a parallel mechanism. See -// `docs/ducklake-materialization.md`. +// The pattern is deliberately open: a verifier is just `(name, violating-rows +// query)` handed to `push_check`. Built-ins differ only in their rows query; +// the `Custom` escape hatch supplies its own (a user SELECT returning the +// violating rows). A sibling annotation family (column-lineage) can emit its +// own checks through the same `push_check` shape rather than bolting on a +// parallel mechanism. See `docs/ducklake-materialization.md`. -use crate::asset_parser::{AssetKind, DataTest}; +use crate::asset_parser::{AssetKind, DataTest, OnSchemaChange}; /// Target context a data-test probe runs against — the materialized table and /// the partition slice (when partitioned, tests are scoped to the slice just @@ -867,19 +1124,31 @@ pub enum DataTestResolved { Custom { path: String, body: String }, } -/// One compiled data-test check: a human-readable `name` and a scalar SQL -/// expression (`violating`) yielding the number of rows that violate it (0 = -/// pass). The materialize summary query embeds every check's count so the -/// worker gets the whole breakdown in one result — all tests run (no -/// abort-on-first) and the worker, not the SQL, decides pass/fail. +/// One compiled data-test check: a human-readable `name` and a one-row probe +/// query yielding `(v, s)` — the violating-row count (0 = pass) and a bounded +/// `to_json` sample of the violating rows as a VARCHAR (NULL when there are +/// none, or when the serialized sample exceeds the size cap). The materialize +/// summary query embeds every check's outcome so the worker gets the whole +/// breakdown in one result — all tests run (no abort-on-first) and the +/// worker, not the SQL, decides pass/fail. The sample is decoration only: +/// enforcement reads `v`, never `s`. #[derive(Debug, Clone, PartialEq, Eq)] pub struct DataTestCheck { pub name: String, - /// Scalar subquery yielding the violating-row count, e.g. - /// `(SELECT count(*) AS v FROM (…))`. - pub violating: String, + /// One-row subquery `SELECT … AS v, … AS s FROM (…)` counting and + /// sampling the check's violating rows in a single scan. + pub probe: String, } +/// Row cap on a data-test sample. Bounded so the sample stays a debugging aid +/// (the full count is still reported); no ORDER BY on the violating rows, so +/// which rows land in the sample is nondeterministic. +const SAMPLE_MAX_ROWS: usize = 20; +/// Byte cap on one serialized sample. Oversized samples are dropped entirely +/// (NULL), never truncated — truncated JSON would fail parsing downstream +/// after paying the bytes anyway. +const SAMPLE_MAX_BYTES: usize = 51_200; + /// The SQL a set of data tests compiles to: referenced-asset `ATTACH` /// statements (resolved by the executor's ATTACH-transform pass) and the /// per-test checks, both in declaration order. @@ -943,11 +1212,62 @@ fn partition_scope(ctx: &DataTestCtx, prefix: &str, table_alias: Option<&str>) - format!("{prefix}{}", conds.join(" AND ")) } -// Record one check: its display `name` plus `count_query` (which yields a -// single-column violating-row count) wrapped as a scalar subquery. -fn push_check(out: &mut DataTestChecks, name: String, count_query: String) { - out.checks - .push(DataTestCheck { name, violating: format!("({count_query})") }); +// Record one check: its display `name` plus `rows_query`, the SELECT of its +// violating rows. The probe counts and samples those rows in one scan: +// `_wm_v` (the subquery alias) referenced as a column is the whole row as a +// struct; `to_json(...)::VARCHAR` keeps the sample a JSON *string* through +// the FFI — expanded rows would be visible to the executor's key-recursive +// `extract_i64(result, "rows"/"snapshot_id")` scans, which a user column of +// the same name could corrupt. `list()` over zero rows and an over-cap +// sample both degrade to NULL (`s` is optional by contract). +fn push_check(out: &mut DataTestChecks, name: String, rows_query: String) { + // `strlen` counts bytes (unlike `length`, characters) — the cap bounds + // payload size on the wire, so bytes are the right unit. + let probe = format!( + "SELECT v, CASE WHEN strlen(s_raw) <= {SAMPLE_MAX_BYTES} THEN s_raw END AS s \ + FROM (SELECT count(*) AS v, \ + to_json(list(_wm_v ORDER BY _wm_rn) FILTER (WHERE _wm_rn <= {SAMPLE_MAX_ROWS}))::VARCHAR AS s_raw \ + FROM (SELECT _wm_v, row_number() OVER () AS _wm_rn FROM ({rows_query}) _wm_v))" + ); + out.checks.push(DataTestCheck { name, probe }); +} + +// `SELECT *` for a sample rows-query, excluding the synthetic physical +// partition column on partitioned targets — it's Windmill's storage detail, +// not part of the producer's logical output (same rule as schema capture). +// `qualifier` scopes the star when the query aliases the target (`_wm_src`). +fn sample_star(ctx: &DataTestCtx, qualifier: Option<&str>) -> String { + let star = match qualifier { + Some(q) => format!("{q}.*"), + None => "*".to_string(), + }; + if ctx.partitioned { + format!("{star} EXCLUDE ({})", quote_ident(ctx.partition_col)) + } else { + star + } +} + +// Self-teaching tail appended to every malformed-custom-test error. It states +// the two rules that aren't documented or scaffolded anywhere else — the body +// is a single SELECT, and it reads the freshly-materialized target through the +// internal `_wm_target.
` alias — and doubles that alias into a copyable +// one-line example. `target_qualified` is already `_wm_target.
`. +fn custom_test_hint(target_qualified: &str) -> String { + format!( + "Write a single SELECT against `{target_qualified}` returning the offending rows, e.g. \ + `SELECT * FROM {target_qualified} WHERE ` — an empty result means the test \ + passes." + ) +} + +// Whether a custom-test statement reads the materialized target through the +// reserved `_wm_target` alias (the only handle the runtime attaches it under). +// SQL identifiers are case-insensitive, so match case-insensitively; +// `split_statements` has already stripped comments, so a match here is a real +// reference, not one buried in a comment. `TARGET_ALIAS` is lowercase. +fn references_target(stmt: &str) -> bool { + stmt.to_lowercase().contains(TARGET_ALIAS) } /// Compile resolved data tests into ATTACH statements + per-test checks for @@ -969,28 +1289,34 @@ pub fn build_data_test_checks( DataTestResolved::BuiltIn(DataTest::Unique { column }) => { let c = quote_ident(column); let scope = partition_scope(ctx, " AND ", None); + // The rows are the GROUP BY result — one `{value, count}` per + // duplicated key — so the count (number of duplicated values, + // not of rows) and the sample share one grain and can't + // contradict each other in the UI. let q = format!( - "SELECT count(*) AS v FROM (SELECT {c} FROM {t} WHERE {c} IS NOT NULL{scope} \ - GROUP BY {c} HAVING count(*) > 1)" + "SELECT {c} AS \"value\", count(*) AS \"count\" FROM {t} \ + WHERE {c} IS NOT NULL{scope} GROUP BY {c} HAVING count(*) > 1" ); push_check(&mut out, format!("unique({column})"), q); } DataTestResolved::BuiltIn(DataTest::NotNull { column }) => { let c = quote_ident(column); let scope = partition_scope(ctx, " AND ", None); - let q = format!("SELECT count(*) AS v FROM {t} WHERE {c} IS NULL{scope}"); + let star = sample_star(ctx, None); + let q = format!("SELECT {star} FROM {t} WHERE {c} IS NULL{scope}"); push_check(&mut out, format!("not_null({column})"), q); } DataTestResolved::BuiltIn(DataTest::AcceptedValues { column, values }) => { let c = quote_ident(column); let scope = partition_scope(ctx, " AND ", None); + let star = sample_star(ctx, None); let list = values .iter() .map(|v| quote_lit(v)) .collect::>() .join(", "); let q = format!( - "SELECT count(*) AS v FROM {t} WHERE {c} IS NOT NULL AND {c} NOT IN ({list}){scope}" + "SELECT {star} FROM {t} WHERE {c} IS NOT NULL AND {c} NOT IN ({list}){scope}" ); push_check(&mut out, format!("accepted_values({column})"), q); } @@ -1054,8 +1380,9 @@ pub fn build_data_test_checks( // segment so the dot stays a schema separator, not a literal. let rt = quote_qualified(ref_table); let scope = partition_scope(ctx, " AND ", Some("_wm_src")); + let star = sample_star(ctx, Some("_wm_src")); let q = format!( - "SELECT count(*) AS v FROM {t} _wm_src \ + "SELECT {star} FROM {t} _wm_src \ WHERE _wm_src.{c} IS NOT NULL{scope} \ AND NOT EXISTS (SELECT 1 FROM {alias}.{rt} _wm_ref \ WHERE _wm_ref.{rc} = _wm_src.{c})" @@ -1074,25 +1401,44 @@ pub fn build_data_test_checks( } DataTestResolved::Custom { path, body } => { // dbt singular-test convention: the body is a *single* SELECT - // (or CTE) returning the violating rows. It is embedded as a - // subquery (`FROM ()`), so a multi-statement body would - // produce invalid SQL — validate up front with an actionable - // error. It runs in the target's connection (can read - // `_wm_target` + the user's attaches); partition substitution is - // already applied by the worker. + // (or CTE) returning the violating rows, reading the + // freshly-materialized target through the internal `_wm_target` + // schema. It is embedded as a subquery (`FROM ()`), so a + // multi-statement or non-SELECT body would produce invalid SQL. + // Neither rule is documented or scaffolded elsewhere, so the + // errors are self-teaching: they name the exact violation and + // append a correct one-line example. It runs in the target's + // connection (can read `_wm_target` + the user's attaches); + // partition substitution is already applied by the worker. + let hint = custom_test_hint(t); let stmts = split_statements(body); if stmts.is_empty() { - return Err(format!("data_test custom `{path}`: empty test body")); + return Err(format!( + "data_test custom `{path}`: empty test body. {hint}" + )); } if stmts.len() > 1 { return Err(format!( - "data_test custom `{path}`: must be a single SELECT returning the \ - violating rows (found {} statements)", + "data_test custom `{path}`: a custom data test must be a single SELECT, \ + but found {} statements. {hint}", stmts.len() )); } - let q = format!("SELECT count(*) AS v FROM ({})", stmts[0]); - push_check(&mut out, format!("custom({path})"), q); + let stmt = &stmts[0]; + if classify_block(stmt) != BlockClass::Output { + return Err(format!( + "data_test custom `{path}`: a custom data test must be a single SELECT, \ + not a write or DDL statement. {hint}" + )); + } + if !references_target(stmt) { + return Err(format!( + "data_test custom `{path}`: the test never reads the freshly-materialized \ + target — reference it through the internal `{TARGET_ALIAS}` schema (as \ + `{t}`), not the table name on its own. {hint}" + )); + } + push_check(&mut out, format!("custom({path})"), stmt.to_string()); } } } @@ -1235,6 +1581,7 @@ mod tests { partition_value_sql: "'2026-06-19'", partitioned: true, strategy: MaterializeStrategy::Replace, + on_schema_change: OnSchemaChange::Warn, }; let st = cg.statements(); assert!(st[0].contains("CREATE TABLE IF NOT EXISTS _wm_target.orders_daily")); @@ -1260,6 +1607,7 @@ mod tests { partition_value_sql: "'2026-06-19'", partitioned: true, strategy: MaterializeStrategy::Merge { unique_key: "order_id".to_string() }, + on_schema_change: OnSchemaChange::Warn, }; let st = cg.statements(); // upsert = delete-by-key (partition-scoped) + insert — NO `MERGE INTO` @@ -1277,6 +1625,42 @@ mod tests { .any(|s| s.starts_with("INSERT INTO _wm_target.orders_daily SELECT *, '2026-06-19'"))); } + #[test] + fn codegen_merge_emits_duplicate_source_key_guard() { + // A keyed merge does not dedup its source, so codegen must emit a guard + // that fails the write when the SELECT has >1 row per key — otherwise + // duplicate source keys silently double-write. + let cg = MaterializeCodegen { + target_qualified: "_wm_target.orders_daily", + select_sql: "SELECT order_id, amount FROM dl.orders", + partition_col: "_wm_partition", + partition_value_sql: "'2026-06-19'", + partitioned: false, + strategy: MaterializeStrategy::Merge { unique_key: "order_id".to_string() }, + on_schema_change: OnSchemaChange::Warn, + }; + let st = cg.statements(); + let guard = st + .iter() + .find(|s| s.contains("_wm_dup_keys")) + .expect("duplicate-key guard stmt"); + // raises via error(), counts non-NULL keys appearing more than once + assert!(guard.contains("error('managed materialize: keyed merge on `order_id` blocked")); + assert!(guard.contains("GROUP BY order_id HAVING count(*) > 1")); + assert!(guard.contains("WHERE order_id IS NOT NULL")); + // must run before the mutations so a violation aborts before any write + let guard_pos = st.iter().position(|s| s.contains("_wm_dup_keys")).unwrap(); + let del_pos = st + .iter() + .position(|s| s.starts_with("DELETE FROM")) + .unwrap(); + let ins_pos = st + .iter() + .position(|s| s.starts_with("INSERT INTO")) + .unwrap(); + assert!(guard_pos < del_pos && guard_pos < ins_pos); + } + #[test] fn codegen_append_inserts_only() { let cg = MaterializeCodegen { @@ -1286,6 +1670,7 @@ mod tests { partition_value_sql: "'2026-06-19'", partitioned: true, strategy: MaterializeStrategy::Append, + on_schema_change: OnSchemaChange::Warn, }; let st = cg.statements(); assert!(st @@ -1307,6 +1692,7 @@ mod tests { partition_value_sql: "''", partitioned: false, strategy: MaterializeStrategy::Replace, + on_schema_change: OnSchemaChange::Warn, }; let st = cg.statements(); assert_eq!( @@ -1332,6 +1718,7 @@ mod tests { track: vec![], close_deleted: false, }, + on_schema_change: OnSchemaChange::Warn, }; let st = cg.statements(); // bootstrap adds the three SCD metadata columns @@ -1388,6 +1775,7 @@ mod tests { track: vec!["name".to_string()], close_deleted: false, }, + on_schema_change: OnSchemaChange::Warn, }; let st = cg.statements(); // only key + tracked cols are compared (addr changes don't rotate a version) @@ -1407,6 +1795,7 @@ mod tests { track: vec![], close_deleted: true, }, + on_schema_change: OnSchemaChange::Warn, }; let st = cg.statements(); // the deleted-key set: current keys absent from the snapshot, captured @@ -1443,6 +1832,169 @@ mod tests { assert!(begin < del_close && del_close < commit); } + fn persist_cg(strategy: MaterializeStrategy, osc: OnSchemaChange) -> Vec { + MaterializeCodegen { + target_qualified: "_wm_target.t", + select_sql: "SELECT a, c FROM dl.src", + partition_col: "_wm_partition", + partition_value_sql: "''", + partitioned: false, + strategy, + on_schema_change: osc, + } + .statements() + } + + #[test] + fn codegen_fail_emits_drift_guard_inside_txn_before_write() { + let st = persist_cg( + MaterializeStrategy::Merge { unique_key: "a".into() }, + OnSchemaChange::Fail, + ); + let begin = st.iter().position(|s| s == "BEGIN TRANSACTION;").unwrap(); + let guard = st + .iter() + .position(|s| s.contains("error(") && s.contains("on_schema_change=fail")) + .expect("fail emits a guard"); + let del = st + .iter() + .position(|s| s.starts_with("DELETE FROM")) + .unwrap(); + let insert = st + .iter() + .position(|s| s.starts_with("INSERT INTO")) + .unwrap(); + let commit = st.iter().position(|s| s == "COMMIT;").unwrap(); + // guard runs right after BEGIN and before any mutation, inside the txn + assert!(begin < guard && guard < del && del < insert && insert < commit); + // both CASE branches are VARCHAR so the planner can't fold error() away + assert!(st[guard].contains("CAST(error(")); + assert!(st[guard].contains("ELSE 'ok' END")); + // fail is positional (no BY NAME) and emits no sync sentinel + assert!(st[insert].starts_with("INSERT INTO _wm_target.t SELECT")); + assert!(!st.iter().any(|s| s == SYNC_ALTER_SENTINEL)); + } + + #[test] + fn fail_guard_drift_is_name_set_based_not_ordered() { + // Documents a deliberate boundary: the fail guard fires on the column + // SET difference (EXCEPT over column_name), so a pure reorder of + // same-named columns is NOT caught here — that is `sync`'s job (BY NAME). + // Keeping this name-set (not an ordered comparison) is what prevents the + // guard from false-positive-aborting a correctly-aligned write. If this + // ever moves to an ordered comparison, update the OnSchemaChange doc. + let guard = + schema_drift_guard_sql("SELECT b, a FROM dl.src", "_wm_target.t", "_wm_partition"); + assert!(guard.contains("column_name")); + assert!(guard.contains("EXCEPT")); + // No positional/ordinal comparison in the guard condition. + assert!(!guard.to_lowercase().contains("ordinal")); + assert!(!guard.to_lowercase().contains("row_number")); + } + + #[test] + fn codegen_sync_uses_by_name_and_sentinel() { + let st = persist_cg(MaterializeStrategy::Append, OnSchemaChange::Sync); + let begin = st.iter().position(|s| s == "BEGIN TRANSACTION;").unwrap(); + let sentinel = st.iter().position(|s| s == SYNC_ALTER_SENTINEL).unwrap(); + let insert = st + .iter() + .position(|s| s.starts_with("INSERT INTO")) + .unwrap(); + // the ALTER-injection slot is right after BEGIN, before the write + assert!(begin < sentinel && sentinel < insert); + // name-mapped insert (a positional insert would cross-wire ALTERed cols) + assert!(st[insert].starts_with("INSERT INTO _wm_target.t BY NAME SELECT")); + assert!(!st.iter().any(|s| s.contains("error("))); + } + + #[test] + fn codegen_ignore_and_warn_write_positionally_no_guard() { + for osc in [OnSchemaChange::Ignore, OnSchemaChange::Warn] { + let st = persist_cg(MaterializeStrategy::Append, osc); + assert!(!st.iter().any(|s| s.contains("error(")), "{osc:?}"); + assert!(!st.iter().any(|s| s == SYNC_ALTER_SENTINEL), "{osc:?}"); + let insert = st.iter().find(|s| s.starts_with("INSERT INTO")).unwrap(); + assert!( + insert.starts_with("INSERT INTO _wm_target.t SELECT"), + "{osc:?}" + ); + } + } + + #[test] + fn codegen_whole_table_replace_and_scd2_ignore_guardrail() { + // Whole-table replace (unpartitioned) and scd2 are not persist-and-mutate: + // fail/sync must not add a guard/sentinel/BY NAME there. + let repl = MaterializeCodegen { + target_qualified: "_wm_target.t", + select_sql: "SELECT a FROM dl.src", + partition_col: "_wm_partition", + partition_value_sql: "''", + partitioned: false, + strategy: MaterializeStrategy::Replace, + on_schema_change: OnSchemaChange::Fail, + }; + assert!(!repl.is_persist_and_mutate()); + let st = repl.statements(); + assert!(!st + .iter() + .any(|s| s.contains("error(") || s == SYNC_ALTER_SENTINEL)); + + let scd2 = MaterializeCodegen { + strategy: MaterializeStrategy::Scd2 { + key: "a".into(), + track: vec![], + close_deleted: false, + }, + on_schema_change: OnSchemaChange::Sync, + ..repl + }; + assert!(!scd2.is_persist_and_mutate()); + let st = scd2.statements(); + assert!(!st.iter().any(|s| s == SYNC_ALTER_SENTINEL)); + assert!(!st.iter().any(|s| s.contains(" BY NAME "))); + } + + #[test] + fn summary_schema_drift_field_only_for_warn_persist_and_mutate() { + // warn + persist-and-mutate ⇒ summary carries the drift column + let warn = plan_for_osc(MaterializeStrategy::Append, false, OnSchemaChange::Warn); + assert!(warn.stmts.last().unwrap().sql.contains("AS schema_drift")); + // ignore / fail / sync ⇒ no summary drift column (they guard the write) + for osc in [ + OnSchemaChange::Ignore, + OnSchemaChange::Fail, + OnSchemaChange::Sync, + ] { + let p = plan_for_osc(MaterializeStrategy::Append, false, osc); + assert!( + !p.stmts.last().unwrap().sql.contains("schema_drift"), + "{osc:?} must not emit the summary drift column" + ); + } + // whole-table replace + warn ⇒ not persist-and-mutate ⇒ no drift column + let repl = plan_for_osc(MaterializeStrategy::Replace, false, OnSchemaChange::Warn); + assert!(!repl.stmts.last().unwrap().sql.contains("schema_drift")); + } + + #[test] + fn fail_guard_is_write_kind_between_txn_markers() { + use MaterializeStmtKind::*; + let plan = plan_for_osc( + MaterializeStrategy::Merge { unique_key: "a".into() }, + false, + OnSchemaChange::Fail, + ); + let begin = kidx(&plan, |s| s.kind == TxnBegin); + let commit = kidx(&plan, |s| s.kind == TxnCommit); + let guard = kidx(&plan, |s| { + s.sql.contains("error(") && s.sql.contains("on_schema_change=fail") + }); + assert_eq!(plan.stmts[guard].kind, Write); + assert!(begin < guard && guard < commit); + } + #[test] fn snapshot_capture_targets_alias() { assert_eq!( @@ -1454,7 +2006,7 @@ mod tests { #[test] fn build_wrap_blocks_orders_setup_attach_codegen_snapshot() { let plan = ok("ATTACH 'ducklake://main' AS dl;\n SELECT a FROM dl.orders WHERE d = '{p}'"); - let blocks = build_wrap_blocks( + let blocks: Vec = build_wrap_blocks( &plan, "ATTACH 'ducklake:postgres:…' AS _wm_target (DATA_PATH 's3://b/p');", "orders_daily", @@ -1463,9 +2015,14 @@ mod tests { "'2026-06-19'", true, MaterializeStrategy::Replace, + OnSchemaChange::Warn, &[], ) - .unwrap(); + .unwrap() + .stmts + .into_iter() + .map(|s| s.sql) + .collect(); // setup block first, then the target ATTACH, then codegen, then result. assert!(blocks[0].starts_with("ATTACH 'ducklake://main' AS dl")); // every setup block must be `;`-terminated so re-splitting can't merge it @@ -1488,6 +2045,112 @@ mod tests { assert!(last.contains("ducklake_snapshots('_wm_target')")); } + // -- materialize plan structure ------------------------------------------ + + fn plan_for(strategy: MaterializeStrategy, partitioned: bool) -> MaterializePlan { + plan_for_osc(strategy, partitioned, OnSchemaChange::Warn) + } + + fn plan_for_osc( + strategy: MaterializeStrategy, + partitioned: bool, + on_schema_change: OnSchemaChange, + ) -> MaterializePlan { + let plan = ok("SELECT a, b FROM src"); + build_wrap_blocks( + &plan, + "ATTACH 'ducklake:…' AS _wm_target;", + "orders", + "main/orders", + "_wm_partition", + "'2026-06-19'", + partitioned, + strategy, + on_schema_change, + &[ + DataTestResolved::BuiltIn(DataTest::NotNull { column: "a".into() }), + DataTestResolved::BuiltIn(DataTest::Unique { column: "b".into() }), + ], + ) + .unwrap() + } + + fn kidx(plan: &MaterializePlan, pred: impl Fn(&MaterializeStmt) -> bool) -> usize { + plan.stmts + .iter() + .position(|s| pred(s)) + .expect("stmt present") + } + + #[test] + fn plan_tags_structure_and_carries_checks() { + use MaterializeStmtKind::*; + let plan = plan_for(MaterializeStrategy::Replace, true); + // leading statements are Setup, ending with the target ATTACH + assert!(plan.stmts[0].kind == Setup); + assert!(plan + .stmts + .iter() + .take_while(|s| s.kind == Setup) + .any(|s| s.sql.contains("_wm_target"))); + // txn markers are tagged, everything between them is Write + let begin = kidx(&plan, |s| s.kind == TxnBegin); + let commit = kidx(&plan, |s| s.kind == TxnCommit); + assert!(begin < commit); + assert!(plan.stmts[begin + 1..commit] + .iter() + .all(|s| s.kind == Write)); + // bootstrap DDL is Write work (it targets the table, not the session) + let bootstrap = kidx(&plan, |s| s.sql.starts_with("CREATE TABLE IF NOT EXISTS")); + assert_eq!(plan.stmts[bootstrap].kind, Write); + // summary is last and carries the breakdown; checks ride along + let last = plan.stmts.last().unwrap(); + assert_eq!(last.kind, Summary); + assert!(last.sql.contains("AS data_tests")); + assert_eq!(plan.checks.len(), 2); + assert!(plan.checks[0].name.contains("not_null(a)")); + } + + #[test] + fn plan_whole_table_replace_has_no_txn_markers() { + use MaterializeStmtKind::*; + let plan = plan_for(MaterializeStrategy::Replace, false); + assert!(!plan.stmts.iter().any(|s| s.kind == TxnBegin)); + assert!(!plan.stmts.iter().any(|s| s.kind == TxnCommit)); + assert_eq!( + plan.stmts.iter().filter(|s| s.kind == Write).count(), + 1, + "single atomic CREATE OR REPLACE" + ); + } + + #[test] + fn plan_scd2_captures_are_write_kind() { + use MaterializeStmtKind::*; + let plan = plan_for( + MaterializeStrategy::Scd2 { key: "a".into(), track: vec![], close_deleted: false }, + false, + ); + let capture = kidx(&plan, |s| s.sql.contains("TEMP TABLE _wm_scd2_changed")); + assert_eq!(plan.stmts[capture].kind, Write); + // no test declared ⇒ empty checks + let plain = ok("SELECT a FROM src"); + let no_tests = build_wrap_blocks( + &plain, + "ATTACH 'ducklake:…' AS _wm_target;", + "orders", + "main/orders", + "_wm_partition", + "''", + false, + MaterializeStrategy::Append, + OnSchemaChange::Warn, + &[], + ) + .unwrap(); + assert!(no_tests.checks.is_empty()); + } + // -- data tests --------------------------------------------------------- fn ctx_partitioned() -> DataTestCtx<'static> { @@ -1519,21 +2182,34 @@ mod tests { // short, asset-free names (the asset is shown once by the breakdown). assert_eq!(sql.checks[0].name, "unique(order_id)"); assert_eq!(sql.checks[1].name, "not_null(user_id)"); - // each `violating` is a scalar count subquery. + // each probe counts and samples the violating rows in one scan, with + // the size guard on the serialized sample. + for c in &sql.checks { + assert!(c + .probe + .starts_with("SELECT v, CASE WHEN strlen(s_raw) <= 51200 THEN s_raw END AS s")); + assert!(c.probe.contains("count(*) AS v")); + assert!(c.probe.contains("FILTER (WHERE _wm_rn <= 20)")); + } + // unique: groups non-null keys within the slice, having count>1; the + // sample is `{value, count}` pairs at the same grain as the count. assert!(sql.checks[0] - .violating - .starts_with("(SELECT count(*) AS v FROM")); - // unique: groups non-null keys within the slice, having count>1 - assert!(sql.checks[0] - .violating + .probe .contains("GROUP BY \"order_id\" HAVING count(*) > 1")); assert!(sql.checks[0] - .violating + .probe + .contains("SELECT \"order_id\" AS \"value\", count(*) AS \"count\"")); + assert!(sql.checks[0] + .probe .contains("\"order_id\" IS NOT NULL AND \"_wm_partition\" = '2026-06-19'")); - // not_null: null rows in the slice + // not_null: null rows in the slice; the sample excludes the synthetic + // partition column (storage detail, not producer output). assert!(sql.checks[1] - .violating + .probe .contains("WHERE \"user_id\" IS NULL AND \"_wm_partition\" = '2026-06-19'")); + assert!(sql.checks[1] + .probe + .contains("SELECT * EXCLUDE (\"_wm_partition\") FROM")); } #[test] @@ -1542,8 +2218,9 @@ mod tests { column: "id".into(), })]; let sql = build_data_test_checks(&tests, &ctx_unpartitioned()).unwrap(); - assert!(sql.checks[0].violating.contains("WHERE \"id\" IS NULL")); - assert!(!sql.checks[0].violating.contains("_wm_partition")); + assert!(sql.checks[0].probe.contains("WHERE \"id\" IS NULL")); + assert!(!sql.checks[0].probe.contains("_wm_partition")); + assert!(!sql.checks[0].probe.contains("EXCLUDE")); } #[test] @@ -1560,14 +2237,14 @@ mod tests { ]; let sql = build_data_test_checks(&tests, &ctx_scd2()).unwrap(); assert!(sql.checks[0] - .violating + .probe .contains("WHERE \"customer_id\" IS NOT NULL AND is_current")); assert!(sql.checks[1] - .violating + .probe .contains("WHERE \"tier\" IS NULL AND is_current")); - assert!(sql.checks[2].violating.contains("AND is_current")); + assert!(sql.checks[2].probe.contains("AND is_current")); // no partition scope leaks in (scd2 is unpartitioned in v1) - assert!(!sql.checks[0].violating.contains("_wm_partition")); + assert!(!sql.checks[0].probe.contains("_wm_partition")); } #[test] @@ -1577,10 +2254,8 @@ mod tests { values: vec!["paid".into(), "o'brien".into()], })]; let sql = build_data_test_checks(&tests, &ctx_unpartitioned()).unwrap(); - assert!(sql.checks[0] - .violating - .contains("NOT IN ('paid', 'o''brien')")); - assert!(sql.checks[0].violating.contains("\"status\" IS NOT NULL")); + assert!(sql.checks[0].probe.contains("NOT IN ('paid', 'o''brien')")); + assert!(sql.checks[0].probe.contains("\"status\" IS NOT NULL")); } #[test] @@ -1604,14 +2279,19 @@ mod tests { assert_eq!(sql.attaches.len(), 1, "same db attached once"); assert_eq!(sql.attaches[0], "ATTACH 'datatable://prod' AS _wm_ref_0;"); assert!(sql.checks[0] - .violating + .probe .contains("NOT EXISTS (SELECT 1 FROM _wm_ref_0.\"users\"")); assert!(sql.checks[1] - .violating + .probe .contains("NOT EXISTS (SELECT 1 FROM _wm_ref_0.\"buyers\"")); assert!(sql.checks[0] - .violating + .probe .contains("_wm_src.\"_wm_partition\" = '2026-06-19'")); + // sample rows come from the aliased target and drop the synthetic + // partition column. + assert!(sql.checks[0] + .probe + .contains("SELECT _wm_src.* EXCLUDE (\"_wm_partition\") FROM")); assert_eq!( sql.checks[0].name, "relationships(user_id -> prod/users.id)" @@ -1648,7 +2328,7 @@ mod tests { "same-lake ref must not ATTACH again" ); assert!(sql.checks[0] - .violating + .probe .contains("NOT EXISTS (SELECT 1 FROM _wm_target.\"users\"")); } @@ -1669,10 +2349,10 @@ mod tests { ); assert!( sql.checks[0] - .violating + .probe .contains("FROM _wm_ref_0.\"main\".\"dim_products\""), "schema-qualified target should be quoted per segment: {}", - sql.checks[0].violating + sql.checks[0].probe ); } @@ -1694,23 +2374,96 @@ mod tests { body: "SELECT * FROM _wm_target.orders WHERE amount < 0;".into(), }]; let sql = build_data_test_checks(&tests, &ctx_unpartitioned()).unwrap(); - // trailing ; stripped, wrapped as a count subquery - assert!(sql.checks[0].violating.contains( - "SELECT count(*) AS v FROM (SELECT * FROM _wm_target.orders WHERE amount < 0)" - )); + // trailing ; stripped, body embedded as the probe's rows query + assert!(sql.checks[0] + .probe + .contains("FROM (SELECT * FROM _wm_target.orders WHERE amount < 0) _wm_v")); + assert!(sql.checks[0].probe.contains("count(*) AS v")); assert_eq!(sql.checks[0].name, "custom(f/tests/amount)"); } #[test] fn data_test_custom_rejects_multi_statement_body() { // The body is embedded as a subquery, so a setup-then-SELECT body would - // produce invalid SQL — reject it up front with an actionable error. + // produce invalid SQL — reject it up front with a self-teaching error + // that names the violation and shows the correct single-SELECT shape. let tests = vec![DataTestResolved::Custom { path: "f/tests/amount".into(), body: "SET threads = 1; SELECT * FROM _wm_target.orders WHERE amount < 0".into(), }]; let err = build_data_test_checks(&tests, &ctx_unpartitioned()).unwrap_err(); assert!(err.contains("single SELECT"), "unexpected error: {err}"); + assert!( + err.contains("found 2 statements"), + "unexpected error: {err}" + ); + // the copyable example points at the internal target alias. + assert!( + err.contains("SELECT * FROM _wm_target.orders WHERE "), + "unexpected error: {err}" + ); + } + + #[test] + fn data_test_custom_rejects_non_select_body() { + // A write/DDL body can't be embedded as `FROM ()`; the error must + // say so and teach the single-SELECT convention. + let tests = vec![DataTestResolved::Custom { + path: "f/tests/amount".into(), + body: "DELETE FROM _wm_target.orders WHERE amount < 0".into(), + }]; + let err = build_data_test_checks(&tests, &ctx_unpartitioned()).unwrap_err(); + assert!(err.contains("single SELECT"), "unexpected error: {err}"); + assert!( + err.contains("SELECT * FROM _wm_target.orders WHERE "), + "unexpected error: {err}" + ); + } + + #[test] + fn data_test_custom_rejects_wrong_target_alias() { + // Referencing the target by its bare table name (not `_wm_target.
`) + // is the most common custom-test mistake — the runtime only attaches the + // freshly-materialized target under `_wm_target`, so the query would fail + // at runtime. Catch it at codegen with a self-teaching error. + let tests = vec![DataTestResolved::Custom { + path: "f/tests/amount".into(), + body: "SELECT * FROM orders WHERE amount < 0".into(), + }]; + let err = build_data_test_checks(&tests, &ctx_unpartitioned()).unwrap_err(); + assert!(err.contains("_wm_target"), "unexpected error: {err}"); + assert!( + err.contains("SELECT * FROM _wm_target.orders WHERE "), + "unexpected error: {err}" + ); + } + + #[test] + fn data_test_custom_accepts_from_first_and_uppercased_alias() { + // DuckDB's FROM-first syntax is a valid Output, and the alias match is + // case-insensitive (SQL identifiers are), so this passes. + let tests = vec![DataTestResolved::Custom { + path: "f/tests/amount".into(), + body: "FROM _WM_TARGET.orders WHERE amount < 0".into(), + }]; + let sql = build_data_test_checks(&tests, &ctx_unpartitioned()).unwrap(); + assert!(sql.checks[0] + .probe + .contains("FROM (FROM _WM_TARGET.orders WHERE amount < 0) _wm_v")); + } + + #[test] + fn data_test_custom_empty_body_teaches_shape() { + let tests = vec![DataTestResolved::Custom { + path: "f/tests/amount".into(), + body: " \n-- just a comment\n".into(), + }]; + let err = build_data_test_checks(&tests, &ctx_unpartitioned()).unwrap_err(); + assert!(err.contains("empty test body"), "unexpected error: {err}"); + assert!( + err.contains("SELECT * FROM _wm_target.orders WHERE "), + "unexpected error: {err}" + ); } #[test] @@ -1724,14 +2477,8 @@ mod tests { #[test] fn materialize_result_sql_embeds_data_tests_breakdown() { let checks = vec![ - DataTestCheck { - name: "unique(order_id)".into(), - violating: "(SELECT count(*) AS v FROM q0)".into(), - }, - DataTestCheck { - name: "custom(f/t)".into(), - violating: "(SELECT count(*) AS v FROM q1)".into(), - }, + DataTestCheck { name: "unique(order_id)".into(), probe: "SELECT v, s FROM q0".into() }, + DataTestCheck { name: "custom(f/t)".into(), probe: "SELECT v, s FROM q1".into() }, ]; let sql = materialize_result_sql( "_wm_target.orders", @@ -1740,11 +2487,19 @@ mod tests { "'2026-06-19'", false, &checks, + None, + ); + // each probe runs once as a one-row CTE; _wm_tr cross-joins them and + // the list-of-struct references the flattened count/sample columns. + assert!(sql.starts_with( + "WITH _wm_t0 AS (SELECT v, s FROM q0), _wm_t1 AS (SELECT v, s FROM q1), \ + _wm_tr AS (SELECT _wm_t0.v AS c0, _wm_t0.s AS s0, _wm_t1.v AS c1, _wm_t1.s AS s1 \ + FROM _wm_t0, _wm_t1)" + )); + assert!(sql.contains("[{'test': 'unique(order_id)', 'violating': c0, 'sample': s0}, ")); + assert!( + sql.contains("{'test': 'custom(f/t)', 'violating': c1, 'sample': s1}] AS data_tests") ); - // counts computed once in a CTE, referenced by the list-of-struct. - assert!(sql.starts_with("WITH _wm_tr AS (SELECT (SELECT count(*) AS v FROM q0) AS c0,")); - assert!(sql.contains("[{'test': 'unique(order_id)', 'violating': c0}, ")); - assert!(sql.contains("{'test': 'custom(f/t)', 'violating': c1}] AS data_tests")); assert!(sql.contains("FROM _wm_tr;")); // no tests -> plain summary, no CTE / data_tests column. let plain = materialize_result_sql( @@ -1754,6 +2509,7 @@ mod tests { "'x'", false, &[], + None, ); assert!(plain.starts_with("SELECT 'ducklake://analytics/orders' AS materialized")); assert!(!plain.contains("data_tests")); @@ -1780,6 +2536,7 @@ mod tests { "'2026-06-19'", true, &[], + None, ); assert!(sql.contains( "FROM (DESCRIBE SELECT * FROM _wm_target.orders_daily) \ diff --git a/backend/parsers/windmill-parser/tests/fixtures/pipeline_annotations.json b/backend/parsers/windmill-parser/tests/fixtures/pipeline_annotations.json index f7977239c1..cabfe20ddf 100644 --- a/backend/parsers/windmill-parser/tests/fixtures/pipeline_annotations.json +++ b/backend/parsers/windmill-parser/tests/fixtures/pipeline_annotations.json @@ -310,6 +310,43 @@ } } }, + { + "name": "materialize on_schema_change=ignore opt", + "code": "// pipeline\n// materialize ducklake://analytics/orders on_schema_change=ignore\nSELECT 1;", + "expected": { + "in_pipeline": true, + "asset_triggers": [], + "native_triggers": [], + "partition": null, + "freshness": null, + "tag": null, + "retry": null, + "materialize": { + "target_kind": "ducklake", + "target_path": "analytics/orders", + "on_schema_change": "ignore" + } + } + }, + { + "name": "materialize on_schema_change unknown value keeps warn default (fail-safe)", + "code": "// materialize ducklake://analytics/orders key=id on_schema_change=bogus\nSELECT 1;", + "expected": { + "in_pipeline": false, + "asset_triggers": [], + "native_triggers": [], + "partition": null, + "freshness": null, + "tag": null, + "retry": null, + "materialize": { + "target_kind": "ducklake", + "target_path": "analytics/orders", + "unique_key": "id", + "on_schema_change": "warn" + } + } + }, { "name": "materialize key without history is plain merge (SCD1, not scd2)", "code": "// materialize ducklake://analytics/dim key=id\nSELECT 1;", @@ -702,5 +739,35 @@ "tag": null, "retry": null } + }, + { + "name": "s3 triple-slash default-storage trigger canonicalizes to bare key", + "code": "// pipeline\n// on s3:///exports/x\nexport function main() {}", + "expected": { + "in_pipeline": true, + "asset_triggers": [ + "s3object:exports/x" + ], + "native_triggers": [], + "partition": null, + "freshness": null, + "tag": null, + "retry": null + } + }, + { + "name": "s3 quad-slash trigger strips all leading slashes to the bare key", + "code": "// pipeline\n// on s3:////x\nexport function main() {}", + "expected": { + "in_pipeline": true, + "asset_triggers": [ + "s3object:x" + ], + "native_triggers": [], + "partition": null, + "freshness": null, + "tag": null, + "retry": null + } } ] diff --git a/backend/parsers/windmill-parser/tests/pipeline_annotations_parity.rs b/backend/parsers/windmill-parser/tests/pipeline_annotations_parity.rs index 2df4acd2e4..7adb07b55c 100644 --- a/backend/parsers/windmill-parser/tests/pipeline_annotations_parity.rs +++ b/backend/parsers/windmill-parser/tests/pipeline_annotations_parity.rs @@ -71,6 +71,13 @@ struct ExpectedMaterialize { track: Vec, #[serde(default)] close_deleted: bool, + // "warn" | "ignore"; absent === "warn" (the default). + #[serde(default = "default_on_schema_change")] + on_schema_change: String, +} + +fn default_on_schema_change() -> String { + "warn".to_string() } #[derive(Deserialize)] @@ -209,6 +216,16 @@ fn pipeline_annotation_fixtures_match() { m.close_deleted, e.close_deleted, "{ctx}: materialize close_deleted" ); + let osc = match m.on_schema_change { + windmill_parser::asset_parser::OnSchemaChange::Warn => "warn", + windmill_parser::asset_parser::OnSchemaChange::Ignore => "ignore", + windmill_parser::asset_parser::OnSchemaChange::Fail => "fail", + windmill_parser::asset_parser::OnSchemaChange::Sync => "sync", + }; + assert_eq!( + osc, e.on_schema_change, + "{ctx}: materialize on_schema_change" + ); } (got, want) => panic!( "{ctx}: materialize mismatch — got {:?}, want present={}", diff --git a/backend/src/monitor.rs b/backend/src/monitor.rs index 817887eaad..60b4024c0a 100644 --- a/backend/src/monitor.rs +++ b/backend/src/monitor.rs @@ -178,6 +178,14 @@ lazy_static::lazy_static! { .and_then(|x| x.parse::().ok()) .unwrap_or(false); + // Ops kill switch for the pipeline freshness watchdog (a background + // pusher — being able to stop it without a redeploy matters more than + // for read-only monitors). + pub static ref DISABLE_FRESHNESS_WATCHDOG: bool = std::env::var("DISABLE_FRESHNESS_WATCHDOG") + .ok() + .and_then(|x| x.parse::().ok()) + .unwrap_or(false); + pub static ref WORKERS_NAMES: Arc>> = Arc::new(RwLock::new(Vec::new())); static ref QUEUE_COUNT_TAGS: Arc>> = Arc::new(RwLock::new(Vec::new())); @@ -2994,6 +3002,28 @@ pub async fn monitor_db( } }; + // run every ~60s (2 iterations * 30s). Enterprise feature: the active + // `// freshness` backstop lives in windmill-queue's `freshness_watchdog` + // (`private`); OSS gets a no-op stub. Runtime-gated on an Enterprise + // license like the audit export above. Safe on concurrent servers — the + // watchdog claims per-script state rows atomically before pushing. + let pipeline_freshness_watchdog_f = async { + if server_mode + && !*DISABLE_FRESHNESS_WATCHDOG + && iteration.is_some() + && iteration.as_ref().unwrap().should_run(2) + { + if let Some(db) = conn.as_sql() { + if matches!( + windmill_common::ee_oss::get_license_plan().await, + windmill_common::ee_oss::LicensePlan::Enterprise + ) { + windmill_queue::freshness_watchdog::tick(db).await; + } + } + } + }; + join!( expired_items_f, zombie_jobs_f, @@ -3021,6 +3051,7 @@ pub async fn monitor_db( manage_audit_partitions_f, export_audit_logs_to_object_store_f, cleanup_scheduled_job_deletions_f, + pipeline_freshness_watchdog_f, ); } diff --git a/backend/tests/freshness_watchdog.rs b/backend/tests/freshness_watchdog.rs new file mode 100644 index 0000000000..17851d0a1e --- /dev/null +++ b/backend/tests/freshness_watchdog.rs @@ -0,0 +1,294 @@ +//! End-to-end tests for the pipeline freshness watchdog (Enterprise). +//! +//! `windmill_queue::freshness_watchdog::tick` is called directly against +//! seeded `script` / `v2_job(_completed)` rows — no worker or API server is +//! needed, since the watchdog's job ends at the push (the pushed job sitting +//! in `v2_job_queue` is itself part of the assertions). Covers: staleness on +//! never-ran and aged-out members, the fresh short-circuit + state reset, +//! the in-flight suppression, the backoff claim, and the skip rules +//! (partitioned, malformed window, non-pipeline scripts). + +#![cfg(feature = "private")] + +use sqlx::{Pool, Postgres}; +use windmill_queue::freshness_watchdog::tick; +use windmill_test_utils::initialize_tracing; + +const WS: &str = "test-workspace"; +const PATH: &str = "u/test-user/freshness_producer"; + +/// Seed a deployed pipeline-member script. Mirrors the deploy path's output: +/// `auto_kind = 'pipeline'`, empty (non-NULL) lock so run-by-path resolution +/// treats it as deployed, hash derived from path+content for uniqueness. +async fn seed_pipeline_script( + db: &Pool, + path: &str, + content: &str, +) -> anyhow::Result<()> { + let mut h = 0i64; + for b in path.bytes().chain(content.bytes()) { + h = h.wrapping_mul(31).wrapping_add(b as i64); + } + sqlx::query( + r#"INSERT INTO script (workspace_id, hash, path, summary, description, content, + created_by, language, tag, lock, auto_kind) + VALUES ($1, $2, $3, '', '', $4, 'test-user', 'bash'::script_lang, 'bash', '', 'pipeline') + ON CONFLICT DO NOTHING"#, + ) + .bind(WS) + .bind(h) + .bind(path) + .bind(content) + .execute(db) + .await?; + // Process-global deployed-script caches are keyed by (workspace, path) / + // (workspace, hash) and would leak between #[sqlx::test] isolated DBs + // that reuse both — resolve everything from this test's own DB. + windmill_common::DEPLOYED_SCRIPT_CACHE_DISABLED + .store(true, std::sync::atomic::Ordering::Relaxed); + Ok(()) +} + +/// Seed a completed root run of `path` that finished `age_s` seconds ago. +async fn seed_completed_run( + db: &Pool, + path: &str, + age_s: i64, + success: bool, +) -> anyhow::Result<()> { + let id = uuid::Uuid::new_v4(); + sqlx::query( + r#"INSERT INTO v2_job (id, workspace_id, runnable_path, kind, created_at, + created_by, permissioned_as, permissioned_as_email, tag) + VALUES ($1, $2, $3, 'script'::job_kind, + now() - ($4::bigint::text || ' seconds')::interval, + 'test-user', 'u/test-user', 'test@windmill.dev', 'bash')"#, + ) + .bind(id) + .bind(WS) + .bind(path) + .bind(age_s) + .execute(db) + .await?; + sqlx::query( + r#"INSERT INTO v2_job_completed (id, workspace_id, duration_ms, status, started_at, completed_at) + VALUES ($1, $2, 0, CASE WHEN $3 THEN 'success'::job_status ELSE 'failure'::job_status END, + now() - ($4::bigint::text || ' seconds')::interval, + now() - ($4::bigint::text || ' seconds')::interval)"#, + ) + .bind(id) + .bind(WS) + .bind(success) + .bind(age_s) + .execute(db) + .await?; + Ok(()) +} + +/// Jobs the watchdog pushed: (path, created_by, args) rows attributed to +/// `trigger_kind = 'freshness'`. +async fn fetch_pushed( + db: &Pool, +) -> anyhow::Result)>> { + let rows = sqlx::query!( + r#"SELECT runnable_path AS "runnable_path!", created_by AS "created_by!", + args AS "args: sqlx::types::Json" + FROM v2_job + WHERE workspace_id = $1 AND trigger_kind = 'freshness' + ORDER BY created_at"#, + WS, + ) + .fetch_all(db) + .await?; + Ok(rows + .into_iter() + .map(|r| (r.runnable_path, r.created_by, r.args.map(|a| a.0))) + .collect()) +} + +async fn state_row(db: &Pool, path: &str) -> anyhow::Result> { + Ok(sqlx::query_scalar!( + "SELECT attempts FROM pipeline_freshness_state WHERE workspace_id = $1 AND script_path = $2", + WS, + path, + ) + .fetch_optional(db) + .await?) +} + +#[sqlx::test(fixtures("base"))] +async fn never_ran_member_is_pushed_once(db: Pool) -> anyhow::Result<()> { + initialize_tracing().await; + seed_pipeline_script(&db, PATH, "# pipeline\n# freshness 30s\necho hi\n").await?; + + tick(&db).await; + + let pushed = fetch_pushed(&db).await?; + assert_eq!(pushed.len(), 1, "one watchdog push expected"); + let (path, created_by, args) = &pushed[0]; + assert_eq!(path, PATH); + assert_eq!(created_by, &format!("freshness-{PATH}")); + let args = args.as_ref().expect("args recorded"); + assert_eq!( + args.get("_wmill_skip_asset_dispatch"), + Some(&serde_json::json!(true)), + "watchdog runs must not re-fire the cascade" + ); + assert_eq!( + args.pointer("/trigger/kind"), + Some(&serde_json::json!("freshness")) + ); + assert_eq!(state_row(&db, PATH).await?, Some(1), "claim row recorded"); + + // Second tick: the pushed job is queued-and-due, so the in-flight guard + // suppresses a duplicate regardless of backoff. + tick(&db).await; + assert_eq!( + fetch_pushed(&db).await?.len(), + 1, + "no duplicate while queued" + ); + + // Simulate the queued job vanishing without a completion: the backoff + // claim (next_attempt_at in the future) now carries the suppression. + sqlx::query!("DELETE FROM v2_job_queue WHERE workspace_id = $1", WS) + .execute(&db) + .await?; + tick(&db).await; + assert_eq!(fetch_pushed(&db).await?.len(), 1, "backoff holds the retry"); + + // Force the backoff window open: the watchdog retries and escalates. + sqlx::query!( + "UPDATE pipeline_freshness_state SET next_attempt_at = now() - interval '1 second' + WHERE workspace_id = $1 AND script_path = $2", + WS, + PATH, + ) + .execute(&db) + .await?; + tick(&db).await; + assert_eq!(fetch_pushed(&db).await?.len(), 2, "due retry pushed"); + assert_eq!(state_row(&db, PATH).await?, Some(2), "attempts escalated"); + Ok(()) +} + +#[sqlx::test(fixtures("base"))] +async fn fresh_member_is_skipped_and_state_reset(db: Pool) -> anyhow::Result<()> { + initialize_tracing().await; + seed_pipeline_script(&db, PATH, "# pipeline\n# freshness 1h\necho hi\n").await?; + seed_completed_run(&db, PATH, 10, true).await?; + // Leftover backoff row from an earlier staleness episode. + sqlx::query!( + "INSERT INTO pipeline_freshness_state (workspace_id, script_path, attempts) VALUES ($1, $2, 3)", + WS, + PATH, + ) + .execute(&db) + .await?; + + tick(&db).await; + + assert!( + fetch_pushed(&db).await?.is_empty(), + "fresh member not pushed" + ); + assert_eq!(state_row(&db, PATH).await?, None, "backoff reset on fresh"); + Ok(()) +} + +#[sqlx::test(fixtures("base"))] +async fn aged_out_member_is_pushed(db: Pool) -> anyhow::Result<()> { + initialize_tracing().await; + seed_pipeline_script(&db, PATH, "# pipeline\n# freshness 1h\necho hi\n").await?; + // Old success outside the window + a recent failure: still stale. + seed_completed_run(&db, PATH, 7200, true).await?; + seed_completed_run(&db, PATH, 60, false).await?; + + tick(&db).await; + + assert_eq!(fetch_pushed(&db).await?.len(), 1, "aged-out member pushed"); + Ok(()) +} + +#[sqlx::test(fixtures("base"))] +async fn partitioned_malformed_and_plain_members_are_skipped( + db: Pool, +) -> anyhow::Result<()> { + initialize_tracing().await; + // Partitioned: freshness means partition-gap detection, out of scope. + seed_pipeline_script( + &db, + "u/test-user/partitioned", + "# pipeline\n# partitioned daily\n# freshness 1h\necho hi\n", + ) + .await?; + // Malformed window: fails safe to unwatched. + seed_pipeline_script( + &db, + "u/test-user/malformed", + "# pipeline\n# freshness soonish\necho hi\n", + ) + .await?; + // Freshness only in prose (parser must reject; ILIKE prefilter passes). + seed_pipeline_script( + &db, + "u/test-user/prose", + "# pipeline\n# ensure freshness of data below\necho hi\n", + ) + .await?; + + tick(&db).await; + + assert!(fetch_pushed(&db).await?.is_empty(), "no member is watched"); + let rows = sqlx::query_scalar!( + r#"SELECT COUNT(*) AS "count!" FROM pipeline_freshness_state WHERE workspace_id = $1"#, + WS, + ) + .fetch_one(&db) + .await?; + assert_eq!(rows, 0, "no state rows for unwatched members"); + Ok(()) +} + +#[sqlx::test(fixtures("base"))] +async fn archived_workspace_is_not_resurrected(db: Pool) -> anyhow::Result<()> { + initialize_tracing().await; + // Workspace archival stops all execution but keeps script rows for + // unarchival — the watchdog must not keep pushing runs there. + seed_pipeline_script(&db, PATH, "# pipeline\n# freshness 30s\necho hi\n").await?; + sqlx::query!("UPDATE workspace SET deleted = true WHERE id = $1", WS) + .execute(&db) + .await?; + + tick(&db).await; + + assert!( + fetch_pushed(&db).await?.is_empty(), + "no pushes into an archived workspace" + ); + assert_eq!( + state_row(&db, PATH).await?, + None, + "no state bookkeeping either" + ); + Ok(()) +} + +#[sqlx::test(fixtures("base"))] +async fn state_of_unwatched_member_is_cleaned_up(db: Pool) -> anyhow::Result<()> { + initialize_tracing().await; + // A stale-bookkeeping row whose script no longer declares freshness + // (e.g. annotation removed and redeployed) must not survive the sweep. + sqlx::query!( + "INSERT INTO pipeline_freshness_state (workspace_id, script_path) VALUES ($1, $2)", + WS, + "u/test-user/gone", + ) + .execute(&db) + .await?; + + tick(&db).await; + + assert_eq!(state_row(&db, "u/test-user/gone").await?, None); + Ok(()) +} diff --git a/backend/windmill-ai/src/ai_bedrock.rs b/backend/windmill-ai/src/ai_bedrock.rs index 8aa9addcae..02ab702d31 100644 --- a/backend/windmill-ai/src/ai_bedrock.rs +++ b/backend/windmill-ai/src/ai_bedrock.rs @@ -857,22 +857,91 @@ pub fn bedrock_stream_event_is_block_stop(event: &ConverseStreamOutput) -> bool matches!(event, ConverseStreamOutput::ContentBlockStop(_)) } -/// Convert accumulated streaming tool calls to OpenAI format -pub fn streaming_tool_calls_to_openai(tool_calls: Vec) -> Vec { +/// Convert accumulated streaming tool calls to OpenAI format. +/// +/// When thinking is enabled, `reasoning` carries the turn's Claude reasoning +/// block; it is attached to the first tool call so the next request replays it +/// before `toolUse` (required by Claude when thinking is enabled — see +/// [`convert_message`]). When `None` (thinking off), tool calls carry no extra +/// content, byte-identical to the pre-feature output. +pub fn streaming_tool_calls_to_openai( + tool_calls: Vec, + reasoning: Option, +) -> Vec { + let mut reasoning = reasoning; tool_calls .into_iter() - .map(|tc| OpenAIToolCall { - id: tc.id, - function: OpenAIFunction { name: tc.name, arguments: tc.arguments }, - r#type: FUNCTION_TYPE.to_string(), - // Worker agent requests never enable thinking, so there is no - // reasoning block to round-trip here (the chat proxy path does — - // see providers/bedrock.rs). - extra_content: None, + .map(|tc| { + // `take` so only the first tool call carries the reasoning block. + let extra_content = reasoning.take().map(|block| crate::ai_types::ExtraContent { + bedrock: Some(block), + ..Default::default() + }); + OpenAIToolCall { + id: tc.id, + function: OpenAIFunction { name: tc.name, arguments: tc.arguments }, + r#type: FUNCTION_TYPE.to_string(), + extra_content, + } }) .collect() } +/// Fold a Bedrock `ReasoningContent` stream delta into an accumulating reasoning +/// block (text + signature, or redacted bytes). Returns the readable text delta +/// when the event carried one, so the caller can stream it as reasoning content. +/// Mirrors the proxy path's accumulation in `providers/bedrock.rs`. +pub fn bedrock_stream_event_to_reasoning_delta( + event: &ConverseStreamOutput, + reasoning: &mut Option, +) -> Option { + let ConverseStreamOutput::ContentBlockDelta(delta_event) = event else { + return None; + }; + let aws_sdk_bedrockruntime::types::ContentBlockDelta::ReasoningContent(rc) = + delta_event.delta()? + else { + return None; + }; + + let entry = reasoning.get_or_insert_with(Default::default); + match rc { + aws_sdk_bedrockruntime::types::ReasoningContentBlockDelta::Text(text) => { + entry + .reasoning_text + .get_or_insert_with(String::new) + .push_str(text); + Some(text.clone()) + } + aws_sdk_bedrockruntime::types::ReasoningContentBlockDelta::Signature(signature) => { + entry + .signature + .get_or_insert_with(String::new) + .push_str(signature); + None + } + aws_sdk_bedrockruntime::types::ReasoningContentBlockDelta::RedactedContent(blob) => { + // Base64 of concatenated fragments != concatenated base64 fragments, + // so accumulate raw bytes and re-encode. + let mut bytes = entry + .redacted_content + .as_deref() + .and_then(|existing| { + base64::Engine::decode(&base64::engine::general_purpose::STANDARD, existing) + .ok() + }) + .unwrap_or_default(); + bytes.extend_from_slice(blob.as_ref()); + entry.redacted_content = Some(base64::Engine::encode( + &base64::engine::general_purpose::STANDARD, + bytes, + )); + None + } + _ => None, + } +} + // ============================================================================ // Tool Configuration Builder // ============================================================================ @@ -1028,6 +1097,101 @@ mod tests { assert!(matches!(&content[1], ContentBlock::ToolUse(_))); } + #[test] + fn streaming_tool_calls_to_openai_attaches_reasoning_to_first_call_only() { + let calls = vec![ + StreamingToolCall { + id: "call_1".to_string(), + name: "a".to_string(), + arguments: "{}".to_string(), + }, + StreamingToolCall { + id: "call_2".to_string(), + name: "b".to_string(), + arguments: "{}".to_string(), + }, + ]; + let reasoning = crate::ai_types::BedrockExtraContent { + reasoning_text: Some("thinking".to_string()), + signature: Some("sig".to_string()), + redacted_content: None, + }; + + let openai = streaming_tool_calls_to_openai(calls, Some(reasoning)); + + let with_reasoning = openai + .iter() + .filter(|tc| { + tc.extra_content + .as_ref() + .and_then(|e| e.bedrock.as_ref()) + .is_some() + }) + .count(); + assert_eq!( + with_reasoning, 1, + "exactly one tool call carries the reasoning block" + ); + let block = openai[0] + .extra_content + .as_ref() + .and_then(|e| e.bedrock.as_ref()) + .expect("first tool call carries reasoning"); + assert_eq!(block.reasoning_text.as_deref(), Some("thinking")); + assert_eq!(block.signature.as_deref(), Some("sig")); + } + + #[test] + fn streaming_tool_calls_to_openai_without_reasoning_has_no_extra_content() { + let calls = vec![StreamingToolCall { + id: "call_1".to_string(), + name: "a".to_string(), + arguments: "{}".to_string(), + }]; + + let openai = streaming_tool_calls_to_openai(calls, None); + + assert!(openai[0].extra_content.is_none()); + } + + #[test] + fn bedrock_reasoning_delta_accumulates_text_and_signature() { + use aws_sdk_bedrockruntime::types::{ + ContentBlockDelta, ContentBlockDeltaEvent, ConverseStreamOutput, + ReasoningContentBlockDelta, + }; + + let mut reasoning = None; + let text_event = ConverseStreamOutput::ContentBlockDelta( + ContentBlockDeltaEvent::builder() + .content_block_index(0) + .delta(ContentBlockDelta::ReasoningContent( + ReasoningContentBlockDelta::Text("let me ".to_string()), + )) + .build() + .unwrap(), + ); + assert_eq!( + bedrock_stream_event_to_reasoning_delta(&text_event, &mut reasoning).as_deref(), + Some("let me ") + ); + + let sig_event = ConverseStreamOutput::ContentBlockDelta( + ContentBlockDeltaEvent::builder() + .content_block_index(0) + .delta(ContentBlockDelta::ReasoningContent( + ReasoningContentBlockDelta::Signature("sig".to_string()), + )) + .build() + .unwrap(), + ); + assert!(bedrock_stream_event_to_reasoning_delta(&sig_event, &mut reasoning).is_none()); + + let block = reasoning.expect("reasoning accumulated"); + assert_eq!(block.reasoning_text.as_deref(), Some("let me ")); + assert_eq!(block.signature.as_deref(), Some("sig")); + } + #[test] fn openai_messages_to_bedrock_skips_cache_points_when_disabled() { let messages = vec![ diff --git a/backend/windmill-ai/src/ai_google.rs b/backend/windmill-ai/src/ai_google.rs index 94d3ef65e0..4374cb8628 100644 --- a/backend/windmill-ai/src/ai_google.rs +++ b/backend/windmill-ai/src/ai_google.rs @@ -312,7 +312,7 @@ impl GeminiToolCallEvent { pub fn to_extra_content(&self) -> Option { self.thought_signature.as_ref().map(|sig| ExtraContent { google: Some(GoogleExtraContent { thought_signature: Some(sig.clone()) }), - bedrock: None, + ..Default::default() }) } } @@ -1001,10 +1001,8 @@ mod tests { #[test] fn gemini_streaming_without_usage_emits_no_usage_chunk() { - let parsed = GeminiParsedEvent { - text: Some("the answer".to_string()), - ..Default::default() - }; + let parsed = + GeminiParsedEvent { text: Some("the answer".to_string()), ..Default::default() }; let mut tool_call_index = 0; let chunks = gemini_event_to_openai_sse_chunks( diff --git a/backend/windmill-ai/src/ai_types.rs b/backend/windmill-ai/src/ai_types.rs index c6f171daf3..b80e237920 100644 --- a/backend/windmill-ai/src/ai_types.rs +++ b/backend/windmill-ai/src/ai_types.rs @@ -118,6 +118,22 @@ pub struct BedrockExtraContent { pub redacted_content: Option, } +/// Native-Anthropic reasoning block emitted in the same assistant turn as a +/// tool call. Like Bedrock, Anthropic requires the thinking block (text + +/// unmodified signature, or redacted bytes) to precede `tool_use` on replay when +/// thinking is enabled, so it is round-tripped through the OpenAI-shaped tool call. +#[derive(Deserialize, Serialize, Clone, Debug, Default)] +pub struct AnthropicExtraContent { + #[serde(skip_serializing_if = "Option::is_none")] + pub thinking: Option, + #[serde(skip_serializing_if = "Option::is_none")] + pub signature: Option, + /// Base64 `data` of a redacted (encrypted) thinking block, when the provider + /// returned one instead of readable text. + #[serde(skip_serializing_if = "Option::is_none")] + pub redacted_thinking: Option, +} + /// Extra content for provider-specific metadata (e.g., Google thought signatures) #[derive(Deserialize, Serialize, Clone, Debug, Default)] pub struct ExtraContent { @@ -125,6 +141,8 @@ pub struct ExtraContent { pub google: Option, #[serde(skip_serializing_if = "Option::is_none")] pub bedrock: Option, + #[serde(skip_serializing_if = "Option::is_none")] + pub anthropic: Option, } #[derive(Deserialize, Serialize, Clone, Debug)] diff --git a/backend/windmill-ai/src/providers/anthropic.rs b/backend/windmill-ai/src/providers/anthropic.rs index c2064f2029..d71de2f35f 100644 --- a/backend/windmill-ai/src/providers/anthropic.rs +++ b/backend/windmill-ai/src/providers/anthropic.rs @@ -92,6 +92,10 @@ pub enum AnthropicRequestContent { #[serde(skip_serializing_if = "Option::is_none")] cache_control: Option, }, + #[serde(rename = "thinking")] + Thinking { thinking: String, signature: String }, + #[serde(rename = "redacted_thinking")] + RedactedThinking { data: String }, #[serde(rename = "image")] Image { source: AnthropicBase64Source }, #[serde(rename = "document")] @@ -131,6 +135,26 @@ pub struct AnthropicMessage { pub content: Vec, } +/// Adaptive thinking config for Anthropic native API. `summarized` display +/// matches the chat proxy path (renders a summarized thinking stream). +#[derive(Serialize, Debug)] +pub struct AnthropicThinking { + pub r#type: &'static str, + pub display: &'static str, +} + +impl AnthropicThinking { + fn adaptive() -> Self { + Self { r#type: "adaptive", display: "summarized" } + } +} + +/// Carries the reasoning effort token alongside adaptive thinking. +#[derive(Serialize, Debug)] +pub struct AnthropicOutputConfig { + pub effort: String, +} + /// Anthropic-specific request structure for standard API #[derive(Serialize)] pub struct AnthropicRequest<'a> { @@ -145,6 +169,10 @@ pub struct AnthropicRequest<'a> { #[serde(skip_serializing_if = "Option::is_none")] pub temperature: Option, #[serde(skip_serializing_if = "Option::is_none")] + pub thinking: Option, + #[serde(skip_serializing_if = "Option::is_none")] + pub output_config: Option, + #[serde(skip_serializing_if = "Option::is_none")] pub max_tokens: Option, pub stream: bool, } @@ -166,6 +194,10 @@ pub struct AnthropicVertexRequest { #[serde(skip_serializing_if = "Option::is_none")] pub temperature: Option, #[serde(skip_serializing_if = "Option::is_none")] + pub thinking: Option, + #[serde(skip_serializing_if = "Option::is_none")] + pub output_config: Option, + #[serde(skip_serializing_if = "Option::is_none")] pub max_tokens: Option, pub stream: bool, } @@ -189,6 +221,25 @@ fn convert_messages_to_anthropic(messages: &[OpenAIMessage]) -> Vec { let mut content: Vec = Vec::new(); + // Replay the turn's thinking block first: when thinking is enabled, + // Claude requires the thinking block (with its unmodified signature) + // to precede tool_use in the assistant turn it was emitted in. It is + // round-tripped on the tool call's extra_content (see AnthropicExtraContent). + if let Some(reasoning) = msg + .tool_calls + .as_ref() + .and_then(|tcs| { + tcs.iter().find_map(|tc| { + tc.extra_content + .as_ref() + .and_then(|ec| ec.anthropic.as_ref()) + }) + }) + .and_then(anthropic_reasoning_block_from_extra) + { + content.push(reasoning); + } + // Add text content if present if let Some(ref c) = msg.content { let text = extract_text_content(c); @@ -246,6 +297,25 @@ fn convert_messages_to_anthropic(messages: &[OpenAIMessage]) -> Vec Option { + if let Some(data) = extra.redacted_thinking.as_deref() { + return Some(AnthropicRequestContent::RedactedThinking { data: data.to_string() }); + } + match (extra.thinking.as_deref(), extra.signature.as_deref()) { + (Some(thinking), Some(signature)) => Some(AnthropicRequestContent::Thinking { + thinking: thinking.to_string(), + signature: signature.to_string(), + }), + _ => None, + } +} + /// Convert OpenAI content to Anthropic content blocks fn convert_content_to_anthropic(content: &Option) -> Vec { let Some(content) = content else { @@ -591,6 +661,17 @@ impl AnthropicQueryBuilder { } } + // Adaptive thinking rejects sampling params, so drop temperature when + // reasoning is on (Anthropic returns a hard 400 otherwise). + let (thinking, output_config, temperature) = match args.reasoning_effort { + Some(effort) => ( + Some(AnthropicThinking::adaptive()), + Some(AnthropicOutputConfig { effort: effort.to_string() }), + None, + ), + None => (None, None, args.temperature), + }; + // Build request based on platform if self.is_vertex() { // For Vertex AI: no model field, anthropic_version in body @@ -600,7 +681,9 @@ impl AnthropicQueryBuilder { messages: anthropic_messages, tools: tools_option, tool_choice, - temperature: args.temperature, + temperature, + thinking, + output_config, max_tokens, stream: true, }; @@ -614,7 +697,9 @@ impl AnthropicQueryBuilder { messages: anthropic_messages, tools: tools_option, tool_choice, - temperature: args.temperature, + temperature, + thinking, + output_config, max_tokens, stream: true, }; @@ -843,6 +928,123 @@ mod tests { assert!(has_header(&request.headers, "anthropic-beta", "some-beta")); } + #[test] + fn convert_messages_replays_thinking_block_before_tool_use() { + let tool_call = crate::ai_types::OpenAIToolCall { + id: "call_1".to_string(), + function: crate::ai_types::OpenAIFunction { + name: "lookup".to_string(), + arguments: "{}".to_string(), + }, + r#type: "function".to_string(), + extra_content: Some(crate::ai_types::ExtraContent { + anthropic: Some(crate::ai_types::AnthropicExtraContent { + thinking: Some("let me think".to_string()), + signature: Some("sig-abc".to_string()), + redacted_thinking: None, + }), + ..Default::default() + }), + }; + let assistant = OpenAIMessage { + role: "assistant".to_string(), + tool_calls: Some(vec![tool_call]), + ..Default::default() + }; + + let messages = convert_messages_to_anthropic(&[assistant]); + let content = &messages[0].content; + match &content[0] { + AnthropicRequestContent::Thinking { thinking, signature } => { + assert_eq!(thinking, "let me think"); + assert_eq!(signature, "sig-abc"); + } + other => panic!("expected thinking block first, got {:?}", other), + } + assert!(matches!( + &content[1], + AnthropicRequestContent::ToolUse { .. } + )); + } + + #[test] + fn convert_messages_drops_thinking_without_signature() { + let tool_call = crate::ai_types::OpenAIToolCall { + id: "call_1".to_string(), + function: crate::ai_types::OpenAIFunction { + name: "lookup".to_string(), + arguments: "{}".to_string(), + }, + r#type: "function".to_string(), + extra_content: Some(crate::ai_types::ExtraContent { + anthropic: Some(crate::ai_types::AnthropicExtraContent { + thinking: Some("unsigned".to_string()), + signature: None, + redacted_thinking: None, + }), + ..Default::default() + }), + }; + let assistant = OpenAIMessage { + role: "assistant".to_string(), + tool_calls: Some(vec![tool_call]), + ..Default::default() + }; + + let messages = convert_messages_to_anthropic(&[assistant]); + // An unsigned thinking block can't be replayed, so only tool_use remains. + assert!(matches!( + messages[0].content[0], + AnthropicRequestContent::ToolUse { .. } + )); + } + + #[test] + fn anthropic_request_serializes_adaptive_thinking_without_temperature() { + let request = AnthropicRequest { + model: "claude-opus-4-8", + system: None, + messages: vec![], + tools: None, + tool_choice: None, + temperature: None, + thinking: Some(AnthropicThinking::adaptive()), + output_config: Some(AnthropicOutputConfig { effort: "high".to_string() }), + max_tokens: Some(64000), + stream: true, + }; + + let body: serde_json::Value = + serde_json::from_str(&serde_json::to_string(&request).unwrap()).unwrap(); + assert_eq!(body["thinking"]["type"], "adaptive"); + assert_eq!(body["thinking"]["display"], "summarized"); + assert_eq!(body["output_config"]["effort"], "high"); + // Sampling params are rejected alongside adaptive thinking. + assert!(body.get("temperature").is_none()); + } + + #[test] + fn anthropic_request_omits_thinking_when_reasoning_off() { + let request = AnthropicRequest { + model: "claude-opus-4-8", + system: None, + messages: vec![], + tools: None, + tool_choice: None, + temperature: Some(0.5), + thinking: None, + output_config: None, + max_tokens: Some(64000), + stream: true, + }; + + let body: serde_json::Value = + serde_json::from_str(&serde_json::to_string(&request).unwrap()).unwrap(); + assert!(body.get("thinking").is_none()); + assert!(body.get("output_config").is_none()); + assert_eq!(body["temperature"], 0.5); + } + #[test] fn rejects_vertex_proxy_request_without_model() { let credentials = credentials(AIPlatform::GoogleVertexAi); diff --git a/backend/windmill-ai/src/providers/bedrock.rs b/backend/windmill-ai/src/providers/bedrock.rs index ddc5f33998..4405a50dcb 100644 --- a/backend/windmill-ai/src/providers/bedrock.rs +++ b/backend/windmill-ai/src/providers/bedrock.rs @@ -9,10 +9,10 @@ use crate::{ ai_bedrock::{ bedrock_model_supports_prompt_caching, bedrock_stream_event_is_block_stop, - bedrock_stream_event_to_text, bedrock_stream_event_to_tool_delta, - bedrock_stream_event_to_tool_delta_with_block_index, bedrock_stream_event_to_tool_start, - bedrock_stream_event_to_tool_start_with_block_index, build_tool_config, - create_inference_config, format_bedrock_error, json_to_document, + bedrock_stream_event_to_reasoning_delta, bedrock_stream_event_to_text, + bedrock_stream_event_to_tool_delta, bedrock_stream_event_to_tool_delta_with_block_index, + bedrock_stream_event_to_tool_start, bedrock_stream_event_to_tool_start_with_block_index, + build_tool_config, create_inference_config, format_bedrock_error, json_to_document, openai_messages_to_bedrock, streaming_tool_calls_to_openai, BearerTokenProvider, BedrockClient, StreamingToolCall, }, @@ -641,57 +641,13 @@ fn bedrock_sse_chunks_for_event( /// Fold a `ReasoningContent` stream delta into the state's pending reasoning /// block. Returns the text delta (for a `reasoning_content` SSE chunk) when the -/// event carried readable reasoning text. +/// event carried readable reasoning text. Shares the worker path's folding so +/// the two never drift. fn accumulate_reasoning_delta( event: &aws_sdk_bedrockruntime::types::ConverseStreamOutput, state: &mut BedrockSseStreamState, ) -> Option { - let aws_sdk_bedrockruntime::types::ConverseStreamOutput::ContentBlockDelta(delta_event) = event - else { - return None; - }; - let aws_sdk_bedrockruntime::types::ContentBlockDelta::ReasoningContent(reasoning) = - delta_event.delta()? - else { - return None; - }; - - let entry = state.reasoning.get_or_insert_with(Default::default); - match reasoning { - aws_sdk_bedrockruntime::types::ReasoningContentBlockDelta::Text(text) => { - entry - .reasoning_text - .get_or_insert_with(String::new) - .push_str(text); - Some(text.clone()) - } - aws_sdk_bedrockruntime::types::ReasoningContentBlockDelta::Signature(signature) => { - entry - .signature - .get_or_insert_with(String::new) - .push_str(signature); - None - } - aws_sdk_bedrockruntime::types::ReasoningContentBlockDelta::RedactedContent(blob) => { - // Base64 of concatenated fragments != concatenated base64 fragments, - // so accumulate raw bytes and re-encode. - let mut bytes = entry - .redacted_content - .as_deref() - .and_then(|existing| { - base64::Engine::decode(&base64::engine::general_purpose::STANDARD, existing) - .ok() - }) - .unwrap_or_default(); - bytes.extend_from_slice(blob.as_ref()); - entry.redacted_content = Some(base64::Engine::encode( - &base64::engine::general_purpose::STANDARD, - bytes, - )); - None - } - _ => None, - } + bedrock_stream_event_to_reasoning_delta(event, &mut state.reasoning) } async fn handle_bedrock_sdk_non_streaming( @@ -942,6 +898,7 @@ impl BedrockQueryBuilder { tools: Option<&[ToolDef]>, model: &str, temperature: Option, + reasoning_effort: Option<&str>, max_tokens: Option, api_key: &str, region: &str, @@ -977,6 +934,9 @@ impl BedrockQueryBuilder { let (bedrock_messages, system_prompts) = openai_messages_to_bedrock(&prepared_messages, enable_prompt_caching)?; + // Adaptive thinking rejects sampling params; drop temperature when reasoning is on. + let temperature = reasoning_effort.is_none().then_some(temperature).flatten(); + // Build inference configuration using shared helper let inference_config = create_inference_config(temperature, max_tokens.map(|t| t as i32)); @@ -994,6 +954,7 @@ impl BedrockQueryBuilder { system_prompts, inference_config, tool_config, + reasoning_effort, stream_event_sink, ) .await @@ -1008,6 +969,7 @@ impl BedrockQueryBuilder { system_prompts: Vec, inference_config: Option, tool_config: Option, + reasoning_effort: Option<&str>, stream_event_sink: Option>, ) -> Result { tracing::debug!( @@ -1035,6 +997,11 @@ impl BedrockQueryBuilder { request_builder = request_builder.set_tool_config(Some(config)); } + if let Some(effort) = reasoning_effort { + request_builder = + request_builder.additional_model_request_fields(bedrock_thinking_fields(effort)); + } + let mut stream = request_builder .send() .await @@ -1053,11 +1020,32 @@ impl BedrockQueryBuilder { let mut accumulated_tool_calls: HashMap = HashMap::new(); let mut current_tool_use_id: Option = None; let mut usage: Option = None; + // Claude reasoning block for the turn (only populated when thinking is on), + // attached to the first tool call for replay before toolUse. + let mut reasoning: Option = None; // Process stream events using shared parsing functions loop { match stream.recv().await { Ok(Some(event)) => { + // Fold reasoning deltas into the turn's reasoning block (for + // replay before toolUse, required by Claude when thinking is + // on) and stream the readable summary as a thinking affordance. + if let Some(reasoning_delta) = + bedrock_stream_event_to_reasoning_delta(&event, &mut reasoning) + { + if let Some(processor) = stream_event_sink.as_ref() { + processor + .send( + StreamingEvent::ReasoningTokenDelta { + content: reasoning_delta, + }, + &mut events_str, + ) + .await?; + } + } + // Handle tool use start using shared parser if let Some(tool_call) = bedrock_stream_event_to_tool_start(&event) { current_tool_use_id = Some(tool_call.id.clone()); @@ -1143,8 +1131,10 @@ impl BedrockQueryBuilder { Some(accumulated_text) }; - let tool_calls = - streaming_tool_calls_to_openai(accumulated_tool_calls.into_values().collect()); + let tool_calls = streaming_tool_calls_to_openai( + accumulated_tool_calls.into_values().collect(), + reasoning, + ); Ok(ParsedResponse::Text { content, diff --git a/backend/windmill-ai/src/providers/google_ai.rs b/backend/windmill-ai/src/providers/google_ai.rs index 650e11c0c7..21428903fa 100644 --- a/backend/windmill-ai/src/providers/google_ai.rs +++ b/backend/windmill-ai/src/providers/google_ai.rs @@ -153,13 +153,18 @@ impl GoogleAIQueryBuilder { (None, None) }; + // Map the effort token onto Gemini's native thinking controls; without + // one, leave provider defaults untouched. + let thinking_config = args + .reasoning_effort + .map(|effort| gemini_thinking_config(args.model, effort)); + build_gemini_generation_config( args.temperature, args.max_tokens, response_mime_type, response_schema, - // Worker AI-agent requests carry no reasoning knob; keep provider defaults. - None, + thinking_config, ) } } diff --git a/backend/windmill-ai/src/providers/openai.rs b/backend/windmill-ai/src/providers/openai.rs index efd7229ebb..bdb882679b 100644 --- a/backend/windmill-ai/src/providers/openai.rs +++ b/backend/windmill-ai/src/providers/openai.rs @@ -192,6 +192,15 @@ pub struct ResponsesApiTextFormat { pub format: ResponsesApiTextFormatConfig, } +/// Reasoning config for the Responses API (`reasoning: { effort }`). +/// The summary is intentionally not requested, mirroring the copilot chat: OpenAI +/// gates reasoning summaries behind organization verification, so asking for one +/// would fail the request for unverified orgs. +#[derive(Serialize)] +pub struct ResponsesApiReasoning { + pub effort: String, +} + #[derive(Serialize)] pub struct ResponsesApiRequest<'a> { pub model: &'a str, @@ -204,6 +213,8 @@ pub struct ResponsesApiRequest<'a> { #[serde(skip_serializing_if = "Option::is_none")] pub temperature: Option, #[serde(skip_serializing_if = "Option::is_none")] + pub reasoning: Option, + #[serde(skip_serializing_if = "Option::is_none")] pub max_output_tokens: Option, #[serde(skip_serializing_if = "Option::is_none")] pub text: Option, @@ -409,6 +420,9 @@ impl OpenAIQueryBuilder { tools, stream: Some(true), temperature: args.temperature, + reasoning: args + .reasoning_effort + .map(|effort| ResponsesApiReasoning { effort: effort.to_string() }), max_output_tokens: args.max_tokens, text, }; @@ -464,6 +478,7 @@ impl OpenAIQueryBuilder { tools, stream: None, // Image generation doesn't use streaming temperature: args.temperature, + reasoning: None, // Image generation models don't take a reasoning effort max_output_tokens: args.max_tokens, text: None, // No structured output for image generation }; diff --git a/backend/windmill-ai/src/providers/other.rs b/backend/windmill-ai/src/providers/other.rs index 7b4dab5ab5..f641daec64 100644 --- a/backend/windmill-ai/src/providers/other.rs +++ b/backend/windmill-ai/src/providers/other.rs @@ -35,6 +35,12 @@ pub struct OpenAICompletionRequest<'a> { #[serde(skip_serializing_if = "Option::is_none")] pub temperature: Option, #[serde(skip_serializing_if = "Option::is_none")] + pub reasoning_effort: Option<&'a str>, + /// DeepSeek disables reasoning via a `thinking` object rather than an effort + /// token (`reasoning_effort: "none"` is rejected by their API). + #[serde(skip_serializing_if = "Option::is_none")] + pub thinking: Option, + #[serde(skip_serializing_if = "Option::is_none")] pub max_completion_tokens: Option, #[serde(skip_serializing_if = "Option::is_none")] pub response_format: Option, @@ -51,6 +57,32 @@ pub struct BuiltRequests { pub without_usage: String, } +/// Provider-specific reasoning translation for OpenAI-compatible providers, +/// mirroring the chat proxy's `applyReasoningToConfig`. Returns the effort token +/// to send, an optional `thinking` object, and the temperature to keep: +/// - DeepSeek disables reasoning via `thinking: {type: disabled}` (the effort +/// token `"none"` is rejected by their API). +/// - Mistral rejects sampling params when reasoning is on (greedy sampling +/// requires `top_p == 1`), so temperature is dropped. +fn provider_reasoning_fields<'a>( + provider: &AIProvider, + reasoning_effort: Option<&'a str>, + temperature: Option, +) -> (Option<&'a str>, Option, Option) { + let (effort, thinking) = match provider { + AIProvider::DeepSeek if reasoning_effort == Some("none") => { + (None, Some(serde_json::json!({ "type": "disabled" }))) + } + _ => (reasoning_effort, None), + }; + let temperature = if *provider == AIProvider::Mistral && reasoning_effort.is_some() { + None + } else { + temperature + }; + (effort, thinking, temperature) +} + /// Query builder for providers using the OpenAI-compatible completion endpoint /// (Mistral, DeepSeek, Groq, TogetherAI, CustomAI, etc.) pub struct OtherQueryBuilder { @@ -109,12 +141,17 @@ impl OtherQueryBuilder { None }; + let (reasoning_effort, thinking, temperature) = + provider_reasoning_fields(&self.provider_kind, args.reasoning_effort, args.temperature); + // Build request with stream_options for usage tracking let request_with_usage = OpenAICompletionRequest { model: args.model, messages: &prepared_messages, tools: args.tools, - temperature: args.temperature, + temperature, + reasoning_effort, + thinking: thinking.clone(), max_completion_tokens: args.max_tokens, response_format: response_format.clone(), tool_choice: tool_choice.clone(), @@ -127,7 +164,9 @@ impl OtherQueryBuilder { model: args.model, messages: &prepared_messages, tools: args.tools, - temperature: args.temperature, + temperature, + reasoning_effort, + thinking, max_completion_tokens: args.max_tokens, response_format, tool_choice, @@ -268,3 +307,50 @@ impl QueryBuilder for OtherQueryBuilder { } } } + +#[cfg(test)] +mod tests { + use super::*; + + #[test] + fn deepseek_off_translates_to_thinking_disabled() { + let (effort, thinking, temperature) = + provider_reasoning_fields(&AIProvider::DeepSeek, Some("none"), Some(0.5)); + // "none" is rejected as an effort token, so it becomes a thinking disable. + assert_eq!(effort, None); + assert_eq!(thinking, Some(serde_json::json!({ "type": "disabled" }))); + assert_eq!(temperature, Some(0.5)); + } + + #[test] + fn deepseek_level_passes_effort_through() { + let (effort, thinking, _) = + provider_reasoning_fields(&AIProvider::DeepSeek, Some("high"), None); + assert_eq!(effort, Some("high")); + assert!(thinking.is_none()); + } + + #[test] + fn mistral_drops_temperature_when_reasoning_on() { + let (effort, thinking, temperature) = + provider_reasoning_fields(&AIProvider::Mistral, Some("high"), Some(0.7)); + assert_eq!(effort, Some("high")); + assert!(thinking.is_none()); + assert_eq!(temperature, None); + } + + #[test] + fn mistral_keeps_temperature_when_reasoning_off() { + let (_, _, temperature) = provider_reasoning_fields(&AIProvider::Mistral, None, Some(0.7)); + assert_eq!(temperature, Some(0.7)); + } + + #[test] + fn other_providers_pass_reasoning_and_temperature_through() { + let (effort, thinking, temperature) = + provider_reasoning_fields(&AIProvider::OpenRouter, Some("high"), Some(0.3)); + assert_eq!(effort, Some("high")); + assert!(thinking.is_none()); + assert_eq!(temperature, Some(0.3)); + } +} diff --git a/backend/windmill-ai/src/query_builder.rs b/backend/windmill-ai/src/query_builder.rs index 9830045f57..79fca4bd7d 100644 --- a/backend/windmill-ai/src/query_builder.rs +++ b/backend/windmill-ai/src/query_builder.rs @@ -12,6 +12,9 @@ pub struct BuildRequestArgs<'a> { pub tools: Option<&'a [ToolDef]>, pub model: &'a str, pub temperature: Option, + /// Provider-native reasoning effort token (e.g. `low`, `high`, `none`). + /// Each `build_request` maps it onto the provider's thinking config. + pub reasoning_effort: Option<&'a str>, pub max_tokens: Option, pub output_schema: Option<&'a OpenAPISchema>, pub output_type: &'a OutputType, diff --git a/backend/windmill-ai/src/sse.rs b/backend/windmill-ai/src/sse.rs index bb1b9d9731..66ff7da9de 100644 --- a/backend/windmill-ai/src/sse.rs +++ b/backend/windmill-ai/src/sse.rs @@ -9,7 +9,9 @@ use windmill_common::{error::Error, utils::rd_string}; use crate::{ ai_google::{parse_gemini_sse_event, GeminiUsageMetadata}, ai_types::UrlCitation, - ai_types::{ExtraContent, GoogleExtraContent, OpenAIFunction, OpenAIToolCall}, + ai_types::{ + AnthropicExtraContent, ExtraContent, GoogleExtraContent, OpenAIFunction, OpenAIToolCall, + }, query_builder::StreamEventSink, types::StreamingEvent, }; @@ -30,6 +32,10 @@ pub struct OpenAIChoiceDeltaToolCall { #[derive(Deserialize)] pub struct OpenAIChoiceDelta { pub content: Option, + /// Reasoning summary streamed by providers that expose it (e.g. DeepSeek's + /// `reasoning_content`). Rendered as a "thinking" affordance. + #[serde(default)] + pub reasoning_content: Option, pub tool_calls: Option>, } @@ -142,6 +148,13 @@ impl SSEParser for OpenAISSEParser { if let Some(mut choices) = event.choices.filter(|s| !s.is_empty()) { if let Some(delta) = choices.remove(0).delta { + if let Some(reasoning) = delta.reasoning_content.filter(|s| !s.is_empty()) { + let event = StreamingEvent::ReasoningTokenDelta { content: reasoning }; + self.stream_event_processor + .send(event, &mut self.events_str) + .await?; + } + if let Some(content) = delta.content.filter(|s| !s.is_empty()) { self.accumulated_content.push_str(&content); let event = StreamingEvent::TokenDelta { content }; @@ -216,6 +229,16 @@ pub enum AnthropicContentBlockStart { id: String, name: String, }, + #[serde(rename = "thinking")] + Thinking { + #[serde(default)] + thinking: String, + }, + #[serde(rename = "redacted_thinking")] + RedactedThinking { + #[serde(default)] + data: String, + }, #[serde(other)] Unknown, } @@ -240,6 +263,10 @@ pub enum AnthropicDelta { InputJsonDelta { partial_json: String }, #[serde(rename = "citations_delta")] CitationsDelta { citation: AnthropicCitationDelta }, + #[serde(rename = "thinking_delta")] + ThinkingDelta { thinking: String }, + #[serde(rename = "signature_delta")] + SignatureDelta { signature: String }, #[serde(other)] Unknown, } @@ -292,6 +319,7 @@ pub enum AnthropicSSEEvent { #[allow(dead_code)] enum ContentBlockState { Text, + Thinking, ToolUse { id: String, name: String }, Unknown, } @@ -310,6 +338,11 @@ pub struct AnthropicSSEParser { pub used_websearch: bool, /// Token usage from message_delta event pub usage: Option, + /// Claude thinking block accumulated from `thinking`/`signature` deltas + /// (or a redacted block). Attached to the first tool call of the turn so it + /// can be replayed before `tool_use` (required by Claude when thinking is on). + pending_reasoning: Option, + reasoning_attached: bool, } impl AnthropicSSEParser { @@ -323,6 +356,8 @@ impl AnthropicSSEParser { annotations: Vec::new(), used_websearch: false, usage: None, + pending_reasoning: None, + reasoning_attached: false, } } } @@ -363,6 +398,17 @@ impl SSEParser for AnthropicSSEParser { self.stream_event_processor .send(event, &mut self.events_str) .await?; + // Attach the turn's thinking block to the first tool call so it + // can be replayed before tool_use on the next request. + let extra_content = if self.reasoning_attached { + None + } else { + self.reasoning_attached = true; + self.pending_reasoning.take().map(|anthropic| ExtraContent { + anthropic: Some(anthropic), + ..Default::default() + }) + }; // Initialize tool call accumulator self.accumulated_tool_calls.insert( index as i64, @@ -370,10 +416,35 @@ impl SSEParser for AnthropicSSEParser { id, function: OpenAIFunction { name, arguments: String::new() }, r#type: "function".to_string(), - extra_content: None, + extra_content, }, ); } + AnthropicContentBlockStart::Thinking { thinking } => { + self.content_blocks + .insert(index, ContentBlockState::Thinking); + let entry = self.pending_reasoning.get_or_insert_with(Default::default); + if !thinking.is_empty() { + entry + .thinking + .get_or_insert_with(String::new) + .push_str(&thinking); + self.stream_event_processor + .send( + StreamingEvent::ReasoningTokenDelta { content: thinking }, + &mut self.events_str, + ) + .await?; + } + } + AnthropicContentBlockStart::RedactedThinking { data } => { + // Redacted blocks arrive whole (no deltas). + self.content_blocks + .insert(index, ContentBlockState::Unknown); + self.pending_reasoning + .get_or_insert_with(Default::default) + .redacted_thinking = Some(data); + } AnthropicContentBlockStart::ServerToolUse { name, .. } => { // Detect websearch tool usage if name == "web_search" { @@ -417,6 +488,34 @@ impl SSEParser for AnthropicSSEParser { title: citation.title, }); } + AnthropicDelta::ThinkingDelta { thinking } => { + if let Some(ContentBlockState::Thinking) = + self.content_blocks.get(&index) + { + self.pending_reasoning + .get_or_insert_with(Default::default) + .thinking + .get_or_insert_with(String::new) + .push_str(&thinking); + self.stream_event_processor + .send( + StreamingEvent::ReasoningTokenDelta { content: thinking }, + &mut self.events_str, + ) + .await?; + } + } + AnthropicDelta::SignatureDelta { signature } => { + if let Some(ContentBlockState::Thinking) = + self.content_blocks.get(&index) + { + self.pending_reasoning + .get_or_insert_with(Default::default) + .signature + .get_or_insert_with(String::new) + .push_str(&signature); + } + } AnthropicDelta::Unknown => {} } } @@ -495,6 +594,15 @@ impl SSEParser for GeminiSSEParser { return Ok(()); }; + if let Some(reasoning) = parsed.reasoning.filter(|s| !s.is_empty()) { + self.stream_event_processor + .send( + StreamingEvent::ReasoningTokenDelta { content: reasoning }, + &mut self.events_str, + ) + .await?; + } + if let Some(text) = parsed.text { self.accumulated_content.push_str(&text); self.stream_event_processor @@ -522,7 +630,7 @@ impl SSEParser for GeminiSSEParser { let extra_content = tool_call.thought_signature.map(|sig| ExtraContent { google: Some(GoogleExtraContent { thought_signature: Some(sig) }), - bedrock: None, + ..Default::default() }); self.accumulated_tool_calls.insert( @@ -805,3 +913,76 @@ impl SSEParser for OpenAIResponsesSSEParser { Ok(()) } } + +#[cfg(test)] +mod tests { + use super::*; + + #[test] + fn reasoning_token_delta_serializes_with_snake_case_tag() { + let event = StreamingEvent::ReasoningTokenDelta { content: "hmm".to_string() }; + let json: serde_json::Value = + serde_json::from_str(&serde_json::to_string(&event).unwrap()).unwrap(); + assert_eq!(json["type"], "reasoning_token_delta"); + assert_eq!(json["content"], "hmm"); + } + + #[test] + fn openai_delta_parses_reasoning_content() { + // DeepSeek and similar stream reasoning under `reasoning_content`. + let delta: OpenAIChoiceDelta = + serde_json::from_str(r#"{"reasoning_content":"let me think"}"#).unwrap(); + assert_eq!(delta.reasoning_content.as_deref(), Some("let me think")); + } + + #[test] + fn parses_anthropic_thinking_events() { + let start: AnthropicSSEEvent = serde_json::from_str( + r#"{"type":"content_block_start","index":0,"content_block":{"type":"thinking","thinking":"x"}}"#, + ) + .unwrap(); + assert!(matches!( + start, + AnthropicSSEEvent::ContentBlockStart { + content_block: AnthropicContentBlockStart::Thinking { .. }, + .. + } + )); + + let thinking_delta: AnthropicSSEEvent = serde_json::from_str( + r#"{"type":"content_block_delta","index":0,"delta":{"type":"thinking_delta","thinking":"more"}}"#, + ) + .unwrap(); + assert!(matches!( + thinking_delta, + AnthropicSSEEvent::ContentBlockDelta { + delta: AnthropicDelta::ThinkingDelta { .. }, + .. + } + )); + + let signature_delta: AnthropicSSEEvent = serde_json::from_str( + r#"{"type":"content_block_delta","index":0,"delta":{"type":"signature_delta","signature":"sig"}}"#, + ) + .unwrap(); + assert!(matches!( + signature_delta, + AnthropicSSEEvent::ContentBlockDelta { + delta: AnthropicDelta::SignatureDelta { .. }, + .. + } + )); + + let redacted: AnthropicSSEEvent = serde_json::from_str( + r#"{"type":"content_block_start","index":0,"content_block":{"type":"redacted_thinking","data":"abc"}}"#, + ) + .unwrap(); + assert!(matches!( + redacted, + AnthropicSSEEvent::ContentBlockStart { + content_block: AnthropicContentBlockStart::RedactedThinking { .. }, + .. + } + )); + } +} diff --git a/backend/windmill-ai/src/types.rs b/backend/windmill-ai/src/types.rs index 8f864468db..6991e0e0d3 100644 --- a/backend/windmill-ai/src/types.rs +++ b/backend/windmill-ai/src/types.rs @@ -203,6 +203,10 @@ pub struct ProviderWithResource { pub kind: AIProvider, pub resource: ProviderResource, pub model: String, + /// Provider-native reasoning effort token (e.g. `low`, `high`, `none`). + /// Belongs to the model selection, so it rides on the provider config. + #[serde(default)] + pub reasoning_effort: Option, } impl ProviderWithResource { @@ -214,6 +218,12 @@ impl ProviderWithResource { &self.model } + /// The reasoning effort to thread to the provider, treating an empty string + /// (e.g. a cleared flow input) as unset. + pub fn get_reasoning_effort(&self) -> Option<&str> { + self.reasoning_effort.as_deref().filter(|s| !s.is_empty()) + } + pub async fn get_base_url(&self, db: &DB) -> Result { self.kind .get_base_url(self.resource.base_url.clone(), db) @@ -365,6 +375,10 @@ pub struct AIAgentResult<'a> { pub enum StreamingEvent { /// Individual token from the AI response TokenDelta { content: String }, + /// Individual token of the model's reasoning / thinking summary. Emitted + /// before the answer when reasoning is enabled; renderable as a "thinking" + /// affordance (thinking tokens bill regardless of whether they are shown). + ReasoningTokenDelta { content: String }, /// Tool call has started ToolCall { call_id: String, function_name: String }, /// Tool call arguments are complete diff --git a/backend/windmill-api-assets/src/lib.rs b/backend/windmill-api-assets/src/lib.rs index b4b0583863..997879e155 100644 --- a/backend/windmill-api-assets/src/lib.rs +++ b/backend/windmill-api-assets/src/lib.rs @@ -614,6 +614,21 @@ struct GraphQuery { struct GraphAssetNode { kind: AssetKind, path: String, + // Fork workspaces only: 'fork' when the fork has materialized this ducklake asset itself, + // 'deferred' when reads fall back to the parent workspace's current table (defer view). + // Absent outside forks, for non-ducklake assets, and for assets never materialized + // anywhere. Lockstep with TS `AssetGraphAssetNode.fork_materialization`. + #[serde(skip_serializing_if = "Option::is_none")] + fork_materialization: Option, + // The base dimension this asset is the SCD2 `_current` companion view + // of — set only on a `ducklake://…/_current` node whose producer + // declares `// materialize … history` on ``. The producer edge already + // links it to that script (both writes are registered at deploy); this lets + // the canvas render it as a derived "current view" of the base rather than an + // unrelated table. Absent for every other asset. Lockstep with TS + // `AssetGraphAssetNode.derived_from`. + #[serde(skip_serializing_if = "Option::is_none")] + derived_from: Option, } #[derive(Serialize, Debug)] @@ -632,6 +647,14 @@ struct GraphRunnableNode { partition_kind: Option, #[serde(skip_serializing_if = "Option::is_none", default)] freshness: Option, + // Completion time of the most recently started successful run of this + // pipeline member. The canvas checks it against the `// freshness` window + // to color the badge fresh/stale. The badge itself is passive; on EE the + // freshness watchdog (windmill-queue) separately re-runs stale + // unpartitioned producers. Absent when no successful run is visible to + // the caller (job RLS applies). + #[serde(skip_serializing_if = "Option::is_none", default)] + last_success_at: Option>, #[serde(skip_serializing_if = "Option::is_none", default)] tag: Option, #[serde(skip_serializing_if = "Option::is_none", default)] @@ -656,6 +679,13 @@ struct GraphRunnableNode { // `merge` / any partitioned write INSERTs into a fixed-schema table. #[serde(skip_serializing_if = "Option::is_none", default)] materialize_strategy: Option, + // `on_schema_change=ignore` on the managed materialize — the producer's + // opt-out from downstream schema-contract warnings. Threaded to the editor + // so its client-side contract mirror suppresses the same warnings the + // server check does. Only serialized when set to `ignore` (default `warn` + // is absent). Lockstep with TS `AssetGraphRunnableNode.materialize_on_schema_change`. + #[serde(skip_serializing_if = "Option::is_none", default)] + materialize_on_schema_change: Option, // Macros this script provides to the workspace registry (deployed // `// macros` library). Drives the library node state + details-pane // signature list. Lockstep with TS `AssetGraphRunnableNode.macros`. @@ -776,6 +806,25 @@ enum TriggerEdge { }, } +// Ordering-only "must-run-after" edge: `runnable_path`'s data test reads +// `asset` (a `// data_test relationships` ref, or a custom test whose body +// reads a known pipeline asset), so the asset's in-pipeline producer must +// materialize before `runnable_path` runs. NOT a data-consumption edge — the +// tested script doesn't ingest the asset's rows, it only needs the table to +// exist at test time. Rendered dashed (like macro edges) and fed into the +// cascade topo-sort so a cold cascade orders the referenced dimension first. +// Only emitted when the referenced asset has a producer in the graph; an +// external table (no producer) adds no edge — the runtime error stands. +#[derive(Serialize, Debug)] +struct TestEdge { + producer_kind: AssetUsageKind, + producer_path: String, + runnable_kind: AssetUsageKind, + runnable_path: String, + asset_kind: AssetKind, + asset_path: String, +} + #[derive(Serialize, Debug)] struct AssetGraphResponse { assets: Vec, @@ -784,12 +833,15 @@ struct AssetGraphResponse { triggers: Vec, #[serde(skip_serializing_if = "Vec::is_empty", default)] macro_edges: Vec, + #[serde(skip_serializing_if = "Vec::is_empty", default)] + test_edges: Vec, } async fn asset_graph( authed: ApiAuthed, Path(w_id): Path, Extension(user_db): Extension, + Extension(db): Extension, Query(q): Query, ) -> JsonResult { let mut tx = user_db.begin(&authed).await?; @@ -920,6 +972,46 @@ async fn asset_graph( .fetch_all(&mut *tx) .await?; + // Newest successful completed run per pipeline member, for the passive + // freshness status on the canvas. Correlated per-path lookup walks + // ix_job_root_job_index_by_path_2 newest-first until the first success, + // so cost is bounded by the member count, not run history. "Newest" is + // by created_at (the index order), not completed_at: with overlapping + // runs of one path this can pick an earlier completion, erring toward + // stale — never toward false-fresh. Inside the user tx so job-visibility + // RLS applies — a caller who can't see the runs gets no timestamp rather + // than leaked completion times. + let member_paths: Vec = pipeline_member_paths + .iter() + .map(|r| r.path.clone()) + .collect(); + let last_success_rows = sqlx::query!( + r#" + SELECT p.path AS "path!", + (SELECT c.completed_at + FROM v2_job j + JOIN v2_job_completed c ON c.id = j.id + WHERE j.workspace_id = $1 + AND j.runnable_path = p.path + AND j.parent_job IS NULL + -- No 'singlestepflow': flows may share a script's path, and + -- a same-path flow run must not read as the script being + -- fresh (false-fresh). Script retries land as native + -- 'script' jobs; only the rare flow-wrapper fallback is + -- missed, which errs stale. Kept in lockstep with the + -- freshness watchdog's queries (freshness_watchdog_ee). + AND j.kind IN ('script', 'preview') + AND c.status = 'success' + ORDER BY j.created_at DESC + LIMIT 1) AS last_success_at + FROM unnest($2::text[]) AS p(path) + "#, + &w_id, + &member_paths, + ) + .fetch_all(&mut *tx) + .await?; + // Existing scripts / flows in the workspace. Used to filter out // orphan trigger rows whose `script_path` no longer resolves — those // would otherwise be added to `runnable_set` below and surface as @@ -1010,6 +1102,11 @@ async fn asset_graph( (r.path.clone(), lineage) }) .collect(); + let last_success_by_path: std::collections::HashMap> = + last_success_rows + .into_iter() + .filter_map(|r| r.last_success_at.map(|t| (r.path, t))) + .collect(); let pipeline_member_script_paths: std::collections::HashSet = pipeline_member_paths.into_iter().map(|r| r.path).collect(); let existing_script_paths: std::collections::HashSet = @@ -1190,9 +1287,204 @@ async fn asset_graph( runnable_set.insert((AssetUsageKind::Script, e.consumer_path.clone())); } + // Data-test ordering edges. A `// data_test relationships -> ` + // (and, best-effort, a custom `// data_test + +{#if !provider || !model} +
Select a model to configure reasoning effort.
+{:else if !capability.supported} +
The selected model does not support reasoning effort.
+{:else} + + {#snippet buttonReplacement()} + + {/snippet} + +{/if} diff --git a/frontend/src/lib/components/CompareDrafts.svelte b/frontend/src/lib/components/CompareDrafts.svelte index 867268cd1a..2614adf445 100644 --- a/frontend/src/lib/components/CompareDrafts.svelte +++ b/frontend/src/lib/components/CompareDrafts.svelte @@ -14,7 +14,13 @@ import { editUrlFor } from './sessions/forkEditUrl' import { AppService, FlowService, ScriptService, type WorkspaceItemDiff } from '$lib/gen' import { sendUserToast } from '$lib/toast' - import { getDraftDiffValues, deployDraft, discardDraft } from '$lib/utils_draft_deploy' + import { + getDraftDiffValues, + deployDraft, + discardDraft, + draftBaseIsStale + } from '$lib/utils_draft_deploy' + import { checkDeployPermission, type DeployPermission } from '$lib/utils_workspace_deploy' import { type DraftItem, useWorkspaceDrafts } from '$lib/workspaceDrafts.svelte' import type { Kind as LayoutKind } from '$lib/utils_deployable' import { userStore } from '$lib/stores' @@ -35,6 +41,15 @@ deployCount?: number updateCount?: number draftCount?: number + /** When set (reached via a session's Review button), preselect only the + * rows this chat modified — `${UserDraftItemKind}:${path}` keys, matching + * Row.key. Undefined → preselect all deployable rows (the default). All rows + * are still shown either way. */ + chatMask?: Set + /** False while the (async) chatMask is still loading. The select-all default + * waits for this so it doesn't race the mask and select everything. Defaults + * to true for callers that don't pass a mask. */ + chatMaskReady?: boolean /** Selecting deploy_to/update asks the page to swap to CompareWorkspaces. */ onModeSelected?: (v: CompareMode) => void /** Fired after a deploy/discard so the page can refresh the *fork* @@ -52,6 +67,8 @@ deployCount = 0, updateCount = 0, draftCount = 0, + chatMask, + chatMaskReady = true, onModeSelected, onChanged }: Props = $props() @@ -216,27 +233,11 @@ path: item.path, getDraft: true }))) as any - // A draft is stale when the version it forked from no longer matches the - // current deployed head: a newer version was deployed after the draft began. - // Scripts compare `parent_hash` vs the deployed `hash`; flows the pinned - // `version_id` vs the deployed head `version_id`; apps the pinned - // `parent_version` vs the deployed head (`versions[last]`). const draftBlob = r.draft as any - const appHead = Array.isArray(r.versions) ? r.versions[r.versions.length - 1] : undefined - const stale = - item.draftKind === 'script' - ? !!r.hash && !!draftBlob?.parent_hash && draftBlob.parent_hash !== r.hash - : item.draftKind === 'flow' - ? r.version_id != null && - draftBlob?.version_id != null && - draftBlob.version_id !== r.version_id - : appHead != null && - draftBlob?.parent_version != null && - draftBlob.parent_version !== appHead summaryCache[item.key] = { deployed: r.summary, draft: draftBlob?.summary, - stale, + stale: draftBaseIsStale(item.draftKind, r), loading: false } } catch (error) { @@ -263,6 +264,21 @@ let selectedItems = $state([]) let deploying = $state(false) + + // Whether the user may deploy drafts into this workspace — fills the + // `RestrictDeployToDeployers` (+ operator) gap via the shared util, same as the + // fork compare page and the session review drawer. Fail-open while resolving. + let deployPerm = $state({ ok: true }) + $effect(() => { + const ws = currentWorkspaceId + // Reset to fail-open on workspace change, and drop a stale resolution — + // otherwise the previous workspace's verdict lingers (or lands last) and + // gates the wrong workspace. + deployPerm = { ok: true } + void checkDeployPermission(ws).then((p) => { + if (ws === currentWorkspaceId) deployPerm = p + }) + }) // Select all on the first non-empty load (deploy-all is the common intent); // only once, so a refetch after a deploy doesn't re-select the leftovers. let hasAutoSelected = $state(false) @@ -286,8 +302,13 @@ }) $effect(() => { - if (!hasAutoSelected && visibleItems.length > 0) { - selectedItems = visibleItems.filter(isSelectable).map((i) => i.key) + if (!hasAutoSelected && chatMaskReady && visibleItems.length > 0) { + // Default intent is deploy-all; when reached from a session's Review + // (chatMask set), preselect only that chat's items instead. + const selectable = visibleItems.filter(isSelectable) + selectedItems = (chatMask ? selectable.filter((i) => chatMask.has(i.key)) : selectable).map( + (i) => i.key + ) hasAutoSelected = true } }) @@ -695,15 +716,19 @@ {/snippet} {#snippet footer()} -
+
+ {#if !deployPerm.ok} + {deployPerm.reason} + {/if}
{/snippet} diff --git a/frontend/src/lib/components/CompareWorkspaces.svelte b/frontend/src/lib/components/CompareWorkspaces.svelte index 8fbef7703c..a2453add27 100644 --- a/frontend/src/lib/components/CompareWorkspaces.svelte +++ b/frontend/src/lib/components/CompareWorkspaces.svelte @@ -37,10 +37,12 @@ import type { Kind } from '$lib/utils_deployable' import { + checkDeployPermission, deployItem, deleteItemInWorkspace, getItemValue, getOnBehalfOf, + type DeployPermission, type DeployResult } from '$lib/utils_workspace_deploy' import { isTriggerOrScheduleKind } from 'windmill-utils-internal' @@ -58,6 +60,7 @@ import { base } from '$lib/base' import CompareModeToggle, { type CompareMode } from './CompareModeToggle.svelte' import { editUrlFor } from './sessions/forkEditUrl' + import { diffInMask } from './sessions/modifiedItemsMask' import DatatableSchemaDiff from './DatatableSchemaDiff.svelte' interface Props { @@ -77,6 +80,14 @@ * out of the default selection — deploying/updating moves the deployed * version, not the draft. The page derives this from the fork drafts. */ draftKeys?: Set + /** When set (reached via a session's Review button), preselect only the + * diffs this chat caused (matched via diffInMask). The deploy_to default + * narrows to these; the update direction (parent→fork) preselects nothing, + * since it's never chat-caused. All rows are still shown. */ + chatMask?: Set + /** False while the (async) chatMask is still loading. The select-all default + * waits for this so it doesn't race the mask. Defaults to true. */ + chatMaskReady?: boolean /** Selecting `draft` asks the page to swap us out for CompareDrafts; * deploy_to/update are handled internally but reported so the page can * remember the direction. */ @@ -95,6 +106,8 @@ updateCount = 0, draftCount = 0, draftKeys = new Set(), + chatMask, + chatMaskReady = true, onModeSelected, onChanged }: Props = $props() @@ -803,11 +816,19 @@ // to parent" flow. The user picks them à la carte by clicking the row. // Items with a pending draft are also left out by default: the deployed // version (not the draft) is what moves, so we make the user opt in. + // The update direction (parent→fork) is never something the chat caused, so + // when scoped to a chat's items (chatMask set) preselect nothing there. + if (chatMask && !mergeIntoParent) { + selectedItems = [] + return + } const filtered = selectableDiffs.filter((d) => !isTriggerOrScheduleKind(d.kind) && !hasDraft(d)) const conflictSafe = mergeIntoParent ? filtered : filtered.filter((d) => !(d.ahead > 0 && d.behind > 0)) - selectedItems = conflictSafe + // When reached from a session's Review, narrow the default to this chat's items. + const scoped = chatMask ? conflictSafe.filter((d) => diffInMask(d, chatMask)) : conflictSafe + selectedItems = scoped .map((d) => getItemKey(d)) .filter((k) => !(deploymentStatus[k]?.status == 'deployed')) } @@ -849,6 +870,21 @@ fetchPermissions() }) + // Can the user actually deploy into the target workspace? Fills the frontend + // gap for the `RestrictDeployToDeployers` rule (+ operator), shared with the + // session review drawer via the same checkDeployPermission util. Cached per + // workspace; `deployPerm` tracks whichever side the current direction targets. + let deployPerms = $state>({}) + const deployPermFetched = new Set() + $effect(() => { + for (const ws of [currentWorkspaceId, parentWorkspaceId]) { + if (!ws || deployPermFetched.has(ws)) continue + deployPermFetched.add(ws) + void checkDeployPermission(ws).then((p) => (deployPerms = { ...deployPerms, [ws]: p })) + } + }) + let deployPerm = $derived(deployPerms[deployTargetWorkspace] ?? { ok: true }) + // Fetch summaries and on_behalf_of_email when comparison data loads $effect(() => { if (comparison?.diffs) { @@ -859,7 +895,7 @@ // Auto-select items on initial load $effect(() => { - if (comparison?.diffs && !hasAutoSelected && selectableDiffs.length > 0) { + if (comparison?.diffs && !hasAutoSelected && chatMaskReady && selectableDiffs.length > 0) { selectDefault() hasAutoSelected = true } @@ -1424,7 +1460,9 @@ deploying || (hasBehindChanges && !allowBehindChangesOverride) || (mergeIntoParent && !canDeployToParent) || + !deployPerm.ok || hasUnselectedOnBehalfOf} + title={!deployPerm.ok ? deployPerm.reason : undefined} loading={deploying} on:click={requestDeploy} > @@ -1435,6 +1473,9 @@ {/if}
+ {#if !deployPerm.ok} + {deployPerm.reason} + {/if} {#if !(mergeIntoParent && !canDeployToParent) && hasUnselectedOnBehalfOf} You must set the "on behalf of" user for all items before deploying diff --git a/frontend/src/lib/components/DataTestsResult.svelte b/frontend/src/lib/components/DataTestsResult.svelte index cba28f3a82..385af187df 100644 --- a/frontend/src/lib/components/DataTestsResult.svelte +++ b/frontend/src/lib/components/DataTestsResult.svelte @@ -1,15 +1,22 @@
    {#each tests as t (t.test)} -
  • - {#if t.violating > 0} - - failed: - {t.test} - — {t.violating} violating row{t.violating === 1 ? '' : 's'} - {:else} - - passed: - {t.test} +
  • +
    + {#if t.violating > 0} + + failed: + {t.test} + — {t.violating} violating row{t.violating === 1 ? '' : 's'} + {#if t.sample && t.sample.length > 0} + + {/if} + {:else} + + passed: + {t.test} + {/if} +
    + {#if t.violating > 0 && t.sample && t.sample.length > 0 && expanded.has(t.test)} +
    +
    + sample of the violating rows ({t.sample.length} of {t.violating}, unordered) +
    + +
    {/if}
  • {/each} diff --git a/frontend/src/lib/components/DisplayResult.svelte b/frontend/src/lib/components/DisplayResult.svelte index cbd81086a9..efb1dd2da1 100644 --- a/frontend/src/lib/components/DisplayResult.svelte +++ b/frontend/src/lib/components/DisplayResult.svelte @@ -574,24 +574,53 @@ // formats are produced by this repo's worker (see duckdb_executor.rs); the // derivation is inert (undefined) for every other DisplayResult use. let dataTests = $derived.by(() => { + // Both structured shapes carry `[{ test, violating, sample? }]`; the + // sample (bounded violating-row rows) may arrive as a JSON string (the + // worker keeps it string-typed through the summary row) and is optional + // by contract — anything malformed degrades to no sample, never to a + // dropped checklist. + const normalize = ( + dt: any + ): Array<{ test: string; violating: number; sample?: Record[] }> | undefined => { + if (typeof dt === 'string') { + try { + dt = JSON.parse(dt) + } catch { + return undefined + } + } + if ( + !Array.isArray(dt) || + dt.length === 0 || + !dt.every((x) => x && typeof x.test === 'string' && typeof x.violating === 'number') + ) { + return undefined + } + return dt.map((x) => { + let sample = x.sample + if (typeof sample === 'string') { + try { + sample = JSON.parse(sample) + } catch { + sample = undefined + } + } + if (!Array.isArray(sample) || !sample.every((r) => r && typeof r === 'object')) { + sample = undefined + } + return { test: x.test, violating: x.violating, sample } + }) + } // Success: structured column on the summary row. const row = Array.isArray(result) ? (result as any)?.[0] : (result as any) - let dt = row?.data_tests - if (typeof dt === 'string') { - try { - dt = JSON.parse(dt) - } catch { - dt = undefined - } - } - if ( - Array.isArray(dt) && - dt.length > 0 && - dt.every((x) => x && typeof x.test === 'string' && typeof x.violating === 'number') - ) { - return dt as Array<{ test: string; violating: number }> - } - // Failure: parse the worker's breakdown out of the error message. + const fromRow = normalize(row?.data_tests) + if (fromRow) return fromRow + // Failure: the worker attaches the same structured breakdown (plus + // per-failed-test samples) to the error payload. + const fromError = normalize((result as any)?.error?.data_tests) + if (fromError) return fromError + // Failure fallback for results predating the structured error payload: + // parse the worker's breakdown out of the error message. const msg = (result as any)?.error?.message if (typeof msg === 'string' && msg.includes('data tests failed on')) { const out: Array<{ test: string; violating: number }> = [] diff --git a/frontend/src/lib/components/DraftBadge.svelte b/frontend/src/lib/components/DraftBadge.svelte index 75cc6cec0b..1527feef73 100644 --- a/frontend/src/lib/components/DraftBadge.svelte +++ b/frontend/src/lib/components/DraftBadge.svelte @@ -38,6 +38,10 @@ /** Offer "Load" alongside "View Diff" on other users' rows. The deploy / * review page sets this false: loading into a fresh editor is moot there. */ allowFork?: boolean + /** Compact variant: render only the avatar circles (no "Draft" pill text), + * slightly smaller — for tight spots like the diff-tree sidebar. The hover + * popover is unchanged. */ + iconOnly?: boolean } let { @@ -49,7 +53,8 @@ itemKind = undefined, path = undefined, onMigrated = undefined, - allowFork = true + allowFork = true, + iconOnly = false }: Props = $props() // Authed user lands first; everyone else keeps the backend's ordering. @@ -193,6 +198,30 @@ } +{#snippet circleStack()} + + + {#each visibleUsers as u, i (i)} + + {initials(u)} + + {/each} + {#if overflowCount > 0} + + +{overflowCount} + + {/if} + +{/snippet} + {#if showBadge} @@ -204,32 +233,23 @@ bind:isOpen={popoverOpen} > {#snippet trigger()} - - {#if orderedUsers.length > 0} - - - {#each visibleUsers as u, i (i)} - - {initials(u)} - - {/each} - {#if overflowCount > 0} - - +{overflowCount} - - {/if} - - {/if} - {draft_only ? 'Draft only' : 'Draft'} - + {#if iconOnly} + + + + {#if orderedUsers.length > 0} + {@render circleStack()} + {/if} + + {:else} + + {#if orderedUsers.length > 0} + {@render circleStack()} + {/if} + {draft_only ? 'Draft only' : 'Draft'} + + {/if} {/snippet} {#snippet content()}
    diff --git a/frontend/src/lib/components/Editor.svelte b/frontend/src/lib/components/Editor.svelte index 2054dc8cf1..8563190a89 100644 --- a/frontend/src/lib/components/Editor.svelte +++ b/frontend/src/lib/components/Editor.svelte @@ -92,6 +92,11 @@ listWorkspaceMacrosCached, macroDefinitionSql } from '$lib/components/assets/workspaceMacros' + import { + fetchLatestSchema, + normalizeAssetPath, + type ContractMarker + } from '$lib/components/assets/AssetGraph/schemaContracts' import * as htmllang from '$lib/svelteMonarch' import { conf, language } from '$lib/vueMonarch' @@ -163,6 +168,11 @@ preparedAssetsSqlQueries?: InferAssetsSqlQueryDetails[] | undefined // To execute preview scripts with the right worker group customTag?: string + // Live schema-contract diagnostics (pipelines gap #2b): owner-scoped + // warning markers computed by the caller (ScriptEditor's contract + // mirror) from the buffer's asset refs vs captured producer schemas. + // Warning severity only — contracts never block; empty clears. + schemaContractMarkers?: ContractMarker[] } let { @@ -195,7 +205,8 @@ enablePreprocessorSnippet = false, rawAppRunnableKey = undefined, preparedAssetsSqlQueries, - customTag + customTag, + schemaContractMarkers = [] }: Props = $props() $effect.pre(() => { @@ -706,6 +717,53 @@ }) } + let schemaContractCompletor: IDisposable | undefined = undefined + + // Column-name completion for pipeline annotation refs (`// column out <- + // ducklake://lake/orders.|`, `// data_test relationships col -> + // ducklake://lake/customers.|`): suggests the referenced asset's *captured* + // columns (with types) so a broken ref never gets typed — the prevention + // side of the schema-contract check. Only fires on annotation comment lines + // with a ducklake URI right before the cursor's `.`; schemas come from the + // short-TTL contract cache, so per-keystroke cost is a map lookup. + function addSchemaContractCompletions() { + schemaContractCompletor?.dispose() + schemaContractCompletor = languages.registerCompletionItemProvider(lang, { + triggerCharacters: ['.'], + provideCompletionItems: async function (model, position) { + // Read the store per request, not at registration — the provider + // outlives a workspace switch. + const workspace = $workspaceStore + if (!workspace) return { suggestions: [] } + const before = model.getLineContent(position.lineNumber).slice(0, position.column - 1) + if (!/^\s*(\/\/|--|#)\s*(column|data_test|on|materialize)\b/.test(before)) { + return { suggestions: [] } + } + const uri = before.match(/ducklake:\/\/([\w/.{}-]+?)\.$/) + if (!uri) return { suggestions: [] } + const schema = await fetchLatestSchema(workspace, normalizeAssetPath(uri[1])) + if (!schema) return { suggestions: [] } + const word = model.getWordUntilPosition(position) + const range = { + startLineNumber: position.lineNumber, + endLineNumber: position.lineNumber, + startColumn: word.startColumn, + endColumn: word.endColumn + } + return { + suggestions: schema.columns.map((c) => ({ + label: c.name, + kind: languages.CompletionItemKind.Field, + detail: `${c.type} · captured schema v${schema.version}`, + insertText: c.name, + range, + sortText: 'a' + c.name + })) + } + } + }) + } + let sqlSchemaCompletor: IDisposable | undefined = undefined async function updateSchema(newSchemaRes: string | undefined) { @@ -1916,6 +1974,7 @@ sqlTypeCompletor && sqlTypeCompletor.dispose() resultCollectionCompletor && resultCollectionCompletor.dispose() workspaceMacroCompletor && workspaceMacroCompletor.dispose() + schemaContractCompletor && schemaContractCompletor.dispose() preprocessorCompletor && preprocessorCompletor.dispose() timeoutModel && clearTimeout(timeoutModel) changeChainStart = undefined @@ -1999,6 +2058,36 @@ : workspaceMacroCompletor?.dispose() }) + // Pipeline annotation grammar is language-agnostic (`//` / `--` / `#` + // comment headers), so contract-ref completions register for every script + // language that can be a pipeline member. The provider line-gates itself, + // so it is inert outside annotation lines. + $effect(() => { + initialized && ['duckdb', 'python3', 'bun', 'deno', 'nativets'].includes(scriptLang ?? '') + ? untrack(() => addSchemaContractCompletions()) + : schemaContractCompletor?.dispose() + }) + + // Schema-contract markers arrive as a prop because the mirror can finish + // computing before Monaco initializes on mount — reacting to `initialized` + // re-applies the pending set once the model exists. The ever-set latch + // keeps unrelated editors from calling setModelMarkers with [] forever. + let contractMarkersEverSet = false + $effect(() => { + const ms = schemaContractMarkers + if (!initialized || (ms.length === 0 && !contractMarkersEverSet)) return + contractMarkersEverSet = true + untrack(() => { + const model = editor?.getModel() + if (!model) return + meditor.setModelMarkers( + model, + 'schema-contracts', + ms.map((m) => ({ ...m, severity: MarkerSeverity.Warning })) + ) + }) + }) + $effect(() => { initialized && canHavePreprocessor(lang) && enablePreprocessorSnippet ? untrack(() => addPreprocessorCompletions(lang)) diff --git a/frontend/src/lib/components/ScriptBuilder.svelte b/frontend/src/lib/components/ScriptBuilder.svelte index fbfa4f1c1b..7d639b2c75 100644 --- a/frontend/src/lib/components/ScriptBuilder.svelte +++ b/frontend/src/lib/components/ScriptBuilder.svelte @@ -41,6 +41,7 @@ } from '$lib/utils' import Path from './Path.svelte' import { invalidateWorkspacePaths } from './PathNameAutocomplete.svelte' + import { notifyContractWarnings } from './assets/AssetGraph/schemaContracts' import ScriptEditor from './ScriptEditor.svelte' import { Alert, Button, Drawer, SecondsInput, Tab, TabContent, Tabs } from './common' import LanguageIcon from './common/languageIcons/LanguageIcon.svelte' @@ -654,6 +655,11 @@ // cache so it shows up immediately instead of after the 60s TTL. invalidateWorkspacePaths($workspaceStore!) + // Authoritative save-time schema-contract check (pipelines gap #2b): + // warn-only, post-commit so a self-produced target resolves to the + // content just deployed. Fire-and-forget — must never gate the deploy. + notifyContractWarnings($workspaceStore!, script.language, script.content) + if (!initialPath) { await CaptureService.moveCapturesAndConfigs({ workspace: $workspaceStore!, diff --git a/frontend/src/lib/components/ScriptEditor.svelte b/frontend/src/lib/components/ScriptEditor.svelte index 575c303b06..8ff211f1c2 100644 --- a/frontend/src/lib/components/ScriptEditor.svelte +++ b/frontend/src/lib/components/ScriptEditor.svelte @@ -105,6 +105,11 @@ import { canHavePreprocessor } from '$lib/script_helpers' import { assetEq, type AssetWithAltAccessType } from './assets/lib' import type { ColumnLineage } from './assets/AssetGraph/parsePipelineAnnotations' + import { + computeContractMarkers, + type ContractMarker, + type SchemaContractGraphContext + } from './assets/AssetGraph/schemaContracts' import { editor as meditor } from 'monaco-editor' import type { ReviewChangesOpts } from './copilot/chat/monaco-adapter' import GitRepoViewer from './GitRepoViewer.svelte' @@ -202,6 +207,11 @@ // regular /scripts/edit route keeps its current open-by-default UX; // the session preview opts in to save vertical real estate. initialTestPanelCollapsed?: boolean + // Producer-side facts for the live schema-contract diagnostics + // (`on_schema_change=ignore` suppression + scd2 `_current` fallback), + // built by the pipeline page from the resolved graph. Absent outside the + // pipeline editor — the check still runs, just without suppression. + schemaContractContext?: SchemaContractGraphContext } let { @@ -242,7 +252,8 @@ previewLayout = 'right', onTestStateChange, onTestJob, - initialTestPanelCollapsed = false + initialTestPanelCollapsed = false, + schemaContractContext = undefined }: Props = $props() $effect(() => { @@ -614,6 +625,35 @@ } ) + // Live schema-contract diagnostics (pipelines gap #2b): diff the buffer's + // asset refs against the captured producer schemas and surface mismatches + // as Monaco warning squiggles — the as-you-type mirror of the authoritative + // save-time check. The result is a prop on Editor (not an imperative call) + // because this can resolve before Monaco initializes on mount. Sequenced so + // a slow schema fetch can't overwrite the markers of a newer keystroke. + let contractMarkers: ContractMarker[] = $state([]) + let contractCheckSeq = 0 + watch([() => inferAssetsRes.current, () => schemaContractContext], () => { + const res = inferAssetsRes.current + const workspace = $workspaceStore + const seq = ++contractCheckSeq + if (!workspace || !res || res.status === 'error') { + contractMarkers = [] + return + } + const bufferCode = code + computeContractMarkers( + workspace, + bufferCode, + (res.assets ?? []) as AssetWithAltAccessType[], + schemaContractContext + ) + .then((markers) => { + if (seq === contractCheckSeq) contractMarkers = markers + }) + .catch((e) => console.error('schema-contract diagnostics failed', e)) + }) + watch([() => code, () => lang], () => { if (lang !== 'ansible') return inferAnsibleExecutionMode(code).then((v) => { @@ -2612,6 +2652,7 @@ bind:code={editorCode} bind:websocketAlive bind:this={editor} + schemaContractMarkers={contractMarkers} {yContent} awareness={wsProvider?.awareness} on:change={(e) => { diff --git a/frontend/src/lib/components/aiProviderStorage.ts b/frontend/src/lib/components/aiProviderStorage.ts index ce6af230bb..7f6d705d3a 100644 --- a/frontend/src/lib/components/aiProviderStorage.ts +++ b/frontend/src/lib/components/aiProviderStorage.ts @@ -50,6 +50,7 @@ export function isSameAsStoredConfig(config: ProviderConfig | undefined): boolea storedConfig !== undefined && storedConfig.kind === config?.kind && storedConfig.resource === config?.resource && - storedConfig.model === config?.model + storedConfig.model === config?.model && + storedConfig.reasoning_effort === config?.reasoning_effort ) } diff --git a/frontend/src/lib/components/assets/AssetGraph/ActivityHistogram.svelte b/frontend/src/lib/components/assets/AssetGraph/ActivityHistogram.svelte index fcfd95a5d3..cd9548de59 100644 --- a/frontend/src/lib/components/assets/AssetGraph/ActivityHistogram.svelte +++ b/frontend/src/lib/components/assets/AssetGraph/ActivityHistogram.svelte @@ -147,9 +147,10 @@ >
    {/if}
- +
- {fmtTime(from)} + from {fmtTime(from)} now
diff --git a/frontend/src/lib/components/assets/AssetGraph/AssetGraphCanvas.svelte b/frontend/src/lib/components/assets/AssetGraph/AssetGraphCanvas.svelte index 1c5cf2dde6..6f8460fa10 100644 --- a/frontend/src/lib/components/assets/AssetGraph/AssetGraphCanvas.svelte +++ b/frontend/src/lib/components/assets/AssetGraph/AssetGraphCanvas.svelte @@ -142,8 +142,8 @@ // editor passes it, so the asset-graph page is unaffected. The page // clears it once the pan has had time to settle. panToNodeId?: string | undefined - // Script paths eligible to *start* a bounded-cascade run (schedule - // roots + manual roots — see boundedCascade.validStarts). When a path is + // Script paths eligible to *start* a bounded-cascade run — roots AND + // mid-DAG models (see boundedCascade.validFromStarts). When a path is // in this set and `onStartBoundedRun` is wired, its node's cascade menu // gains a "Run downstream up to…" entry. validStartPaths?: ReadonlySet @@ -222,6 +222,7 @@ | 'add-anchor' | 'data-test' | 'macro' + | 'test-dependency' unsaved?: boolean // Edge from a missing-trigger placeholder — styled red dashed to // signal "this script declared `// on kafka` but no trigger row @@ -322,19 +323,59 @@ // own adjacency (boundedCascade.buildLineageDownstreamMap). const hasLineageDownstream = new Set(buildLineageDownstreamMap(g).keys()) + // Producers that declare `// data_test` checks, keyed by runnable id — the + // asset node uses this (plus the producer's run state) to render the + // data-test outcome badge. Tests only assert on ducklake `// materialize` + // targets (v1), so guard status is a ducklake-only concept below. + const producerHasTests = new Set() + // Producer → its declared `// materialize` target, so a multi-output + // producer's guard badge lands only on the table its tests assert on — + // not on its other ducklake outputs. Mirrors the write-edge badge anchor. + const guardMaterializeTarget = new Map< + string, + NonNullable + >() + for (const r of g.runnables) { + if (r.data_tests && r.data_tests.length > 0) producerHasTests.add(`${r.usage_kind}:${r.path}`) + if (r.materialize_target) + guardMaterializeTarget.set(`${r.usage_kind}:${r.path}`, r.materialize_target) + } + for (const a of g.assets) { const assetId = `asset:${a.kind}:${a.path}` + // Guard/outcome badge inputs: is a producer of this (ducklake) asset + // test-guarded, and did that guarded producer's latest run fail? + let dataTestGuarded = false + let producerFailed = false + if (a.kind === 'ducklake') { + for (const p of producersByAsset.get(`${a.kind}:${a.path}`) ?? []) { + const rid = `${p.kind}:${p.path}` + if (!producerHasTests.has(rid)) continue + // Tests assert on the producer's `// materialize` target; when the + // producer declares one, only that table is guarded (a multi-output + // producer must not badge its other ducklake writes). No declared + // target → single-output producer, so its lone ducklake write is it. + const mt = guardMaterializeTarget.get(rid) + if (mt && !(mt.kind === a.kind && mt.path === a.path)) continue + dataTestGuarded = true + if (runStates?.get(rid)?.status === 'failure') producerFailed = true + } + } nodes.push({ id: assetId, type: 'asset', data: { asset_kind: a.kind, path: a.path, + fork_materialization: a.fork_materialization, + derived_from: a.derived_from, onAddScript: onAddScriptForAsset, pathPrefix, defaultPathSuffix, producers: producersByAsset.get(`${a.kind}:${a.path}`) ?? [], - onRunProducer + onRunProducer, + dataTestGuarded, + producerFailed } }) } @@ -401,6 +442,7 @@ in_pipeline: r.in_pipeline ?? false, partition_kind: r.partition_kind, freshness: r.freshness, + last_success_at: r.last_success_at, tag: r.tag, retry: r.retry, macros: r.macros, @@ -521,6 +563,22 @@ }) } + // Data-test ordering edges (producer → tested script): the referenced + // asset must be materialized before the test runs, so the cascade orders + // the producer first. Rendered as a dashed "must run after" link, distinct + // from data flow — the tested script doesn't consume the asset's rows. + for (const te of g.test_edges ?? []) { + const producerId = `${te.producer_kind}:${te.producer_path}` + const testedId = `${te.runnable_kind}:${te.runnable_path}` + if (!runnableNodeIds.has(producerId) || !runnableNodeIds.has(testedId)) continue + edges.push({ + id: `testdep:${producerId}->${testedId}`, + source: producerId, + target: testedId, + kind: 'test-dependency' + }) + } + // Non-asset triggers (schedule + native) are rendered as source nodes // above the pipeline script. Real (non-missing) nodes are // deduplicated per (kind, ref) tuple so a single schedule shared @@ -949,6 +1007,16 @@ label = e.via_use ? 'uses lib' : 'macros' labelStyle = 'fill: rgb(139 92 246); font-size: 10px; font-weight: 600;' break + case 'test-dependency': + // Producer → tested script: amber dashed ordering link. Not + // data flow (blue/gray) nor execution trigger (gray "triggers") + // — it only says "the test needs this asset to exist first". + style = 'stroke: rgb(217 119 6); stroke-width: 1.25px;' + strokeDasharray = '5 3' + markerColor = 'rgb(217 119 6)' + label = 'test needs' + labelStyle = 'fill: rgb(217 119 6); font-size: 10px; font-weight: 600;' + break default: style = '' } diff --git a/frontend/src/lib/components/assets/AssetGraph/AssetGraphDetailsPane.svelte b/frontend/src/lib/components/assets/AssetGraph/AssetGraphDetailsPane.svelte index 1abe21f76d..095169e3b7 100644 --- a/frontend/src/lib/components/assets/AssetGraph/AssetGraphDetailsPane.svelte +++ b/frontend/src/lib/components/assets/AssetGraph/AssetGraphDetailsPane.svelte @@ -1,5 +1,5 @@ + {#if data.fork_materialization === 'deferred'} + + + parent + + {:else if data.fork_materialization === 'fork'} + + + fork + + {/if} + + {#if data.derived_from} + + + + {/if} + {#if showGuardBadge} + +
+ +
+ {/if} {#if showActions} + {#if calendarPicker && materializeTarget?.kind === 'ducklake' && value} + {#if selectedMaterialized} + + + {value} is already materialized — running replaces it. + + {:else} + + + {value} not materialized yet — running creates it. + + {/if} + {/if} + + + {#if upstreamMissing} + + + + No upstream data for {value} yet — this run may materialize an + empty partition. + + + {/if} + + + {#if recentlyMissing.length > 0} +
+ + {recentlyMissing.length} of the last {recentWindow(spec.kind)} + {spec.kind} partitions are not materialized: + +
+ {#each recentlyMissing.slice(0, MAX_MISSING_CHIPS) as b (b)} + + {/each} + {#if recentlyMissing.length > MAX_MISSING_CHIPS} + + +{recentlyMissing.length - MAX_MISSING_CHIPS} more + + {/if} +
+
+ {/if} + diff --git a/frontend/src/lib/components/assets/AssetGraph/PipelineGraphEditor.svelte b/frontend/src/lib/components/assets/AssetGraph/PipelineGraphEditor.svelte index 8dca18bd8e..6e6a694e82 100644 --- a/frontend/src/lib/components/assets/AssetGraph/PipelineGraphEditor.svelte +++ b/frontend/src/lib/components/assets/AssetGraph/PipelineGraphEditor.svelte @@ -22,6 +22,7 @@ import type { RunnableRunState, PipelineEvent } from './activeRunnables.svelte' import type { PipelineOutputKind } from './pipelineTemplates' import type { PipelineEditorState } from './pipelineEditorState.svelte' + import type { SchemaContractGraphContext } from './schemaContracts' type RunProducer = { kind: 'script' | 'flow'; path: string; unsaved?: boolean; cascade?: boolean } @@ -73,6 +74,8 @@ selectionProducers = [], selectionColumnGraph, schemaCanEvolve = true, + selectionForkMaterialization = undefined, + schemaContractContext = undefined, downstreamSubscribers = 0, onStartBoundedRunForOpen, canBoundedRunOpenScript = false, @@ -170,6 +173,11 @@ /** Transitive column-lineage trace for a selected ducklake asset (route page). */ selectionColumnGraph?: ColumnLineageGraph schemaCanEvolve?: boolean + /** Fork workspaces: data-environment state of the selected ducklake asset (route page). */ + selectionForkMaterialization?: 'fork' | 'deferred' + /** Producer-side facts for the editor's live schema-contract diagnostics + * (ignore suppression + scd2 `_current` fallback), from the route page. */ + schemaContractContext?: SchemaContractGraphContext downstreamSubscribers?: number onStartBoundedRunForOpen?: (path: string) => void canBoundedRunOpenScript?: boolean @@ -486,6 +494,8 @@ selectionProducers={activeDraft ? [] : selectionProducers} {selectionColumnGraph} {schemaCanEvolve} + {selectionForkMaterialization} + {schemaContractContext} {runsRefreshKey} {runsPendingJobId} {activeRunnable} diff --git a/frontend/src/lib/components/assets/AssetGraph/PipelineRunForm.svelte b/frontend/src/lib/components/assets/AssetGraph/PipelineRunForm.svelte index fc5e7ad945..4384f125ba 100644 --- a/frontend/src/lib/components/assets/AssetGraph/PipelineRunForm.svelte +++ b/frontend/src/lib/components/assets/AssetGraph/PipelineRunForm.svelte @@ -1,6 +1,9 @@ - +
+ {#if partitionSpec} + args.partition, (v) => (args.partition = v)} + {workspace} + {materializeTarget} + {upstreamAssets} + /> + {/if} + +
diff --git a/frontend/src/lib/components/assets/AssetGraph/PipelineScriptView.svelte b/frontend/src/lib/components/assets/AssetGraph/PipelineScriptView.svelte index cadd3c6a94..18564c6edd 100644 --- a/frontend/src/lib/components/assets/AssetGraph/PipelineScriptView.svelte +++ b/frontend/src/lib/components/assets/AssetGraph/PipelineScriptView.svelte @@ -1,11 +1,13 @@ + +{#if show} +
+
+ +
+

Finish setting up pipelines

+

+ Pipelines materialize data into DuckLake tables backed by object storage. Configure the + following before your first pipeline can run. +

+
+
+ +
    + {#each steps as step (step.title)} + {@const Icon = step.icon} +
  • + {#if step.done === true} + + {:else if step.done === false} + + {:else} + + {/if} +
    + {step.title} + {step.description} +
    + {#if step.done !== true} + + {step.cta} + + + {:else} + Configured + {/if} +
  • + {/each} +
+
+{/if} diff --git a/frontend/src/lib/components/assets/AssetGraph/RunnableNode.svelte b/frontend/src/lib/components/assets/AssetGraph/RunnableNode.svelte index 8b5e90170c..4cb65ee8f3 100644 --- a/frontend/src/lib/components/assets/AssetGraph/RunnableNode.svelte +++ b/frontend/src/lib/components/assets/AssetGraph/RunnableNode.svelte @@ -22,12 +22,13 @@ import { preventDefault, stopPropagation } from 'svelte/legacy' import type { GraphUsageKind } from './types' import type { RunnableRunState } from './activeRunnables.svelte' + import { parseDurationSecs } from './parsePipelineAnnotations' import { NODE } from '$lib/components/graph/util' import DropdownV2 from '$lib/components/DropdownV2.svelte' import Popover from '$lib/components/meltComponents/Popover.svelte' import type { Item } from '$lib/utils' import { workspaceStore } from '$lib/stores' - import { sendUserToast } from '$lib/utils' + import { sendUserToast, msToReadableTimeShort } from '$lib/utils' interface Props { data: { @@ -36,6 +37,9 @@ in_pipeline?: boolean partition_kind?: 'daily' | 'hourly' | 'weekly' | 'monthly' | 'dynamic' freshness?: string + // Completion time (ISO) of the newest successful run visible to + // the caller. With `freshness`, drives the fresh/stale chip state. + last_success_at?: string tag?: string retry?: { count: number; delay?: string } // Macros this script provides (deployed/drafted `// macros` library). @@ -122,6 +126,44 @@ } } + // Freshness verdict: newest successful run (server `last_success_at`, + // or a newer one the session poll observed) vs the `// freshness` + // window. No verdict (undefined) for drafts — no run history — and for + // unparseable windows; the chip then stays neutral like the other + // annotation chips. + let freshnessWindowS = $derived(data.freshness ? parseDurationSecs(data.freshness) : undefined) + // Ticks so a node crosses fresh→stale while the canvas stays open (the + // graph payload is static between refetches). Armed only when a verdict + // is rendered. + let nowMs = $state(Date.now()) + $effect(() => { + if (freshnessWindowS === undefined || data.unsaved) return + const id = setInterval(() => (nowMs = Date.now()), 30_000) + return () => clearInterval(id) + }) + let lastSuccessMs = $derived.by(() => { + const server = data.last_success_at ? new Date(data.last_success_at).getTime() : undefined + const polled = data.runState?.lastSuccessAt + ? new Date(data.runState.lastSuccessAt).getTime() + : undefined + if (server === undefined) return polled + return polled === undefined ? server : Math.max(server, polled) + }) + let freshnessState = $derived.by((): 'fresh' | 'stale' | undefined => { + if (freshnessWindowS === undefined || data.unsaved) return undefined + if (lastSuccessMs === undefined) return 'stale' + return nowMs - lastSuccessMs <= freshnessWindowS * 1000 ? 'fresh' : 'stale' + }) + let freshnessTooltip = $derived.by(() => { + const base = `// freshness ${data.freshness}` + if (freshnessState === undefined) return base + if (lastSuccessMs === undefined) return `${base} — stale: no successful run yet` + const ago = msToReadableTimeShort(Math.max(0, nowMs - lastSuccessMs)) + return freshnessState === 'fresh' + ? `${base} — fresh: last successful run ${ago} ago` + : `${base} — stale: last successful run ${ago} ago` + }) + // Cascade + bounded-run options live on the Run button's caret popover // (whenever there's a cascade OR a bounded-run start — see `hasCaret` // below), so the kebab menu stays focused on lifecycle actions only. @@ -164,8 +206,8 @@
+ guidelines). Only the freshness chip (when it has a verdict) + and the run-state chip below use semantic colors. --> {#if data.partition_kind}
{data.partition_kind}
{/if} + {#if data.freshness}
{data.freshness} @@ -356,10 +410,10 @@ >
- Run downstream up to… + Run + downstream… - Pick end node(s) on the graph, then run only the cascade between this script - and them. + Run this script and everything downstream. Or pick end node(s) on the graph to + bound the cascade between this script and them.
diff --git a/frontend/src/lib/components/assets/AssetGraph/TriggerNode.svelte b/frontend/src/lib/components/assets/AssetGraph/TriggerNode.svelte index caf7d90482..b888ae226a 100644 --- a/frontend/src/lib/components/assets/AssetGraph/TriggerNode.svelte +++ b/frontend/src/lib/components/assets/AssetGraph/TriggerNode.svelte @@ -183,7 +183,7 @@ ...(data.onStartBoundedRun ? [ { - displayName: 'Run downstream up to…', + displayName: 'Run + downstream…', icon: Target, action: () => data.onStartBoundedRun?.() } diff --git a/frontend/src/lib/components/assets/AssetGraph/activeRunnables.svelte.ts b/frontend/src/lib/components/assets/AssetGraph/activeRunnables.svelte.ts index 917e21766a..f6682804c7 100644 --- a/frontend/src/lib/components/assets/AssetGraph/activeRunnables.svelte.ts +++ b/frontend/src/lib/components/assets/AssetGraph/activeRunnables.svelte.ts @@ -1,8 +1,15 @@ import { JobService } from '$lib/gen' export type RunStatus = 'running' | 'success' | 'failure' -/** Per-runnable badge state: latest run status + runs observed this session. */ -export type RunnableRunState = { status: RunStatus; runs: number } +/** + * Per-runnable badge state: latest run status + runs observed this session. + * `lastSuccessAt` is the completion time (start + duration when the listing + * carries it, else start as a conservative lower bound) of the newest + * successful run seen by the poll — lets the freshness chip go green right + * after an in-session run, ahead of the next graph refetch (whose + * `last_success_at` would carry it). + */ +export type RunnableRunState = { status: RunStatus; runs: number; lastSuccessAt?: string } export type EventStatus = 'queued' | 'running' | 'success' | 'failure' /** One folder activity-log row (a job observed by the poll). */ @@ -14,6 +21,12 @@ export type PipelineEvent = { /** What started it, as far as the job listing reveals. */ source: 'schedule' | 'run' at: string + /** + * Completion time (start + duration) for completed rows. The freshness + * chip compares against completion — `at` is the start time and would + * read a long run as older than its output actually is. + */ + completedAt?: string /** * Queued jobs: when the job is due to start. A future value means a * scheduled run waiting for its cron tick, not pipeline activity. @@ -45,7 +58,8 @@ function statesEq(a: Map, b: Map() const countedJobIds = new Set() // Job ids we've observed in-flight at least once. The catch-up pulse is @@ -241,10 +255,25 @@ export function useActiveRunnableIds( const prev = completedHistory.get(id) const status: RunStatus = (j as any).success === true ? 'success' : 'failure' const ts = startedTs ?? new Date(pollStartedMs).toISOString() + // Freshness compares against COMPLETION time (that's + // when the output materialized — the server-side + // last_success_at is completed_at too). The listing + // only carries started_at, so add duration_ms; when + // absent, the start is a conservative lower bound + // (errs stale, never false-fresh). + const durationMs = (j as any).duration_ms + const doneTs = + typeof durationMs === 'number' && startedTs + ? new Date(new Date(startedTs).getTime() + durationMs).toISOString() + : ts completedHistory.set(id, { runs: (prev?.runs ?? 0) + 1, lastStatus: !prev || ts >= prev.lastTs ? status : prev.lastStatus, - lastTs: !prev || ts >= prev.lastTs ? ts : prev.lastTs + lastTs: !prev || ts >= prev.lastTs ? ts : prev.lastTs, + lastSuccessTs: + status === 'success' && (!prev?.lastSuccessTs || doneTs >= prev.lastSuccessTs) + ? doneTs + : prev?.lastSuccessTs }) } } @@ -267,6 +296,10 @@ export function useActiveRunnableIds( : 'failure', source: (j as any).schedule_path ? 'schedule' : 'run', at: startedTs ?? new Date(pollStartedMs).toISOString(), + completedAt: + !isQueued && typeof (j as any).duration_ms === 'number' && startedTs + ? new Date(new Date(startedTs).getTime() + (j as any).duration_ms).toISOString() + : undefined, scheduledFor: isQueued ? ((j as any).scheduled_for as string | undefined) : undefined }) } @@ -289,7 +322,11 @@ export function useActiveRunnableIds( // previous badge state until a worker picks the job up. const snap = new Map() for (const [id, h] of completedHistory) { - snap.set(id, { status: runningThisTick.has(id) ? 'running' : h.lastStatus, runs: h.runs }) + snap.set(id, { + status: runningThisTick.has(id) ? 'running' : h.lastStatus, + runs: h.runs, + lastSuccessAt: h.lastSuccessTs + }) } for (const id of runningThisTick) { if (!snap.has(id)) snap.set(id, { status: 'running', runs: 0 }) diff --git a/frontend/src/lib/components/assets/AssetGraph/boundedCascade.test.ts b/frontend/src/lib/components/assets/AssetGraph/boundedCascade.test.ts index fa735682b7..e4692de6b8 100644 --- a/frontend/src/lib/components/assets/AssetGraph/boundedCascade.test.ts +++ b/frontend/src/lib/components/assets/AssetGraph/boundedCascade.test.ts @@ -7,24 +7,29 @@ import { buildLineageDag, buildLineageDownstreamMap, descendants, + nonAutorunTriggerScripts, + reachableCutting, scriptNodeId, scriptsOf, - validStarts + validStarts, + validFromStarts } from './boundedCascade' import { computeInducedSchedule } from './graphTraversal' type W = [script: string, asset: string] // producer write edge (datatable) type R = [script: string, asset: string] // pure-read edge (datatable) type S = [script: string, asset: string] // `// on ` subscription +type T = [producer: string, tested: string, asset: string] // `// data_test` ordering edge function graph(opts: { scripts?: string[] writes?: W[] reads?: R[] subs?: S[] + tests?: T[] native?: Array<[kind: NativeTriggerKind, script: string]> }): AssetGraphResponse { - const { scripts = [], writes = [], reads = [], subs = [], native = [] } = opts + const { scripts = [], writes = [], reads = [], subs = [], tests = [], native = [] } = opts const triggers: AssetGraphTrigger[] = [ ...subs.map( ([s, a]) => @@ -60,7 +65,15 @@ function graph(opts: { access_type: 'r' as const })) ], - triggers + triggers, + test_edges: tests.map(([producer, tested, a]) => ({ + producer_kind: 'script' as const, + producer_path: producer, + runnable_kind: 'script' as const, + runnable_path: tested, + asset_kind: 'datatable' as const, + asset_path: a + })) } } @@ -95,6 +108,22 @@ describe('buildLineageDag', () => { expect([...(dag.down.get(sn('u')) ?? [])]).toEqual([asset('x')]) expect(dag.up.get(sn('u'))).toBeUndefined() // asset is not upstream of its own writer }) + + it('routes a data_test ordering edge through the referenced asset', () => { + // prod writes x; tested has a `// data_test` against x (prod → tested edge). + // The DAG must place x (and thus prod) upstream of tested so a cascade + // materializes x first. + const g = graph({ + scripts: ['prod', 'tested'], + writes: [['prod', 'x']], + tests: [['prod', 'tested', 'x']] + }) + const dag = buildLineageDag(g) + // asset x → tested (routed through the asset, not a direct prod → tested hop) + expect([...(dag.down.get(asset('x')) ?? [])]).toEqual([sn('tested')]) + // prod → x → tested makes prod an ancestor of tested. + expect(ancestors(dag, sn('tested'))).toEqual(new Set([asset('x'), sn('prod')])) + }) }) describe('ancestors / descendants', () => { @@ -247,6 +276,165 @@ describe('validStarts', () => { }) }) +describe('validFromStarts (mid-DAG selective execution)', () => { + // a → x → sub (subscriber) → y → reader (pure read). `k` is event-triggered. + const g = () => + graph({ + scripts: ['a', 'sub', 'reader', 'k'], + writes: [ + ['a', 'x'], + ['sub', 'y'] + ], + reads: [['reader', 'y']], + subs: [['sub', 'x']], + native: [['kafka', 'k']] + }) + + it('includes mid-DAG subscribers and pure readers, not just roots', () => { + const from = validFromStarts(g()) + expect(from.has(sn('a'))).toBe(true) // root + expect(from.has(sn('sub'))).toBe(true) // mid-DAG subscriber — NOT a validStart + expect(from.has(sn('reader'))).toBe(true) // pure reader + // The old root-only gate would have rejected the mid-DAG nodes. + expect(validStarts(g()).has(sn('sub'))).toBe(false) + }) + + it('excludes event-triggered scripts (no run-now gesture)', () => { + expect(validFromStarts(g()).has(sn('k'))).toBe(false) + }) + + it('excludes webhook/data_upload mid-DAG subscribers (need caller input)', () => { + // a → x → upload_mid (subscribes x AND `// on data_upload`) → y → consumer. + // upload_mid must NOT be an eligible start (empty-arg run has no S3Object), + // and must be a barrier so `consumer` isn't run with a skipped producer. + const g2 = graph({ + scripts: ['a', 'upload_mid', 'hook_mid', 'consumer'], + writes: [ + ['a', 'x'], + ['upload_mid', 'y'] + ], + subs: [ + ['upload_mid', 'x'], + ['hook_mid', 'x'], + ['consumer', 'y'] + ], + native: [ + ['data_upload', 'upload_mid'], + ['webhook', 'hook_mid'] + ] + }) + const from = validFromStarts(g2) + expect(from.has(sn('upload_mid'))).toBe(false) + expect(from.has(sn('hook_mid'))).toBe(false) + expect(from.has(sn('a'))).toBe(true) // the plain root is still eligible + // and they're barriers, so running downstream from `a` cuts them + consumer. + const bars = nonAutorunTriggerScripts(g2) + expect(bars.has(sn('upload_mid'))).toBe(true) + expect(bars.has(sn('hook_mid'))).toBe(true) + expect(scriptsOf(reachableCutting(buildLineageDag(g2), [sn('a')], bars)).sort()).toEqual(['a']) + }) + + it('keeps a scheduled root that also carries an event trigger', () => { + // schedule wins over the secondary kafka trigger in validStarts, so the + // scheduled root stays --from-eligible (regression guard). + const g2 = graph({ + scripts: ['sched_evt', 'consumer'], + writes: [['sched_evt', 'x']], + subs: [['consumer', 'x']], + native: [ + ['schedule', 'sched_evt'], + ['kafka', 'sched_evt'] + ] + }) + expect(validStarts(g2).has(sn('sched_evt'))).toBe(true) + expect(nonAutorunTriggerScripts(g2).has(sn('sched_evt'))).toBe(true) + expect(validFromStarts(g2).has(sn('sched_evt'))).toBe(true) // union with roots + }) + + it('runs a mid-DAG start plus its downstream WITHOUT re-running upstream', () => { + // Starting at `sub`, the unbounded downstream is {sub, y, reader} — `a`/`x` + // upstream are never pulled in (dbt `--select sub+`). + const dag = buildLineageDag(g()) + const downstream = new Set([sn('sub'), ...descendants(dag, sn('sub'))]) + expect(scriptsOf(downstream).sort()).toEqual(['reader', 'sub']) + expect(downstream.has(sn('a'))).toBe(false) + expect(downstream.has(asset('x'))).toBe(false) + }) +}) + +describe('reachableCutting (barrier cut for "Run + downstream")', () => { + // a → x → k(kafka, also reads x) → z → consumer. Starting at `a` and running + // downstream must cut the event handler `k` AND `consumer` (only reachable + // through it) — else they'd launch with empty args. Mirrors the CLI cut. + const g = () => + graph({ + scripts: ['a', 'k', 'consumer'], + writes: [ + ['a', 'x'], + ['k', 'z'] + ], + reads: [['k', 'x']], + subs: [['consumer', 'z']], + native: [['kafka', 'k']] + }) + + it('detects event-triggered scripts as barriers', () => { + expect(nonAutorunTriggerScripts(g()).has(sn('k'))).toBe(true) + expect(nonAutorunTriggerScripts(g()).has(sn('a'))).toBe(false) + }) + + it('cuts an event descendant and its event-only downstream from a run set', () => { + const dag = buildLineageDag(g()) + const barriers = nonAutorunTriggerScripts(g()) + const runNodes = reachableCutting(dag, [sn('a')], barriers) + expect(scriptsOf(runNodes).sort()).toEqual(['a']) // k + consumer cut + expect(runNodes.has(sn('k'))).toBe(false) + expect(runNodes.has(sn('consumer'))).toBe(false) + }) + + it('protects an explicit start even if it carries an event trigger', () => { + const dag = buildLineageDag(g()) + // start = k itself (user named it); it runs, and so does its downstream. + const barriers = new Set([...nonAutorunTriggerScripts(g())].filter((id) => id !== sn('k'))) + const runNodes = reachableCutting(dag, [sn('k')], barriers) + expect(scriptsOf(runNodes).sort()).toEqual(['consumer', 'k']) + }) + + it('keeps a scheduled event root (and its downstream) reachable from an upstream start', () => { + // a → x → sched_evt(schedule + kafka) → y → consumer. Running downstream + // from `a`, sched_evt is a scheduled root so it must NOT be a barrier even + // though it carries an event trigger — matching the CLI, which excludes all + // valid roots (`starts`), not just the picked start. Mirror of the page's + // barrier construction: nonAutorunTriggerScripts minus validStarts minus start. + const g2 = graph({ + scripts: ['a', 'sched_evt', 'consumer'], + writes: [ + ['a', 'x'], + ['sched_evt', 'y'] + ], + subs: [ + ['sched_evt', 'x'], + ['consumer', 'y'] + ], + native: [ + ['schedule', 'sched_evt'], + ['kafka', 'sched_evt'] + ] + }) + const dag = buildLineageDag(g2) + const roots = validStarts(g2) + const barriers = new Set( + [...nonAutorunTriggerScripts(g2)].filter((id) => !roots.has(id) && id !== sn('a')) + ) + const runNodes = reachableCutting(dag, [sn('a')], barriers) + expect(scriptsOf(runNodes).sort()).toEqual(['a', 'consumer', 'sched_evt']) + // Without the validStarts exclusion, sched_evt (a kafka handler) would be a + // barrier and `consumer` would be dropped — the bug this guards. + const naiveBarriers = new Set([...nonAutorunTriggerScripts(g2)].filter((id) => id !== sn('a'))) + expect(scriptsOf(reachableCutting(dag, [sn('a')], naiveBarriers)).sort()).toEqual(['a']) + }) +}) + describe('buildLineageDownstreamMap (read-aware scheduling)', () => { // a writes x; c only *reads* x (no `// on x`). c must still run after a. const g = graph({ @@ -271,6 +459,47 @@ describe('buildLineageDownstreamMap (read-aware scheduling)', () => { expect(readAware.indegree.get('c')).toBe(1) expect(readAware.nodes).toEqual(['a', 'c']) }) + + it('orders a disjoint-root producer before a data_test that references it', () => { + // Two disjoint roots (the HD-1 repro): `dim` produces the dimension + // `dimc`; `fct` produces `fcto` and has a `// data_test relationships` + // against `dimc`. Without the test edge they are unordered and a cold + // cascade can run `fct` first ("table dimc does not exist"). The test + // edge must place `dim` strictly before `fct`. + const g = graph({ + scripts: ['dim', 'fct'], + writes: [ + ['dim', 'dimc'], + ['fct', 'fcto'] + ], + tests: [['dim', 'fct', 'dimc']] + }) + const selected = new Set(['dim', 'fct']) + const map = buildLineageDownstreamMap(g) + expect([...(map.get('dim') ?? [])]).toEqual(['fct']) + const schedule = computeInducedSchedule(g, selected, map) + expect(schedule.roots).toEqual(['dim']) + expect(schedule.indegree.get('fct')).toBe(1) + expect(schedule.nodes).toEqual(['dim', 'fct']) + expect(schedule.cyclic).toEqual([]) + }) + + it('adds no ordering when the referenced asset has no in-pipeline producer', () => { + // `fct` tests against an external `ext` table nothing produces — the + // backend emits no test edge, so the frontend sees none and `fct` stays + // an independent root (the runtime error stands, as designed). + const g = graph({ + scripts: ['dim', 'fct'], + writes: [['fct', 'fcto']], + tests: [] // no producer for `ext` ⇒ backend omitted the edge + }) + const schedule = computeInducedSchedule( + g, + new Set(['dim', 'fct']), + buildLineageDownstreamMap(g) + ) + expect(schedule.roots.sort()).toEqual(['dim', 'fct']) + }) }) describe('assetUriToNodeId', () => { @@ -280,4 +509,15 @@ describe('assetUriToNodeId', () => { expect(assetUriToNodeId('ducklake://lake/t')).toBe('ducklake:lake/t') expect(assetUriToNodeId('not-a-uri')).toBeUndefined() }) + it('strips leading slashes from S3 keys so s3:/// and s3:// share a node', () => { + // Mirror of Rust `parse_asset_syntax`: `--to s3:///exports/x` must resolve + // to the same canonical node as the graph's `s3object:exports/x`. + expect(assetUriToNodeId('s3:///exports/x')).toBe('s3object:exports/x') + expect(assetUriToNodeId('s3:///exports/x')).toBe(assetUriToNodeId('s3://exports/x')) + // All leading slashes are stripped so a canonical key never starts with + // `/` (the quad-slash `S3Object(s3="/x")` form collapses to `x`). + expect(assetUriToNodeId('s3:////x')).toBe('s3object:x') + // Hive-partition keys and non-S3 kinds are untouched. + expect(assetUriToNodeId('s3:///t/y=2024/f.parquet')).toBe('s3object:t/y=2024/f.parquet') + }) }) diff --git a/frontend/src/lib/components/assets/AssetGraph/boundedCascade.ts b/frontend/src/lib/components/assets/AssetGraph/boundedCascade.ts index 44a26d9c71..6fcabbe71f 100644 --- a/frontend/src/lib/components/assets/AssetGraph/boundedCascade.ts +++ b/frontend/src/lib/components/assets/AssetGraph/boundedCascade.ts @@ -38,14 +38,16 @@ export function assetUriToNodeId(uri: string): string | undefined { // `s3` is the URI prefix for the `s3object` asset kind (mirrors the CLI // `assetUri` and the canvas). All other kinds use their name verbatim. const kind = prefix === 's3' ? 's3object' : prefix - return `${kind}:${m[2]}` + // Mirror Rust `parse_asset_syntax`: strip all leading slashes from S3 keys so + // `s3:///key` (default storage) and `s3://key` resolve to the same node id + // and a canonical key never starts with `/`. + const path = kind === 's3object' ? m[2].replace(/^\/+/, '') : m[2] + return `${kind}:${path}` } // Native trigger kinds that fan out *per event*: a single event always flows // through the whole reactive downstream, so "run up to X now" is not a // meaningful gesture and these are never offered as bounded-run starts. -// `webhook`/`data_upload` have no trigger row in `/assets/graph`, so a root -// whose only entry is one of those reads as a *manual* root below. const EVENT_TRIGGER_KINDS: ReadonlySet = new Set([ 'kafka', 'mqtt', @@ -56,6 +58,20 @@ const EVENT_TRIGGER_KINDS: ReadonlySet = new Set([ 'email' ]) +// Trigger kinds a cascade must NOT auto-run: the per-event kinds above PLUS the +// input-only entrypoints `webhook`/`data_upload`, which need caller-supplied +// input (a request body / an uploaded S3Object) and would run the wrong thing +// with empty args. Mirror of the CLI `NON_AUTORUN_TRIGGER_KINDS`. When the +// deployed `/assets/graph` omits a `webhook`/`data_upload` row (it has none for +// them), such a script simply reads as a manual root here — but whenever the +// marker IS visible (editor overlay / draft), it's excluded from bounded-run +// starts and cut as a barrier, exactly as the CLI does. +const NON_AUTORUN_TRIGGER_KINDS: ReadonlySet = new Set([ + ...EVENT_TRIGGER_KINDS, + 'webhook', + 'data_upload' +]) + export type LineageDag = { /** upstream node id → set of direct downstream node ids. */ down: Map> @@ -70,10 +86,17 @@ export type LineageDag = { * - producer script → asset (write / rw edges) * - asset → reader script (pure-read edges — a data dependency) * - asset → subscriber script (`// on ` triggers) + * - asset → testing script (`// data_test` ordering edges) * * An `rw` edge is treated as production only (script → asset); emitting the * reverse asset → script too would make every upsert a 2-cycle through its own * asset. + * + * `test_edges` are modeled through the referenced asset (asset → testing + * script), NOT as a direct producer → testing-script hop: the producer already + * has a write edge to that asset, so this yields producer → asset → testing + * script and keeps the two-hop (script → asset → script) invariant that + * `buildLineageDownstreamMap` relies on. */ export function buildLineageDag(g: AssetGraphResponse): LineageDag { const down = new Map>() @@ -107,6 +130,13 @@ export function buildLineageDag(g: AssetGraphResponse): LineageDag { if (t.trigger_kind !== 'asset' || t.runnable_kind !== 'script') continue addEdge(assetKey(t), scriptNodeId(t.runnable_path)) } + // Data-test ordering edges: the referenced asset must exist before the + // tested script runs. Routed through the asset node so the existing + // producer → asset write edge extends into producer → asset → testing script. + for (const t of g.test_edges ?? []) { + if (t.runnable_kind !== 'script') continue + addEdge(assetKey(t), scriptNodeId(t.runnable_path)) + } return { down, up, nodes } } @@ -138,6 +168,37 @@ export function ancestors(dag: LineageDag, n: string): Set { return closure(dag.up, n) } +/** + * Nodes reachable from `starts` over the lineage DAG, treating `barriers` as cut + * points: a barrier node is neither included NOR traversed through, so a node + * reachable ONLY via a barrier is excluded while one also reachable via another + * path stays. Mirror of the CLI `reachableCutting`. Used to keep event handlers + * (and their event-only downstream) out of a cascade run — running such a + * consumer whose producer was skipped would feed it missing/stale inputs. + */ +export function reachableCutting( + dag: LineageDag, + starts: Iterable, + barriers: Set +): Set { + const seen = new Set() + const queue: string[] = [] + for (const s of starts) { + if (barriers.has(s) || seen.has(s)) continue + seen.add(s) + queue.push(s) + } + while (queue.length > 0) { + const n = queue.shift()! + for (const next of dag.down.get(n) ?? []) { + if (barriers.has(next) || seen.has(next)) continue + seen.add(next) + queue.push(next) + } + } + return seen +} + export type BoundedResult = { /** Path-between node set (scripts + assets), always including `start`. */ nodes: Set @@ -201,6 +262,56 @@ export function validStarts(g: AssetGraphResponse): Set { return out } +/** + * Script node ids eligible as an EXPLICIT bounded-run start from *anywhere* in + * the DAG (dbt's `--select model+`): every script that can run with empty args — + * i.e. all scripts except event-triggered ones (kafka/mqtt/nats/postgres/sqs/ + * gcp/email fan out per event and have no "run now" gesture). Unlike + * `validStarts` (schedule/manual roots only), this INCLUDES mid-DAG asset + * subscribers and pure readers, so "Run + downstream" can begin at any model — + * that node plus its transitive downstream runs, upstream is never re-run. + * (`webhook`/`data_upload` have no trigger row here — same as `validStarts` they + * read as manual roots and are already included.) + */ +export function validFromStarts(g: AssetGraphResponse): Set { + const nonAutorunScripts = new Set() + for (const t of g.triggers ?? []) { + if (t.runnable_kind !== 'script') continue + if (NON_AUTORUN_TRIGGER_KINDS.has(t.trigger_kind)) nonAutorunScripts.add(t.runnable_path) + } + // Seed with schedule/manual roots: `validStarts` lets a schedule identity win + // over a secondary non-autorun trigger, so a scheduled root that also carries + // e.g. a `// on kafka` stays `--from`-eligible. The mid-DAG loop then adds only + // scripts that can run with empty args — a `webhook`/`data_upload` subscriber + // is NOT added (it needs caller-supplied input). + const out = new Set(validStarts(g)) + for (const r of g.runnables ?? []) { + if (r.usage_kind !== 'script') continue + if (!nonAutorunScripts.has(r.path)) out.add(scriptNodeId(r.path)) + } + return out +} + +/** + * Script node ids carrying a non-autorun trigger (event kinds kafka/mqtt/…/email + * PLUS input-only webhook/data_upload) — they fan out per event or need + * caller-supplied input, so a cascade must cut them (as `barriers` for + * `reachableCutting`) even when they're a lineage descendant of the start. + * Mirror of the CLI `nonAutorunTriggerScripts`. Only detects what the graph + * surfaces: the deployed `/assets/graph` omits `webhook`/`data_upload` rows, so + * such a handler is only cut when its marker is visible (editor overlay / draft) + * — the same limitation as `validStarts`; the CLI closes it via graph enrichment. + */ +export function nonAutorunTriggerScripts(g: AssetGraphResponse): Set { + const out = new Set() + for (const t of g.triggers ?? []) { + if (t.runnable_kind === 'script' && NON_AUTORUN_TRIGGER_KINDS.has(t.trigger_kind)) { + out.add(scriptNodeId(t.runnable_path)) + } + } + return out +} + /** Project a node-id set to the script paths it contains (run targets). */ export function scriptsOf(nodes: Iterable): string[] { const out: string[] = [] diff --git a/frontend/src/lib/components/assets/AssetGraph/cascadeOrchestrator.test.ts b/frontend/src/lib/components/assets/AssetGraph/cascadeOrchestrator.test.ts index 92d7a977fe..3a0fd4fbd3 100644 --- a/frontend/src/lib/components/assets/AssetGraph/cascadeOrchestrator.test.ts +++ b/frontend/src/lib/components/assets/AssetGraph/cascadeOrchestrator.test.ts @@ -201,7 +201,48 @@ describe('runSelection', () => { expect(r.launched.indexOf('c')).toBeGreaterThan(r.launched.indexOf('b')) }) - it('stops scheduling after a failure', async () => { + it('skips a failed node’s descendants but keeps independent branches running', async () => { + // Two independent chains: a → b and c → d. `a` fails; `b` (its descendant) + // must be skipped, but `d` depends only on the successful `c`, so it must + // still run — a failure must not stall unrelated branches. + const sched = schedule( + [ + ['a', 'b'], + ['c', 'd'] + ], + ['a', 'b', 'c', 'd'], + ['a', 'c'] + ) + const r = fakeRunner({ a: 'failure' }) + const res = await runSelection({ schedule: sched, ...r }) + expect(res.ok).toBe(false) + expect(res.statuses.get('a')?.status).toBe('failure') + expect(res.statuses.get('b')?.status).toBe('skipped') + expect(res.statuses.get('c')?.status).toBe('success') + expect(res.statuses.get('d')?.status).toBe('success') + expect(r.launched).toContain('d') + expect(r.launched).not.toContain('b') + }) + + it('skips a join node when any one of its upstreams fails', async () => { + // {a, b} → c. `a` fails; `c` needs both, so it must be skipped even though + // `b` succeeds — a poisoned lineage isn’t rescued by a sibling success. + const sched = schedule( + [ + ['a', 'c'], + ['b', 'c'] + ], + ['a', 'b', 'c'], + ['a', 'b'] + ) + const r = fakeRunner({ a: 'failure' }) + const res = await runSelection({ schedule: sched, ...r }) + expect(res.ok).toBe(false) + expect(res.statuses.get('c')?.status).toBe('skipped') + expect(r.launched).not.toContain('c') + }) + + it('stops scheduling a failed node’s chain', async () => { const sched = schedule([['a', 'b']], ['a', 'b'], ['a']) const r = fakeRunner({ a: 'failure' }) const res = await runSelection({ schedule: sched, ...r }) diff --git a/frontend/src/lib/components/assets/AssetGraph/cascadeOrchestrator.ts b/frontend/src/lib/components/assets/AssetGraph/cascadeOrchestrator.ts index 16d357af00..424f1581ec 100644 --- a/frontend/src/lib/components/assets/AssetGraph/cascadeOrchestrator.ts +++ b/frontend/src/lib/components/assets/AssetGraph/cascadeOrchestrator.ts @@ -128,8 +128,9 @@ export type SelectionRunOptions = { * Execute an arbitrary selected set of scripts (e.g. a bounded-cascade * selection) in topological order. Unlike `runCascade` there is no single * privileged root — every `schedule.roots` entry is seeded at once and a node - * runs as soon as its in-set upstreams all succeed. Failure stops *scheduling* - * (in-flight jobs finish); everything not yet started ends 'skipped'. + * runs as soon as its in-set upstreams all succeed. A failure abandons only + * that node's lineage (its transitive descendants end 'skipped'); INDEPENDENT + * branches keep running. In-flight jobs always finish. */ export async function runSelection(opts: SelectionRunOptions): Promise { const { schedule, launch, waitTerminal, onUpdate } = opts @@ -137,10 +138,29 @@ export async function runSelection(opts: SelectionRunOptions): Promise() const inFlight = new Set>() const emit = () => onUpdate?.(new Map(statuses)) + function poison(path: string) { + const stack = [path] + while (stack.length > 0) { + const n = stack.pop()! + for (const s of schedule.edges.get(n) ?? []) { + if (!poisoned.has(s)) { + poisoned.add(s) + stack.push(s) + } + } + } + } + function schedule_(path: string) { const p = runNode(path).finally(() => inFlight.delete(p)) inFlight.add(p) @@ -159,6 +179,7 @@ export async function runSelection(opts: SelectionRunOptions): Promise { expect(mergeColumnLineage([], annotated)).toEqual(annotated) }) }) + +// Mirror of the Rust `parse_duration_secs` tests (windmill-common assets.rs) +// — the freshness chip's staleness verdict depends on identical parsing. +describe('parseDurationSecs', () => { + it('parses suffixed durations', () => { + expect(parseDurationSecs('30s')).toBe(30) + expect(parseDurationSecs('5m')).toBe(300) + expect(parseDurationSecs('2h')).toBe(7200) + expect(parseDurationSecs('1d')).toBe(86400) + }) + + it('bare integer means seconds', () => { + expect(parseDurationSecs('45')).toBe(45) + }) + + it('tolerates surrounding whitespace', () => { + expect(parseDurationSecs(' 5 m ')).toBe(300) + }) + + it('accepts an explicit plus sign (Rust i64 parsing does)', () => { + expect(parseDurationSecs('+5m')).toBe(300) + expect(parseDurationSecs('+45')).toBe(45) + }) + + it('rejects malformed / non-positive input', () => { + expect(parseDurationSecs('')).toBeUndefined() + expect(parseDurationSecs('h')).toBeUndefined() + expect(parseDurationSecs('1.5h')).toBeUndefined() + expect(parseDurationSecs('-5m')).toBeUndefined() + expect(parseDurationSecs('0')).toBeUndefined() + expect(parseDurationSecs('fast')).toBeUndefined() + }) + + it('rejects values beyond i32 seconds (mirrors backend cap)', () => { + expect(parseDurationSecs('999999999d')).toBeUndefined() + }) +}) diff --git a/frontend/src/lib/components/assets/AssetGraph/parsePipelineAnnotations.ts b/frontend/src/lib/components/assets/AssetGraph/parsePipelineAnnotations.ts index f33a03fff5..47eea81246 100644 --- a/frontend/src/lib/components/assets/AssetGraph/parsePipelineAnnotations.ts +++ b/frontend/src/lib/components/assets/AssetGraph/parsePipelineAnnotations.ts @@ -67,6 +67,28 @@ export type FreshnessSpec = { duration: string } +// Mirrors backend `parse_duration_secs` (windmill-common assets.rs): a bare +// integer means seconds, otherwise `` with an `s`/`m`/`h`/`d` suffix +// (e.g. `30s`, `5m`, `2h`, `1d`). Returns undefined for malformed or +// non-positive input so a typo'd `// freshness` window fails safe (the chip +// stays neutral instead of guessing a staleness verdict). +export function parseDurationSecs(s: string): number | undefined { + const t = s.trim() + if (!t) return undefined + const last = t[t.length - 1] + const mult = + last === 's' ? 1 : last === 'm' ? 60 : last === 'h' ? 3600 : last === 'd' ? 86400 : undefined + const num = (mult !== undefined ? t.slice(0, -1) : t).trim() + // `+?`: Rust's i64 parsing accepts an explicit plus sign (`+5m`), so the + // mirror must too — divergence here would leave the chip neutral for a + // window the deploy path and watchdog honor. + if (mult === undefined && !/^\+?\d+$/.test(t)) return undefined + if (!/^\+?\d+$/.test(num)) return undefined + const secs = Number(num) * (mult ?? 1) + if (!Number.isSafeInteger(secs) || secs <= 0 || secs > 2147483647) return undefined + return secs +} + // `// retry []` — see backend RetrySpec. Delay is kept as the // raw duration string and resolved to seconds at deploy. export type RetrySpec = { @@ -95,6 +117,20 @@ export type MaterializeSpec = { track?: string[] // SCD2 hard-delete-close (`deletes=close`): close absent keys; absent === false closeDeleted?: boolean + // `on_schema_change=ignore` opts the produced asset out of downstream + // schema-contract warnings (save-time metadata only). Default `warn`; + // `fail` is deliberately unrecognized in v1 (saves never hard-block). + onSchemaChange?: 'warn' | 'ignore' +} + +// The `_current` SCD2 companion view this managed materialize also +// produces, or `undefined` when it isn't a managed scd2 target. Mirrors Rust +// `MaterializeSpec::scd2_current_target`: managed scd2 creates the base table +// *and* the `_current` view each run; `manual` mode owns its own DDL and creates +// no companion. The graph surfaces register it as a second write of the producer +// so a read of the view links back instead of orphaning. +export function scd2CurrentTargetPath(m: MaterializeSpec): string | undefined { + return m.scd2 && !m.manual ? `${m.targetPath}_current` : undefined } // `// data_test …` — a data-quality assertion run against the @@ -221,7 +257,18 @@ function parseKvOpts(s: string): Map { function parseAssetSyntax(s: string): PipelineTriggerAsset | undefined { for (const [prefix, kind] of ASSET_PREFIXES) { if (s.startsWith(prefix)) { - return { kind, path: s.slice(prefix.length) } + let path = s.slice(prefix.length) + // Mirror the Rust `parse_asset_syntax` S3 canonicalization: strip all + // leading slashes so the SDK object form (`s3:///key`, default + // storage) and DuckDB / `// on s3://key` share one asset path, and a + // canonical key never starts with `/` (so ref reconstruction + // round-trips). Without this the live graph preview would show + // disconnected `/key` and `key` nodes. S3-only; leading slashes only, + // so Hive-partition keys are untouched. + if (kind === 's3object') { + path = path.replace(/^\/+/, '') + } + return { kind, path } } } return undefined @@ -242,7 +289,8 @@ function parseAssetSyntaxDefault(s: string): PipelineTriggerAsset | undefined { // managed mode; a leading `scd2` word is an alias for the `history` flag; the // next token is the target asset URI (default-syntax shorthands enabled); the // remainder are strategy options (`append` flag, `key=
`, `history` flag, -// `track=`, `deletes=close`). Missing/empty target → undefined (dropped). +// `track=`, `deletes=close`, `on_schema_change=ignore`). Missing/empty +// target → undefined (dropped). function parseMaterializeSpec(s: string): MaterializeSpec | undefined { // One optional leading mode keyword: `manual` (track-only) or `scd2` (an alias // for the `history` flag below). @@ -282,6 +330,9 @@ function parseMaterializeSpec(s: string): MaterializeSpec | undefined { // `deletes=close` (scd2 only) opts into hard-delete-close; any other value // (or absence) keeps the soft-delete default. const closeDeleted = opts.get('deletes') === 'close' + // `on_schema_change=ignore` suppresses downstream contract warnings; any + // other value (or absence) keeps the `warn` default, fail-safe like `deletes=`. + const onSchemaChange = opts.get('on_schema_change') === 'ignore' ? 'ignore' : 'warn' return { targetKind: asset.kind, targetPath: asset.path, @@ -290,7 +341,8 @@ function parseMaterializeSpec(s: string): MaterializeSpec | undefined { uniqueKey, scd2, track, - closeDeleted + closeDeleted, + onSchemaChange } } diff --git a/frontend/src/lib/components/assets/AssetGraph/partitionBuckets.test.ts b/frontend/src/lib/components/assets/AssetGraph/partitionBuckets.test.ts new file mode 100644 index 0000000000..a8a47af259 --- /dev/null +++ b/frontend/src/lib/components/assets/AssetGraph/partitionBuckets.test.ts @@ -0,0 +1,161 @@ +import { describe, expect, it } from 'vitest' +import { + bucketFor, + bucketFromInputValue, + defaultBucket, + inputValueFromBucket, + isBeforeStart, + isValidStart, + isValidTimeZone, + partitionInputType, + partitionMetadataError, + recentBuckets, + startBucketOf, + usesCalendarPicker +} from './partitionBuckets' +import type { PartitionSpec } from './parsePipelineAnnotations' + +const spec = (kind: PartitionSpec['kind'], extra: Partial = {}): PartitionSpec => + ({ kind, ...extra }) as PartitionSpec + +// A fixed UTC instant: 2026-07-05 14:37Z (a Sunday — ISO week 27 of 2026). +// Explicit UTC so the assertions are independent of the test runner's TZ, and +// they exercise the same UTC default the backend uses when `spec.tz` is absent. +const at = new Date(Date.UTC(2026, 6, 5, 14, 37, 0)) + +describe('bucketFor mirrors backend default_format (UTC)', () => { + it('daily -> %Y-%m-%d', () => { + expect(bucketFor(spec('daily'), at)).toBe('2026-07-05') + }) + it('hourly -> %Y-%m-%dT%H', () => { + expect(bucketFor(spec('hourly'), at)).toBe('2026-07-05T14') + }) + it('monthly -> %Y-%m', () => { + expect(bucketFor(spec('monthly'), at)).toBe('2026-07') + }) + it('weekly -> ISO %G-W%V', () => { + expect(bucketFor(spec('weekly'), at)).toBe('2026-W27') + }) +}) + +describe('bucketFor honours spec.tz', () => { + // 2026-07-05 02:30Z is still 2026-07-04 in America/New_York (UTC-4 in July). + const nearMidnight = new Date(Date.UTC(2026, 6, 5, 2, 30, 0)) + it('shifts the day boundary by the producer tz', () => { + expect(bucketFor(spec('daily'), nearMidnight)).toBe('2026-07-05') // UTC default + expect(bucketFor(spec('daily', { tz: 'America/New_York' }), nearMidnight)).toBe('2026-07-04') + }) + it('shifts the hour bucket by the producer tz', () => { + // 02:30Z -> 22 (previous day) in New York. + expect(bucketFor(spec('hourly', { tz: 'America/New_York' }), nearMidnight)).toBe( + '2026-07-04T22' + ) + }) +}) + +describe('defaultBucket honours the start= anchor', () => { + // `at` is 2026-07-05. + it('seeds the current bucket when at or after start', () => { + expect(defaultBucket(spec('daily'), at)).toBe('2026-07-05') + expect(defaultBucket(spec('daily', { start: '2026-01-01' }), at)).toBe('2026-07-05') + expect(defaultBucket(spec('daily', { start: '2026-07-05' }), at)).toBe('2026-07-05') // == start, not before + }) + it('seeds the start bucket (never a pre-start one) when before start', () => { + expect(defaultBucket(spec('daily', { start: '2026-08-01' }), at)).toBe('2026-08-01') + expect(defaultBucket(spec('monthly', { start: '2026-08-01' }), at)).toBe('2026-08') + expect(defaultBucket(spec('hourly', { start: '2026-08-01' }), at)).toBe('2026-08-01T00') + }) + it('isBeforeStart mirrors the backend date comparison', () => { + expect(isBeforeStart(spec('daily', { start: '2026-08-01' }), at)).toBe(true) + expect(isBeforeStart(spec('daily', { start: '2026-07-05' }), at)).toBe(false) + expect(isBeforeStart(spec('daily'), at)).toBe(false) + }) + it('startBucketOf renders the anchor in the cadence, undefined when unset', () => { + expect(startBucketOf(spec('daily', { start: '2026-08-01' }))).toBe('2026-08-01') + expect(startBucketOf(spec('monthly', { start: '2026-08-15' }))).toBe('2026-08') + expect(startBucketOf(spec('hourly', { start: '2026-08-01' }))).toBe('2026-08-01T00') + expect(startBucketOf(spec('daily'))).toBeUndefined() + }) +}) + +describe('malformed metadata fails safe (parity with backend validation)', () => { + it('isValidTimeZone rejects garbage, accepts real zones and absence', () => { + expect(isValidTimeZone(undefined)).toBe(true) + expect(isValidTimeZone('UTC')).toBe(true) + expect(isValidTimeZone('America/New_York')).toBe(true) + expect(isValidTimeZone('Not/AZone')).toBe(false) + expect(isValidTimeZone('garbage')).toBe(false) + }) + it('isValidStart rejects malformed and JS-normalized dates', () => { + expect(isValidStart(undefined)).toBe(true) + expect(isValidStart('2026-08-01')).toBe(true) + expect(isValidStart('2026-02-31')).toBe(false) // JS would roll to Mar 3 + expect(isValidStart('2026-13-01')).toBe(false) + expect(isValidStart('08/01/2026')).toBe(false) + }) + it('partitionMetadataError reports the first problem, else undefined', () => { + expect(partitionMetadataError(spec('daily'))).toBeUndefined() + expect( + partitionMetadataError(spec('daily', { tz: 'America/New_York', start: '2026-08-01' })) + ).toBeUndefined() + expect(partitionMetadataError(spec('daily', { tz: 'Not/AZone' }))).toContain('timezone') + expect(partitionMetadataError(spec('daily', { start: '2026-02-31' }))).toContain('start date') + }) + it('bucketFor never throws on an invalid tz (falls back to UTC)', () => { + expect(bucketFor(spec('daily', { tz: 'Not/AZone' }), at)).toBe('2026-07-05') + }) + it('an invalid start is treated as no anchor (isBeforeStart/startBucketOf)', () => { + expect(isBeforeStart(spec('daily', { start: '2026-02-31' }), at)).toBe(false) + expect(startBucketOf(spec('daily', { start: '2026-02-31' }))).toBeUndefined() + }) +}) + +describe('partitionInputType', () => { + it('maps each calendar kind to its native input', () => { + expect(partitionInputType(spec('daily'))).toBe('date') + expect(partitionInputType(spec('hourly'))).toBe('datetime-local') + expect(partitionInputType(spec('weekly'))).toBe('week') + expect(partitionInputType(spec('monthly'))).toBe('month') + }) + it('falls back to text for dynamic and custom-format specs', () => { + expect(partitionInputType(spec('dynamic', { key: '$.tenant' }))).toBe('text') + expect(partitionInputType(spec('daily', { format: '%Y/%m/%d' }))).toBe('text') + expect(usesCalendarPicker(spec('dynamic', { key: '$.tenant' }))).toBe(false) + expect(usesCalendarPicker(spec('daily', { format: '%Y/%m/%d' }))).toBe(false) + }) +}) + +describe('native input <-> bucket round-trip', () => { + it('hourly truncates the datetime-local minutes and restores :00', () => { + expect(bucketFromInputValue(spec('hourly'), '2026-07-05T14:37')).toBe('2026-07-05T14') + expect(inputValueFromBucket(spec('hourly'), '2026-07-05T14')).toBe('2026-07-05T14:00') + }) + it('non-hourly kinds pass through unchanged', () => { + expect(bucketFromInputValue(spec('daily'), '2026-07-05')).toBe('2026-07-05') + expect(inputValueFromBucket(spec('weekly'), '2026-W27')).toBe('2026-W27') + }) + it('empty stays empty', () => { + expect(bucketFromInputValue(spec('hourly'), '')).toBe('') + expect(inputValueFromBucket(spec('daily'), '')).toBe('') + }) +}) + +describe('recentBuckets (UTC)', () => { + it('walks back day by day, most recent first', () => { + expect(recentBuckets(spec('daily'), at, 3)).toEqual(['2026-07-05', '2026-07-04', '2026-07-03']) + }) + it('walks back hour by hour', () => { + expect(recentBuckets(spec('hourly'), at, 3)).toEqual([ + '2026-07-05T14', + '2026-07-05T13', + '2026-07-05T12' + ]) + }) + it('walks back month by month across a year boundary without day roll-over', () => { + const jan31 = new Date(Date.UTC(2026, 0, 31, 0, 0, 0)) + expect(recentBuckets(spec('monthly'), jan31, 3)).toEqual(['2026-01', '2025-12', '2025-11']) + }) + it('walks back week by week', () => { + expect(recentBuckets(spec('weekly'), at, 3)).toEqual(['2026-W27', '2026-W26', '2026-W25']) + }) +}) diff --git a/frontend/src/lib/components/assets/AssetGraph/partitionBuckets.ts b/frontend/src/lib/components/assets/AssetGraph/partitionBuckets.ts new file mode 100644 index 0000000000..401ef82cae --- /dev/null +++ b/frontend/src/lib/components/assets/AssetGraph/partitionBuckets.ts @@ -0,0 +1,261 @@ +// Client-side partition-bucket math for the run form's partition picker. +// Produces the same canonical bucket strings the backend renders in +// `windmill-common/src/partition_ee.rs` (`resolve_time_partition` / +// `default_format`): the instant is localized to the spec's timezone +// (`spec.tz`, defaulting to UTC — NOT the browser's zone) and then formatted. +// Getting the zone right matters — a browser in a non-UTC zone would otherwise +// seed a default bucket, and compare against materialized rows, off by a +// day/hour near every boundary and always for an explicit `tz=` spec. +// +// daily %Y-%m-%d -> 2026-07-05 +// hourly %Y-%m-%dT%H -> 2026-07-05T14 +// weekly %G-W%V (ISO) -> 2026-W27 +// monthly %Y-%m -> 2026-07 +// +// Pure module (no Svelte runes) so the mapping is unit-testable. + +import type { PartitionSpec } from './parsePipelineAnnotations' + +export type PartitionInputType = 'date' | 'month' | 'week' | 'datetime-local' | 'text' + +// A spec carrying a custom strftime `format` can't be reproduced by the native +// date pickers (arbitrary strftime), and `dynamic` partitions are a free-form +// key extracted from the payload — both fall back to a plain text input. +export function usesCalendarPicker(spec: PartitionSpec): boolean { + return spec.kind !== 'dynamic' && !spec.format +} + +export function partitionInputType(spec: PartitionSpec): PartitionInputType { + if (!usesCalendarPicker(spec)) return 'text' + switch (spec.kind) { + case 'monthly': + return 'month' + case 'weekly': + return 'week' + case 'hourly': + return 'datetime-local' + default: + return 'date' + } +} + +function pad(n: number): string { + return String(n).padStart(2, '0') +} + +const ZONED_PARTS_OPTS: Intl.DateTimeFormatOptions = { + year: 'numeric', + month: '2-digit', + day: '2-digit', + hour: '2-digit', + minute: '2-digit', + second: '2-digit', + hourCycle: 'h23' +} + +// Re-express `at` as a Date whose UTC fields equal the wall-clock in `tz`, so +// all downstream field reads / calendar arithmetic can use the UTC getters and +// stay in the producer's zone. `hourCycle: 'h23'` keeps hours 00–23. A +// malformed `tz` (rejected by `Intl`) falls back to UTC rather than throwing — +// the backend validates `tz=` and is the source of truth for the error; the +// picker must never crash the graph view (`partitionMetadataError` gates +// auto-seeding so a bogus bucket isn't silently sent). +function zonedAsUtc(at: Date, tz: string): Date { + let parts: Intl.DateTimeFormatPart[] + try { + parts = new Intl.DateTimeFormat('en-US', { ...ZONED_PARTS_OPTS, timeZone: tz }).formatToParts( + at + ) + } catch { + parts = new Intl.DateTimeFormat('en-US', { + ...ZONED_PARTS_OPTS, + timeZone: 'UTC' + }).formatToParts(at) + } + const g = (t: string) => Number(parts.find((p) => p.type === t)?.value) + return new Date( + Date.UTC(g('year'), g('month') - 1, g('day'), g('hour'), g('minute'), g('second')) + ) +} + +// `tz=` is valid iff `Intl` accepts it (absent === UTC === valid). +export function isValidTimeZone(tz?: string): boolean { + if (!tz) return true + try { + new Intl.DateTimeFormat('en-US', { timeZone: tz }) + return true + } catch { + return false + } +} + +// `start=` is valid iff it's a real `YYYY-MM-DD` calendar date. The round-trip +// check rejects dates JS would silently normalize (e.g. `2026-02-31` → Mar 3), +// which the backend's `NaiveDate::parse_from_str` also rejects. +export function isValidStart(start?: string): boolean { + if (!start) return true + const m = start.match(/^(\d{4})-(\d{2})-(\d{2})$/) + if (!m) return false + const y = Number(m[1]) + const mo = Number(m[2]) + const d = Number(m[3]) + const dt = new Date(Date.UTC(y, mo - 1, d)) + return dt.getUTCFullYear() === y && dt.getUTCMonth() === mo - 1 && dt.getUTCDate() === d +} + +// A human-readable reason the `// partitioned` metadata is unusable, or +// undefined when it's sound. Mirrors the backend's `tz=` / `start=` validation +// so the picker can refuse to auto-seed a bucket the backend would reject. +export function partitionMetadataError(spec: PartitionSpec): string | undefined { + if (!isValidTimeZone(spec.tz)) return `invalid timezone "${spec.tz}"` + if (!isValidStart(spec.start)) return `invalid start date "${spec.start}" (want YYYY-MM-DD)` + return undefined +} + +// ISO 8601 week-numbering year + week (chrono's %G / %V) of a UTC-substituted +// date. Standard "nearest Thursday" algorithm, all in UTC. +function isoWeekOf(d: Date): { isoYear: number; week: number } { + const date = new Date(Date.UTC(d.getUTCFullYear(), d.getUTCMonth(), d.getUTCDate())) + const dayNum = (date.getUTCDay() + 6) % 7 // Mon=0 … Sun=6 + date.setUTCDate(date.getUTCDate() - dayNum + 3) // Thursday of this week + const isoYear = date.getUTCFullYear() + const firstThursday = new Date(Date.UTC(isoYear, 0, 4)) + const fdNum = (firstThursday.getUTCDay() + 6) % 7 + firstThursday.setUTCDate(firstThursday.getUTCDate() - fdNum + 3) + const week = 1 + Math.round((date.getTime() - firstThursday.getTime()) / (7 * 86400000)) + return { isoYear, week } +} + +// Render a UTC-substituted date into its canonical bucket for the cadence. +function fmtBucket(kind: PartitionSpec['kind'], d: Date): string { + const y = d.getUTCFullYear() + const m = d.getUTCMonth() + 1 + const day = d.getUTCDate() + const h = d.getUTCHours() + switch (kind) { + case 'hourly': + return `${y}-${pad(m)}-${pad(day)}T${pad(h)}` + case 'monthly': + return `${y}-${pad(m)}` + case 'weekly': { + const { isoYear, week } = isoWeekOf(d) + return `${isoYear}-W${pad(week)}` + } + default: + return `${y}-${pad(m)}-${pad(day)}` + } +} + +export function bucketFor(spec: PartitionSpec, at: Date): string { + return fmtBucket(spec.kind, zonedAsUtc(at, spec.tz ?? 'UTC')) +} + +// The zoned start date (`spec.start`, `YYYY-MM-DD`) as a UTC-substituted Date at +// 00:00, or undefined if unset/malformed. `start` is a plain date in the +// producer's tz — the backend parses it as a NaiveDate and compares by date. +function startDate(spec: PartitionSpec): Date | undefined { + if (!spec.start || !isValidStart(spec.start)) return undefined + const m = spec.start.match(/^(\d{4})-(\d{2})-(\d{2})$/)! + return new Date(Date.UTC(Number(m[1]), Number(m[2]) - 1, Number(m[3]))) +} + +// True when `at` (localized to the producer tz) falls on a date before the +// `start=` anchor — exactly the backend's `local.date_naive() < start_date` +// check (`resolve_time_partition`), which resolves such an instant to no +// partition. +export function isBeforeStart(spec: PartitionSpec, at: Date): boolean { + const start = startDate(spec) + if (!start) return false + const zoned = zonedAsUtc(at, spec.tz ?? 'UTC') + const zonedDate = Date.UTC(zoned.getUTCFullYear(), zoned.getUTCMonth(), zoned.getUTCDate()) + return zonedDate < start.getTime() +} + +// The canonical bucket of the `start=` anchor (its date at 00:00), or undefined +// if unset. Buckets sort lexicographically within a cadence, so callers can +// compare against it to drop pre-start buckets. +export function startBucketOf(spec: PartitionSpec): string | undefined { + const start = startDate(spec) + return start ? fmtBucket(spec.kind, start) : undefined +} + +// The bucket to pre-fill the picker with. Normally the current bucket (matching +// the backend's "absent partition arg -> current bucket" resolution), but when +// the current instant is before the `start=` anchor the backend would resolve +// to NO partition — so default to the first valid bucket (the start) rather +// than a pre-start one the worker would take verbatim and materialize early. +export function defaultBucket(spec: PartitionSpec, at: Date): string { + if (isBeforeStart(spec, at)) { + const start = startDate(spec) + if (start) return fmtBucket(spec.kind, start) + } + return bucketFor(spec, at) +} + +// Native input value -> canonical bucket. Only hourly differs: datetime-local +// carries a minute component the hourly bucket drops. The picked wall-clock is +// taken verbatim as the bucket (the user picks in the producer's frame), so no +// timezone conversion happens here. +export function bucketFromInputValue(spec: PartitionSpec, inputValue: string): string { + if (!inputValue) return '' + if (spec.kind === 'hourly') { + const m = inputValue.match(/^(\d{4}-\d{2}-\d{2}T\d{2})/) + return m ? m[1] : inputValue + } + return inputValue +} + +// Canonical bucket -> native input value. Only hourly differs: datetime-local +// needs a minute component the bucket omits. +export function inputValueFromBucket(spec: PartitionSpec, bucket: string): string { + if (!bucket) return '' + if (spec.kind === 'hourly') { + return /T\d{2}$/.test(bucket) ? `${bucket}:00` : bucket + } + return bucket +} + +// The last `count` buckets ending at (and including) `now`, most-recent first, +// localized to `spec.tz`. Arithmetic walks calendar units in the zoned frame, +// so it's exact across DST (no ±1 drift). Undefined for non-calendar specs +// (the caller guards on `usesCalendarPicker`). +export function recentBuckets(spec: PartitionSpec, now: Date, count: number): string[] { + const base = zonedAsUtc(now, spec.tz ?? 'UTC') + const out: string[] = [] + for (let i = 0; i < count; i++) { + const d = new Date(base) + switch (spec.kind) { + case 'hourly': + d.setUTCHours(d.getUTCHours() - i) + break + case 'weekly': + d.setUTCDate(d.getUTCDate() - 7 * i) + break + case 'monthly': + // Normalize to the 1st first so subtracting months can't roll over + // a short target month (e.g. Mar 31 − 1mo → Mar 3). + d.setUTCDate(1) + d.setUTCMonth(d.getUTCMonth() - i) + break + default: + d.setUTCDate(d.getUTCDate() - i) + } + out.push(fmtBucket(spec.kind, d)) + } + return out +} + +// How many recent buckets the "missing partitions" hint scans, per kind — a +// window that reads as "recent" for each cadence without flooding the hint. +export function recentWindow(kind: PartitionSpec['kind']): number { + switch (kind) { + case 'hourly': + return 24 + case 'weekly': + return 8 + case 'monthly': + return 6 + default: + return 14 + } +} diff --git a/frontend/src/lib/components/assets/AssetGraph/pipelineHistory.svelte.ts b/frontend/src/lib/components/assets/AssetGraph/pipelineHistory.svelte.ts index 50ad3e1d6e..d654085179 100644 --- a/frontend/src/lib/components/assets/AssetGraph/pipelineHistory.svelte.ts +++ b/frontend/src/lib/components/assets/AssetGraph/pipelineHistory.svelte.ts @@ -80,7 +80,15 @@ export function usePipelineHistory( kind: j.job_kind.startsWith('flow') ? 'flow' : 'script', status: j.success ? 'success' : 'failure', source: j.schedule_path ? 'schedule' : 'run', - at: j.started_at ?? j.created_at + at: j.started_at ?? j.created_at, + // Same completion-time derivation as the live poll — + // the freshness chip compares against completion, and + // `at` (start) would read a long run as older than its + // output actually is. + completedAt: + j.started_at != undefined + ? new Date(new Date(j.started_at).getTime() + j.duration_ms).toISOString() + : undefined }) } sawFullPage = rows.length === PER_PAGE diff --git a/frontend/src/lib/components/assets/AssetGraph/pipelineTemplates.dataTest.test.ts b/frontend/src/lib/components/assets/AssetGraph/pipelineTemplates.dataTest.test.ts new file mode 100644 index 0000000000..2acdf2abfb --- /dev/null +++ b/frontend/src/lib/components/assets/AssetGraph/pipelineTemplates.dataTest.test.ts @@ -0,0 +1,56 @@ +import { describe, expect, it } from 'vitest' +import { + autoOutputAsset, + compatibleOutputKinds, + generatePipelineDraft, + PIPELINE_OUTPUT_KINDS +} from './pipelineTemplates' + +// The `data_test` output kind scaffolds a *custom* (singular) data test: a +// standalone DuckDB script referenced from a materialize script's +// `-- data_test ` line. It must be a single SELECT that reads the +// freshly-materialized target through the internal `_wm_target` schema — the +// two rules the backend's self-teaching errors enforce. +describe('data_test scaffold', () => { + it('is a DuckDB-only output kind exposed in the picker', () => { + expect(compatibleOutputKinds('duckdb')).toContain('data_test') + expect(compatibleOutputKinds('python3')).not.toContain('data_test') + expect(PIPELINE_OUTPUT_KINDS.map((k) => k.id)).toContain('data_test') + }) + + it('produces no output asset (it asserts against an existing target)', () => { + expect(autoOutputAsset('data_test', 'folder', 'duckdb')).toBeUndefined() + }) + + it('scaffolds a single SELECT against `_wm_target.
`', () => { + const src = generatePipelineDraft({ + language: 'duckdb', + outputKind: 'data_test', + triggers: [] + }) + // starter body is a single SELECT against the internal target alias. + expect(src).toContain('SELECT * FROM _wm_target.your_table WHERE your_condition;') + // exactly one SQL statement (single SELECT) — count statement lines, not + // the word "SELECT" that also appears in the guidance comment. + const stmtLines = src.split('\n').filter((l) => /^\s*SELECT\b/i.test(l)) + expect(stmtLines).toHaveLength(1) + // no `-- materialize ` output annotation — a data test declares no + // asset (the word still appears in the guidance comment, which is fine). + expect(src).not.toMatch(/^--\s*materialize\s/m) + // teaches how to wire it up + the offending-rows convention. + expect(src).toContain('-- data_test ') + expect(src).toContain('offending rows') + }) + + it('seeds the table name from an upstream ducklake asset when present', () => { + const src = generatePipelineDraft({ + language: 'duckdb', + outputKind: 'data_test', + input: { kind: 'ducklake', path: 'analytics/orders' }, + triggers: [] + }) + expect(src).toContain('SELECT * FROM _wm_target.orders WHERE your_condition;') + // no ATTACH of the input — the runtime attaches the target as `_wm_target`. + expect(src).not.toContain('ATTACH') + }) +}) diff --git a/frontend/src/lib/components/assets/AssetGraph/pipelineTemplates.test.ts b/frontend/src/lib/components/assets/AssetGraph/pipelineTemplates.test.ts new file mode 100644 index 0000000000..262d36e6cf --- /dev/null +++ b/frontend/src/lib/components/assets/AssetGraph/pipelineTemplates.test.ts @@ -0,0 +1,60 @@ +import { describe, expect, it } from 'vitest' +import type { ScriptLang } from '$lib/gen' +import { + autoOutputAsset, + generatePipelineDraft, + type PipelineOutputKind +} from './pipelineTemplates' + +// The seeded draft asset (`autoOutputAsset`, stored as `outputAssets` and used +// by resolveGraph for inactive-draft node identity) must match the asset +// identity the deploy-time / wasm parser infers from the generated body. The +// parser canonicalizes any S3 URI by stripping the `s3://` prefix and all +// leading slashes (see backend `parse_asset_syntax`); if the seed carried a +// leading slash while the body wrote `s3:///key`, the preview would render a +// duplicate `/key` node and a phantom post-deploy drift. This pins the two in +// lockstep so that class of drift can't regress. + +// Mirror of the parser's S3 canonicalization for a raw `s3://…` URI. +function canonicalS3Key(uri: string): string { + const rest = uri.replace(/^s3:\/\//, '') + return rest.replace(/^\/+/, '') +} + +const S3_KINDS: PipelineOutputKind[] = ['s3_parquet', 's3_object'] +const LANGS: ScriptLang[] = ['bun', 'python3', 'duckdb'] + +describe('pipelineTemplates S3 seed/body parity', () => { + for (const language of LANGS) { + for (const outputKind of S3_KINDS) { + it(`${language} ${outputKind}: seeded asset path matches the body's S3 write URI`, () => { + const output = autoOutputAsset(outputKind, 'demo', language) + expect(output).toBeDefined() + const asset = output! + + // The seed must be a canonical slashless key so it matches the + // identity the parser infers from the generated body. + expect(asset.kind).toBe('s3object') + expect(asset.path.startsWith('/')).toBe(false) + + const body = generatePipelineDraft({ + language, + outputKind, + output: asset, + triggers: [] + }) + + // Every S3 URI the generated body emits must canonicalize back to + // the seeded asset path — the write target especially. + const uris = body.match(/s3:\/\/[^'"`)\s]+/g) ?? [] + expect(uris.length).toBeGreaterThan(0) + for (const uri of uris) { + // Runtime form must stay triple-slash (default storage); a bare + // `s3://key` would target a named storage `key` at run time. + expect(uri.startsWith('s3:///')).toBe(true) + expect(canonicalS3Key(uri)).toBe(asset.path) + } + }) + } + } +}) diff --git a/frontend/src/lib/components/assets/AssetGraph/pipelineTemplates.ts b/frontend/src/lib/components/assets/AssetGraph/pipelineTemplates.ts index 63a90ef099..04d4232dd0 100644 --- a/frontend/src/lib/components/assets/AssetGraph/pipelineTemplates.ts +++ b/frontend/src/lib/components/assets/AssetGraph/pipelineTemplates.ts @@ -17,6 +17,7 @@ export type PipelineOutputKind = | 'datatable' | 'ducklake' | 'materialize' + | 'data_test' | 's3_parquet' | 's3_object' | 'macros' @@ -37,6 +38,11 @@ export const PIPELINE_OUTPUT_KINDS: PipelineOutputKindMeta[] = [ label: 'Materialized table', description: 'Managed DuckLake table — idempotent, versioned, tracked' }, + { + id: 'data_test', + label: 'Data test', + description: 'Custom assertion — a single SELECT returning offending rows (empty = pass)' + }, { id: 'datatable', label: 'Data table', @@ -83,7 +89,16 @@ const LANG_COMPATIBILITY: Record = { // single SELECT. The Python/TS `wmll.ducklake` helper currently takes a SQL // SELECT (not in-memory rows), so a polyglot managed materialize is a // separate follow-up — those langs keep the `ducklake` raw-write kind. - duckdb: ['materialize', 'datatable', 'ducklake', 's3_parquet', 's3_object', 'macros', 'none'], + duckdb: [ + 'materialize', + 'data_test', + 'datatable', + 'ducklake', + 's3_parquet', + 's3_object', + 'macros', + 'none' + ], postgresql: ['datatable', 'none'], mysql: ['none'], mssql: ['none'], @@ -160,16 +175,16 @@ export function autoOutputAsset( case 'ducklake': case 'materialize': return { kind: 'ducklake', path: `main/${adj}_${pick(TABLE_NOUNS)}_${slug}` } - // s3 paths carry the canonical leading slash of a default-storage - // object (`s3:///` parses to path `/`). The deploy-time - // parser stores writes in that form — a slashless seeded path would - // never match it, and the post-deploy drift check would report the - // output as lost (it isn't; the key differs by one '/'). Bodies that - // take a bare key (`{ s3: ... }`) strip the slash via `s3Key`. + // s3 outputs use the canonical slashless key. `parse_asset_syntax` + // normalizes `s3:///` (default storage) and `s3://` to the + // bare ``, so the seeded draft asset must be slashless to match the + // deploy-time inferred identity — otherwise the post-deploy drift check + // would flag the output as a phantom `/`-prefixed node. The generated + // bodies still emit the `s3:///` default-storage URI for runtime I/O. case 's3_parquet': return { kind: 's3object', - path: `/pipelines/${folder}/${adj}_${pick(DATASET_NOUNS)}_${slug}.parquet` + path: `pipelines/${folder}/${adj}_${pick(DATASET_NOUNS)}_${slug}.parquet` } case 's3_object': { // duckdb's natural output for a generic blob is CSV (one COPY TO @@ -179,12 +194,14 @@ export function autoOutputAsset( const ext = language === 'duckdb' ? 'csv' : 'json' return { kind: 's3object', - path: `/pipelines/${folder}/${adj}_${pick(FILE_NOUNS)}_${slug}.${ext}` + path: `pipelines/${folder}/${adj}_${pick(FILE_NOUNS)}_${slug}.${ext}` } } // A macro library produces no asset — its "output" is the registry - // entries the deploy records. + // entries the deploy records. A custom data test produces no asset + // either — it asserts against an existing materialized target. case 'macros': + case 'data_test': case 'none': return undefined } @@ -205,10 +222,11 @@ export function assetUri(asset: { kind: AssetKind; path: string }): string { return `${ASSET_URI_PREFIX[asset.kind]}${asset.path}` } -// Bare object key for the SDK's `{ s3: }` forms: the canonical asset -// path of a default-storage object has a leading slash, the key must not. +// Bare object key for the SDK's `{ s3: }` / `s3:///` forms. Asset +// paths are already canonical slashless keys; strip stray leading slashes +// defensively so the emitted key never starts with '/'. function s3Key(path: string): string { - return path.replace(/^\//, '') + return path.replace(/^\/+/, '') } // Splits a datatable asset path (`/
` or `/.
`) @@ -316,6 +334,17 @@ export type TemplateContext = { function header(ctx: TemplateContext): string { const { language, triggers, output, outputKind } = ctx const p = commentPrefix(language) + // A custom data test is a standalone script, not a graph node that produces + // an asset — so it gets no `// pipeline` / output annotation. Instead, tell + // the author how to wire it up (the `data_test` reference) and the two rules + // that aren't obvious: single SELECT, returning the offending rows. + if (outputKind === 'data_test') { + return [ + `${p} Custom data test — reference it from a materialize script with \`${p} data_test \`.`, + `${p} It must be a single SELECT returning the offending rows; the run fails if any row comes back.`, + '' + ].join('\n') + } const lines = triggers.map((t) => { switch (t.kind) { case 'asset': @@ -391,8 +420,11 @@ function bodyTs(ctx: TemplateContext): string { if (!input) return '' switch (input.kind) { case 's3object': + // `s3:///` URI — one spelling shared with the `// on + // s3:///…` annotation form (the object literal `{ s3: }` + // is equivalent). return [ - ` const buf = await wmill.loadS3File({ s3: ${JSON.stringify(s3Key(input.path))} })`, + ` const buf = await wmill.loadS3File(${JSON.stringify(`s3:///${s3Key(input.path)}`)})`, ` const rows = JSON.parse(new TextDecoder().decode(buf))`, `` ].join('\n') @@ -418,9 +450,10 @@ function bodyTs(ctx: TemplateContext): string { switch (outputKind) { case 's3_parquet': case 's3_object': + // `s3:///` URI — see the loadS3File note above. return [ ` const payload = new TextEncoder().encode(JSON.stringify(rows))`, - ` await wmill.writeS3File({ s3: ${JSON.stringify(s3Key(output.path))} }, payload)` + ` await wmill.writeS3File(${JSON.stringify(`s3:///${s3Key(output.path)}`)}, payload)` ].join('\n') case 'datatable': { const dbName = output.path.split('/')[0] ?? 'main' @@ -474,8 +507,11 @@ function bodyPython(ctx: TemplateContext): string { if (!input) return '' switch (input.kind) { case 's3object': + // `s3:///` URI — SDK string params must be s3:// URIs + // (bare keys are rejected), and this form matches the + // `# on s3:///…` annotation spelling. return [ - ` buf = wmill.load_s3_file(${JSON.stringify(s3Key(input.path))})`, + ` buf = wmill.load_s3_file(${JSON.stringify(`s3:///${s3Key(input.path)}`)})`, ` import json; rows = json.loads(buf.decode("utf-8"))` ].join('\n') case 'datatable': @@ -498,9 +534,10 @@ function bodyPython(ctx: TemplateContext): string { switch (outputKind) { case 's3_parquet': case 's3_object': + // `s3:///` URI — see the load_s3_file note above. return [ ` import json`, - ` wmill.write_s3_file(${JSON.stringify(s3Key(output.path))}, json.dumps(rows).encode("utf-8"))` + ` wmill.write_s3_file(${JSON.stringify(`s3:///${s3Key(output.path)}`)}, json.dumps(rows).encode("utf-8"))` ].join('\n') case 'datatable': { const dbName = output.path.split('/')[0] ?? 'main' @@ -536,6 +573,22 @@ function bodyPython(ctx: TemplateContext): string { function bodyDuckdb(ctx: TemplateContext): string { const { input, output, outputKind } = ctx const dataUpload = isDataUpload(ctx.triggers) + if (outputKind === 'data_test') { + // Standalone custom data test: it reads ONLY the freshly-materialized + // target, which the runtime attaches under the internal `_wm_target` + // schema — so it emits no ATTACH / input load of its own. When the test + // was created off a ducklake asset, seed its table name; otherwise a + // clear placeholder. This exact shape (single SELECT vs `_wm_target`) is + // what the backend's self-teaching errors ask for. + const testTable = input?.kind === 'ducklake' ? catalogTableRef(input.path) : 'your_table' + return [ + '', + `-- Return the rows that VIOLATE your assertion; an empty result means the test passes.`, + `-- \`_wm_target\` is the freshly-materialized target, attached by the runtime.`, + `SELECT * FROM _wm_target.${testTable} WHERE your_condition;`, + '' + ].join('\n') + } const lines: string[] = [] if (dataUpload) { // `(s3object)` param declaration → the run form renders the S3 picker @@ -584,7 +637,7 @@ function bodyDuckdb(ctx: TemplateContext): string { if (!input) return null switch (input.kind) { case 's3object': - return `read_parquet('s3://${input.path}')` + return `read_parquet('s3:///${input.path}')` case 'datatable': // `pg` is the attached Postgres catalog (see ATTACH above). // Use a 2-part `pg.
` ref so the asset parser maps it @@ -607,7 +660,7 @@ function bodyDuckdb(ctx: TemplateContext): string { `COPY (`, ` SELECT *`, ` FROM ${fromExpr}`, - `) TO 's3://${output.path}' (FORMAT 'parquet');` + `) TO 's3:///${output.path}' (FORMAT 'parquet');` ) } break @@ -618,7 +671,7 @@ function bodyDuckdb(ctx: TemplateContext): string { `COPY (`, ` SELECT *`, ` FROM ${fromExpr}`, - `) TO 's3://${output.path}' (FORMAT 'csv', HEADER);` + `) TO 's3:///${output.path}' (FORMAT 'csv', HEADER);` ) } break diff --git a/frontend/src/lib/components/assets/AssetGraph/resolveGraph.test.ts b/frontend/src/lib/components/assets/AssetGraph/resolveGraph.test.ts index 3c98d57602..2216d37b0f 100644 --- a/frontend/src/lib/components/assets/AssetGraph/resolveGraph.test.ts +++ b/frontend/src/lib/components/assets/AssetGraph/resolveGraph.test.ts @@ -99,6 +99,50 @@ describe('resolveGraph', () => { }) }) + it('scd2 materialize draft: writes both the base dim and its _current companion view', () => { + const drafts = new Map([ + [ + 'f/x/dim', + { + script: { + content: '-- materialize ducklake://main/dim_customers key=id history\nselect 1' + } + } + ] + ]) + const r = resolveGraph(input({ drafts })) + // Base dimension: a plain write output node. + expect(r.assets).toContainEqual({ kind: 'ducklake', path: 'main/dim_customers' }) + // Companion `_current` view: same producer, marked derived from the base. + expect(r.assets).toContainEqual({ + kind: 'ducklake', + path: 'main/dim_customers_current', + derived_from: 'main/dim_customers' + }) + for (const path of ['main/dim_customers', 'main/dim_customers_current']) { + expect(r.edges).toContainEqual({ + runnable_path: 'f/x/dim', + runnable_kind: 'script', + asset_kind: 'ducklake', + asset_path: path, + access_type: 'w', + unsaved: true + }) + } + }) + + it('non-scd2 materialize draft: writes only the base dim, no _current companion', () => { + const drafts = new Map([ + [ + 'f/x/dim', + { script: { content: '-- materialize ducklake://main/dim_customers key=id\nselect 1' } } + ] + ]) + const r = resolveGraph(input({ drafts })) + expect(r.assets).toContainEqual({ kind: 'ducklake', path: 'main/dim_customers' }) + expect(r.assets.some((a) => a.path === 'main/dim_customers_current')).toBe(false) + }) + it('active draft: live body writes are authoritative over the snapshot', () => { const drafts = new Map([ [ @@ -341,6 +385,55 @@ describe('resolveGraph', () => { expect(r.edges.some((e) => e.asset_path === 'main/out' && e.access_type === 'w')).toBe(true) }) + it('editing a saved scd2 producer keeps both the base and _current persisted write edges', () => { + // Deploy persists a write to both `main/dim` and `main/dim_current`. + // Opening the producer for editing must not judge the companion `_current` + // write stale — otherwise a consumer of only the view orphans mid-edit. + const base = baseGraph({ + runnables: [{ path: 'f/x/dim', usage_kind: 'script' }], + edges: [ + { + runnable_path: 'f/x/dim', + runnable_kind: 'script', + asset_kind: 'ducklake', + asset_path: 'main/dim_customers', + access_type: 'w' + }, + { + runnable_path: 'f/x/dim', + runnable_kind: 'script', + asset_kind: 'ducklake', + asset_path: 'main/dim_customers_current', + access_type: 'w' + } + ] + }) + const r = resolveGraph( + input({ + base, + liveAnnotations: { + scriptPath: 'f/x/dim', + annotations: ann({ + materialize: { + targetKind: 'ducklake', + targetPath: 'main/dim_customers', + uniqueKey: 'id', + scd2: true + } + }) + }, + liveBodyAssets: { scriptPath: 'f/x/dim', assets: [] } + }) + ) + for (const path of ['main/dim_customers', 'main/dim_customers_current']) { + expect( + r.edges.some( + (e) => e.runnable_path === 'f/x/dim' && e.asset_path === path && e.access_type === 'w' + ) + ).toBe(true) + } + }) + it('selecting a saved script unchanged drops nothing (no stale removal)', () => { const base = baseGraph({ runnables: [{ path: 'f/x/prod', usage_kind: 'script' }], diff --git a/frontend/src/lib/components/assets/AssetGraph/resolveGraph.ts b/frontend/src/lib/components/assets/AssetGraph/resolveGraph.ts index 849849c555..328535ef87 100644 --- a/frontend/src/lib/components/assets/AssetGraph/resolveGraph.ts +++ b/frontend/src/lib/components/assets/AssetGraph/resolveGraph.ts @@ -2,6 +2,7 @@ import type { AssetGraphMacroEdge, AssetGraphResponse, NativeTriggerKind } from import { mergeColumnLineage, parsePipelineAnnotations, + scd2CurrentTargetPath, type ColumnLineage, type PipelineAnnotations } from './parsePipelineAnnotations' @@ -265,9 +266,16 @@ function makeContext(input: ResolveGraphInput): ResolveContext { // lives in an annotation (not the SQL body), so neither triggerAssets // (inputs) nor the body-inferred assets cover it. Without this its // persisted write-edge is judged stale and dropped the moment the - // script is selected/edited — leaving the output asset unlinked. + // script is selected/edited — leaving the output asset unlinked. A + // managed scd2 producer persists a second write to the `_current` + // companion view, so keep that too or a consumer of only the view + // orphans while the producer is open for editing. const m = liveAnnotations.annotations.materialize - if (m) liveRefKeys.add(`${m.targetKind}:${m.targetPath}`) + if (m) { + liveRefKeys.add(`${m.targetKind}:${m.targetPath}`) + const currentPath = scd2CurrentTargetPath(m) + if (currentPath) liveRefKeys.add(`${m.targetKind}:${currentPath}`) + } } for (const a of liveBodyAssets.assets) liveRefKeys.add(`${a.kind}:${a.path}`) } @@ -373,7 +381,7 @@ function seedDraftOverlays(acc: Accumulator, input: ResolveGraphInput) { // creation/last edit, or the seeded output for a fresh draft // whose body doesn't yet write anything inferable). const liveForThisDraft = liveBodyAssets.scriptPath === path - const writeOuts: Array<{ kind: AssetKind; path: string }> = [] + const writeOuts: Array<{ kind: AssetKind; path: string; derivedFrom?: string }> = [] if (liveForThisDraft) { writeOuts.push(...extractWrites(liveBodyAssets.assets)) } else if (d.outputAssets) { @@ -382,16 +390,35 @@ function seedDraftOverlays(acc: Accumulator, input: ResolveGraphInput) { // `// materialize ` declares a write output via annotation, not // the SQL body, so the body-inference tiers above miss it. Add it from // the live-parsed annotations so an edited materialize script keeps its - // output edge (the loop below dedups against existing assets/edges). + // output edge (the loop below dedups against existing assets/edges). A + // managed scd2 materialize also produces the `_current` companion + // view — add it too (mirrors the deploy path) so a draft consuming only + // the view links back to this producer instead of orphaning. if (parsed.materialize) { writeOuts.push({ kind: parsed.materialize.targetKind, path: parsed.materialize.targetPath }) + const currentPath = scd2CurrentTargetPath(parsed.materialize) + if (currentPath) { + writeOuts.push({ + kind: parsed.materialize.targetKind, + path: currentPath, + derivedFrom: parsed.materialize.targetPath + }) + } } for (const out of writeOuts) { - const hasAsset = assets.some((a) => a.kind === out.kind && a.path === out.path) - if (!hasAsset) assets.push({ kind: out.kind, path: out.path }) + const existing = assets.find((a) => a.kind === out.kind && a.path === out.path) + if (!existing) { + assets.push({ + kind: out.kind, + path: out.path, + ...(out.derivedFrom ? { derived_from: out.derivedFrom } : {}) + }) + } else if (out.derivedFrom && existing.derived_from == undefined) { + existing.derived_from = out.derivedFrom + } // Dedup against edges already in the overlay (this draft's base // edges were dropped above, so this only guards against duplicate // writeOuts entries — not against the persisted version). diff --git a/frontend/src/lib/components/assets/AssetGraph/schemaContracts.test.ts b/frontend/src/lib/components/assets/AssetGraph/schemaContracts.test.ts new file mode 100644 index 0000000000..9247b23d9b --- /dev/null +++ b/frontend/src/lib/components/assets/AssetGraph/schemaContracts.test.ts @@ -0,0 +1,276 @@ +import { describe, expect, it } from 'vitest' +import { + buildSchemaContractContext, + diffSchemaContracts, + mapWarningsToMarkers, + normalizeAssetPath, + referencedDucklakePaths, + type CapturedSchemaLite +} from './schemaContracts' +import { parsePipelineAnnotations } from './parsePipelineAnnotations' +import type { AssetWithAltAccessType } from '../lib' + +// Mirrors backend/windmill-common/src/schema_contracts.rs unit tests — the two +// diffs must apply the same rules or the editor previews a different verdict +// than the save-time check returns. + +function schema(cols: [string, string][], version = 2): CapturedSchemaLite { + return { + columns: cols.map(([name, type]) => ({ name, type })), + version, + capturedAt: '2026-01-01T00:00:00Z' + } +} + +function readAsset(path: string, cols: string[]): AssetWithAltAccessType { + return { + path, + kind: 'ducklake', + access_type: 'r', + columns: Object.fromEntries(cols.map((c) => [c, 'r' as const])) + } +} + +const NO_ANN = { columnLineage: [], dataTests: [] } + +describe('diffSchemaContracts', () => { + it('warns on a missing read column, matching case-insensitively', () => { + const schemas = new Map([ + [ + 'lake/orders', + schema([ + ['Order_ID', 'BIGINT'], + ['amount_usd', 'DOUBLE'] + ]) + ] + ]) + const w = diffSchemaContracts({ + ...NO_ANN, + assets: [readAsset('lake/orders', ['order_id', 'amount'])], + schemas, + ignored: new Set() + }) + expect(w).toHaveLength(1) + expect(w[0].kind).toBe('missing_column') + expect(w[0].column).toBe('amount') + expect(w[0].schema_version).toBe(2) + }) + + it('skips unknown-column assets, "*" and reserved columns', () => { + const schemas = new Map([['lake/orders', schema([['id', 'BIGINT']])]]) + const noColumns: AssetWithAltAccessType = { + path: 'lake/orders', + kind: 'ducklake', + access_type: 'r' + } + expect( + diffSchemaContracts({ ...NO_ANN, assets: [noColumns], schemas, ignored: new Set() }) + ).toEqual([]) + expect( + diffSchemaContracts({ + ...NO_ANN, + assets: [readAsset('lake/orders', ['*', '_wm_partition', 'id'])], + schemas, + ignored: new Set() + }) + ).toEqual([]) + }) + + it('is silent for assets without a captured schema', () => { + expect( + diffSchemaContracts({ + ...NO_ANN, + assets: [readAsset('lake/unknown', ['whatever'])], + schemas: new Map(), + ignored: new Set() + }) + ).toEqual([]) + }) + + it('normalizes the {partition} token before lookup', () => { + const schemas = new Map([['lake/orders', schema([['id', 'BIGINT']])]]) + const w = diffSchemaContracts({ + ...NO_ANN, + assets: [readAsset('lake/orders/{partition}', ['gone'])], + schemas, + ignored: new Set() + }) + expect(w).toHaveLength(1) + expect(w[0].asset_path).toBe('lake/orders') + }) + + it('warns on broken // column lineage refs', () => { + const ann = parsePipelineAnnotations( + '// column total <- ducklake://lake/orders.amount\nSELECT 1;' + ) + const schemas = new Map([['lake/orders', schema([['id', 'BIGINT']])]]) + const w = diffSchemaContracts({ + assets: [], + columnLineage: ann.columnLineage, + dataTests: [], + schemas, + ignored: new Set() + }) + expect(w).toHaveLength(1) + expect(w[0].kind).toBe('missing_lineage_source') + }) + + it('flags missing relationship columns and captured-type differences', () => { + const ann = parsePipelineAnnotations( + '// materialize ducklake://lake/orders\n' + + '// data_test relationships customer_id -> ducklake://lake/customers.id\n' + + '// data_test relationships customer_id -> ducklake://lake/customers.uuid\n' + + 'SELECT 1;' + ) + const schemas = new Map([ + ['lake/customers', schema([['id', 'VARCHAR']])], + ['lake/orders', schema([['customer_id', 'BIGINT']])] + ]) + const w = diffSchemaContracts({ + assets: [], + columnLineage: ann.columnLineage, + dataTests: ann.dataTests, + materialize: ann.materialize, + schemas, + ignored: new Set() + }) + expect(w).toHaveLength(2) + expect( + w.some( + (x) => + x.kind === 'relationship_type_mismatch' && + x.expected_type === 'BIGINT' && + x.found_type === 'VARCHAR' + ) + ).toBe(true) + expect(w.some((x) => x.kind === 'missing_relationship_column' && x.column === 'uuid')).toBe( + true + ) + }) + + it('suppresses ignored assets down to one informational note', () => { + const schemas = new Map([['lake/orders', schema([['id', 'BIGINT']])]]) + const ignored = new Set(['lake/orders']) + const w = diffSchemaContracts({ + ...NO_ANN, + assets: [readAsset('lake/orders', ['a', 'b'])], + schemas, + ignored + }) + expect(w).toHaveLength(1) + expect(w[0].kind).toBe('suppressed') + expect( + diffSchemaContracts({ + ...NO_ANN, + assets: [readAsset('lake/orders', ['id'])], + schemas, + ignored + }) + ).toEqual([]) + }) +}) + +describe('referencedDucklakePaths', () => { + it('collects paths from reads, lineage, relationships and materialize', () => { + const ann = parsePipelineAnnotations( + '// materialize ducklake://lake/out\n' + + '// column total <- ducklake://lake/a.amount\n' + + '// data_test relationships k -> ducklake://lake/b.id\n' + + 'SELECT 1;' + ) + const refs = referencedDucklakePaths({ + assets: [readAsset('lake/c/{partition}', ['x'])], + columnLineage: ann.columnLineage, + dataTests: ann.dataTests, + materialize: ann.materialize + }) + expect(refs.sort()).toEqual(['lake/a', 'lake/b', 'lake/c', 'lake/out']) + }) +}) + +describe('buildSchemaContractContext', () => { + it('derives ignored assets and scd2 _current bases from graph runnables', () => { + const ctx = buildSchemaContractContext([ + { + materialize_target: { kind: 'ducklake', path: 'lake/dim' }, + materialize_strategy: 'scd2', + materialize_on_schema_change: 'ignore' + }, + { + materialize_target: { kind: 'ducklake', path: 'lake/orders' }, + materialize_strategy: 'replace' + }, + // non-ducklake and absent targets are ignored + { materialize_target: { kind: 's3object', path: 'x/y' }, materialize_strategy: 'scd2' }, + {} + ]) + expect(ctx.ignoredAssets).toEqual(['lake/dim', 'lake/dim_current']) + expect(ctx.scd2CurrentBases).toEqual({ 'lake/dim_current': 'lake/dim' }) + }) + + it('ignores _current only for scd2 producers (backend spec.scd2 gate)', () => { + const ctx = buildSchemaContractContext([ + { + materialize_target: { kind: 'ducklake', path: 'lake/t' }, + materialize_strategy: 'replace', + materialize_on_schema_change: 'ignore' + } + ]) + // a non-scd2 producer's `_current` is an unrelated asset — it must + // keep warning, exactly like the server-side check + expect(ctx.ignoredAssets).toEqual(['lake/t']) + expect(ctx.scd2CurrentBases).toEqual({}) + }) +}) + +describe('mapWarningsToMarkers', () => { + it('anchors annotation warnings to their lines and body reads to the identifier', () => { + const code = + '-- pipeline\n' + + '-- on ducklake://lake/orders\n' + + '-- column total <- ducklake://lake/orders.amount\n' + + '-- data_test relationships k -> ducklake://lake/customers.uuid\n' + + 'SELECT amount FROM dl.orders;' + const markers = mapWarningsToMarkers(code, [ + { + kind: 'missing_lineage_source', + asset_path: 'lake/orders', + column: 'amount', + message: 'm1' + }, + { + kind: 'missing_relationship_column', + asset_path: 'lake/customers', + column: 'uuid', + message: 'm2' + }, + { kind: 'missing_column', asset_path: 'lake/orders', column: 'amount', message: 'm3' }, + { kind: 'suppressed', asset_path: 'lake/orders', message: 'hidden' } + ]) + expect(markers).toHaveLength(3) + expect(markers[0].startLineNumber).toBe(3) + expect(markers[1].startLineNumber).toBe(4) + // body-read warning anchors to the first occurrence of the identifier, + // which is the annotation line mentioning `amount` (line 3) + expect(markers[2].startLineNumber).toBe(3) + // token range is tight around the identifier, not the whole line + const line3 = '-- column total <- ducklake://lake/orders.amount' + expect(markers[2].startColumn).toBe(line3.indexOf('amount') + 1) + }) + + it('falls back to the line mentioning the asset path', () => { + const code = '# pipeline\n# on ducklake://lake/orders\nprint(1)' + const markers = mapWarningsToMarkers(code, [ + { kind: 'missing_column', asset_path: 'lake/orders', column: 'zzz', message: 'm' } + ]) + expect(markers[0].startLineNumber).toBe(2) + }) +}) + +describe('normalizeAssetPath', () => { + it('strips the partition token and trailing slashes', () => { + expect(normalizeAssetPath('lake/orders/{partition}')).toBe('lake/orders') + expect(normalizeAssetPath('lake/orders_{partition}')).toBe('lake/orders_') + expect(normalizeAssetPath('lake/orders/')).toBe('lake/orders') + }) +}) diff --git a/frontend/src/lib/components/assets/AssetGraph/schemaContracts.ts b/frontend/src/lib/components/assets/AssetGraph/schemaContracts.ts new file mode 100644 index 0000000000..4a1de04e13 --- /dev/null +++ b/frontend/src/lib/components/assets/AssetGraph/schemaContracts.ts @@ -0,0 +1,440 @@ +// Client-side mirror of the save-time schema-contract check (pipelines gap +// #2b, backend/windmill-common/src/schema_contracts.rs). The backend endpoint +// (`checkSchemaContracts`) is the authoritative check run on save; this mirror +// drives the *live* editor surface (Monaco warning markers + completions) from +// the WASM parse that already runs on the open buffer, so the two must apply +// the same rules: ducklake-only, case-insensitive column names, `columns` +// absent ⇒ skip, `_wm_partition` whitelisted, `{partition}` token stripped, +// annotation-declared lineage only, asset without captured schema ⇒ silent. + +import { AssetService, ScriptService, type ContractWarning, type ScriptLang } from '$lib/gen' +import { sendUserToast } from '$lib/toast' +import type { AssetWithAltAccessType } from '../lib' +import { + parsePipelineAnnotations, + type ColumnLineage, + type DataTest, + type MaterializeSpec +} from './parsePipelineAnnotations' + +// Columns the materialize engine manages; excluded from the captured schema on +// purpose, so reads of them must not warn. +const RESERVED_COLUMNS = ['_wm_partition'] + +const PARTITION_TOKEN = '{partition}' + +export type CapturedSchemaLite = { + columns: { name: string; type: string }[] + version: number + capturedAt: string +} + +// Strip the `{partition}` token a declared URI may carry so lookups hit the +// captured path (mirrors `normalize_asset_path`). +export function normalizeAssetPath(path: string): string { + return path + .replaceAll('/' + PARTITION_TOKEN, '') + .replaceAll(PARTITION_TOKEN, '') + .replace(/\/+$/, '') +} + +function isReserved(name: string): boolean { + return RESERVED_COLUMNS.some((r) => r.toLowerCase() === name.toLowerCase()) +} + +function findColumn( + schema: CapturedSchemaLite, + name: string +): { name: string; type: string } | undefined { + const lower = name.toLowerCase() + return schema.columns.find((c) => c.name.toLowerCase() === lower) +} + +export type SchemaContractInputs = { + // Per-asset column reads/writes from the WASM parse (entries without a + // `columns` map are skipped — wildcard/unknown access). + assets: AssetWithAltAccessType[] + // Annotation-declared `// column` lineage ONLY (not merged AST-inferred + // lineage — redundant with body reads and alias-attribution can misfire). + columnLineage: ColumnLineage[] + dataTests: DataTest[] + materialize?: MaterializeSpec + // Latest captured schema per normalized ducklake path (after any + // `_current` → base-table fallback the caller resolved). + schemas: Map + // Normalized paths whose producer declares `on_schema_change=ignore`. + ignored: Set +} + +// Mirrors backend `diff_contract` — same warning kinds and suppression +// semantics, minus the human message wording (the editor renders its own). +export function diffSchemaContracts(input: SchemaContractInputs): ContractWarning[] { + const { assets, columnLineage, dataTests, materialize, schemas, ignored } = input + const warnings: ContractWarning[] = [] + + // W1 — body-read/written columns missing from the captured schema. + for (const a of assets) { + if (a.kind !== 'ducklake' || a.columns == undefined) continue + const path = normalizeAssetPath(a.path) + const schema = schemas.get(path) + if (!schema) continue + for (const col of Object.keys(a.columns)) { + if (col === '*' || isReserved(col)) continue + if (!findColumn(schema, col)) { + warnings.push({ + kind: 'missing_column', + asset_path: path, + column: col, + schema_version: schema.version, + captured_at: schema.capturedAt, + message: `column \`${col}\` of ducklake://${path} is not in its captured schema (v${schema.version}, columns: ${schema.columns.map((c) => c.name).join(', ')})` + }) + } + } + } + + // W2 — `// column` lineage source refs. + for (const cl of columnLineage) { + for (const input of cl.inputs) { + if (input.from_kind !== 'ducklake' || isReserved(input.from_column)) continue + const path = normalizeAssetPath(input.from_path) + const schema = schemas.get(path) + if (!schema) continue + if (!findColumn(schema, input.from_column)) { + warnings.push({ + kind: 'missing_lineage_source', + asset_path: path, + column: input.from_column, + schema_version: schema.version, + captured_at: schema.capturedAt, + message: `\`// column ${cl.column}\` reads \`${input.from_column}\` from ducklake://${path}, which is not in its captured schema (v${schema.version})` + }) + } + } + } + + // W3 — relationships refs: missing column, and captured-type difference + // when the consumer's own materialize target has a capture. Types still + // coerce at run time, so a difference is "differs", never "will fail". + const ownSchema = + materialize?.targetKind === 'ducklake' + ? schemas.get(normalizeAssetPath(materialize.targetPath)) + : undefined + for (const dt of dataTests) { + if (dt.type !== 'relationships' || dt.to_kind !== 'ducklake') continue + const path = normalizeAssetPath(dt.to_path) + const schema = schemas.get(path) + if (!schema) continue + const refCol = findColumn(schema, dt.to_column) + if (!refCol) { + warnings.push({ + kind: 'missing_relationship_column', + asset_path: path, + column: dt.to_column, + schema_version: schema.version, + captured_at: schema.capturedAt, + message: `\`// data_test relationships ${dt.column}\` references ducklake://${path}.${dt.to_column}, which is not in its captured schema (v${schema.version})` + }) + } else { + const ownCol = ownSchema && findColumn(ownSchema, dt.column) + if (ownCol && ownCol.type.toLowerCase() !== refCol.type.toLowerCase()) { + warnings.push({ + kind: 'relationship_type_mismatch', + asset_path: path, + column: dt.to_column, + expected_type: ownCol.type, + found_type: refCol.type, + schema_version: schema.version, + captured_at: schema.capturedAt, + message: `\`// data_test relationships ${dt.column}\` joins \`${dt.column}\` (${ownCol.type}) to ducklake://${path}.${dt.to_column} (${refCol.type}) — captured types differ` + }) + } + } + } + + // W4 — producer `on_schema_change=ignore`: drop the asset's warnings, + // leaving one informational entry per suppressed asset. + if (ignored.size > 0) { + const suppressed: string[] = [] + const kept = warnings.filter((w) => { + if (ignored.has(w.asset_path)) { + if (!suppressed.includes(w.asset_path)) suppressed.push(w.asset_path) + return false + } + return true + }) + warnings.length = 0 + warnings.push(...kept) + for (const path of suppressed) { + warnings.push({ + kind: 'suppressed', + asset_path: path, + message: `schema mismatches on ducklake://${path} suppressed by its producer's \`on_schema_change=ignore\`` + }) + } + } + + return warnings +} + +// The ducklake paths a buffer references in ways the contract check inspects — +// what the editor needs captured schemas for. +export function referencedDucklakePaths( + input: Pick +): string[] { + const paths = new Set() + for (const a of input.assets) { + if (a.kind === 'ducklake' && a.columns != undefined) paths.add(normalizeAssetPath(a.path)) + } + for (const cl of input.columnLineage) { + for (const i of cl.inputs) { + if (i.from_kind === 'ducklake') paths.add(normalizeAssetPath(i.from_path)) + } + } + for (const dt of input.dataTests) { + if (dt.type === 'relationships' && dt.to_kind === 'ducklake') + paths.add(normalizeAssetPath(dt.to_path)) + } + if (input.materialize?.targetKind === 'ducklake') + paths.add(normalizeAssetPath(input.materialize.targetPath)) + return [...paths] +} + +// --- Captured-schema cache ------------------------------------------------ + +// Short-TTL cache so per-keystroke recomputes and completion requests don't +// re-fetch. Captured schemas only change when a producer materializes, so a +// briefly stale hit is fine — the authoritative save-time check re-reads. +const SCHEMA_TTL_MS = 30_000 +const schemaCache = new Map() + +export async function fetchLatestSchema( + workspace: string, + path: string +): Promise { + const key = `${workspace}:${path}` + const hit = schemaCache.get(key) + if (hit && Date.now() - hit.at < SCHEMA_TTL_MS) return hit.value + let value: CapturedSchemaLite | undefined = undefined + try { + const versions = await AssetService.listAssetSchemas({ workspace, path }) + const latest = versions[0] + if (latest) { + value = { + columns: latest.columns, + version: latest.version, + capturedAt: latest.captured_at + } + } + } catch (e) { + console.error('failed to fetch captured asset schema', path, e) + } + schemaCache.set(key, { at: Date.now(), value }) + return value +} + +// Resolve the schema map for a set of referenced paths, applying the scd2 +// `_current` → base-table fallback when the graph identifies the view's +// producer as a managed scd2 materializer (the view is `SELECT * … WHERE +// is_current`, so columns are identical). +export async function fetchSchemasForPaths( + workspace: string, + paths: string[], + scd2CurrentBase?: (path: string) => string | undefined +): Promise> { + const out = new Map() + await Promise.all( + paths.map(async (p) => { + let schema = await fetchLatestSchema(workspace, p) + if (!schema && p.endsWith('_current')) { + const base = scd2CurrentBase?.(p) + if (base) schema = await fetchLatestSchema(workspace, base) + } + if (schema) out.set(p, schema) + }) + ) + return out +} + +// --- Pipeline-graph context --------------------------------------------------- + +// Producer-side facts the contract mirror needs but cannot derive from the +// open buffer: which assets are muted (`on_schema_change=ignore`) and which +// `_current` views map to an scd2 base table. Built by the pipeline page +// from the resolved graph; absent outside the pipeline editor (standalone +// script editor), where suppression simply doesn't apply client-side — the +// save-time server check remains authoritative either way. +export type SchemaContractGraphContext = { + // Normalized asset paths whose producer declares `on_schema_change=ignore`. + ignoredAssets: string[] + // `_current` → base for managed scd2 producers in the graph. + scd2CurrentBases: Record +} + +export function buildSchemaContractContext( + runnables: Pick< + import('./types').AssetGraphRunnableNode, + 'materialize_target' | 'materialize_strategy' | 'materialize_on_schema_change' + >[] +): SchemaContractGraphContext { + const ignoredAssets: string[] = [] + const scd2CurrentBases: Record = {} + for (const r of runnables) { + const t = r.materialize_target + if (!t || t.kind !== 'ducklake') continue + const base = normalizeAssetPath(t.path) + if (r.materialize_on_schema_change === 'ignore') { + ignoredAssets.push(base) + // The `_current` companion is the producer's own view only for scd2 — + // mirroring the backend's `spec.scd2` gate; for any other strategy a + // `_current` ref is an unrelated asset that must keep warning. + if (r.materialize_strategy === 'scd2') { + ignoredAssets.push(`${base}_current`) + } + } + if (r.materialize_strategy === 'scd2') { + scd2CurrentBases[`${base}_current`] = base + } + } + return { ignoredAssets, scd2CurrentBases } +} + +// --- Save-time surface ------------------------------------------------------ + +// Run the authoritative backend check for just-deployed content and toast the +// result. Never throws — a failed check must not taint a successful deploy. +export async function notifyContractWarnings( + workspace: string, + language: ScriptLang, + content: string +): Promise { + // Every checkable ref carries the `ducklake` token (URIs and the bare + // default-syntax shorthand alike) — skip the round-trip for the vast + // majority of saves that can't produce a warning. + if (!content.includes('ducklake')) return + try { + const { warnings } = await ScriptService.checkSchemaContracts({ + workspace, + requestBody: { language, content } + }) + const real = warnings.filter((w) => w.kind !== 'suppressed') + if (real.length === 0) return + sendUserToast( + `Schema contract: ${real.length} warning${real.length > 1 ? 's' : ''}`, + 'warning', + [], + real.map((w) => `• ${w.message}`).join('\n'), + 10000 + ) + } catch (e) { + console.error('schema-contract check failed', e) + } +} + +// End-to-end live-editor check: parse the buffer's annotations, resolve the +// captured schemas for everything it references, diff, and anchor the result +// to source positions. Cheap per keystroke — the annotation parse is a line +// scan and schema fetches hit the short-TTL cache. +export async function computeContractMarkers( + workspace: string, + code: string, + assets: AssetWithAltAccessType[], + context?: SchemaContractGraphContext +): Promise { + const ann = parsePipelineAnnotations(code) + const inputs = { + assets, + // Annotation-declared lineage only — body-inferred lineage is redundant + // with the body-read check and its alias attribution can misfire. + columnLineage: ann.columnLineage, + dataTests: ann.dataTests, + materialize: ann.materialize + } + const refs = referencedDucklakePaths(inputs) + if (refs.length === 0) return [] + const schemas = await fetchSchemasForPaths( + workspace, + refs, + context ? (p) => context.scd2CurrentBases[p] : undefined + ) + if (schemas.size === 0) return [] + const warnings = diffSchemaContracts({ + ...inputs, + schemas, + ignored: new Set(context?.ignoredAssets ?? []) + }) + return mapWarningsToMarkers(code, warnings) +} + +// --- Editor marker mapping --------------------------------------------------- + +export type ContractMarker = { + message: string + startLineNumber: number + startColumn: number + endLineNumber: number + endColumn: number +} + +// Best-effort source anchoring: annotation-family warnings anchor to their +// annotation line; body-read warnings anchor to the first occurrence of the +// column identifier; fallback is the `// on`/first line mentioning the asset. +export function mapWarningsToMarkers(code: string, warnings: ContractWarning[]): ContractMarker[] { + const lines = code.split('\n') + + function lineMatching(pred: (line: string) => boolean): number | undefined { + const idx = lines.findIndex(pred) + return idx >= 0 ? idx + 1 : undefined + } + + function tokenRange( + lineNumber: number, + token: string + ): { startColumn: number; endColumn: number } { + const line = lines[lineNumber - 1] ?? '' + const idx = line.toLowerCase().indexOf(token.toLowerCase()) + if (idx < 0) return { startColumn: 1, endColumn: line.length + 1 } + return { startColumn: idx + 1, endColumn: idx + 1 + token.length } + } + + return warnings + .filter((w) => w.kind !== 'suppressed') + .map((w) => { + let lineNumber: number | undefined + let token: string | undefined = w.column ?? undefined + switch (w.kind) { + case 'missing_lineage_source': + lineNumber = lineMatching( + (l) => /^\s*(\/\/|--|#)\s*column\s/.test(l) && !!w.column && l.includes(w.column) + ) + break + case 'missing_relationship_column': + case 'relationship_type_mismatch': + lineNumber = lineMatching( + (l) => + /^\s*(\/\/|--|#)\s*data_test\s+relationships\s/.test(l) && l.includes(w.asset_path) + ) + break + case 'missing_column': { + // first body occurrence of the column identifier + const re = new RegExp(`\\b${w.column?.replace(/[.*+?^${}()|[\]\\]/g, '\\$&')}\\b`, 'i') + lineNumber = w.column ? lineMatching((l) => re.test(l)) : undefined + break + } + } + if (lineNumber == undefined) { + // fallback: the `// on …` (or any) line mentioning the asset path + lineNumber = lineMatching((l) => l.includes(w.asset_path)) ?? 1 + token = w.asset_path + } + const range = token + ? tokenRange(lineNumber, token) + : { startColumn: 1, endColumn: (lines[lineNumber - 1]?.length ?? 0) + 1 } + return { + message: w.message, + startLineNumber: lineNumber, + endLineNumber: lineNumber, + ...range + } + }) +} diff --git a/frontend/src/lib/components/assets/AssetGraph/types.ts b/frontend/src/lib/components/assets/AssetGraph/types.ts index 0db259a389..0e055abca2 100644 --- a/frontend/src/lib/components/assets/AssetGraph/types.ts +++ b/frontend/src/lib/components/assets/AssetGraph/types.ts @@ -6,6 +6,16 @@ export type GraphUsageKind = 'script' | 'flow' export interface AssetGraphAssetNode { kind: AssetKind path: string + // Fork workspaces only: 'fork' when this ducklake asset was materialized in + // the fork itself, 'deferred' when reads fall back to the parent workspace's + // current table via a defer view. Absent outside forks / for other kinds / + // when never materialized anywhere. Lockstep with Rust `GraphAssetNode`. + fork_materialization?: 'fork' | 'deferred' + // Base dimension path this node is the SCD2 `_current` companion view of + // (its producer declares `// materialize … history` on ``). Set only on + // the `_current` node; lets the canvas mark it as a derived "current view" + // rather than an unrelated table. Lockstep with Rust `GraphAssetNode`. + derived_from?: string } export interface AssetGraphRunnableNode { @@ -22,6 +32,11 @@ export interface AssetGraphRunnableNode { // Raw `// freshness ` value, e.g. "1h", "30m". Surfaced for // the badge; the runtime parses it as needed. freshness?: string + // Completion time (ISO) of the newest successful run of this pipeline + // member visible to the caller. The freshness chip compares it against + // the `// freshness` window to render fresh/stale. Absent = no + // successful run found (or none visible under job RLS). + last_success_at?: string // `// tag ` worker-tag override. Surfaced for the badge so users // can see which worker pool will pick this script up at a glance. tag?: string @@ -44,8 +59,15 @@ export interface AssetGraphRunnableNode { // Managed `// materialize` write strategy. Absent for non-materializing or // `manual` scripts. Used (with `partition_kind`) to decide whether a // produced asset's schema can evolve: only whole-table `replace` can, since - // `append`/`merge`/partitioned writes INSERT into a fixed-schema table. - materialize_strategy?: 'replace' | 'append' | 'merge' + // `append`/`merge`/`scd2`/partitioned writes INSERT into a fixed-schema + // table. `scd2` also identifies the producer of a `_current` companion + // view for the schema-contract `_current` → base-table fallback. + materialize_strategy?: 'replace' | 'append' | 'merge' | 'scd2' + // `on_schema_change=ignore` on the managed materialize — the producer's + // opt-out from downstream schema-contract warnings. Only present when set + // to `ignore` (default `warn` is absent). Threaded into the editor's + // contract mirror so it suppresses the same warnings the server check does. + materialize_on_schema_change?: string // Macros this script provides to the workspace registry (deployed // `// macros` library). Non-empty marks the node as a macro library; // drives the "defines N macros" badge and the details-pane signature @@ -130,12 +152,29 @@ export interface AssetGraphMacroEdge { unsaved?: boolean } +// Ordering-only "must-run-after" edge: `runnable_path`'s `// data_test` +// (a `relationships` ref, or a custom test reading a pipeline asset) needs +// `asset` materialized before the tested script runs — but the tested script +// doesn't consume the asset's rows, so this is NOT a lineage edge. Resolved +// server-side to the referenced asset's in-pipeline producer; fed into the +// cascade topo-sort (buildLineageDag) so a cold cascade orders the referenced +// dimension first, and rendered dashed on the canvas (like macro edges). +export interface AssetGraphTestEdge { + producer_kind: GraphUsageKind + producer_path: string + runnable_kind: GraphUsageKind + runnable_path: string + asset_kind: AssetKind + asset_path: string +} + export interface AssetGraphResponse { assets: AssetGraphAssetNode[] runnables: AssetGraphRunnableNode[] edges: AssetGraphEdge[] triggers: AssetGraphTrigger[] macro_edges?: AssetGraphMacroEdge[] + test_edges?: AssetGraphTestEdge[] } export type AssetGraphNodeData = diff --git a/frontend/src/lib/components/common/badge/Badge.svelte b/frontend/src/lib/components/common/badge/Badge.svelte index cf7a7cad66..c57d41029c 100644 --- a/frontend/src/lib/components/common/badge/Badge.svelte +++ b/frontend/src/lib/components/common/badge/Badge.svelte @@ -90,7 +90,7 @@ const hovers: Partial> = { gray: 'hover:bg-surface-hover', - blue: 'hover:bg-blue-200 dark:hover:bg-blue-700/40', + blue: 'hover:bg-blue-100 dark:hover:bg-blue-700/60', red: 'hover:bg-red-200 dark:hover:bg-red-500/25', green: 'hover:bg-green-200 dark:hover:bg-green-500/25', yellow: 'hover:bg-yellow-200 dark:hover:bg-yellow-500/25', diff --git a/frontend/src/lib/components/copilot/chat/AIChatManager.svelte.ts b/frontend/src/lib/components/copilot/chat/AIChatManager.svelte.ts index 5f5835bd13..c67bcd17b5 100644 --- a/frontend/src/lib/components/copilot/chat/AIChatManager.svelte.ts +++ b/frontend/src/lib/components/copilot/chat/AIChatManager.svelte.ts @@ -44,6 +44,9 @@ import { buildSummaryMessageContent } from './compactionPrompt' import { dfs } from '$lib/components/flows/previousResults' +import { SvelteSet } from 'svelte/reactivity' +import type { UserDraftItemKind } from '$lib/gen' +import { maskKey } from '$lib/components/sessions/modifiedItemsMask' import { getStringError } from './utils' import { type PasteAttachment } from './pasteTokens' import { chatDraft, expanded } from './chatDraft' @@ -343,6 +346,75 @@ export class AIChatManager { // session rather than the UI-active one — keeps backgrounded sessions isolated. sessionId: string | undefined = undefined + // Fired whenever the active chat id changes away from the one the consumer + // knows (a "/clear" rotation or a history switch). Session runtimes wire this + // to keep the session record's chatId aligned — the compare-page handoff + // (`from_session`) reads it, and a stale id would preselect the previous + // chat's items. Set here (not imported) to avoid a copilot→sessions cycle. + onChatRotated: ((chatId: string) => void) | undefined = undefined + + // Workspace items the CURRENT chat modified via AI tool calls, as + // `${UserDraftItemKind}:${storagePath}` keys (see modifiedItemsMask.ts). + // undefined = untracked: the global side-panel chat (never initialised) and + // loaded legacy chats with no stored mask, both of which fall back to the + // show-all bar. A SvelteSet (even empty) = tracked. Reactive so the session + // bar updates as tools record mid-turn. + modifiedItems = $state | undefined>(undefined) + + // Start tracking for a brand-new session chat (empty = "tracked, nothing yet"). + initModifiedItemsTracking() { + this.modifiedItems = new SvelteSet() + } + + // Record an item an AI tool call created/edited/deleted. No-op when untracked + // (the global singleton never initialises the set), so it stays unaffected. + recordModifiedItem(itemKind: UserDraftItemKind, storagePath: string) { + this.modifiedItems?.add(maskKey(itemKind, storagePath)) + } + + // Un-record an item whose chat-made change was discarded — without this the + // still-existing deployed item would keep reading as this chat's "Deployed" + // edit. Persisted immediately: unlike recordModifiedItem (whose persistence + // rides on the turn's saveChat), a discard can fire from the review dock + // outside any turn, and waiting would resurrect the entry on reload. + async removeModifiedItem(itemKind: UserDraftItemKind, storagePath: string) { + if (!this.modifiedItems?.delete(maskKey(itemKind, storagePath))) return + await this.#persistModifiedItems() + } + + // Move a mask entry to the path a draft actually deployed to. A draft-only + // flow/app parks at a synthetic `draft_{uuid}` storage path and deploys to + // its chosen path — without the move, the existence check at the synthetic + // path fails after reload and the deployed row vanishes from the dock. + async renameModifiedItem(itemKind: UserDraftItemKind, fromPath: string, toPath: string) { + if (fromPath === toPath) return + if (!this.modifiedItems?.delete(maskKey(itemKind, fromPath))) return + this.modifiedItems.add(maskKey(itemKind, toPath)) + await this.#persistModifiedItems() + } + + // Serialized, snapshot-at-write-time persistence: two rapid dock actions + // would otherwise race their saveChat writes, and the earlier (staler) + // snapshot could land last — dropping the later mutation until the next + // turn-end save. + #maskPersistQueue: Promise = Promise.resolve() + #persistModifiedItems(): Promise { + this.#maskPersistQueue = this.#maskPersistQueue.then(() => + this.historyManager + .saveChat( + this.displayMessages, + this.messages, + this.contextUsage, + this.modifiedItems ? [...this.modifiedItems] : undefined + ) + // Swallow (and log) a failed write so it can't wedge the queue as a + // rejected link — the next persist snapshots the full current set, so + // a lost write self-heals on the next mutation or turn-end save. + .catch((e) => console.error('Failed to persist modified-items mask', e)) + ) + return this.#maskPersistQueue + } + // Workspace AI skills (name + description) advertised in the GLOBAL system // prompt and surfaced as slash commands in session chat. Loaded // asynchronously when entering GLOBAL mode; the system message is rebuilt @@ -659,7 +731,12 @@ export class AIChatManager { ) switch (result) { case 'ok': - await this.historyManager.saveChat(this.displayMessages, this.messages, this.contextUsage) + await this.historyManager.saveChat( + this.displayMessages, + this.messages, + this.contextUsage, + this.modifiedItems ? [...this.modifiedItems] : undefined + ) sendUserToast('Conversation compacted.') break case 'empty': @@ -1636,7 +1713,12 @@ export class AIChatManager { const projectedContextTokens = this.contextTokens + this.estimateMessagesTokens([userMessage]) this.messages.push(userMessage) - await this.historyManager.saveChat(this.displayMessages, this.messages, this.contextUsage) + await this.historyManager.saveChat( + this.displayMessages, + this.messages, + this.contextUsage, + this.modifiedItems ? [...this.modifiedItems] : undefined + ) this.currentReply = '' this.currentReasoning = '' @@ -1672,7 +1754,12 @@ export class AIChatManager { this.contextUsage = Math.max(0, this.contextUsage - freed) } } - await this.historyManager.saveChat(this.displayMessages, this.messages, this.contextUsage) + await this.historyManager.saveChat( + this.displayMessages, + this.messages, + this.contextUsage, + this.modifiedItems ? [...this.modifiedItems] : undefined + ) } } // Rollback anchors for restoreUnsentTurn: captured after compaction so @@ -1758,7 +1845,10 @@ export class AIChatManager { }, requestConfirmation: this.requestConfirmation, shouldAutoAcceptToolConfirmations: () => this.autoAcceptToolConfirmationsActive, - requestUserQuestion: this.requestUserQuestion + requestUserQuestion: this.requestUserQuestion, + onItemModified: (kind, path) => this.recordModifiedItem(kind, path), + onItemDeployed: (kind, from, to) => void this.renameModifiedItem(kind, from, to), + onItemDiscarded: (kind, path) => void this.removeModifiedItem(kind, path) } } @@ -1794,7 +1884,12 @@ export class AIChatManager { this.contextUsage = result?.lastIterationUsage ? result.lastIterationUsage.prompt + result.lastIterationUsage.completion : undefined - await this.historyManager.saveChat(this.displayMessages, this.messages, this.contextUsage) + await this.historyManager.saveChat( + this.displayMessages, + this.messages, + this.contextUsage, + this.modifiedItems ? [...this.modifiedItems] : undefined + ) // Still counts as the saved first turn — skipping the hook here would // permanently miss it (the next turn isn't "first" anymore). if (isFirstUserTurn && this.afterFirstTurnSaved) { @@ -1824,7 +1919,12 @@ export class AIChatManager { // user message on reload. Remove it instead. this.historyManager.deletePastChat(this.historyManager.getCurrentChatId()) } else { - await this.historyManager.saveChat(this.displayMessages, this.messages, this.contextUsage) + await this.historyManager.saveChat( + this.displayMessages, + this.messages, + this.contextUsage, + this.modifiedItems ? [...this.modifiedItems] : undefined + ) } if (!wasAborted) { sendUserToast('The model returned no response — your message was restored to the input.') @@ -1843,7 +1943,12 @@ export class AIChatManager { if (this.autoAcceptEditsActive) { this.acceptPendingFlowEdits() } - await this.historyManager.saveChat(this.displayMessages, this.messages, this.contextUsage) + await this.historyManager.saveChat( + this.displayMessages, + this.messages, + this.contextUsage, + this.modifiedItems ? [...this.modifiedItems] : undefined + ) // Only this branch is a clean send: the queued-message flush below // auto-sends the next message after it (set after saveChat so a // persistence failure falls through to the restore path instead). @@ -1867,7 +1972,12 @@ export class AIChatManager { // compaction on the next send instead of failing the same way again. this.contextUsage = undefined try { - await this.historyManager.saveChat(this.displayMessages, this.messages, this.contextUsage) + await this.historyManager.saveChat( + this.displayMessages, + this.messages, + this.contextUsage, + this.modifiedItems ? [...this.modifiedItems] : undefined + ) } catch (saveErr) { console.error('Failed to persist partial chat after error', saveErr) } @@ -1996,15 +2106,25 @@ export class AIChatManager { // Drop any message queued in this conversation so it can't auto-send into // the fresh chat or linger as a card across the switch. this.queuedMessage = '' - await this.historyManager.save(this.displayMessages, this.messages, this.contextUsage) + await this.historyManager.save( + this.displayMessages, + this.messages, + this.contextUsage, + this.modifiedItems ? [...this.modifiedItems] : undefined + ) this.displayMessages = [] this.messages = [] this.contextUsage = undefined + // The mask belongs to the conversation just saved — the fresh chat starts + // its own (empty) tracking; carrying entries over would claim the previous + // conversation's edits for the new one. Untracked chats stay untracked. + if (this.modifiedItems) this.modifiedItems = new SvelteSet() // In an AI session, linked files are session-scoped: they persist across conversations // (cleared only when the session is deleted). The ephemeral global side-panel chat has no // session, so "New chat" must clear them — otherwise the next, unrelated conversation // would still get the previous file roster and could read/search it. if (!this.isSessionChat) this.attachedFiles.clear() + this.onChatRotated?.(this.historyManager.getCurrentChatId()) } loadPastChat = async (id: string) => { @@ -2019,7 +2139,16 @@ export class AIChatManager { this.displayMessages = chat.displayMessages this.messages = chat.actualMessages this.contextUsage = normalizeContextUsage(chat.contextUsage) + // Seed the modified-items mask from the stored chat. A stored array + // (even empty) → tracked; a legacy chat with no field stays untracked + // (undefined) so the session bar falls back to showing all drafts. The + // global side-panel chat never tracks, so leave it untouched there. + if (this.isSessionChat) { + const stored = this.historyManager.getModifiedItems(id) + this.modifiedItems = stored !== undefined ? new SvelteSet(stored) : undefined + } this.#automaticScroll = true + this.onChatRotated?.(id) } } diff --git a/frontend/src/lib/components/copilot/chat/AIChatManager.test.ts b/frontend/src/lib/components/copilot/chat/AIChatManager.test.ts index d23cbf96a1..a415377a7b 100644 --- a/frontend/src/lib/components/copilot/chat/AIChatManager.test.ts +++ b/frontend/src/lib/components/copilot/chat/AIChatManager.test.ts @@ -693,7 +693,9 @@ describe('AIChatManager context compaction', () => { expect(manager.messages[0]).toMatchObject({ role: 'user', content: 'c'.repeat(400) }) // Mid-turn, the report is debited by the freed estimate (visible in the // compaction-time save) so a rolled-back turn keeps a consistent value - expect(saveChat).toHaveBeenCalledWith(expect.anything(), expect.anything(), 650_000) + // 4th arg: the modified-items mask rides on every save (undefined here — + // this bare manager never initialised tracking). + expect(saveChat).toHaveBeenCalledWith(expect.anything(), expect.anything(), 650_000, undefined) // At commit, the no-report turn clears the stored value; the readable // number falls back to estimating the now-tiny compacted history expect(manager.contextUsage).toBeUndefined() diff --git a/frontend/src/lib/components/copilot/chat/HistoryManager.svelte.ts b/frontend/src/lib/components/copilot/chat/HistoryManager.svelte.ts index 9173e4c441..0db0751293 100644 --- a/frontend/src/lib/components/copilot/chat/HistoryManager.svelte.ts +++ b/frontend/src/lib/components/copilot/chat/HistoryManager.svelte.ts @@ -25,6 +25,12 @@ interface ChatSchema extends IDBSchema { // New writes store the plain reported token count; chats persisted by // earlier versions may still hold the legacy anchor object. contextUsage?: PersistedContextUsage + // Workspace items this chat modified via AI tool calls, as + // `${UserDraftItemKind}:${storagePath}` keys. Persisted out-of-band from + // the message arrays so it survives compaction. Absent (undefined) on + // chats predating this feature → consumers fall back to showing all + // workspace drafts; a defined array (even empty) means "tracked". + modifiedItems?: string[] } } } @@ -80,6 +86,29 @@ export function __resetLegacyChatClaimForTesting(): void { legacyChatClaim = undefined } +// Read a chat's modified-items mask by chatId WITHOUT mounting an AIChatManager, +// for the standalone /forks/compare route. Returns undefined for a legacy chat +// (no field) so the page falls back to selecting all items; a defined array +// (even empty) narrows the preselection. Opens a throwaway user-scoped handle; +// the `get` is O(1) on the `id` keyPath. +export async function readChatModifiedItems(chatId: string): Promise { + const dbh = userScopedDb(DB_NAME, { + version: 1, + upgrade: createChatStore, + migrate: migrateLegacyChatDb + }) + try { + const db = await dbh.whenReady() + const chat = await db?.get('chats', chatId) + return chat?.modifiedItems + } catch (err) { + console.error('Could not read chat modified items', err) + return undefined + } finally { + dbh.close() + } +} + export default class HistoryManager { // Per-instance handle to the shared per-user DB lifecycle. There is one // HistoryManager per AIChatManager (the singleton + one per session runtime), @@ -100,6 +129,7 @@ export default class HistoryManager { lastModified: number sessionId?: string contextUsage?: PersistedContextUsage + modifiedItems?: string[] } > = $state({}) @@ -173,10 +203,15 @@ export default class HistoryManager { return Object.values(this.savedChats) } + getModifiedItems(id: string): string[] | undefined { + return this.savedChats[id]?.modifiedItems + } + async saveChat( displayMessages: DisplayMessage[], messages: ChatCompletionMessageParam[], - contextUsage?: number + contextUsage?: number, + modifiedItems?: string[] ) { if (displayMessages.length > 0) { // Compaction replaces the original first message with a summary boundary. @@ -203,7 +238,18 @@ export default class HistoryManager { id: this.currentChatId, lastModified: Date.now(), ...(this.sessionId ? { sessionId: this.sessionId } : {}), - ...(contextUsage !== undefined ? { contextUsage } : {}) + ...(contextUsage !== undefined ? { contextUsage } : {}), + // Only persist when the caller passes a defined array. Loaded legacy + // chats keep their accumulator undefined, so we never retroactively + // stamp them with [] (which would flip them to the filtered view). + // But since `put` replaces the whole record, a caller that omits the + // argument must not ERASE a tracked chat's stored mask — fall back to + // the previously saved field. + ...(modifiedItems !== undefined + ? { modifiedItems } + : this.savedChats[this.currentChatId]?.modifiedItems !== undefined + ? { modifiedItems: this.savedChats[this.currentChatId].modifiedItems } + : {}) } this.savedChats = { ...this.savedChats, @@ -218,9 +264,10 @@ export default class HistoryManager { async save( displayMessages: DisplayMessage[], messages: ChatCompletionMessageParam[], - contextUsage?: number + contextUsage?: number, + modifiedItems?: string[] ) { - await this.saveChat(displayMessages, messages, contextUsage) + await this.saveChat(displayMessages, messages, contextUsage, modifiedItems) this.currentChatId = createLongHash() } diff --git a/frontend/src/lib/components/copilot/chat/HistoryManager.test.ts b/frontend/src/lib/components/copilot/chat/HistoryManager.test.ts index 38fb7ecf0d..540dfaf72e 100644 --- a/frontend/src/lib/components/copilot/chat/HistoryManager.test.ts +++ b/frontend/src/lib/components/copilot/chat/HistoryManager.test.ts @@ -141,3 +141,30 @@ describe('HistoryManager title across compaction', () => { expect(hm.getAllSavedChats().find((c) => c.id === id)?.title).toBe('original first question') }) }) + +describe('HistoryManager modified-items mask persistence', () => { + const msgs = [{ role: 'user', content: 'hello', index: 0 }] as DisplayMessage[] + + it('a save without the argument preserves a previously stored mask', async () => { + const hm = new HistoryManager() + await hm.init() + const id = hm.getCurrentChatId() + + await hm.saveChat(msgs, [] as ChatCompletionMessageParam[], undefined, ['script:u/a/x']) + expect(hm.getModifiedItems(id)).toEqual(['script:u/a/x']) + + // e.g. manual compaction re-saving the transcript: the whole record is + // rewritten, but the tracked mask must survive. + await hm.saveChat(msgs, [] as ChatCompletionMessageParam[]) + expect(hm.getModifiedItems(id)).toEqual(['script:u/a/x']) + }) + + it('never retroactively stamps an untracked chat', async () => { + const hm = new HistoryManager() + await hm.init() + const id = hm.getCurrentChatId() + + await hm.saveChat(msgs, [] as ChatCompletionMessageParam[]) + expect(hm.getModifiedItems(id)).toBeUndefined() + }) +}) diff --git a/frontend/src/lib/components/copilot/chat/global/core.ts b/frontend/src/lib/components/copilot/chat/global/core.ts index 90dcb655f4..24045280d0 100644 --- a/frontend/src/lib/components/copilot/chat/global/core.ts +++ b/frontend/src/lib/components/copilot/chat/global/core.ts @@ -114,6 +114,7 @@ import { getEphemeralSecretVariableDraftValue, getGlobalDraft, getGlobalDraftStoragePath, + itemKindFor, listGlobalDrafts, persistGlobalDraft, readGlobalDraftValue, @@ -2779,6 +2780,7 @@ function finishAppDraftWrite( ): string { const failure = draftWriteFailure(result, ctx) if (failure) return failure + ctx.toolCallbacks.onItemModified?.(result.itemKind, result.storagePath) const { content, message } = onSaved() ctx.toolCallbacks.setToolStatus(ctx.toolId, { content, result: 'Saved as draft' }) return JSON.stringify({ success: true, message }, null, 2) @@ -2791,6 +2793,7 @@ function finishDraftWrite( ): string { const failure = draftWriteFailure(result, ctx) if (failure) return failure + ctx.toolCallbacks.onItemModified?.(result.itemKind, result.storagePath) const stored = result.item const verb = existed ? 'Updated' : 'Created' // Don't echo the flow value back: the model just sent it in the write call, @@ -3898,6 +3901,16 @@ async function discardLocalDraft( await deleteGlobalDraft(workspace, type, path, triggerKind) + // The chat's touch on the item is undone — drop it from the mask so a + // pre-existing deployed item doesn't keep reading as this chat's edit. + const discardedKind = itemKindFor(type, triggerKind) + if (discardedKind) { + toolCallbacks.onItemDiscarded?.( + discardedKind, + getGlobalDraftStoragePath(workspace, type, path, triggerKind) + ) + } + toolCallbacks.setToolStatus(toolId, { content: `Discarded ${type} "${path}" draft`, result: 'Draft discarded' @@ -4237,6 +4250,9 @@ async function deployDraft( }) let actions: ToolDisplayAction[] | undefined + // Where the deploy actually lands — the app branch can resolve a different + // target from the draft's own path fields; the mask rename below must track it. + let deployedPath = path if (type === 'script' || type === 'flow') { // Promote the full persisted draft via the shared deploy module — the same @@ -4429,6 +4445,7 @@ async function deployDraft( throw e } } + deployedPath = targetPath if (await AppService.existsApp({ workspace, path: targetPath })) { // Omit custom_path on update for now. The backend preserves it when absent, while // sending it requires admin privileges; this chat deploy path does not yet mirror @@ -4478,6 +4495,18 @@ async function deployDraft( await deleteGlobalDraft(workspace, type, path, triggerKind, { preserveLiveDraft: true }) + // Move the chat's mask entry to the deployed path: a draft-only item's + // synthetic storage key never exists deployed, so the entry would otherwise + // stop matching anything after the draft is gone. + const deployedKind = itemKindFor(type, triggerKind) + if (deployedKind) { + toolCallbacks.onItemDeployed?.( + deployedKind, + getGlobalDraftStoragePath(workspace, type, path, triggerKind), + deployedPath + ) + } + // Reload the session preview if it's open on the deployed item. Map the // deploy type to the preview kind — a raw app deploys under 'app' but the // preview addresses it as 'raw_app'; non-previewable types map to undefined. @@ -4548,6 +4577,17 @@ async function deleteWorkspaceItem( await deleteGlobalDraft(workspace, type, path, triggerKind) + // Record the deletion in the chat's modified-items mask. In a fork this leaves a + // reviewable "removed" diff vs the parent that stays scoped to this chat. Keyed + // by the same (itemKind, storagePath) as writes so it joins the draft/fork lists. + const deletedKind = itemKindFor(type, triggerKind) + if (deletedKind) { + toolCallbacks.onItemModified?.( + deletedKind, + getGlobalDraftStoragePath(workspace, type, path, triggerKind) + ) + } + toolCallbacks.setToolStatus(toolId, { content: `Deleted ${type} "${path}"`, result: 'Deleted' diff --git a/frontend/src/lib/components/copilot/chat/global/userDraftAdapter.ts b/frontend/src/lib/components/copilot/chat/global/userDraftAdapter.ts index 87ca767904..2717d41608 100644 --- a/frontend/src/lib/components/copilot/chat/global/userDraftAdapter.ts +++ b/frontend/src/lib/components/copilot/chat/global/userDraftAdapter.ts @@ -108,7 +108,7 @@ function clearEphemeralSecretVariableDraftValues(workspace: string): void { secretVariableDraftValues.delete(workspace) } -function itemKindFor( +export function itemKindFor( type: WorkspaceItemType, triggerKind?: TriggerKind ): UserDraftItemKind | undefined { @@ -377,10 +377,25 @@ export async function readGlobalDraftValue( return (await fetchBackendDraftValue(workspace, itemKind, storagePath)) as V | undefined } +// `itemKind` + `storagePath` are the canonical identity of the persisted draft +// (NOT item.path, which is the friendly display path). Callers use them to record +// the chat's modified-items mask. export type DraftPersistResult = - | { status: 'saved'; item: WorkspaceItem } - | { status: 'conflict'; item: WorkspaceItem; serverTimestamp?: string } - | { status: 'error'; item: WorkspaceItem; message: string } + | { status: 'saved'; item: WorkspaceItem; itemKind: UserDraftItemKind; storagePath: string } + | { + status: 'conflict' + item: WorkspaceItem + itemKind: UserDraftItemKind + storagePath: string + serverTimestamp?: string + } + | { + status: 'error' + item: WorkspaceItem + itemKind: UserDraftItemKind + storagePath: string + message: string + } // Persist a built draft value. `UserDraft.seed` reflects it into an open editor's // cell WITHOUT a double-POST (no-ops if no cell; its seedNextWrite suppresses the @@ -417,14 +432,20 @@ export async function persistGlobalDraft( // the chat "saved" while the DB-backed source of truth was never updated. const saveState = UserDraftDbSyncer.getState({ workspace, itemKind, path: storagePath }) if (saveState.state === 'failed') { - return { status: 'error', item, message: saveState.failureMessage ?? 'Draft save failed' } + return { + status: 'error', + item, + itemKind, + storagePath, + message: saveState.failureMessage ?? 'Draft save failed' + } } const conflict = opts.force ? undefined : UserDraftDbSyncer.getConflict({ workspace, itemKind, path: storagePath }).conflict return conflict - ? { status: 'conflict', item, serverTimestamp: conflict.serverTimestamp } - : { status: 'saved', item } + ? { status: 'conflict', item, itemKind, storagePath, serverTimestamp: conflict.serverTimestamp } + : { status: 'saved', item, itemKind, storagePath } } export async function getGlobalDraft( diff --git a/frontend/src/lib/components/copilot/chat/shared.ts b/frontend/src/lib/components/copilot/chat/shared.ts index 7abdc75168..c454f15fe6 100644 --- a/frontend/src/lib/components/copilot/chat/shared.ts +++ b/frontend/src/lib/components/copilot/chat/shared.ts @@ -3,6 +3,7 @@ import type { ChatCompletionMessageFunctionToolCall, ChatCompletionMessageParam } from 'openai/resources/chat/completions.mjs' +import type { UserDraftItemKind } from '$lib/gen' /** * Special module IDs used throughout the flow system @@ -759,6 +760,15 @@ export interface ToolCallbacks { toolId: string, question: UserQuestionDisplay ) => Promise + /** Records a workspace item the tool call created/edited/deleted, by its + * canonical (itemKind, storagePath). Session chats wire this to accumulate the + * chat's modified-items mask; the global side-panel chat omits it (no-op). */ + onItemModified?: (itemKind: UserDraftItemKind, storagePath: string) => void + /** A tool deployed a draft: the mask entry moves from the draft's storage path + * to the deployed path (they differ for synthetic draft-only storage keys). */ + onItemDeployed?: (itemKind: UserDraftItemKind, storagePath: string, deployedPath: string) => void + /** A tool discarded a draft: the chat's touch on the item is undone. */ + onItemDiscarded?: (itemKind: UserDraftItemKind, storagePath: string) => void } export function createToolDef( diff --git a/frontend/src/lib/components/copilot/reasoningRegistry.ts b/frontend/src/lib/components/copilot/reasoningRegistry.ts index 4776e64c79..519c1a4021 100644 --- a/frontend/src/lib/components/copilot/reasoningRegistry.ts +++ b/frontend/src/lib/components/copilot/reasoningRegistry.ts @@ -319,7 +319,7 @@ export const DEEPSEEK_OFF_SENTINEL: ReasoningEffort = 'none' * model that reasons *by default* — omitting the field would silently keep * the default-on behavior. Undefined means omission is the correct off. */ -function explicitOffToken(provider: AIProvider, model: string): ReasoningEffort | undefined { +export function explicitOffToken(provider: AIProvider, model: string): ReasoningEffort | undefined { switch (reasoningProviderFamily(provider, model)) { case 'googleai': // Gemini 2.5/3 think by default (dynamic budget / level). The backend diff --git a/frontend/src/lib/components/sessions/DraftDiffDrawer.svelte b/frontend/src/lib/components/sessions/DraftDiffDrawer.svelte deleted file mode 100644 index 0b529b2056..0000000000 --- a/frontend/src/lib/components/sessions/DraftDiffDrawer.svelte +++ /dev/null @@ -1,204 +0,0 @@ - - - buildEditUrl(d as unknown as WorkspaceItemDiff, workspaceId)} -> - {#snippet titleExtra()} -
- - {ws?.name ?? workspaceId} -
- {/snippet} -
diff --git a/frontend/src/lib/components/sessions/ForkDiffDrawer.svelte b/frontend/src/lib/components/sessions/ForkDiffDrawer.svelte deleted file mode 100644 index e03fc5a49c..0000000000 --- a/frontend/src/lib/components/sessions/ForkDiffDrawer.svelte +++ /dev/null @@ -1,125 +0,0 @@ - - - buildEditUrl(d as unknown as WorkspaceItemDiff, forkWorkspaceId)} -> - {#snippet titleExtra()} -
- - {forkWs?.name ?? forkWorkspaceId} - - {parentWs?.name ?? parentWorkspaceId} - {#if comparison} - - {comparison.summary.total_diffs} item{comparison.summary.total_diffs !== 1 ? 's' : ''} - - {#if comparison.summary.conflicts > 0} - - - {comparison.summary.conflicts} conflict{comparison.summary.conflicts !== 1 ? 's' : ''} - - {/if} - {/if} -
- {/snippet} -
diff --git a/frontend/src/lib/components/sessions/SessionChangesBar.svelte b/frontend/src/lib/components/sessions/SessionChangesBar.svelte new file mode 100644 index 0000000000..9f59d1f320 --- /dev/null +++ b/frontend/src/lib/components/sessions/SessionChangesBar.svelte @@ -0,0 +1,264 @@ + + +{#snippet dock()} + +
+ {#if dockCounts.draft > 0} + diffDrawer?.open()}> + {dockCounts.draft} draft{dockCounts.draft === 1 ? '' : 's'} + + {/if} + {#if dockCounts.deployed > 0} + diffDrawer?.open()}> + {dockCounts.deployed} deployed + + {/if} + {#if deletionOnly && compareHref} + + + Review deletions on compare page + + + {/if} +
+{/snippet} + +{#if committedId && isUnavailable} + +
+
+ +
+ The {committedIsFork ? 'fork' : 'workspace'} has been archived or deleted + + Move this session to another workspace, or discard it. + {committedId} + +
+
+
+ onMove?.(workspaceId)} + onCreateFork={async (fork) => { + await onCreateForkAndMove?.(fork) + }} + createForkCaption="Created immediately and the session moved into it." + > + {#snippet trigger()} + + {/snippet} + + +
+
+{:else if showBar && committedId} + +
+
+ + Edits +
+ {@render dock()} +
+{/if} + + +{#if committedId && !isUnavailable} + + void runtime?.manager.renameModifiedItem(item.draftKind, item.path, item.displayPath)} + onItemDiscarded={(item) => void runtime?.manager.removeModifiedItem(item.draftKind, item.path)} + /> +{/if} diff --git a/frontend/src/lib/components/sessions/SessionDiffDrawer.svelte b/frontend/src/lib/components/sessions/SessionDiffDrawer.svelte new file mode 100644 index 0000000000..1a6692048c --- /dev/null +++ b/frontend/src/lib/components/sessions/SessionDiffDrawer.svelte @@ -0,0 +1,107 @@ + + + + {#snippet titleExtra()} +
+ {#if isFork} + + + {ws?.name ?? workspaceId} + + + + {parentWs?.name ?? parentWorkspaceId} + + {:else} + + + {ws?.name ?? workspaceId} + + {/if} +
+ {/snippet} +
diff --git a/frontend/src/lib/components/sessions/SessionDraftBar.svelte b/frontend/src/lib/components/sessions/SessionDraftBar.svelte deleted file mode 100644 index d23070151a..0000000000 --- a/frontend/src/lib/components/sessions/SessionDraftBar.svelte +++ /dev/null @@ -1,77 +0,0 @@ - - -{#if committedId && count > 0} -
-
- - - {count} draft{count === 1 ? '' : 's'} - {#snippet text()} - Tracks all unsaved draft changes in this workspace — including edits made outside this - chat (e.g. in the editor), not only changes made by the assistant. - {/snippet} - -
-
- drawer?.open()} /> - -
-
- - -{/if} diff --git a/frontend/src/lib/components/sessions/SessionForkBar.svelte b/frontend/src/lib/components/sessions/SessionForkBar.svelte deleted file mode 100644 index a0aafee456..0000000000 --- a/frontend/src/lib/components/sessions/SessionForkBar.svelte +++ /dev/null @@ -1,212 +0,0 @@ - - -{#if committedId && isUnavailable} - -
-
- -
- The {committedIsFork ? 'fork' : 'workspace'} has been archived or deleted - - Move this session to another workspace, or discard it. - {committedId} - -
-
-
- onMove?.(workspaceId)} - onCreateFork={async (fork) => { - await onCreateForkAndMove?.(fork) - }} - createForkCaption="Created immediately and the session moved into it." - > - {#snippet trigger()} - - {/snippet} - - -
-
-{:else if forksAllowed && isFork && sessionWorkspace && parentWorkspace && parentWorkspaceId && committedId} - {@const StatusIcon = - forkStatus === 'ahead' - ? GitPullRequestArrow - : forkStatus === 'diverged' - ? GitCompareArrows - : GitFork} - {@const statusColor = - forkStatus === 'ahead' - ? 'text-blue-500' - : forkStatus === 'diverged' - ? 'text-amber-500' - : 'text-secondary'} - {@const statusTitle = - forkStatus === 'ahead' - ? 'Ahead of parent' - : forkStatus === 'diverged' - ? 'Diverged from parent' - : forkStatus === 'in_sync' - ? 'In sync with parent' - : 'Fork'} -
-
- - - - - {sessionWorkspace.name} - - - - {parentWorkspace.name} - -
-
- diffDrawer?.open()} - /> - -
-
- - -{/if} diff --git a/frontend/src/lib/components/sessions/SessionPicker.svelte b/frontend/src/lib/components/sessions/SessionPicker.svelte index aad3de26c3..0342fd53e1 100644 --- a/frontend/src/lib/components/sessions/SessionPicker.svelte +++ b/frontend/src/lib/components/sessions/SessionPicker.svelte @@ -21,7 +21,6 @@ import { slide } from 'svelte/transition' import { createSession, - deriveForkStatus, deleteSessionsForWorkspace, isForkSession, reconcileAfterWorkspaceChange, @@ -53,15 +52,6 @@ import { sendUserToast } from '$lib/toast' import { currentWorkspaceRootId, workspaceRootId } from './sessionScope.svelte' - // Look up the cached fork comparison for a session through its runtime - // (if any). The deriveForkStatus helper handles the "no runtime yet" - // and "comparison not loaded" cases by returning undefined; we render - // a neutral fork icon in that interim, then upgrade to the proper - // status icon once the comparison lands. - function forkStatusFor(session: Session) { - return deriveForkStatus(session, $userWorkspaces, getRuntime(session.id)?.forkComparison.val) - } - function isForkFor(session: Session): boolean { return isForkSession(session, $userWorkspaces) } @@ -124,8 +114,8 @@ } // Flat list passing the archive + scope filters. Grouping for display happens - // in `sessionGroups`; this flat view drives the runtime / fork-comparison - // effects, the unread total, and keyboard navigation. + // in `sessionGroups`; this flat view drives the runtime effect, the unread + // total, and keyboard navigation. const visibleSessions = $derived( sessionState.sessions.filter((s) => { if (s.transient) return false @@ -211,23 +201,6 @@ } }) - // Pre-fetch the fork comparison for every visible fork session so the - // sidebar icons reflect the right ahead/diverged state without - // requiring the user to click into each session. Cheap enough at - // typical session counts; falls back to a plain dot until the - // fetch lands. - $effect(() => { - if (sectionCollapsed.val) return - for (const session of visibleSessions) { - if (!session.workspace_id) continue - const ws = $userWorkspaces.find((w) => w.id === session.workspace_id) - if (!ws?.parent_workspace_id) continue - const rt = getRuntime(session.id) - if (!rt) continue - void rt.ensureForkComparison(ws.parent_workspace_id, session.workspace_id) - } - }) - function isUnavailableFork(session: Session): boolean { return !!session.workspace_id && !$userWorkspaces.find((w) => w.id === session.workspace_id) } @@ -241,10 +214,6 @@ if (!isUnavailableFork(session)) { syncWorkspaceTo(session.workspace_id) } - // Refresh the fork diff count — users typically click back into a - // session after editing items elsewhere in the SPA, where neither - // the visibility-change nor the AI-loading signal would fire. - void getRuntime(session.id)?.refreshForkComparison() await goto(`/sessions?session_name=${encodeURIComponent(session.name)}`) if (restoreFocus) { // goto() resets focus to — put it back on the active session button @@ -436,7 +405,7 @@ {session.summary ?? 'Untitled session'} {#if draft || unread > 0} diff --git a/frontend/src/lib/components/sessions/SessionStatusDot.svelte b/frontend/src/lib/components/sessions/SessionStatusDot.svelte index b47816ab19..35028c80cc 100644 --- a/frontend/src/lib/components/sessions/SessionStatusDot.svelte +++ b/frontend/src/lib/components/sessions/SessionStatusDot.svelte @@ -3,19 +3,16 @@ AlertCircle, AlertTriangle, Building, - GitCompareArrows, GitFork, - GitPullRequestArrow, GitPullRequestClosed } from 'lucide-svelte' import type { SessionChatStatus } from './sessionRuntime.svelte' - import type { ForkStatus } from './sessionState.svelte' let { status, isFork, - forkStatus - }: { status: SessionChatStatus; isFork: boolean; forkStatus?: ForkStatus } = $props() + unavailable = false + }: { status: SessionChatStatus; isFork: boolean; unavailable?: boolean } = $props() const statusTooltip: Record = { idle: 'No chat activity', @@ -26,13 +23,6 @@ error: 'Last message had an error' } - const forkTooltip: Record = { - in_sync: 'Fork — in sync with parent', - ahead: 'Fork — ahead of parent', - diverged: 'Fork — diverged from parent', - unavailable: 'Fork — no longer available' - } - // Live chat signals take precedence over the persistent kind/fork // indicator: streaming, needs-confirmation, and error are time-critical // and warrant briefly hijacking the icon slot. @@ -41,7 +31,11 @@ ) const persistentTitle = $derived( - isFork ? (forkStatus ? forkTooltip[forkStatus] : 'Fork session') : 'Root workspace session' + isFork + ? unavailable + ? 'Fork — no longer available' + : 'Fork session' + : 'Root workspace session' ) const title = $derived(liveOverride ? statusTooltip[status] : persistentTitle) @@ -59,11 +53,7 @@ {:else if status === 'error'} {:else if isFork} - {#if forkStatus === 'ahead'} - - {:else if forkStatus === 'diverged'} - - {:else if forkStatus === 'unavailable'} + {#if unavailable} {:else} diff --git a/frontend/src/lib/components/sessions/SessionWrapper.svelte b/frontend/src/lib/components/sessions/SessionWrapper.svelte index c9fdb6454f..ee8cbed6e0 100644 --- a/frontend/src/lib/components/sessions/SessionWrapper.svelte +++ b/frontend/src/lib/components/sessions/SessionWrapper.svelte @@ -30,8 +30,7 @@ import RawAppEditorView from './RawAppEditorView.svelte' import PipelineEditorView from './PipelineEditorView.svelte' import SessionWorkspaceBar from './SessionWorkspaceBar.svelte' - import SessionForkBar from './SessionForkBar.svelte' - import SessionDraftBar from './SessionDraftBar.svelte' + import SessionChangesBar from './SessionChangesBar.svelte' import { createSession, deleteSessionsForWorkspace, @@ -235,7 +234,7 @@ // True when the session committed to a workspace that's no longer in // the user's list (deleted / archived / access revoked). The chat is - // disabled and SessionForkBar shows a move/discard banner. + // disabled and SessionChangesBar shows a move/discard banner. const isUnavailable = $derived( !!session?.workspace_id && !$userWorkspaces.find((w) => w.id === session!.workspace_id) ) @@ -274,16 +273,16 @@ {#if !hasFirstUserMessage} {/if} - +
{#if session.archived && !isUnavailable}
{/if} - moveAndActivate(workspaceId)} onCreateForkAndMove={(fork) => createForkAndMove(fork)} onArchive={() => archiveAndReset()} onDelete={() => (deleteConfirmOpen = true)} /> -
{/snippet} diff --git a/frontend/src/lib/components/sessions/WorkspaceDiffDrawer.svelte b/frontend/src/lib/components/sessions/WorkspaceDiffDrawer.svelte index e6b4e5fd2f..50c6653b0a 100644 --- a/frontend/src/lib/components/sessions/WorkspaceDiffDrawer.svelte +++ b/frontend/src/lib/components/sessions/WorkspaceDiffDrawer.svelte @@ -1,45 +1,19 @@ - - searchableText(d)} + items={displayEntries} + bind:filteredItems={searchedEntries} + f={(e: DisplayEntry) => searchableText(e)} /> -{#snippet renderTreeNode(node: TreeNode, depth: number)} + +{#snippet rowBadge(item: DeployItem)} + {#if badgeOf(item) === 'draft'} + {#if model.staleOf(item.key)} + + + {#snippet trigger()} + + {/snippet} + {#snippet content()} +
+ Started from an older deployed version. A newer version was deployed after this draft + began. Review the latest deploy before deploying. +
+ {/snippet} +
+ {/if} + + {item.draftOnly ? 'Draft only' : 'Draft'} + + {:else} + + + + + + + {/if} +{/snippet} + + +{#snippet deployFailed(item: DeployItem)} + {@const s = model.statusOf(item.key)} + {#if s?.status === 'failed'} + + Failed + + + {/if} +{/snippet} + +{#snippet renderTreeNode(node: TreeNode, depth: number)} {#if node.type === 'folder'} {@const isUserScope = node.isScope && node.name.startsWith('u/')} {@const fkey = node.key} @@ -417,36 +613,62 @@ {@const isHl = fkey === highlightedKey}
(folderOpen[fkey] = (e.currentTarget as HTMLDetailsElement).open)} + ontoggle={(e) => { + // Record real user toggles only; skip the echo fired when the `open` + // attribute is driven by state (search force-open, expandApp). + const domOpen = (e.currentTarget as HTMLDetailsElement).open + if (domOpen !== isFolderOpen(fkey)) folderOpen[fkey] = domOpen + }} class="select-none" > {#if node.app} - {@const appSummary = summaries[node.app.summaryKey] ?? node.app.summary} - + {@const appItem = segmentItems.find((it) => it.key === node.app?.summaryKey)} + setHoverHighlight(fkey)} + onclick={(e) => { + e.preventDefault() + if (appItem) revealDiff(appItem, fkey) + }} title={node.fullPath} - class="flex items-center gap-1.5 px-3 py-1.5 cursor-pointer hover:bg-surface-hover list-none [&::-webkit-details-marker]:hidden tree-summary {isHl + class="flex items-center gap-2 pl-3 pr-1 py-2 rounded-md cursor-pointer hover:bg-surface-hover list-none [&::-webkit-details-marker]:hidden tree-summary {isHl ? 'bg-surface-hover' : ''}" style="padding-left: {depth * 12 + 8}px" > - {appSummary ?? node.name} + {node.app.summary ?? node.name} - - + {#if appItem} + {@render rowBadge(staged[appItem.key] ?? appItem)} + {/if} + + {:else} setHoverHighlight(fkey)} - class="flex items-center gap-1.5 px-3 py-1.5 cursor-pointer text-xs font-normal font-mono text-secondary hover:bg-surface-hover list-none [&::-webkit-details-marker]:hidden tree-summary {isHl + class="flex items-center gap-2 pl-3 pr-1 py-2 rounded-md cursor-pointer text-xs font-normal font-mono text-secondary hover:bg-surface-hover list-none [&::-webkit-details-marker]:hidden tree-summary {isHl ? 'bg-surface-hover' : ''}" style="padding-left: {depth * 12 + 8}px" @@ -465,14 +687,13 @@ {/if} {node.name} - - + + + + {/if}
-
- { - highlightedKey = key - scrollToDiff(d) - }} - onmouseenter={() => setHoverHighlight(key)} + +
+ {#if isSynthetic(d)} + void revealSynthetic(d, key)} + onmouseenter={() => setHoverHighlight(key)} + > + {#snippet extras()} + + {/snippet} + + {:else} + revealDiff(d, key)} + onmouseenter={() => setHoverHighlight(key)} + > + {#snippet extras()} + {@render rowBadge(staged[d.key] ?? d)} + {#if d.deployKind === 'raw_app'} + + { + e.stopPropagation() + expandApp(d) + }} + > + {#if loadedDiffs[d.key]?.state === 'loading'} + + {:else} + + {/if} + + {:else} + + {/if} + {/snippet} + + {/if} +
+ {/if} +{/snippet} + + +{#snippet diffBlock(item: DeployItem)} + {@const loaded = loadedDiffs[item.key]} + {#if !mountedRows[item.key]} +
+ + Diff loads on scroll… +
+ {:else if !loaded || loaded.state === 'loading'} +
+ + Loading diff… +
+ {:else if loaded.state === 'error'} +
{loaded.error}
+ {:else if item.deployKind === 'raw_app'} +
+ {#each rawAppItems(item, loaded) as sub (displayKey(sub))} + +
+
+ {sub.path} +
+ {#if sub.kind === 'raw_app_file'} + + {:else} + {@const runnable = sub as RawAppRunnableItem} + + {/if} +
+ {/each} +
+ {:else} + {/if} {/snippet} @@ -540,178 +869,215 @@ /> {/snippet} - {/snippet} -
- {#if diffs.length > 0} - - {/if} -
-
- {#if loading && diffs.length === 0} -
- - Loading comparison... +
+
+ {#if model.items.length > 0} + + {/if} +
+
+ {#if model.loading && model.items.length === 0} +
+ + Loading changes... +
+ {:else if model.error} +
{model.error}
+ {:else if model.items.length === 0} +
No changes.
+ {:else if orderedItems.length === 0} +
No files match.
+ {:else} +
+ {#each orderedItems as d (d.key)} + + {@const view = staged[d.key] ?? d} + {@const action = actionFor(view)} + {@const editUrl = editUrlFor?.(d)} + {@const status = model.statusOf(d.key)} +
- - -
- {#if editUrl} - + +
+ {#if editUrl} + + {d.displayPath} + + {:else} +
+ {d.displayPath} +
+ {/if} +
+
+ {@render rowBadge(view)} + {#if status?.status === 'failed'} + {@render deployFailed(d)} + {:else if action.op !== 'none'} + {#if action.secondary?.length} + + {/if} +
+ + {#if staged[d.key]} + +
+ +
+ {/if} +
+ {/if} +
+
+
+ + {#if view.done} +
- {dpath} - + Deployed — no pending changes. +
{:else} -
- {dpath} +
+ {@render diffBlock(view)}
{/if}
-
- {#if d.ahead && d.ahead > 0} - {d.ahead} ahead - {/if} - {#if d.behind && d.behind > 0} - {d.behind} behind - {/if} - - - {status} - -
- -
- {#if !mountedRows[key]} - -
- - Diff loads on scroll… -
- {:else if d.kind === 'raw_app_file'} - - {@const rawFile = d as RawAppFileItem} - - {:else if 'appPath' in d} - - {@const runnable = d as RawAppRunnableItem} - - {:else if !loaded || loaded.state === 'loading'} -
- - Loading diff… -
- {:else if loaded.state === 'error'} -
{loaded.error}
- {:else if loaded.state === 'ready'} - - {/if}
-
- {/each} - - {/if} - + {/each} + + + + {/if} + + + + + {#if model.items.length > 0 && compareSessionHref} + + {/if} + +